[==========] Running 14 tests from 1 test suite.
[----------] Global test environment set-up.
[----------] 14 tests from AutoRebalancerTest
[ RUN ] AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing
WARNING: Logging before InitGoogleLogging() is written to STDERR
I20250112 15:05:51.074505 17651 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.17.60.254:37549,127.17.60.253:46031,127.17.60.252:46213
I20250112 15:05:51.074823 17651 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250112 15:05:51.074949 17651 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20250112 15:05:51.079495 17651 server_base.cc:1034] running on GCE node
W20250112 15:05:51.079783 17656 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250112 15:05:51.080386 17657 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250112 15:05:51.081256 17659 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250112 15:05:51.081465 17651 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250112 15:05:51.081511 17651 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250112 15:05:51.081539 17651 hybrid_clock.cc:648] HybridClock initialized: now 1736694351081539 us; error 0 us; skew 500 ppm
I20250112 15:05:51.082490 17651 webserver.cc:458] Webserver started at http://127.17.60.254:46231/ using document root <none> and password file <none>
I20250112 15:05:51.082629 17651 fs_manager.cc:362] Metadata directory not provided
I20250112 15:05:51.082715 17651 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250112 15:05:51.082790 17651 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250112 15:05:51.083871 17651 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task95dLSD/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1736694351070047-17651-0/minicluster-data/master-0-root/instance:
uuid: "7a02131677894a6a94c870ffbab58fef"
format_stamp: "Formatted at 2025-01-12 15:05:51 on dist-test-slave-b4nz"
I20250112 15:05:51.085129 17651 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20250112 15:05:51.085675 17664 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250112 15:05:51.085814 17651 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20250112 15:05:51.085875 17651 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task95dLSD/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1736694351070047-17651-0/minicluster-data/master-0-root
uuid: "7a02131677894a6a94c870ffbab58fef"
format_stamp: "Formatted at 2025-01-12 15:05:51 on dist-test-slave-b4nz"
I20250112 15:05:51.085928 17651 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task95dLSD/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1736694351070047-17651-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-task95dLSD/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1736694351070047-17651-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-task95dLSD/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1736694351070047-17651-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250112 15:05:51.097211 17651 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250112 15:05:51.097448 17651 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250112 15:05:51.097572 17651 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250112 15:05:51.101341 17651 rpc_server.cc:307] RPC server started. Bound to: 127.17.60.254:37549
I20250112 15:05:51.101372 17716 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.17.60.254:37549 every 8 connection(s)
I20250112 15:05:51.101740 17651 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20250112 15:05:51.101938 17717 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
W20250112 15:05:51.103005 17719 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250112 15:05:51.103612 17722 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250112 15:05:51.104099 17720 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250112 15:05:51.104833 17717 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.17.60.254" port: 37549 } has no permanent_uuid. Determining permanent_uuid...
I20250112 15:05:51.105073 17651 server_base.cc:1034] running on GCE node
I20250112 15:05:51.105165 17651 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250112 15:05:51.105199 17651 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250112 15:05:51.105213 17651 hybrid_clock.cc:648] HybridClock initialized: now 1736694351105214 us; error 0 us; skew 500 ppm
I20250112 15:05:51.105872 17651 webserver.cc:458] Webserver started at http://127.17.60.253:41099/ using document root <none> and password file <none>
I20250112 15:05:51.105952 17651 fs_manager.cc:362] Metadata directory not provided
I20250112 15:05:51.105998 17651 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250112 15:05:51.106060 17651 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250112 15:05:51.106277 17651 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task95dLSD/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1736694351070047-17651-0/minicluster-data/master-1-root/instance:
uuid: "f420d939a3c94dbf91632b7e5fad770a"
format_stamp: "Formatted at 2025-01-12 15:05:51 on dist-test-slave-b4nz"
I20250112 15:05:51.107120 17651 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20250112 15:05:51.107534 17729 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250112 15:05:51.107656 17651 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20250112 15:05:51.107728 17651 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task95dLSD/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1736694351070047-17651-0/minicluster-data/master-1-root
uuid: "f420d939a3c94dbf91632b7e5fad770a"
format_stamp: "Formatted at 2025-01-12 15:05:51 on dist-test-slave-b4nz"
I20250112 15:05:51.107771 17651 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task95dLSD/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1736694351070047-17651-0/minicluster-data/master-1-root
metadata directory: /tmp/dist-test-task95dLSD/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1736694351070047-17651-0/minicluster-data/master-1-root
1 data directories: /tmp/dist-test-task95dLSD/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1736694351070047-17651-0/minicluster-data/master-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250112 15:05:51.110280 17717 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.17.60.253" port: 46031 } has no permanent_uuid. Determining permanent_uuid...
W20250112 15:05:51.110594 17666 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.17.60.253:46031: connect: Connection refused (error 111)
W20250112 15:05:51.110937 17717 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.17.60.253:46031: Network error: Client connection negotiation failed: client connection to 127.17.60.253:46031: connect: Connection refused (error 111)
I20250112 15:05:51.139166 17651 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250112 15:05:51.139401 17651 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250112 15:05:51.142500 17651 rpc_server.cc:307] RPC server started. Bound to: 127.17.60.253:46031
I20250112 15:05:51.142540 17782 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.17.60.253:46031 every 8 connection(s)
I20250112 15:05:51.142818 17783 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 15:05:51.142853 17651 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20250112 15:05:51.143743 17783 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.17.60.254" port: 37549 } has no permanent_uuid. Determining permanent_uuid...
W20250112 15:05:51.144186 17785 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250112 15:05:51.144714 17786 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250112 15:05:51.145251 17788 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250112 15:05:51.146250 17651 server_base.cc:1034] running on GCE node
I20250112 15:05:51.146348 17651 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250112 15:05:51.146376 17651 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250112 15:05:51.146395 17651 hybrid_clock.cc:648] HybridClock initialized: now 1736694351146395 us; error 0 us; skew 500 ppm
I20250112 15:05:51.146642 17783 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.17.60.253" port: 46031 } has no permanent_uuid. Determining permanent_uuid...
I20250112 15:05:51.146989 17651 webserver.cc:458] Webserver started at http://127.17.60.252:39969/ using document root <none> and password file <none>
I20250112 15:05:51.147071 17651 fs_manager.cc:362] Metadata directory not provided
I20250112 15:05:51.147107 17651 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250112 15:05:51.147135 17651 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250112 15:05:51.147361 17651 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task95dLSD/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1736694351070047-17651-0/minicluster-data/master-2-root/instance:
uuid: "ab6589ccae414a1e966e89830ebaca6b"
format_stamp: "Formatted at 2025-01-12 15:05:51 on dist-test-slave-b4nz"
I20250112 15:05:51.148120 17651 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20250112 15:05:51.148622 17795 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250112 15:05:51.148730 17651 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20250112 15:05:51.148773 17651 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task95dLSD/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1736694351070047-17651-0/minicluster-data/master-2-root
uuid: "ab6589ccae414a1e966e89830ebaca6b"
format_stamp: "Formatted at 2025-01-12 15:05:51 on dist-test-slave-b4nz"
I20250112 15:05:51.148834 17651 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task95dLSD/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1736694351070047-17651-0/minicluster-data/master-2-root
metadata directory: /tmp/dist-test-task95dLSD/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1736694351070047-17651-0/minicluster-data/master-2-root
1 data directories: /tmp/dist-test-task95dLSD/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1736694351070047-17651-0/minicluster-data/master-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250112 15:05:51.149428 17783 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.17.60.252" port: 46213 } has no permanent_uuid. Determining permanent_uuid...
W20250112 15:05:51.149978 17783 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.17.60.252:46213: Network error: Client connection negotiation failed: client connection to 127.17.60.252:46213: connect: Connection refused (error 111)
I20250112 15:05:51.153873 17651 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250112 15:05:51.154094 17651 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250112 15:05:51.157313 17651 rpc_server.cc:307] RPC server started. Bound to: 127.17.60.252:46213
I20250112 15:05:51.157366 17848 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.17.60.252:46213 every 8 connection(s)
I20250112 15:05:51.157513 17651 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20250112 15:05:51.157687 17849 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 15:05:51.158332 17849 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.17.60.254" port: 37549 } has no permanent_uuid. Determining permanent_uuid...
I20250112 15:05:51.160094 17717 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.17.60.253" port: 46031 } attempt: 1
I20250112 15:05:51.160930 17849 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.17.60.253" port: 46031 } has no permanent_uuid. Determining permanent_uuid...
I20250112 15:05:51.162945 17717 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.17.60.252" port: 46213 } has no permanent_uuid. Determining permanent_uuid...
I20250112 15:05:51.164038 17849 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.17.60.252" port: 46213 } has no permanent_uuid. Determining permanent_uuid...
I20250112 15:05:51.167498 17717 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7a02131677894a6a94c870ffbab58fef: Bootstrap starting.
I20250112 15:05:51.167498 17849 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ab6589ccae414a1e966e89830ebaca6b: Bootstrap starting.
I20250112 15:05:51.168081 17717 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7a02131677894a6a94c870ffbab58fef: Neither blocks nor log segments found. Creating new log.
I20250112 15:05:51.168084 17849 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ab6589ccae414a1e966e89830ebaca6b: Neither blocks nor log segments found. Creating new log.
I20250112 15:05:51.168327 17849 log.cc:826] T 00000000000000000000000000000000 P ab6589ccae414a1e966e89830ebaca6b: Log is configured to *not* fsync() on all Append() calls
I20250112 15:05:51.168821 17717 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7a02131677894a6a94c870ffbab58fef: No bootstrap required, opened a new log
I20250112 15:05:51.168824 17849 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ab6589ccae414a1e966e89830ebaca6b: No bootstrap required, opened a new log
I20250112 15:05:51.170084 17849 raft_consensus.cc:357] T 00000000000000000000000000000000 P ab6589ccae414a1e966e89830ebaca6b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a02131677894a6a94c870ffbab58fef" member_type: VOTER last_known_addr { host: "127.17.60.254" port: 37549 } } peers { permanent_uuid: "f420d939a3c94dbf91632b7e5fad770a" member_type: VOTER last_known_addr { host: "127.17.60.253" port: 46031 } } peers { permanent_uuid: "ab6589ccae414a1e966e89830ebaca6b" member_type: VOTER last_known_addr { host: "127.17.60.252" port: 46213 } }
I20250112 15:05:51.170074 17717 raft_consensus.cc:357] T 00000000000000000000000000000000 P 7a02131677894a6a94c870ffbab58fef [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a02131677894a6a94c870ffbab58fef" member_type: VOTER last_known_addr { host: "127.17.60.254" port: 37549 } } peers { permanent_uuid: "f420d939a3c94dbf91632b7e5fad770a" member_type: VOTER last_known_addr { host: "127.17.60.253" port: 46031 } } peers { permanent_uuid: "ab6589ccae414a1e966e89830ebaca6b" member_type: VOTER last_known_addr { host: "127.17.60.252" port: 46213 } }
I20250112 15:05:51.170181 17849 raft_consensus.cc:383] T 00000000000000000000000000000000 P ab6589ccae414a1e966e89830ebaca6b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 15:05:51.170199 17717 raft_consensus.cc:383] T 00000000000000000000000000000000 P 7a02131677894a6a94c870ffbab58fef [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 15:05:51.170210 17849 raft_consensus.cc:738] T 00000000000000000000000000000000 P ab6589ccae414a1e966e89830ebaca6b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ab6589ccae414a1e966e89830ebaca6b, State: Initialized, Role: FOLLOWER
I20250112 15:05:51.170225 17717 raft_consensus.cc:738] T 00000000000000000000000000000000 P 7a02131677894a6a94c870ffbab58fef [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a02131677894a6a94c870ffbab58fef, State: Initialized, Role: FOLLOWER
I20250112 15:05:51.170333 17717 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7a02131677894a6a94c870ffbab58fef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a02131677894a6a94c870ffbab58fef" member_type: VOTER last_known_addr { host: "127.17.60.254" port: 37549 } } peers { permanent_uuid: "f420d939a3c94dbf91632b7e5fad770a" member_type: VOTER last_known_addr { host: "127.17.60.253" port: 46031 } } peers { permanent_uuid: "ab6589ccae414a1e966e89830ebaca6b" member_type: VOTER last_known_addr { host: "127.17.60.252" port: 46213 } }
I20250112 15:05:51.170327 17849 consensus_queue.cc:260] T 00000000000000000000000000000000 P ab6589ccae414a1e966e89830ebaca6b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a02131677894a6a94c870ffbab58fef" member_type: VOTER last_known_addr { host: "127.17.60.254" port: 37549 } } peers { permanent_uuid: "f420d939a3c94dbf91632b7e5fad770a" member_type: VOTER last_known_addr { host: "127.17.60.253" port: 46031 } } peers { permanent_uuid: "ab6589ccae414a1e966e89830ebaca6b" member_type: VOTER last_known_addr { host: "127.17.60.252" port: 46213 } }
I20250112 15:05:51.170601 17717 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7a02131677894a6a94c870ffbab58fef [sys.catalog]: configured and running, proceeding with master startup.
I20250112 15:05:51.170606 17849 sys_catalog.cc:564] T 00000000000000000000000000000000 P ab6589ccae414a1e966e89830ebaca6b [sys.catalog]: configured and running, proceeding with master startup.
I20250112 15:05:51.170575 17857 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7a02131677894a6a94c870ffbab58fef [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a02131677894a6a94c870ffbab58fef" member_type: VOTER last_known_addr { host: "127.17.60.254" port: 37549 } } peers { permanent_uuid: "f420d939a3c94dbf91632b7e5fad770a" member_type: VOTER last_known_addr { host: "127.17.60.253" port: 46031 } } peers { permanent_uuid: "ab6589ccae414a1e966e89830ebaca6b" member_type: VOTER last_known_addr { host: "127.17.60.252" port: 46213 } } }
I20250112 15:05:51.170673 17857 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7a02131677894a6a94c870ffbab58fef [sys.catalog]: This master's current role is: FOLLOWER
I20250112 15:05:51.171944 17651 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 1
I20250112 15:05:51.172056 17858 sys_catalog.cc:455] T 00000000000000000000000000000000 P ab6589ccae414a1e966e89830ebaca6b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a02131677894a6a94c870ffbab58fef" member_type: VOTER last_known_addr { host: "127.17.60.254" port: 37549 } } peers { permanent_uuid: "f420d939a3c94dbf91632b7e5fad770a" member_type: VOTER last_known_addr { host: "127.17.60.253" port: 46031 } } peers { permanent_uuid: "ab6589ccae414a1e966e89830ebaca6b" member_type: VOTER last_known_addr { host: "127.17.60.252" port: 46213 } } }
I20250112 15:05:51.172122 17858 sys_catalog.cc:458] T 00000000000000000000000000000000 P ab6589ccae414a1e966e89830ebaca6b [sys.catalog]: This master's current role is: FOLLOWER
W20250112 15:05:51.172358 17878 catalog_manager.cc:1559] T 00000000000000000000000000000000 P 7a02131677894a6a94c870ffbab58fef: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20250112 15:05:51.172361 17880 catalog_manager.cc:1559] T 00000000000000000000000000000000 P ab6589ccae414a1e966e89830ebaca6b: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20250112 15:05:51.172399 17878 catalog_manager.cc:874] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
W20250112 15:05:51.172436 17880 catalog_manager.cc:874] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20250112 15:05:51.192807 17857 raft_consensus.cc:491] T 00000000000000000000000000000000 P 7a02131677894a6a94c870ffbab58fef [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 15:05:51.192914 17857 raft_consensus.cc:513] T 00000000000000000000000000000000 P 7a02131677894a6a94c870ffbab58fef [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a02131677894a6a94c870ffbab58fef" member_type: VOTER last_known_addr { host: "127.17.60.254" port: 37549 } } peers { permanent_uuid: "f420d939a3c94dbf91632b7e5fad770a" member_type: VOTER last_known_addr { host: "127.17.60.253" port: 46031 } } peers { permanent_uuid: "ab6589ccae414a1e966e89830ebaca6b" member_type: VOTER last_known_addr { host: "127.17.60.252" port: 46213 } }
I20250112 15:05:51.193207 17857 leader_election.cc:290] T 00000000000000000000000000000000 P 7a02131677894a6a94c870ffbab58fef [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f420d939a3c94dbf91632b7e5fad770a (127.17.60.253:46031), ab6589ccae414a1e966e89830ebaca6b (127.17.60.252:46213)
I20250112 15:05:51.193513 17823 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "7a02131677894a6a94c870ffbab58fef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ab6589ccae414a1e966e89830ebaca6b" is_pre_election: true
I20250112 15:05:51.193614 17823 raft_consensus.cc:2463] T 00000000000000000000000000000000 P ab6589ccae414a1e966e89830ebaca6b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7a02131677894a6a94c870ffbab58fef in term 0.
I20250112 15:05:51.193811 17665 leader_election.cc:304] T 00000000000000000000000000000000 P 7a02131677894a6a94c870ffbab58fef [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7a02131677894a6a94c870ffbab58fef, ab6589ccae414a1e966e89830ebaca6b; no voters:
I20250112 15:05:51.193938 17857 raft_consensus.cc:2798] T 00000000000000000000000000000000 P 7a02131677894a6a94c870ffbab58fef [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250112 15:05:51.194003 17857 raft_consensus.cc:491] T 00000000000000000000000000000000 P 7a02131677894a6a94c870ffbab58fef [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250112 15:05:51.194032 17857 raft_consensus.cc:3054] T 00000000000000000000000000000000 P 7a02131677894a6a94c870ffbab58fef [term 0 FOLLOWER]: Advancing to term 1
I20250112 15:05:51.194849 17857 raft_consensus.cc:513] T 00000000000000000000000000000000 P 7a02131677894a6a94c870ffbab58fef [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a02131677894a6a94c870ffbab58fef" member_type: VOTER last_known_addr { host: "127.17.60.254" port: 37549 } } peers { permanent_uuid: "f420d939a3c94dbf91632b7e5fad770a" member_type: VOTER last_known_addr { host: "127.17.60.253" port: 46031 } } peers { permanent_uuid: "ab6589ccae414a1e966e89830ebaca6b" member_type: VOTER last_known_addr { host: "127.17.60.252" port: 46213 } }
I20250112 15:05:51.194981 17857 leader_election.cc:290] T 00000000000000000000000000000000 P 7a02131677894a6a94c870ffbab58fef [CANDIDATE]: Term 1 election: Requested vote from peers f420d939a3c94dbf91632b7e5fad770a (127.17.60.253:46031), ab6589ccae414a1e966e89830ebaca6b (127.17.60.252:46213)
I20250112 15:05:51.195160 17823 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "7a02131677894a6a94c870ffbab58fef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ab6589ccae414a1e966e89830ebaca6b"
I20250112 15:05:51.195225 17823 raft_consensus.cc:3054] T 00000000000000000000000000000000 P ab6589ccae414a1e966e89830ebaca6b [term 0 FOLLOWER]: Advancing to term 1
I20250112 15:05:51.195894 17823 raft_consensus.cc:2463] T 00000000000000000000000000000000 P ab6589ccae414a1e966e89830ebaca6b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7a02131677894a6a94c870ffbab58fef in term 1.
I20250112 15:05:51.196044 17665 leader_election.cc:304] T 00000000000000000000000000000000 P 7a02131677894a6a94c870ffbab58fef [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7a02131677894a6a94c870ffbab58fef, ab6589ccae414a1e966e89830ebaca6b; no voters:
I20250112 15:05:51.196127 17857 raft_consensus.cc:2798] T 00000000000000000000000000000000 P 7a02131677894a6a94c870ffbab58fef [term 1 FOLLOWER]: Leader election won for term 1
I20250112 15:05:51.196285 17857 raft_consensus.cc:695] T 00000000000000000000000000000000 P 7a02131677894a6a94c870ffbab58fef [term 1 LEADER]: Becoming Leader. State: Replica: 7a02131677894a6a94c870ffbab58fef, State: Running, Role: LEADER
I20250112 15:05:51.196388 17857 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7a02131677894a6a94c870ffbab58fef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a02131677894a6a94c870ffbab58fef" member_type: VOTER last_known_addr { host: "127.17.60.254" port: 37549 } } peers { permanent_uuid: "f420d939a3c94dbf91632b7e5fad770a" member_type: VOTER last_known_addr { host: "127.17.60.253" port: 46031 } } peers { permanent_uuid: "ab6589ccae414a1e966e89830ebaca6b" member_type: VOTER last_known_addr { host: "127.17.60.252" port: 46213 } }
I20250112 15:05:51.196836 17882 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7a02131677894a6a94c870ffbab58fef [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7a02131677894a6a94c870ffbab58fef. Latest consensus state: current_term: 1 leader_uuid: "7a02131677894a6a94c870ffbab58fef" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a02131677894a6a94c870ffbab58fef" member_type: VOTER last_known_addr { host: "127.17.60.254" port: 37549 } } peers { permanent_uuid: "f420d939a3c94dbf91632b7e5fad770a" member_type: VOTER last_known_addr { host: "127.17.60.253" port: 46031 } } peers { permanent_uuid: "ab6589ccae414a1e966e89830ebaca6b" member_type: VOTER last_known_addr { host: "127.17.60.252" port: 46213 } } }
I20250112 15:05:51.196965 17882 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7a02131677894a6a94c870ffbab58fef [sys.catalog]: This master's current role is: LEADER
I20250112 15:05:51.197204 17884 catalog_manager.cc:1476] Loading table and tablet metadata into memory...
I20250112 15:05:51.197407 17884 catalog_manager.cc:1485] Initializing Kudu cluster ID...
I20250112 15:05:51.198694 17823 raft_consensus.cc:1270] T 00000000000000000000000000000000 P ab6589ccae414a1e966e89830ebaca6b [term 1 FOLLOWER]: Refusing update from remote peer 7a02131677894a6a94c870ffbab58fef: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250112 15:05:51.198982 17882 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 7a02131677894a6a94c870ffbab58fef [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab6589ccae414a1e966e89830ebaca6b" member_type: VOTER last_known_addr { host: "127.17.60.252" port: 46213 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250112 15:05:51.199576 17858 sys_catalog.cc:455] T 00000000000000000000000000000000 P ab6589ccae414a1e966e89830ebaca6b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7a02131677894a6a94c870ffbab58fef. Latest consensus state: current_term: 1 leader_uuid: "7a02131677894a6a94c870ffbab58fef" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a02131677894a6a94c870ffbab58fef" member_type: VOTER last_known_addr { host: "127.17.60.254" port: 37549 } } peers { permanent_uuid: "f420d939a3c94dbf91632b7e5fad770a" member_type: VOTER last_known_addr { host: "127.17.60.253" port: 46031 } } peers { permanent_uuid: "ab6589ccae414a1e966e89830ebaca6b" member_type: VOTER last_known_addr { host: "127.17.60.252" port: 46213 } } }
I20250112 15:05:51.199659 17858 sys_catalog.cc:458] T 00000000000000000000000000000000 P ab6589ccae414a1e966e89830ebaca6b [sys.catalog]: This master's current role is: FOLLOWER
I20250112 15:05:51.200387 17858 sys_catalog.cc:455] T 00000000000000000000000000000000 P ab6589ccae414a1e966e89830ebaca6b [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "7a02131677894a6a94c870ffbab58fef" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a02131677894a6a94c870ffbab58fef" member_type: VOTER last_known_addr { host: "127.17.60.254" port: 37549 } } peers { permanent_uuid: "f420d939a3c94dbf91632b7e5fad770a" member_type: VOTER last_known_addr { host: "127.17.60.253" port: 46031 } } peers { permanent_uuid: "ab6589ccae414a1e966e89830ebaca6b" member_type: VOTER last_known_addr { host: "127.17.60.252" port: 46213 } } }
I20250112 15:05:51.200438 17858 sys_catalog.cc:458] T 00000000000000000000000000000000 P ab6589ccae414a1e966e89830ebaca6b [sys.catalog]: This master's current role is: FOLLOWER
I20250112 15:05:51.200502 17885 mvcc.cc:204] Tried to move back new op lower bound from 7113500062508138496 to 7113500062501224448. Current Snapshot: MvccSnapshot[applied={T|T < 7113500062508138496}]
I20250112 15:05:51.200290 17857 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7a02131677894a6a94c870ffbab58fef [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "7a02131677894a6a94c870ffbab58fef" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a02131677894a6a94c870ffbab58fef" member_type: VOTER last_known_addr { host: "127.17.60.254" port: 37549 } } peers { permanent_uuid: "f420d939a3c94dbf91632b7e5fad770a" member_type: VOTER last_known_addr { host: "127.17.60.253" port: 46031 } } peers { permanent_uuid: "ab6589ccae414a1e966e89830ebaca6b" member_type: VOTER last_known_addr { host: "127.17.60.252" port: 46213 } } }
I20250112 15:05:51.200599 17857 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7a02131677894a6a94c870ffbab58fef [sys.catalog]: This master's current role is: LEADER
I20250112 15:05:51.201081 17884 catalog_manager.cc:1348] Generated new cluster ID: e6fdd5e7d15344eb963048544bcd65ae
I20250112 15:05:51.201126 17884 catalog_manager.cc:1496] Initializing Kudu internal certificate authority...
I20250112 15:05:51.202103 17783 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.17.60.252" port: 46213 } attempt: 1
I20250112 15:05:51.206180 17783 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f420d939a3c94dbf91632b7e5fad770a: Bootstrap starting.
I20250112 15:05:51.206647 17783 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f420d939a3c94dbf91632b7e5fad770a: Neither blocks nor log segments found. Creating new log.
I20250112 15:05:51.207414 17783 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f420d939a3c94dbf91632b7e5fad770a: No bootstrap required, opened a new log
I20250112 15:05:51.207588 17783 raft_consensus.cc:357] T 00000000000000000000000000000000 P f420d939a3c94dbf91632b7e5fad770a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a02131677894a6a94c870ffbab58fef" member_type: VOTER last_known_addr { host: "127.17.60.254" port: 37549 } } peers { permanent_uuid: "f420d939a3c94dbf91632b7e5fad770a" member_type: VOTER last_known_addr { host: "127.17.60.253" port: 46031 } } peers { permanent_uuid: "ab6589ccae414a1e966e89830ebaca6b" member_type: VOTER last_known_addr { host: "127.17.60.252" port: 46213 } }
I20250112 15:05:51.207653 17783 raft_consensus.cc:383] T 00000000000000000000000000000000 P f420d939a3c94dbf91632b7e5fad770a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 15:05:51.207679 17783 raft_consensus.cc:738] T 00000000000000000000000000000000 P f420d939a3c94dbf91632b7e5fad770a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f420d939a3c94dbf91632b7e5fad770a, State: Initialized, Role: FOLLOWER
I20250112 15:05:51.207746 17783 consensus_queue.cc:260] T 00000000000000000000000000000000 P f420d939a3c94dbf91632b7e5fad770a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a02131677894a6a94c870ffbab58fef" member_type: VOTER last_known_addr { host: "127.17.60.254" port: 37549 } } peers { permanent_uuid: "f420d939a3c94dbf91632b7e5fad770a" member_type: VOTER last_known_addr { host: "127.17.60.253" port: 46031 } } peers { permanent_uuid: "ab6589ccae414a1e966e89830ebaca6b" member_type: VOTER last_known_addr { host: "127.17.60.252" port: 46213 } }
I20250112 15:05:51.207921 17783 sys_catalog.cc:564] T 00000000000000000000000000000000 P f420d939a3c94dbf91632b7e5fad770a [sys.catalog]: configured and running, proceeding with master startup.
I20250112 15:05:51.207947 17755 raft_consensus.cc:3054] T 00000000000000000000000000000000 P f420d939a3c94dbf91632b7e5fad770a [term 0 FOLLOWER]: Advancing to term 1
I20250112 15:05:51.208607 17757 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "7a02131677894a6a94c870ffbab58fef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f420d939a3c94dbf91632b7e5fad770a" is_pre_election: true
I20250112 15:05:51.208916 17756 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "7a02131677894a6a94c870ffbab58fef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f420d939a3c94dbf91632b7e5fad770a"
I20250112 15:05:51.209383 17755 raft_consensus.cc:1270] T 00000000000000000000000000000000 P f420d939a3c94dbf91632b7e5fad770a [term 1 FOLLOWER]: Refusing update from remote peer 7a02131677894a6a94c870ffbab58fef: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250112 15:05:51.209556 17857 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 7a02131677894a6a94c870ffbab58fef [LEADER]: Connected to new peer: Peer: permanent_uuid: "f420d939a3c94dbf91632b7e5fad770a" member_type: VOTER last_known_addr { host: "127.17.60.253" port: 46031 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250112 15:05:51.209617 17651 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 2
I20250112 15:05:51.209888 17651 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20250112 15:05:51.210485 17891 sys_catalog.cc:455] T 00000000000000000000000000000000 P f420d939a3c94dbf91632b7e5fad770a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a02131677894a6a94c870ffbab58fef" member_type: VOTER last_known_addr { host: "127.17.60.254" port: 37549 } } peers { permanent_uuid: "f420d939a3c94dbf91632b7e5fad770a" member_type: VOTER last_known_addr { host: "127.17.60.253" port: 46031 } } peers { permanent_uuid: "ab6589ccae414a1e966e89830ebaca6b" member_type: VOTER last_known_addr { host: "127.17.60.252" port: 46213 } } }
I20250112 15:05:51.210575 17891 sys_catalog.cc:458] T 00000000000000000000000000000000 P f420d939a3c94dbf91632b7e5fad770a [sys.catalog]: This master's current role is: FOLLOWER
I20250112 15:05:51.210615 17891 sys_catalog.cc:455] T 00000000000000000000000000000000 P f420d939a3c94dbf91632b7e5fad770a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7a02131677894a6a94c870ffbab58fef. Latest consensus state: current_term: 1 leader_uuid: "7a02131677894a6a94c870ffbab58fef" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a02131677894a6a94c870ffbab58fef" member_type: VOTER last_known_addr { host: "127.17.60.254" port: 37549 } } peers { permanent_uuid: "f420d939a3c94dbf91632b7e5fad770a" member_type: VOTER last_known_addr { host: "127.17.60.253" port: 46031 } } peers { permanent_uuid: "ab6589ccae414a1e966e89830ebaca6b" member_type: VOTER last_known_addr { host: "127.17.60.252" port: 46213 } } }
I20250112 15:05:51.210675 17891 sys_catalog.cc:458] T 00000000000000000000000000000000 P f420d939a3c94dbf91632b7e5fad770a [sys.catalog]: This master's current role is: FOLLOWER
I20250112 15:05:51.210712 17891 sys_catalog.cc:455] T 00000000000000000000000000000000 P f420d939a3c94dbf91632b7e5fad770a [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "7a02131677894a6a94c870ffbab58fef" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a02131677894a6a94c870ffbab58fef" member_type: VOTER last_known_addr { host: "127.17.60.254" port: 37549 } } peers { permanent_uuid: "f420d939a3c94dbf91632b7e5fad770a" member_type: VOTER last_known_addr { host: "127.17.60.253" port: 46031 } } peers { permanent_uuid: "ab6589ccae414a1e966e89830ebaca6b" member_type: VOTER last_known_addr { host: "127.17.60.252" port: 46213 } } }
I20250112 15:05:51.210765 17891 sys_catalog.cc:458] T 00000000000000000000000000000000 P f420d939a3c94dbf91632b7e5fad770a [sys.catalog]: This master's current role is: FOLLOWER
W20250112 15:05:51.210937 17902 catalog_manager.cc:1559] T 00000000000000000000000000000000 P f420d939a3c94dbf91632b7e5fad770a: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20250112 15:05:51.210968 17902 catalog_manager.cc:874] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
W20250112 15:05:51.214040 17908 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250112 15:05:51.214617 17910 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250112 15:05:51.214993 17907 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250112 15:05:51.215698 17651 server_base.cc:1034] running on GCE node
I20250112 15:05:51.215838 17651 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250112 15:05:51.215864 17651 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250112 15:05:51.215888 17651 hybrid_clock.cc:648] HybridClock initialized: now 1736694351215887 us; error 0 us; skew 500 ppm
I20250112 15:05:51.216650 17651 webserver.cc:458] Webserver started at http://127.17.60.193:36667/ using document root <none> and password file <none>
I20250112 15:05:51.216734 17651 fs_manager.cc:362] Metadata directory not provided
I20250112 15:05:51.216779 17651 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250112 15:05:51.216830 17651 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250112 15:05:51.217078 17651 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task95dLSD/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1736694351070047-17651-0/minicluster-data/ts-0-root/instance:
uuid: "293268888c03422ca22674275564c902"
format_stamp: "Formatted at 2025-01-12 15:05:51 on dist-test-slave-b4nz"
I20250112 15:05:51.217963 17651 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20250112 15:05:51.218430 17916 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250112 15:05:51.218537 17651 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20250112 15:05:51.218573 17651 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task95dLSD/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1736694351070047-17651-0/minicluster-data/ts-0-root
uuid: "293268888c03422ca22674275564c902"
format_stamp: "Formatted at 2025-01-12 15:05:51 on dist-test-slave-b4nz"
I20250112 15:05:51.218616 17651 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task95dLSD/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1736694351070047-17651-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-task95dLSD/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1736694351070047-17651-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-task95dLSD/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1736694351070047-17651-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250112 15:05:51.227138 17857 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7a02131677894a6a94c870ffbab58fef [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "7a02131677894a6a94c870ffbab58fef" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a02131677894a6a94c870ffbab58fef" member_type: VOTER last_known_addr { host: "127.17.60.254" port: 37549 } } peers { permanent_uuid: "f420d939a3c94dbf91632b7e5fad770a" member_type: VOTER last_known_addr { host: "127.17.60.253" port: 46031 } } peers { permanent_uuid: "ab6589ccae414a1e966e89830ebaca6b" member_type: VOTER last_known_addr { host: "127.17.60.252" port: 46213 } } }
I20250112 15:05:51.227238 17857 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7a02131677894a6a94c870ffbab58fef [sys.catalog]: This master's current role is: LEADER
I20250112 15:05:51.228256 17884 catalog_manager.cc:1371] Generated new certificate authority record
I20250112 15:05:51.228958 17884 catalog_manager.cc:1505] Loading token signing keys...
I20250112 15:05:51.236232 17651 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250112 15:05:51.236351 17884 catalog_manager.cc:5899] T 00000000000000000000000000000000 P 7a02131677894a6a94c870ffbab58fef: Generated new TSK 0
I20250112 15:05:51.236549 17884 catalog_manager.cc:1515] Initializing in-progress tserver states...
I20250112 15:05:51.236608 17651 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250112 15:05:51.236809 17651 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250112 15:05:51.237125 17651 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250112 15:05:51.237159 17651 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250112 15:05:51.237183 17651 ts_tablet_manager.cc:610] Registered 0 tablets
I20250112 15:05:51.237210 17651 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250112 15:05:51.241717 17651 rpc_server.cc:307] RPC server started. Bound to: 127.17.60.193:45763
I20250112 15:05:51.241782 17981 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.17.60.193:45763 every 8 connection(s)
I20250112 15:05:51.242260 17651 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250112 15:05:51.244537 17991 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250112 15:05:51.246336 17983 heartbeater.cc:346] Connected to a master server at 127.17.60.254:37549
I20250112 15:05:51.246351 17982 heartbeater.cc:346] Connected to a master server at 127.17.60.252:46213
I20250112 15:05:51.246477 17982 heartbeater.cc:463] Registering TS with master...
I20250112 15:05:51.246662 17982 heartbeater.cc:510] Master 127.17.60.252:46213 requested a full tablet report, sending...
I20250112 15:05:51.246829 17984 heartbeater.cc:346] Connected to a master server at 127.17.60.253:46031
I20250112 15:05:51.246878 17984 heartbeater.cc:463] Registering TS with master...
I20250112 15:05:51.246917 17983 heartbeater.cc:463] Registering TS with master...
I20250112 15:05:51.246964 17984 heartbeater.cc:510] Master 127.17.60.253:46031 requested a full tablet report, sending...
I20250112 15:05:51.246995 17983 heartbeater.cc:510] Master 127.17.60.254:37549 requested a full tablet report, sending...
I20250112 15:05:51.247210 17813 ts_manager.cc:194] Registered new tserver with Master: 293268888c03422ca22674275564c902 (127.17.60.193:45763)
W20250112 15:05:51.247224 17989 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250112 15:05:51.247334 17681 ts_manager.cc:194] Registered new tserver with Master: 293268888c03422ca22674275564c902 (127.17.60.193:45763)
I20250112 15:05:51.247450 17747 ts_manager.cc:194] Registered new tserver with Master: 293268888c03422ca22674275564c902 (127.17.60.193:45763)
W20250112 15:05:51.247634 17988 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250112 15:05:51.248203 17651 server_base.cc:1034] running on GCE node
I20250112 15:05:51.248296 17651 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
I20250112 15:05:51.248315 17681 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:60216
W20250112 15:05:51.248323 17651 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250112 15:05:51.248409 17651 hybrid_clock.cc:648] HybridClock initialized: now 1736694351248409 us; error 0 us; skew 500 ppm
I20250112 15:05:51.249085 17651 webserver.cc:458] Webserver started at http://127.17.60.194:39485/ using document root <none> and password file <none>
I20250112 15:05:51.249181 17651 fs_manager.cc:362] Metadata directory not provided
I20250112 15:05:51.249226 17651 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250112 15:05:51.249271 17651 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250112 15:05:51.249552 17651 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task95dLSD/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1736694351070047-17651-0/minicluster-data/ts-1-root/instance:
uuid: "496d3e69ee014a9d9de3519ff7dbf4b3"
format_stamp: "Formatted at 2025-01-12 15:05:51 on dist-test-slave-b4nz"
I20250112 15:05:51.250466 17651 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20250112 15:05:51.250892 17997 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250112 15:05:51.251034 17651 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20250112 15:05:51.251076 17651 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task95dLSD/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1736694351070047-17651-0/minicluster-data/ts-1-root
uuid: "496d3e69ee014a9d9de3519ff7dbf4b3"
format_stamp: "Formatted at 2025-01-12 15:05:51 on dist-test-slave-b4nz"
I20250112 15:05:51.251130 17651 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task95dLSD/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1736694351070047-17651-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-task95dLSD/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1736694351070047-17651-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-task95dLSD/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1736694351070047-17651-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250112 15:05:51.262761 17651 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250112 15:05:51.263008 17651 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250112 15:05:51.263177 17651 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250112 15:05:51.263368 17651 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250112 15:05:51.263396 17651 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250112 15:05:51.263430 17651 ts_tablet_manager.cc:610] Registered 0 tablets
I20250112 15:05:51.263460 17651 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250112 15:05:51.266752 17651 rpc_server.cc:307] RPC server started. Bound to: 127.17.60.194:43477
I20250112 15:05:51.266871 18061 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.17.60.194:43477 every 8 connection(s)
I20250112 15:05:51.267279 17651 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250112 15:05:51.270604 18071 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250112 15:05:51.270895 18062 heartbeater.cc:346] Connected to a master server at 127.17.60.252:46213
I20250112 15:05:51.270954 18062 heartbeater.cc:463] Registering TS with master...
I20250112 15:05:51.271054 18062 heartbeater.cc:510] Master 127.17.60.252:46213 requested a full tablet report, sending...
W20250112 15:05:51.271266 18069 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250112 15:05:51.271514 17813 ts_manager.cc:194] Registered new tserver with Master: 496d3e69ee014a9d9de3519ff7dbf4b3 (127.17.60.194:43477)
I20250112 15:05:51.271548 18063 heartbeater.cc:346] Connected to a master server at 127.17.60.254:37549
I20250112 15:05:51.271598 18063 heartbeater.cc:463] Registering TS with master...
I20250112 15:05:51.271687 18063 heartbeater.cc:510] Master 127.17.60.254:37549 requested a full tablet report, sending...
W20250112 15:05:51.271816 18068 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250112 15:05:51.272146 17651 server_base.cc:1034] running on GCE node
I20250112 15:05:51.272150 17681 ts_manager.cc:194] Registered new tserver with Master: 496d3e69ee014a9d9de3519ff7dbf4b3 (127.17.60.194:43477)
I20250112 15:05:51.272330 17651 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250112 15:05:51.272364 17651 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250112 15:05:51.272382 17651 hybrid_clock.cc:648] HybridClock initialized: now 1736694351272382 us; error 0 us; skew 500 ppm
I20250112 15:05:51.272655 17681 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:60220
I20250112 15:05:51.273207 17651 webserver.cc:458] Webserver started at http://127.17.60.195:35909/ using document root <none> and password file <none>
I20250112 15:05:51.273298 17651 fs_manager.cc:362] Metadata directory not provided
I20250112 15:05:51.273331 17651 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250112 15:05:51.273360 17651 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250112 15:05:51.273643 17651 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task95dLSD/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1736694351070047-17651-0/minicluster-data/ts-2-root/instance:
uuid: "76b66f65b69e49b396532320a307bcd0"
format_stamp: "Formatted at 2025-01-12 15:05:51 on dist-test-slave-b4nz"
I20250112 15:05:51.274163 18064 heartbeater.cc:346] Connected to a master server at 127.17.60.253:46031
I20250112 15:05:51.274211 18064 heartbeater.cc:463] Registering TS with master...
I20250112 15:05:51.274257 18064 heartbeater.cc:510] Master 127.17.60.253:46031 requested a full tablet report, sending...
I20250112 15:05:51.274506 17747 ts_manager.cc:194] Registered new tserver with Master: 496d3e69ee014a9d9de3519ff7dbf4b3 (127.17.60.194:43477)
I20250112 15:05:51.274737 17651 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20250112 15:05:51.275499 18077 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250112 15:05:51.275655 17651 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20250112 15:05:51.275717 17651 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task95dLSD/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1736694351070047-17651-0/minicluster-data/ts-2-root
uuid: "76b66f65b69e49b396532320a307bcd0"
format_stamp: "Formatted at 2025-01-12 15:05:51 on dist-test-slave-b4nz"
I20250112 15:05:51.275761 17651 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task95dLSD/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1736694351070047-17651-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-task95dLSD/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1736694351070047-17651-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-task95dLSD/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1736694351070047-17651-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250112 15:05:51.294461 17651 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250112 15:05:51.294721 17651 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250112 15:05:51.294881 17651 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250112 15:05:51.295095 17651 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250112 15:05:51.295121 17651 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250112 15:05:51.295142 17651 ts_tablet_manager.cc:610] Registered 0 tablets
I20250112 15:05:51.295169 17651 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250112 15:05:51.299352 17651 rpc_server.cc:307] RPC server started. Bound to: 127.17.60.195:45367
I20250112 15:05:51.299477 18140 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.17.60.195:45367 every 8 connection(s)
I20250112 15:05:51.303457 18143 heartbeater.cc:346] Connected to a master server at 127.17.60.253:46031
I20250112 15:05:51.303712 18143 heartbeater.cc:463] Registering TS with master...
I20250112 15:05:51.303875 18141 heartbeater.cc:346] Connected to a master server at 127.17.60.252:46213
I20250112 15:05:51.303957 18141 heartbeater.cc:463] Registering TS with master...
I20250112 15:05:51.304059 18141 heartbeater.cc:510] Master 127.17.60.252:46213 requested a full tablet report, sending...
I20250112 15:05:51.304235 18143 heartbeater.cc:510] Master 127.17.60.253:46031 requested a full tablet report, sending...
I20250112 15:05:51.304301 17813 ts_manager.cc:194] Registered new tserver with Master: 76b66f65b69e49b396532320a307bcd0 (127.17.60.195:45367)
I20250112 15:05:51.304544 17747 ts_manager.cc:194] Registered new tserver with Master: 76b66f65b69e49b396532320a307bcd0 (127.17.60.195:45367)
I20250112 15:05:51.306424 18142 heartbeater.cc:346] Connected to a master server at 127.17.60.254:37549
I20250112 15:05:51.306473 18142 heartbeater.cc:463] Registering TS with master...
I20250112 15:05:51.306553 18142 heartbeater.cc:510] Master 127.17.60.254:37549 requested a full tablet report, sending...
I20250112 15:05:51.306758 17681 ts_manager.cc:194] Registered new tserver with Master: 76b66f65b69e49b396532320a307bcd0 (127.17.60.195:45367)
I20250112 15:05:51.307190 17681 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:60232
I20250112 15:05:51.307528 17651 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.007874818s
I20250112 15:05:52.172791 17880 catalog_manager.cc:1260] Loaded cluster ID: e6fdd5e7d15344eb963048544bcd65ae
I20250112 15:05:52.172873 17880 catalog_manager.cc:1553] T 00000000000000000000000000000000 P ab6589ccae414a1e966e89830ebaca6b: loading cluster ID for follower catalog manager: success
I20250112 15:05:52.173653 17880 catalog_manager.cc:1575] T 00000000000000000000000000000000 P ab6589ccae414a1e966e89830ebaca6b: acquiring CA information for follower catalog manager: success
I20250112 15:05:52.173897 17880 catalog_manager.cc:1603] T 00000000000000000000000000000000 P ab6589ccae414a1e966e89830ebaca6b: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20250112 15:05:52.211385 17902 catalog_manager.cc:1260] Loaded cluster ID: e6fdd5e7d15344eb963048544bcd65ae
I20250112 15:05:52.211460 17902 catalog_manager.cc:1553] T 00000000000000000000000000000000 P f420d939a3c94dbf91632b7e5fad770a: loading cluster ID for follower catalog manager: success
I20250112 15:05:52.212098 17902 catalog_manager.cc:1575] T 00000000000000000000000000000000 P f420d939a3c94dbf91632b7e5fad770a: acquiring CA information for follower catalog manager: success
I20250112 15:05:52.212236 17902 catalog_manager.cc:1603] T 00000000000000000000000000000000 P f420d939a3c94dbf91632b7e5fad770a: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20250112 15:05:52.248927 17983 heartbeater.cc:502] Master 127.17.60.254:37549 was elected leader, sending a full tablet report...
I20250112 15:05:52.273204 18063 heartbeater.cc:502] Master 127.17.60.254:37549 was elected leader, sending a full tablet report...
I20250112 15:05:52.307706 18142 heartbeater.cc:502] Master 127.17.60.254:37549 was elected leader, sending a full tablet report...
I20250112 15:05:53.171978 17877 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20250112 15:05:53.357697 17651 test_util.cc:274] Using random seed: -1442583684
I20250112 15:05:53.370781 17681 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:60248:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 1
split_rows_range_bounds {
rows: "\004\001\000\377\377\377\037\004\001\000\376\377\377?\004\001\000\375\377\377_""\004\001\000\377\377\377\037\004\001\000\376\377\377?\004\001\000\375\377\377_"
indirect_data: """"
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250112 15:05:53.378135 17946 tablet_service.cc:1467] Processing CreateTablet for tablet 9202a8ac823243a59bcc9db743b299ec (DEFAULT_TABLE table=test-workload [id=2ed6531222be4f0d80cf00c1f18c2d5e]), partition=RANGE (key) PARTITION 1073741822 <= VALUES < 1610612733
I20250112 15:05:53.378254 18026 tablet_service.cc:1467] Processing CreateTablet for tablet 2d3106e9e16349fd8737dc1fda0d1427 (DEFAULT_TABLE table=test-workload [id=2ed6531222be4f0d80cf00c1f18c2d5e]), partition=RANGE (key) PARTITION VALUES < 536870911
I20250112 15:05:53.378358 17946 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9202a8ac823243a59bcc9db743b299ec. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 15:05:53.378384 18026 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2d3106e9e16349fd8737dc1fda0d1427. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 15:05:53.379518 18178 tablet_bootstrap.cc:492] T 9202a8ac823243a59bcc9db743b299ec P 293268888c03422ca22674275564c902: Bootstrap starting.
I20250112 15:05:53.379630 18105 tablet_service.cc:1467] Processing CreateTablet for tablet 6292bb4b6efd418a8f700a903e3af221 (DEFAULT_TABLE table=test-workload [id=2ed6531222be4f0d80cf00c1f18c2d5e]), partition=RANGE (key) PARTITION 536870911 <= VALUES < 1073741822
I20250112 15:05:53.379734 18105 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6292bb4b6efd418a8f700a903e3af221. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 15:05:53.379941 18178 tablet_bootstrap.cc:654] T 9202a8ac823243a59bcc9db743b299ec P 293268888c03422ca22674275564c902: Neither blocks nor log segments found. Creating new log.
I20250112 15:05:53.380230 18177 tablet_bootstrap.cc:492] T 2d3106e9e16349fd8737dc1fda0d1427 P 496d3e69ee014a9d9de3519ff7dbf4b3: Bootstrap starting.
I20250112 15:05:53.380439 18178 tablet_bootstrap.cc:492] T 9202a8ac823243a59bcc9db743b299ec P 293268888c03422ca22674275564c902: No bootstrap required, opened a new log
I20250112 15:05:53.380522 18178 ts_tablet_manager.cc:1397] T 9202a8ac823243a59bcc9db743b299ec P 293268888c03422ca22674275564c902: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20250112 15:05:53.380705 18178 raft_consensus.cc:357] T 9202a8ac823243a59bcc9db743b299ec P 293268888c03422ca22674275564c902 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "293268888c03422ca22674275564c902" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 45763 } }
I20250112 15:05:53.380759 18178 raft_consensus.cc:383] T 9202a8ac823243a59bcc9db743b299ec P 293268888c03422ca22674275564c902 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 15:05:53.380781 18177 tablet_bootstrap.cc:654] T 2d3106e9e16349fd8737dc1fda0d1427 P 496d3e69ee014a9d9de3519ff7dbf4b3: Neither blocks nor log segments found. Creating new log.
I20250112 15:05:53.380784 18178 raft_consensus.cc:738] T 9202a8ac823243a59bcc9db743b299ec P 293268888c03422ca22674275564c902 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 293268888c03422ca22674275564c902, State: Initialized, Role: FOLLOWER
I20250112 15:05:53.380918 18178 consensus_queue.cc:260] T 9202a8ac823243a59bcc9db743b299ec P 293268888c03422ca22674275564c902 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "293268888c03422ca22674275564c902" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 45763 } }
I20250112 15:05:53.381016 18180 tablet_bootstrap.cc:492] T 6292bb4b6efd418a8f700a903e3af221 P 76b66f65b69e49b396532320a307bcd0: Bootstrap starting.
I20250112 15:05:53.381031 18178 raft_consensus.cc:397] T 9202a8ac823243a59bcc9db743b299ec P 293268888c03422ca22674275564c902 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250112 15:05:53.381174 18178 raft_consensus.cc:491] T 9202a8ac823243a59bcc9db743b299ec P 293268888c03422ca22674275564c902 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250112 15:05:53.381254 18178 raft_consensus.cc:3054] T 9202a8ac823243a59bcc9db743b299ec P 293268888c03422ca22674275564c902 [term 0 FOLLOWER]: Advancing to term 1
I20250112 15:05:53.378135 17945 tablet_service.cc:1467] Processing CreateTablet for tablet c7297c6c590e42879b745e235f34748e (DEFAULT_TABLE table=test-workload [id=2ed6531222be4f0d80cf00c1f18c2d5e]), partition=RANGE (key) PARTITION 1610612733 <= VALUES
I20250112 15:05:53.381345 18177 tablet_bootstrap.cc:492] T 2d3106e9e16349fd8737dc1fda0d1427 P 496d3e69ee014a9d9de3519ff7dbf4b3: No bootstrap required, opened a new log
I20250112 15:05:53.381412 17945 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c7297c6c590e42879b745e235f34748e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 15:05:53.381384 18177 ts_tablet_manager.cc:1397] T 2d3106e9e16349fd8737dc1fda0d1427 P 496d3e69ee014a9d9de3519ff7dbf4b3: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20250112 15:05:53.381393 18180 tablet_bootstrap.cc:654] T 6292bb4b6efd418a8f700a903e3af221 P 76b66f65b69e49b396532320a307bcd0: Neither blocks nor log segments found. Creating new log.
I20250112 15:05:53.381608 18177 raft_consensus.cc:357] T 2d3106e9e16349fd8737dc1fda0d1427 P 496d3e69ee014a9d9de3519ff7dbf4b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "496d3e69ee014a9d9de3519ff7dbf4b3" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 43477 } }
I20250112 15:05:53.381652 18177 raft_consensus.cc:383] T 2d3106e9e16349fd8737dc1fda0d1427 P 496d3e69ee014a9d9de3519ff7dbf4b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 15:05:53.381669 18177 raft_consensus.cc:738] T 2d3106e9e16349fd8737dc1fda0d1427 P 496d3e69ee014a9d9de3519ff7dbf4b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 496d3e69ee014a9d9de3519ff7dbf4b3, State: Initialized, Role: FOLLOWER
I20250112 15:05:53.381705 18177 consensus_queue.cc:260] T 2d3106e9e16349fd8737dc1fda0d1427 P 496d3e69ee014a9d9de3519ff7dbf4b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "496d3e69ee014a9d9de3519ff7dbf4b3" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 43477 } }
I20250112 15:05:53.381736 18177 raft_consensus.cc:397] T 2d3106e9e16349fd8737dc1fda0d1427 P 496d3e69ee014a9d9de3519ff7dbf4b3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250112 15:05:53.381753 18177 raft_consensus.cc:491] T 2d3106e9e16349fd8737dc1fda0d1427 P 496d3e69ee014a9d9de3519ff7dbf4b3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250112 15:05:53.381770 18177 raft_consensus.cc:3054] T 2d3106e9e16349fd8737dc1fda0d1427 P 496d3e69ee014a9d9de3519ff7dbf4b3 [term 0 FOLLOWER]: Advancing to term 1
I20250112 15:05:53.381916 18178 raft_consensus.cc:513] T 9202a8ac823243a59bcc9db743b299ec P 293268888c03422ca22674275564c902 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "293268888c03422ca22674275564c902" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 45763 } }
I20250112 15:05:53.381966 18178 leader_election.cc:304] T 9202a8ac823243a59bcc9db743b299ec P 293268888c03422ca22674275564c902 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 293268888c03422ca22674275564c902; no voters:
I20250112 15:05:53.382050 18178 leader_election.cc:290] T 9202a8ac823243a59bcc9db743b299ec P 293268888c03422ca22674275564c902 [CANDIDATE]: Term 1 election: Requested vote from peers
I20250112 15:05:53.382095 18183 raft_consensus.cc:2798] T 9202a8ac823243a59bcc9db743b299ec P 293268888c03422ca22674275564c902 [term 1 FOLLOWER]: Leader election won for term 1
I20250112 15:05:53.382186 18178 ts_tablet_manager.cc:1428] T 9202a8ac823243a59bcc9db743b299ec P 293268888c03422ca22674275564c902: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250112 15:05:53.382195 18183 raft_consensus.cc:695] T 9202a8ac823243a59bcc9db743b299ec P 293268888c03422ca22674275564c902 [term 1 LEADER]: Becoming Leader. State: Replica: 293268888c03422ca22674275564c902, State: Running, Role: LEADER
I20250112 15:05:53.382283 18183 consensus_queue.cc:237] T 9202a8ac823243a59bcc9db743b299ec P 293268888c03422ca22674275564c902 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "293268888c03422ca22674275564c902" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 45763 } }
I20250112 15:05:53.382328 18177 raft_consensus.cc:513] T 2d3106e9e16349fd8737dc1fda0d1427 P 496d3e69ee014a9d9de3519ff7dbf4b3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "496d3e69ee014a9d9de3519ff7dbf4b3" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 43477 } }
I20250112 15:05:53.382383 18177 leader_election.cc:304] T 2d3106e9e16349fd8737dc1fda0d1427 P 496d3e69ee014a9d9de3519ff7dbf4b3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 496d3e69ee014a9d9de3519ff7dbf4b3; no voters:
I20250112 15:05:53.382485 18177 leader_election.cc:290] T 2d3106e9e16349fd8737dc1fda0d1427 P 496d3e69ee014a9d9de3519ff7dbf4b3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20250112 15:05:53.382503 18187 raft_consensus.cc:2798] T 2d3106e9e16349fd8737dc1fda0d1427 P 496d3e69ee014a9d9de3519ff7dbf4b3 [term 1 FOLLOWER]: Leader election won for term 1
I20250112 15:05:53.382597 18177 ts_tablet_manager.cc:1428] T 2d3106e9e16349fd8737dc1fda0d1427 P 496d3e69ee014a9d9de3519ff7dbf4b3: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20250112 15:05:53.382669 18178 tablet_bootstrap.cc:492] T c7297c6c590e42879b745e235f34748e P 293268888c03422ca22674275564c902: Bootstrap starting.
I20250112 15:05:53.382597 18187 raft_consensus.cc:695] T 2d3106e9e16349fd8737dc1fda0d1427 P 496d3e69ee014a9d9de3519ff7dbf4b3 [term 1 LEADER]: Becoming Leader. State: Replica: 496d3e69ee014a9d9de3519ff7dbf4b3, State: Running, Role: LEADER
I20250112 15:05:53.382752 18187 consensus_queue.cc:237] T 2d3106e9e16349fd8737dc1fda0d1427 P 496d3e69ee014a9d9de3519ff7dbf4b3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "496d3e69ee014a9d9de3519ff7dbf4b3" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 43477 } }
I20250112 15:05:53.383014 18178 tablet_bootstrap.cc:654] T c7297c6c590e42879b745e235f34748e P 293268888c03422ca22674275564c902: Neither blocks nor log segments found. Creating new log.
I20250112 15:05:53.383384 17681 catalog_manager.cc:5526] T 2d3106e9e16349fd8737dc1fda0d1427 P 496d3e69ee014a9d9de3519ff7dbf4b3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 496d3e69ee014a9d9de3519ff7dbf4b3 (127.17.60.194). New cstate: current_term: 1 leader_uuid: "496d3e69ee014a9d9de3519ff7dbf4b3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "496d3e69ee014a9d9de3519ff7dbf4b3" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 43477 } health_report { overall_health: HEALTHY } } }
I20250112 15:05:53.383698 17680 catalog_manager.cc:5526] T 9202a8ac823243a59bcc9db743b299ec P 293268888c03422ca22674275564c902 reported cstate change: term changed from 0 to 1, leader changed from <none> to 293268888c03422ca22674275564c902 (127.17.60.193). New cstate: current_term: 1 leader_uuid: "293268888c03422ca22674275564c902" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "293268888c03422ca22674275564c902" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 45763 } health_report { overall_health: HEALTHY } } }
I20250112 15:05:53.385331 18178 tablet_bootstrap.cc:492] T c7297c6c590e42879b745e235f34748e P 293268888c03422ca22674275564c902: No bootstrap required, opened a new log
I20250112 15:05:53.385397 18178 ts_tablet_manager.cc:1397] T c7297c6c590e42879b745e235f34748e P 293268888c03422ca22674275564c902: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20250112 15:05:53.385732 18178 raft_consensus.cc:357] T c7297c6c590e42879b745e235f34748e P 293268888c03422ca22674275564c902 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "293268888c03422ca22674275564c902" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 45763 } }
I20250112 15:05:53.385783 18178 raft_consensus.cc:383] T c7297c6c590e42879b745e235f34748e P 293268888c03422ca22674275564c902 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 15:05:53.385802 18178 raft_consensus.cc:738] T c7297c6c590e42879b745e235f34748e P 293268888c03422ca22674275564c902 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 293268888c03422ca22674275564c902, State: Initialized, Role: FOLLOWER
I20250112 15:05:53.385838 18178 consensus_queue.cc:260] T c7297c6c590e42879b745e235f34748e P 293268888c03422ca22674275564c902 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "293268888c03422ca22674275564c902" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 45763 } }
I20250112 15:05:53.385880 18178 raft_consensus.cc:397] T c7297c6c590e42879b745e235f34748e P 293268888c03422ca22674275564c902 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250112 15:05:53.385902 18178 raft_consensus.cc:491] T c7297c6c590e42879b745e235f34748e P 293268888c03422ca22674275564c902 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250112 15:05:53.385926 18178 raft_consensus.cc:3054] T c7297c6c590e42879b745e235f34748e P 293268888c03422ca22674275564c902 [term 0 FOLLOWER]: Advancing to term 1
I20250112 15:05:53.386335 18178 raft_consensus.cc:513] T c7297c6c590e42879b745e235f34748e P 293268888c03422ca22674275564c902 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "293268888c03422ca22674275564c902" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 45763 } }
I20250112 15:05:53.386384 18178 leader_election.cc:304] T c7297c6c590e42879b745e235f34748e P 293268888c03422ca22674275564c902 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 293268888c03422ca22674275564c902; no voters:
I20250112 15:05:53.386411 18178 leader_election.cc:290] T c7297c6c590e42879b745e235f34748e P 293268888c03422ca22674275564c902 [CANDIDATE]: Term 1 election: Requested vote from peers
I20250112 15:05:53.386453 18185 raft_consensus.cc:2798] T c7297c6c590e42879b745e235f34748e P 293268888c03422ca22674275564c902 [term 1 FOLLOWER]: Leader election won for term 1
I20250112 15:05:53.386483 18178 ts_tablet_manager.cc:1428] T c7297c6c590e42879b745e235f34748e P 293268888c03422ca22674275564c902: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20250112 15:05:53.386502 18185 raft_consensus.cc:695] T c7297c6c590e42879b745e235f34748e P 293268888c03422ca22674275564c902 [term 1 LEADER]: Becoming Leader. State: Replica: 293268888c03422ca22674275564c902, State: Running, Role: LEADER
I20250112 15:05:53.386538 18185 consensus_queue.cc:237] T c7297c6c590e42879b745e235f34748e P 293268888c03422ca22674275564c902 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "293268888c03422ca22674275564c902" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 45763 } }
I20250112 15:05:53.386701 18180 tablet_bootstrap.cc:492] T 6292bb4b6efd418a8f700a903e3af221 P 76b66f65b69e49b396532320a307bcd0: No bootstrap required, opened a new log
I20250112 15:05:53.386754 18180 ts_tablet_manager.cc:1397] T 6292bb4b6efd418a8f700a903e3af221 P 76b66f65b69e49b396532320a307bcd0: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.000s
I20250112 15:05:53.386899 18180 raft_consensus.cc:357] T 6292bb4b6efd418a8f700a903e3af221 P 76b66f65b69e49b396532320a307bcd0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "76b66f65b69e49b396532320a307bcd0" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 45367 } }
I20250112 15:05:53.386952 18180 raft_consensus.cc:383] T 6292bb4b6efd418a8f700a903e3af221 P 76b66f65b69e49b396532320a307bcd0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 15:05:53.386973 18180 raft_consensus.cc:738] T 6292bb4b6efd418a8f700a903e3af221 P 76b66f65b69e49b396532320a307bcd0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 76b66f65b69e49b396532320a307bcd0, State: Initialized, Role: FOLLOWER
I20250112 15:05:53.387014 18180 consensus_queue.cc:260] T 6292bb4b6efd418a8f700a903e3af221 P 76b66f65b69e49b396532320a307bcd0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "76b66f65b69e49b396532320a307bcd0" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 45367 } }
I20250112 15:05:53.387044 18180 raft_consensus.cc:397] T 6292bb4b6efd418a8f700a903e3af221 P 76b66f65b69e49b396532320a307bcd0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250112 15:05:53.387064 18180 raft_consensus.cc:491] T 6292bb4b6efd418a8f700a903e3af221 P 76b66f65b69e49b396532320a307bcd0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250112 15:05:53.387086 18180 raft_consensus.cc:3054] T 6292bb4b6efd418a8f700a903e3af221 P 76b66f65b69e49b396532320a307bcd0 [term 0 FOLLOWER]: Advancing to term 1
I20250112 15:05:53.387243 17680 catalog_manager.cc:5526] T c7297c6c590e42879b745e235f34748e P 293268888c03422ca22674275564c902 reported cstate change: term changed from 0 to 1, leader changed from <none> to 293268888c03422ca22674275564c902 (127.17.60.193). New cstate: current_term: 1 leader_uuid: "293268888c03422ca22674275564c902" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "293268888c03422ca22674275564c902" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 45763 } health_report { overall_health: HEALTHY } } }
I20250112 15:05:53.401186 18180 raft_consensus.cc:513] T 6292bb4b6efd418a8f700a903e3af221 P 76b66f65b69e49b396532320a307bcd0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "76b66f65b69e49b396532320a307bcd0" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 45367 } }
I20250112 15:05:53.401278 18180 leader_election.cc:304] T 6292bb4b6efd418a8f700a903e3af221 P 76b66f65b69e49b396532320a307bcd0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 76b66f65b69e49b396532320a307bcd0; no voters:
I20250112 15:05:53.401405 18180 leader_election.cc:290] T 6292bb4b6efd418a8f700a903e3af221 P 76b66f65b69e49b396532320a307bcd0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20250112 15:05:53.401464 18195 raft_consensus.cc:2798] T 6292bb4b6efd418a8f700a903e3af221 P 76b66f65b69e49b396532320a307bcd0 [term 1 FOLLOWER]: Leader election won for term 1
I20250112 15:05:53.401544 18180 ts_tablet_manager.cc:1428] T 6292bb4b6efd418a8f700a903e3af221 P 76b66f65b69e49b396532320a307bcd0: Time spent starting tablet: real 0.015s user 0.001s sys 0.000s
I20250112 15:05:53.401625 18195 raft_consensus.cc:695] T 6292bb4b6efd418a8f700a903e3af221 P 76b66f65b69e49b396532320a307bcd0 [term 1 LEADER]: Becoming Leader. State: Replica: 76b66f65b69e49b396532320a307bcd0, State: Running, Role: LEADER
I20250112 15:05:53.401686 18195 consensus_queue.cc:237] T 6292bb4b6efd418a8f700a903e3af221 P 76b66f65b69e49b396532320a307bcd0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "76b66f65b69e49b396532320a307bcd0" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 45367 } }
I20250112 15:05:53.402189 17680 catalog_manager.cc:5526] T 6292bb4b6efd418a8f700a903e3af221 P 76b66f65b69e49b396532320a307bcd0 reported cstate change: term changed from 0 to 1, leader changed from <none> to 76b66f65b69e49b396532320a307bcd0 (127.17.60.195). New cstate: current_term: 1 leader_uuid: "76b66f65b69e49b396532320a307bcd0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "76b66f65b69e49b396532320a307bcd0" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 45367 } health_report { overall_health: HEALTHY } } }
I20250112 15:05:53.403599 17651 tablet_server.cc:178] TabletServer@127.17.60.193:0 shutting down...
I20250112 15:05:53.406152 17651 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250112 15:05:53.406286 17651 tablet_replica.cc:331] T c7297c6c590e42879b745e235f34748e P 293268888c03422ca22674275564c902: stopping tablet replica
I20250112 15:05:53.406369 17651 raft_consensus.cc:2238] T c7297c6c590e42879b745e235f34748e P 293268888c03422ca22674275564c902 [term 1 LEADER]: Raft consensus shutting down.
I20250112 15:05:53.406419 17651 raft_consensus.cc:2267] T c7297c6c590e42879b745e235f34748e P 293268888c03422ca22674275564c902 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 15:05:53.406741 17651 tablet_replica.cc:331] T 9202a8ac823243a59bcc9db743b299ec P 293268888c03422ca22674275564c902: stopping tablet replica
I20250112 15:05:53.406780 17651 raft_consensus.cc:2238] T 9202a8ac823243a59bcc9db743b299ec P 293268888c03422ca22674275564c902 [term 1 LEADER]: Raft consensus shutting down.
I20250112 15:05:53.406801 17651 raft_consensus.cc:2267] T 9202a8ac823243a59bcc9db743b299ec P 293268888c03422ca22674275564c902 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 15:05:53.418898 17651 tablet_server.cc:195] TabletServer@127.17.60.193:0 shutdown complete.
I20250112 15:05:53.419849 17651 tablet_server.cc:178] TabletServer@127.17.60.194:0 shutting down...
I20250112 15:05:53.422613 17651 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250112 15:05:53.422734 17651 tablet_replica.cc:331] T 2d3106e9e16349fd8737dc1fda0d1427 P 496d3e69ee014a9d9de3519ff7dbf4b3: stopping tablet replica
I20250112 15:05:53.422793 17651 raft_consensus.cc:2238] T 2d3106e9e16349fd8737dc1fda0d1427 P 496d3e69ee014a9d9de3519ff7dbf4b3 [term 1 LEADER]: Raft consensus shutting down.
I20250112 15:05:53.422840 17651 raft_consensus.cc:2267] T 2d3106e9e16349fd8737dc1fda0d1427 P 496d3e69ee014a9d9de3519ff7dbf4b3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 15:05:53.434880 17651 tablet_server.cc:195] TabletServer@127.17.60.194:0 shutdown complete.
I20250112 15:05:53.435648 17651 tablet_server.cc:178] TabletServer@127.17.60.195:0 shutting down...
I20250112 15:05:53.437804 17651 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250112 15:05:53.437934 17651 tablet_replica.cc:331] T 6292bb4b6efd418a8f700a903e3af221 P 76b66f65b69e49b396532320a307bcd0: stopping tablet replica
I20250112 15:05:53.437995 17651 raft_consensus.cc:2238] T 6292bb4b6efd418a8f700a903e3af221 P 76b66f65b69e49b396532320a307bcd0 [term 1 LEADER]: Raft consensus shutting down.
I20250112 15:05:53.438045 17651 raft_consensus.cc:2267] T 6292bb4b6efd418a8f700a903e3af221 P 76b66f65b69e49b396532320a307bcd0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 15:05:53.449813 17651 tablet_server.cc:195] TabletServer@127.17.60.195:0 shutdown complete.
I20250112 15:05:53.450593 17651 master.cc:537] Master@127.17.60.254:37549 shutting down...
I20250112 15:05:53.452659 17651 raft_consensus.cc:2238] T 00000000000000000000000000000000 P 7a02131677894a6a94c870ffbab58fef [term 1 LEADER]: Raft consensus shutting down.
I20250112 15:05:53.452745 17651 raft_consensus.cc:2267] T 00000000000000000000000000000000 P 7a02131677894a6a94c870ffbab58fef [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 15:05:53.452771 17651 tablet_replica.cc:331] T 00000000000000000000000000000000 P 7a02131677894a6a94c870ffbab58fef: stopping tablet replica
I20250112 15:05:53.465065 17651 master.cc:559] Master@127.17.60.254:37549 shutdown complete.
I20250112 15:05:53.466356 17651 master.cc:537] Master@127.17.60.253:46031 shutting down...
I20250112 15:05:53.468432 17651 raft_consensus.cc:2238] T 00000000000000000000000000000000 P f420d939a3c94dbf91632b7e5fad770a [term 1 FOLLOWER]: Raft consensus shutting down.
I20250112 15:05:53.468492 17651 raft_consensus.cc:2267] T 00000000000000000000000000000000 P f420d939a3c94dbf91632b7e5fad770a [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 15:05:53.468523 17651 tablet_replica.cc:331] T 00000000000000000000000000000000 P f420d939a3c94dbf91632b7e5fad770a: stopping tablet replica
I20250112 15:05:53.480316 17651 master.cc:559] Master@127.17.60.253:46031 shutdown complete.
I20250112 15:05:53.481513 17651 master.cc:537] Master@127.17.60.252:46213 shutting down...
I20250112 15:05:53.483785 17651 raft_consensus.cc:2238] T 00000000000000000000000000000000 P ab6589ccae414a1e966e89830ebaca6b [term 1 FOLLOWER]: Raft consensus shutting down.
I20250112 15:05:53.483848 17651 raft_consensus.cc:2267] T 00000000000000000000000000000000 P ab6589ccae414a1e966e89830ebaca6b [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 15:05:53.483866 17651 tablet_replica.cc:331] T 00000000000000000000000000000000 P ab6589ccae414a1e966e89830ebaca6b: stopping tablet replica
I20250112 15:05:53.495905 17651 master.cc:559] Master@127.17.60.252:46213 shutdown complete.
[ OK ] AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing (2426 ms)
[ RUN ] AutoRebalancerTest.AutoRebalancingTurnOffAndOn
I20250112 15:05:53.500895 17651 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.17.60.254:45251
I20250112 15:05:53.501053 17651 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250112 15:05:53.502333 18201 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250112 15:05:53.503015 18202 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250112 15:05:53.503890 18204 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250112 15:05:53.504731 17651 server_base.cc:1034] running on GCE node
I20250112 15:05:53.504796 17651 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250112 15:05:53.504818 17651 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250112 15:05:53.504830 17651 hybrid_clock.cc:648] HybridClock initialized: now 1736694353504831 us; error 0 us; skew 500 ppm
I20250112 15:05:53.505393 17651 webserver.cc:458] Webserver started at http://127.17.60.254:41213/ using document root <none> and password file <none>
I20250112 15:05:53.505455 17651 fs_manager.cc:362] Metadata directory not provided
I20250112 15:05:53.505488 17651 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250112 15:05:53.505520 17651 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250112 15:05:53.505748 17651 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task95dLSD/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1736694351070047-17651-0/minicluster-data/master-0-root/instance:
uuid: "5d4bbf880f214236b21b3546f0554eef"
format_stamp: "Formatted at 2025-01-12 15:05:53 on dist-test-slave-b4nz"
I20250112 15:05:53.506525 17651 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20250112 15:05:53.506930 18209 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250112 15:05:53.507046 17651 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20250112 15:05:53.507086 17651 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task95dLSD/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1736694351070047-17651-0/minicluster-data/master-0-root
uuid: "5d4bbf880f214236b21b3546f0554eef"
format_stamp: "Formatted at 2025-01-12 15:05:53 on dist-test-slave-b4nz"
I20250112 15:05:53.507149 17651 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task95dLSD/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1736694351070047-17651-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-task95dLSD/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1736694351070047-17651-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-task95dLSD/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1736694351070047-17651-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250112 15:05:53.527408 17651 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250112 15:05:53.527609 17651 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250112 15:05:53.529981 17651 rpc_server.cc:307] RPC server started. Bound to: 127.17.60.254:45251
I20250112 15:05:53.532274 18261 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.17.60.254:45251 every 8 connection(s)
I20250112 15:05:53.532491 18262 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 15:05:53.533497 18262 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5d4bbf880f214236b21b3546f0554eef: Bootstrap starting.
I20250112 15:05:53.533843 18262 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5d4bbf880f214236b21b3546f0554eef: Neither blocks nor log segments found. Creating new log.
I20250112 15:05:53.534339 18262 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5d4bbf880f214236b21b3546f0554eef: No bootstrap required, opened a new log
I20250112 15:05:53.534452 18262 raft_consensus.cc:357] T 00000000000000000000000000000000 P 5d4bbf880f214236b21b3546f0554eef [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5d4bbf880f214236b21b3546f0554eef" member_type: VOTER }
I20250112 15:05:53.534500 18262 raft_consensus.cc:383] T 00000000000000000000000000000000 P 5d4bbf880f214236b21b3546f0554eef [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 15:05:53.534513 18262 raft_consensus.cc:738] T 00000000000000000000000000000000 P 5d4bbf880f214236b21b3546f0554eef [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5d4bbf880f214236b21b3546f0554eef, State: Initialized, Role: FOLLOWER
I20250112 15:05:53.534564 18262 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5d4bbf880f214236b21b3546f0554eef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5d4bbf880f214236b21b3546f0554eef" member_type: VOTER }
I20250112 15:05:53.534600 18262 raft_consensus.cc:397] T 00000000000000000000000000000000 P 5d4bbf880f214236b21b3546f0554eef [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250112 15:05:53.534628 18262 raft_consensus.cc:491] T 00000000000000000000000000000000 P 5d4bbf880f214236b21b3546f0554eef [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250112 15:05:53.534652 18262 raft_consensus.cc:3054] T 00000000000000000000000000000000 P 5d4bbf880f214236b21b3546f0554eef [term 0 FOLLOWER]: Advancing to term 1
I20250112 15:05:53.535086 18262 raft_consensus.cc:513] T 00000000000000000000000000000000 P 5d4bbf880f214236b21b3546f0554eef [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5d4bbf880f214236b21b3546f0554eef" member_type: VOTER }
I20250112 15:05:53.535146 18262 leader_election.cc:304] T 00000000000000000000000000000000 P 5d4bbf880f214236b21b3546f0554eef [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5d4bbf880f214236b21b3546f0554eef; no voters:
I20250112 15:05:53.535250 18262 leader_election.cc:290] T 00000000000000000000000000000000 P 5d4bbf880f214236b21b3546f0554eef [CANDIDATE]: Term 1 election: Requested vote from peers
I20250112 15:05:53.535300 18265 raft_consensus.cc:2798] T 00000000000000000000000000000000 P 5d4bbf880f214236b21b3546f0554eef [term 1 FOLLOWER]: Leader election won for term 1
I20250112 15:05:53.535374 18262 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5d4bbf880f214236b21b3546f0554eef [sys.catalog]: configured and running, proceeding with master startup.
I20250112 15:05:53.535426 18265 raft_consensus.cc:695] T 00000000000000000000000000000000 P 5d4bbf880f214236b21b3546f0554eef [term 1 LEADER]: Becoming Leader. State: Replica: 5d4bbf880f214236b21b3546f0554eef, State: Running, Role: LEADER
I20250112 15:05:53.535475 18265 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5d4bbf880f214236b21b3546f0554eef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5d4bbf880f214236b21b3546f0554eef" member_type: VOTER }
I20250112 15:05:53.535638 18266 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5d4bbf880f214236b21b3546f0554eef [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5d4bbf880f214236b21b3546f0554eef" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5d4bbf880f214236b21b3546f0554eef" member_type: VOTER } }
I20250112 15:05:53.535698 18266 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5d4bbf880f214236b21b3546f0554eef [sys.catalog]: This master's current role is: LEADER
I20250112 15:05:53.535918 18267 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5d4bbf880f214236b21b3546f0554eef [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5d4bbf880f214236b21b3546f0554eef. Latest consensus state: current_term: 1 leader_uuid: "5d4bbf880f214236b21b3546f0554eef" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5d4bbf880f214236b21b3546f0554eef" member_type: VOTER } }
I20250112 15:05:53.535939 18269 catalog_manager.cc:1476] Loading table and tablet metadata into memory...
I20250112 15:05:53.535975 18267 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5d4bbf880f214236b21b3546f0554eef [sys.catalog]: This master's current role is: LEADER
I20250112 15:05:53.536087 18269 catalog_manager.cc:1485] Initializing Kudu cluster ID...
I20250112 15:05:53.536592 18269 catalog_manager.cc:1348] Generated new cluster ID: b30ecac84b2c4246afef7e805c40cba0
I20250112 15:05:53.536643 18269 catalog_manager.cc:1496] Initializing Kudu internal certificate authority...
I20250112 15:05:53.536667 17651 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20250112 15:05:53.542972 18269 catalog_manager.cc:1371] Generated new certificate authority record
I20250112 15:05:53.543340 18269 catalog_manager.cc:1505] Loading token signing keys...
I20250112 15:05:53.553766 18269 catalog_manager.cc:5899] T 00000000000000000000000000000000 P 5d4bbf880f214236b21b3546f0554eef: Generated new TSK 0
I20250112 15:05:53.553846 18269 catalog_manager.cc:1515] Initializing in-progress tserver states...
I20250112 15:05:53.568601 17651 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250112 15:05:53.570397 18285 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250112 15:05:53.570757 18284 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250112 15:05:53.571370 18287 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250112 15:05:53.572052 17651 server_base.cc:1034] running on GCE node
I20250112 15:05:53.572144 17651 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250112 15:05:53.572170 17651 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250112 15:05:53.572189 17651 hybrid_clock.cc:648] HybridClock initialized: now 1736694353572189 us; error 0 us; skew 500 ppm
I20250112 15:05:53.572840 17651 webserver.cc:458] Webserver started at http://127.17.60.193:44385/ using document root <none> and password file <none>
I20250112 15:05:53.572929 17651 fs_manager.cc:362] Metadata directory not provided
I20250112 15:05:53.572993 17651 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250112 15:05:53.573045 17651 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250112 15:05:53.573325 17651 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task95dLSD/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1736694351070047-17651-0/minicluster-data/ts-0-root/instance:
uuid: "79518508ccd644b28675a5de1e7280e2"
format_stamp: "Formatted at 2025-01-12 15:05:53 on dist-test-slave-b4nz"
I20250112 15:05:53.574193 17651 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20250112 15:05:53.574571 18293 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250112 15:05:53.574683 17651 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20250112 15:05:53.574728 17651 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task95dLSD/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1736694351070047-17651-0/minicluster-data/ts-0-root
uuid: "79518508ccd644b28675a5de1e7280e2"
format_stamp: "Formatted at 2025-01-12 15:05:53 on dist-test-slave-b4nz"
I20250112 15:05:53.574774 17651 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task95dLSD/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1736694351070047-17651-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-task95dLSD/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1736694351070047-17651-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-task95dLSD/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1736694351070047-17651-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250112 15:05:53.588157 17651 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250112 15:05:53.588333 17651 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250112 15:05:53.588472 17651 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250112 15:05:53.588716 17651 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250112 15:05:53.588745 17651 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250112 15:05:53.588778 17651 ts_tablet_manager.cc:610] Registered 0 tablets
I20250112 15:05:53.588804 17651 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250112 15:05:53.591360 17651 rpc_server.cc:307] RPC server started. Bound to: 127.17.60.193:38531
I20250112 15:05:53.591423 18356 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.17.60.193:38531 every 8 connection(s)
I20250112 15:05:53.591719 17651 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250112 15:05:53.593151 18361 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250112 15:05:53.593734 18364 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250112 15:05:53.594018 18362 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250112 15:05:53.594888 18357 heartbeater.cc:346] Connected to a master server at 127.17.60.254:45251
I20250112 15:05:53.594942 18357 heartbeater.cc:463] Registering TS with master...
I20250112 15:05:53.594957 17651 server_base.cc:1034] running on GCE node
I20250112 15:05:53.595069 18357 heartbeater.cc:510] Master 127.17.60.254:45251 requested a full tablet report, sending...
I20250112 15:05:53.595109 17651 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250112 15:05:53.595153 17651 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250112 15:05:53.595170 17651 hybrid_clock.cc:648] HybridClock initialized: now 1736694353595170 us; error 0 us; skew 500 ppm
I20250112 15:05:53.595314 18226 ts_manager.cc:194] Registered new tserver with Master: 79518508ccd644b28675a5de1e7280e2 (127.17.60.193:38531)
I20250112 15:05:53.595788 17651 webserver.cc:458] Webserver started at http://127.17.60.194:45877/ using document root <none> and password file <none>
I20250112 15:05:53.595851 18226 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:51720
I20250112 15:05:53.595865 17651 fs_manager.cc:362] Metadata directory not provided
I20250112 15:05:53.595934 17651 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250112 15:05:53.595978 17651 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250112 15:05:53.596307 17651 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task95dLSD/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1736694351070047-17651-0/minicluster-data/ts-1-root/instance:
uuid: "aeb32dce70ca494ca326c528e640c9ce"
format_stamp: "Formatted at 2025-01-12 15:05:53 on dist-test-slave-b4nz"
I20250112 15:05:53.597334 17651 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20250112 15:05:53.597767 18369 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250112 15:05:53.597919 17651 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20250112 15:05:53.597975 17651 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task95dLSD/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1736694351070047-17651-0/minicluster-data/ts-1-root
uuid: "aeb32dce70ca494ca326c528e640c9ce"
format_stamp: "Formatted at 2025-01-12 15:05:53 on dist-test-slave-b4nz"
I20250112 15:05:53.598017 17651 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task95dLSD/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1736694351070047-17651-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-task95dLSD/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1736694351070047-17651-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-task95dLSD/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1736694351070047-17651-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250112 15:05:53.604789 17651 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250112 15:05:53.604954 17651 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250112 15:05:53.605086 17651 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250112 15:05:53.605243 17651 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250112 15:05:53.605268 17651 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250112 15:05:53.605288 17651 ts_tablet_manager.cc:610] Registered 0 tablets
I20250112 15:05:53.605315 17651 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250112 15:05:53.607889 17651 rpc_server.cc:307] RPC server started. Bound to: 127.17.60.194:35351
I20250112 15:05:53.607939 18432 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.17.60.194:35351 every 8 connection(s)
I20250112 15:05:53.608201 17651 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250112 15:05:53.610214 18439 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250112 15:05:53.610780 18437 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250112 15:05:53.610796 18436 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250112 15:05:53.610983 18433 heartbeater.cc:346] Connected to a master server at 127.17.60.254:45251
I20250112 15:05:53.611023 18433 heartbeater.cc:463] Registering TS with master...
I20250112 15:05:53.611100 18433 heartbeater.cc:510] Master 127.17.60.254:45251 requested a full tablet report, sending...
I20250112 15:05:53.611308 18226 ts_manager.cc:194] Registered new tserver with Master: aeb32dce70ca494ca326c528e640c9ce (127.17.60.194:35351)
I20250112 15:05:53.611502 17651 server_base.cc:1034] running on GCE node
I20250112 15:05:53.611608 17651 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250112 15:05:53.611634 17651 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250112 15:05:53.611645 17651 hybrid_clock.cc:648] HybridClock initialized: now 1736694353611645 us; error 0 us; skew 500 ppm
I20250112 15:05:53.611788 18226 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:51736
I20250112 15:05:53.612221 17651 webserver.cc:458] Webserver started at http://127.17.60.195:33669/ using document root <none> and password file <none>
I20250112 15:05:53.612299 17651 fs_manager.cc:362] Metadata directory not provided
I20250112 15:05:53.612344 17651 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250112 15:05:53.612388 17651 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250112 15:05:53.612670 17651 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task95dLSD/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1736694351070047-17651-0/minicluster-data/ts-2-root/instance:
uuid: "90a0b30f5377440b8d72889b9d2f0a70"
format_stamp: "Formatted at 2025-01-12 15:05:53 on dist-test-slave-b4nz"
I20250112 15:05:53.613533 17651 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20250112 15:05:53.613953 18444 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250112 15:05:53.614090 17651 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20250112 15:05:53.614145 17651 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task95dLSD/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1736694351070047-17651-0/minicluster-data/ts-2-root
uuid: "90a0b30f5377440b8d72889b9d2f0a70"
format_stamp: "Formatted at 2025-01-12 15:05:53 on dist-test-slave-b4nz"
I20250112 15:05:53.614190 17651 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task95dLSD/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1736694351070047-17651-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-task95dLSD/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1736694351070047-17651-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-task95dLSD/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1736694351070047-17651-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250112 15:05:53.621296 17651 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250112 15:05:53.621483 17651 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250112 15:05:53.621618 17651 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250112 15:05:53.621786 17651 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250112 15:05:53.621817 17651 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250112 15:05:53.621850 17651 ts_tablet_manager.cc:610] Registered 0 tablets
I20250112 15:05:53.621876 17651 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250112 15:05:53.625025 17651 rpc_server.cc:307] RPC server started. Bound to: 127.17.60.195:35603
I20250112 15:05:53.625067 18507 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.17.60.195:35603 every 8 connection(s)
I20250112 15:05:53.627712 18508 heartbeater.cc:346] Connected to a master server at 127.17.60.254:45251
I20250112 15:05:53.627764 18508 heartbeater.cc:463] Registering TS with master...
I20250112 15:05:53.627858 18508 heartbeater.cc:510] Master 127.17.60.254:45251 requested a full tablet report, sending...
I20250112 15:05:53.628039 18226 ts_manager.cc:194] Registered new tserver with Master: 90a0b30f5377440b8d72889b9d2f0a70 (127.17.60.195:35603)
I20250112 15:05:53.628350 18226 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:51740
I20250112 15:05:53.628492 17651 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.00330377s
I20250112 15:05:54.596592 18357 heartbeater.cc:502] Master 127.17.60.254:45251 was elected leader, sending a full tablet report...
I20250112 15:05:54.612413 18433 heartbeater.cc:502] Master 127.17.60.254:45251 was elected leader, sending a full tablet report...
I20250112 15:05:54.628860 18508 heartbeater.cc:502] Master 127.17.60.254:45251 was elected leader, sending a full tablet report...
I20250112 15:05:54.653949 17651 test_util.cc:274] Using random seed: -1441287431
I20250112 15:05:54.658854 18226 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51744:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "\004\001\000\377\377\377\017\004\001\000\376\377\377\037\004\001\000\375\377\377/\004\001\000\374\377\377?\004\001\000\373\377\377O\004\001\000\372\377\377_\004\001\000\371\377\377o""\004\001\000\377\377\377\017\004\001\000\376\377\377\037\004\001\000\375\377\377/\004\001\000\374\377\377?\004\001\000\373\377\377O\004\001\000\372\377\377_\004\001\000\371\377\377o"
indirect_data: """"
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 15:05:54.659046 18226 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-workload in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 15:05:54.664073 18395 tablet_service.cc:1467] Processing CreateTablet for tablet 9032f9edb5a746729802355e7d212c95 (DEFAULT_TABLE table=test-workload [id=98ba48bb661c4e378ec9a1bd2e16b275]), partition=RANGE (key) PARTITION 536870910 <= VALUES < 805306365
I20250112 15:05:54.664160 18397 tablet_service.cc:1467] Processing CreateTablet for tablet e311741568d04cca81089b8bad66b805 (DEFAULT_TABLE table=test-workload [id=98ba48bb661c4e378ec9a1bd2e16b275]), partition=RANGE (key) PARTITION VALUES < 268435455
I20250112 15:05:54.664232 18395 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9032f9edb5a746729802355e7d212c95. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 15:05:54.664301 18397 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e311741568d04cca81089b8bad66b805. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 15:05:54.664597 18472 tablet_service.cc:1467] Processing CreateTablet for tablet e311741568d04cca81089b8bad66b805 (DEFAULT_TABLE table=test-workload [id=98ba48bb661c4e378ec9a1bd2e16b275]), partition=RANGE (key) PARTITION VALUES < 268435455
I20250112 15:05:54.664637 18321 tablet_service.cc:1467] Processing CreateTablet for tablet e311741568d04cca81089b8bad66b805 (DEFAULT_TABLE table=test-workload [id=98ba48bb661c4e378ec9a1bd2e16b275]), partition=RANGE (key) PARTITION VALUES < 268435455
I20250112 15:05:54.664698 18472 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e311741568d04cca81089b8bad66b805. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 15:05:54.664746 18321 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e311741568d04cca81089b8bad66b805. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 15:05:54.665884 18528 tablet_bootstrap.cc:492] T e311741568d04cca81089b8bad66b805 P 79518508ccd644b28675a5de1e7280e2: Bootstrap starting.
I20250112 15:05:54.665941 18394 tablet_service.cc:1467] Processing CreateTablet for tablet cf6c409b42d842cfa25a995f95544e5e (DEFAULT_TABLE table=test-workload [id=98ba48bb661c4e378ec9a1bd2e16b275]), partition=RANGE (key) PARTITION 805306365 <= VALUES < 1073741820
I20250112 15:05:54.666062 18318 tablet_service.cc:1467] Processing CreateTablet for tablet cf6c409b42d842cfa25a995f95544e5e (DEFAULT_TABLE table=test-workload [id=98ba48bb661c4e378ec9a1bd2e16b275]), partition=RANGE (key) PARTITION 805306365 <= VALUES < 1073741820
I20250112 15:05:54.666079 18394 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cf6c409b42d842cfa25a995f95544e5e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 15:05:54.666142 18318 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cf6c409b42d842cfa25a995f95544e5e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 15:05:54.666261 18529 tablet_bootstrap.cc:492] T e311741568d04cca81089b8bad66b805 P aeb32dce70ca494ca326c528e640c9ce: Bootstrap starting.
I20250112 15:05:54.666489 18528 tablet_bootstrap.cc:654] T e311741568d04cca81089b8bad66b805 P 79518508ccd644b28675a5de1e7280e2: Neither blocks nor log segments found. Creating new log.
I20250112 15:05:54.666772 18529 tablet_bootstrap.cc:654] T e311741568d04cca81089b8bad66b805 P aeb32dce70ca494ca326c528e640c9ce: Neither blocks nor log segments found. Creating new log.
I20250112 15:05:54.667376 18528 tablet_bootstrap.cc:492] T e311741568d04cca81089b8bad66b805 P 79518508ccd644b28675a5de1e7280e2: No bootstrap required, opened a new log
I20250112 15:05:54.667414 18317 tablet_service.cc:1467] Processing CreateTablet for tablet b3b76094dfae41f38c01fd10b38c7a28 (DEFAULT_TABLE table=test-workload [id=98ba48bb661c4e378ec9a1bd2e16b275]), partition=RANGE (key) PARTITION 1073741820 <= VALUES < 1342177275
I20250112 15:05:54.667433 18528 ts_tablet_manager.cc:1397] T e311741568d04cca81089b8bad66b805 P 79518508ccd644b28675a5de1e7280e2: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20250112 15:05:54.667503 18317 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b3b76094dfae41f38c01fd10b38c7a28. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 15:05:54.667717 18316 tablet_service.cc:1467] Processing CreateTablet for tablet 2319a98d89eb4234aa86cc3930853fce (DEFAULT_TABLE table=test-workload [id=98ba48bb661c4e378ec9a1bd2e16b275]), partition=RANGE (key) PARTITION 1342177275 <= VALUES < 1610612730
I20250112 15:05:54.667552 18393 tablet_service.cc:1467] Processing CreateTablet for tablet b3b76094dfae41f38c01fd10b38c7a28 (DEFAULT_TABLE table=test-workload [id=98ba48bb661c4e378ec9a1bd2e16b275]), partition=RANGE (key) PARTITION 1073741820 <= VALUES < 1342177275
I20250112 15:05:54.667833 18316 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2319a98d89eb4234aa86cc3930853fce. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 15:05:54.667923 18470 tablet_service.cc:1467] Processing CreateTablet for tablet 9032f9edb5a746729802355e7d212c95 (DEFAULT_TABLE table=test-workload [id=98ba48bb661c4e378ec9a1bd2e16b275]), partition=RANGE (key) PARTITION 536870910 <= VALUES < 805306365
I20250112 15:05:54.667944 18529 tablet_bootstrap.cc:492] T e311741568d04cca81089b8bad66b805 P aeb32dce70ca494ca326c528e640c9ce: No bootstrap required, opened a new log
I20250112 15:05:54.668011 18470 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9032f9edb5a746729802355e7d212c95. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 15:05:54.668038 18529 ts_tablet_manager.cc:1397] T e311741568d04cca81089b8bad66b805 P aeb32dce70ca494ca326c528e640c9ce: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20250112 15:05:54.668177 18529 raft_consensus.cc:357] T e311741568d04cca81089b8bad66b805 P aeb32dce70ca494ca326c528e640c9ce [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90a0b30f5377440b8d72889b9d2f0a70" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 35603 } } peers { permanent_uuid: "79518508ccd644b28675a5de1e7280e2" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 38531 } } peers { permanent_uuid: "aeb32dce70ca494ca326c528e640c9ce" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 35351 } }
I20250112 15:05:54.668254 18529 raft_consensus.cc:383] T e311741568d04cca81089b8bad66b805 P aeb32dce70ca494ca326c528e640c9ce [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 15:05:54.668274 18529 raft_consensus.cc:738] T e311741568d04cca81089b8bad66b805 P aeb32dce70ca494ca326c528e640c9ce [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aeb32dce70ca494ca326c528e640c9ce, State: Initialized, Role: FOLLOWER
I20250112 15:05:54.668319 18529 consensus_queue.cc:260] T e311741568d04cca81089b8bad66b805 P aeb32dce70ca494ca326c528e640c9ce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90a0b30f5377440b8d72889b9d2f0a70" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 35603 } } peers { permanent_uuid: "79518508ccd644b28675a5de1e7280e2" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 38531 } } peers { permanent_uuid: "aeb32dce70ca494ca326c528e640c9ce" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 35351 } }
I20250112 15:05:54.668489 18529 ts_tablet_manager.cc:1428] T e311741568d04cca81089b8bad66b805 P aeb32dce70ca494ca326c528e640c9ce: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20250112 15:05:54.668545 18529 tablet_bootstrap.cc:492] T cf6c409b42d842cfa25a995f95544e5e P aeb32dce70ca494ca326c528e640c9ce: Bootstrap starting.
I20250112 15:05:54.668650 18315 tablet_service.cc:1467] Processing CreateTablet for tablet 45204e59d67b4abe9ed00311ca03b73d (DEFAULT_TABLE table=test-workload [id=98ba48bb661c4e378ec9a1bd2e16b275]), partition=RANGE (key) PARTITION 1610612730 <= VALUES < 1879048185
I20250112 15:05:54.668756 18315 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 45204e59d67b4abe9ed00311ca03b73d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 15:05:54.669157 18469 tablet_service.cc:1467] Processing CreateTablet for tablet cf6c409b42d842cfa25a995f95544e5e (DEFAULT_TABLE table=test-workload [id=98ba48bb661c4e378ec9a1bd2e16b275]), partition=RANGE (key) PARTITION 805306365 <= VALUES < 1073741820
I20250112 15:05:54.669240 18469 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cf6c409b42d842cfa25a995f95544e5e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 15:05:54.669888 18314 tablet_service.cc:1467] Processing CreateTablet for tablet 4e45a9f365b344c2aa0703adea508eeb (DEFAULT_TABLE table=test-workload [id=98ba48bb661c4e378ec9a1bd2e16b275]), partition=RANGE (key) PARTITION 1879048185 <= VALUES
I20250112 15:05:54.669974 18314 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4e45a9f365b344c2aa0703adea508eeb. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 15:05:54.667843 18390 tablet_service.cc:1467] Processing CreateTablet for tablet 4e45a9f365b344c2aa0703adea508eeb (DEFAULT_TABLE table=test-workload [id=98ba48bb661c4e378ec9a1bd2e16b275]), partition=RANGE (key) PARTITION 1879048185 <= VALUES
I20250112 15:05:54.667868 18393 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b3b76094dfae41f38c01fd10b38c7a28. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 15:05:54.667862 18528 raft_consensus.cc:357] T e311741568d04cca81089b8bad66b805 P 79518508ccd644b28675a5de1e7280e2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90a0b30f5377440b8d72889b9d2f0a70" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 35603 } } peers { permanent_uuid: "79518508ccd644b28675a5de1e7280e2" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 38531 } } peers { permanent_uuid: "aeb32dce70ca494ca326c528e640c9ce" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 35351 } }
I20250112 15:05:54.670248 18390 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4e45a9f365b344c2aa0703adea508eeb. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 15:05:54.670266 18528 raft_consensus.cc:383] T e311741568d04cca81089b8bad66b805 P 79518508ccd644b28675a5de1e7280e2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 15:05:54.670291 18528 raft_consensus.cc:738] T e311741568d04cca81089b8bad66b805 P 79518508ccd644b28675a5de1e7280e2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 79518508ccd644b28675a5de1e7280e2, State: Initialized, Role: FOLLOWER
I20250112 15:05:54.670325 18528 consensus_queue.cc:260] T e311741568d04cca81089b8bad66b805 P 79518508ccd644b28675a5de1e7280e2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90a0b30f5377440b8d72889b9d2f0a70" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 35603 } } peers { permanent_uuid: "79518508ccd644b28675a5de1e7280e2" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 38531 } } peers { permanent_uuid: "aeb32dce70ca494ca326c528e640c9ce" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 35351 } }
I20250112 15:05:54.670473 18528 ts_tablet_manager.cc:1428] T e311741568d04cca81089b8bad66b805 P 79518508ccd644b28675a5de1e7280e2: Time spent starting tablet: real 0.003s user 0.000s sys 0.000s
I20250112 15:05:54.670535 18528 tablet_bootstrap.cc:492] T cf6c409b42d842cfa25a995f95544e5e P 79518508ccd644b28675a5de1e7280e2: Bootstrap starting.
I20250112 15:05:54.670956 18528 tablet_bootstrap.cc:654] T cf6c409b42d842cfa25a995f95544e5e P 79518508ccd644b28675a5de1e7280e2: Neither blocks nor log segments found. Creating new log.
I20250112 15:05:54.665900 18320 tablet_service.cc:1467] Processing CreateTablet for tablet ded9df74a71c4d059719cb5401e11822 (DEFAULT_TABLE table=test-workload [id=98ba48bb661c4e378ec9a1bd2e16b275]), partition=RANGE (key) PARTITION 268435455 <= VALUES < 536870910
I20250112 15:05:54.671288 18320 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ded9df74a71c4d059719cb5401e11822. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 15:05:54.667884 18471 tablet_service.cc:1467] Processing CreateTablet for tablet ded9df74a71c4d059719cb5401e11822 (DEFAULT_TABLE table=test-workload [id=98ba48bb661c4e378ec9a1bd2e16b275]), partition=RANGE (key) PARTITION 268435455 <= VALUES < 536870910
I20250112 15:05:54.671568 18471 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ded9df74a71c4d059719cb5401e11822. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 15:05:54.671944 18529 tablet_bootstrap.cc:654] T cf6c409b42d842cfa25a995f95544e5e P aeb32dce70ca494ca326c528e640c9ce: Neither blocks nor log segments found. Creating new log.
I20250112 15:05:54.667708 18392 tablet_service.cc:1467] Processing CreateTablet for tablet 2319a98d89eb4234aa86cc3930853fce (DEFAULT_TABLE table=test-workload [id=98ba48bb661c4e378ec9a1bd2e16b275]), partition=RANGE (key) PARTITION 1342177275 <= VALUES < 1610612730
I20250112 15:05:54.672493 18392 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2319a98d89eb4234aa86cc3930853fce. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 15:05:54.672600 18471 tablet_service.cc:1467] Processing CreateTablet for tablet b3b76094dfae41f38c01fd10b38c7a28 (DEFAULT_TABLE table=test-workload [id=98ba48bb661c4e378ec9a1bd2e16b275]), partition=RANGE (key) PARTITION 1073741820 <= VALUES < 1342177275
I20250112 15:05:54.672675 18471 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b3b76094dfae41f38c01fd10b38c7a28. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 15:05:54.672783 18529 tablet_bootstrap.cc:492] T cf6c409b42d842cfa25a995f95544e5e P aeb32dce70ca494ca326c528e640c9ce: No bootstrap required, opened a new log
I20250112 15:05:54.672863 18529 ts_tablet_manager.cc:1397] T cf6c409b42d842cfa25a995f95544e5e P aeb32dce70ca494ca326c528e640c9ce: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20250112 15:05:54.673141 18529 raft_consensus.cc:357] T cf6c409b42d842cfa25a995f95544e5e P aeb32dce70ca494ca326c528e640c9ce [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79518508ccd644b28675a5de1e7280e2" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 38531 } } peers { permanent_uuid: "aeb32dce70ca494ca326c528e640c9ce" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 35351 } } peers { permanent_uuid: "90a0b30f5377440b8d72889b9d2f0a70" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 35603 } }
I20250112 15:05:54.673291 18529 raft_consensus.cc:383] T cf6c409b42d842cfa25a995f95544e5e P aeb32dce70ca494ca326c528e640c9ce [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 15:05:54.673357 18529 raft_consensus.cc:738] T cf6c409b42d842cfa25a995f95544e5e P aeb32dce70ca494ca326c528e640c9ce [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aeb32dce70ca494ca326c528e640c9ce, State: Initialized, Role: FOLLOWER
I20250112 15:05:54.673439 18529 consensus_queue.cc:260] T cf6c409b42d842cfa25a995f95544e5e P aeb32dce70ca494ca326c528e640c9ce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79518508ccd644b28675a5de1e7280e2" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 38531 } } peers { permanent_uuid: "aeb32dce70ca494ca326c528e640c9ce" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 35351 } } peers { permanent_uuid: "90a0b30f5377440b8d72889b9d2f0a70" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 35603 } }
I20250112 15:05:54.673576 18529 ts_tablet_manager.cc:1428] T cf6c409b42d842cfa25a995f95544e5e P aeb32dce70ca494ca326c528e640c9ce: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20250112 15:05:54.665891 18396 tablet_service.cc:1467] Processing CreateTablet for tablet ded9df74a71c4d059719cb5401e11822 (DEFAULT_TABLE table=test-workload [id=98ba48bb661c4e378ec9a1bd2e16b275]), partition=RANGE (key) PARTITION 268435455 <= VALUES < 536870910
I20250112 15:05:54.673682 18472 tablet_service.cc:1467] Processing CreateTablet for tablet 4e45a9f365b344c2aa0703adea508eeb (DEFAULT_TABLE table=test-workload [id=98ba48bb661c4e378ec9a1bd2e16b275]), partition=RANGE (key) PARTITION 1879048185 <= VALUES
I20250112 15:05:54.673753 18396 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ded9df74a71c4d059719cb5401e11822. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 15:05:54.673801 18472 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4e45a9f365b344c2aa0703adea508eeb. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 15:05:54.665972 18319 tablet_service.cc:1467] Processing CreateTablet for tablet 9032f9edb5a746729802355e7d212c95 (DEFAULT_TABLE table=test-workload [id=98ba48bb661c4e378ec9a1bd2e16b275]), partition=RANGE (key) PARTITION 536870910 <= VALUES < 805306365
I20250112 15:05:54.673674 18529 tablet_bootstrap.cc:492] T 4e45a9f365b344c2aa0703adea508eeb P aeb32dce70ca494ca326c528e640c9ce: Bootstrap starting.
I20250112 15:05:54.674875 18319 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9032f9edb5a746729802355e7d212c95. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 15:05:54.667770 18391 tablet_service.cc:1467] Processing CreateTablet for tablet 45204e59d67b4abe9ed00311ca03b73d (DEFAULT_TABLE table=test-workload [id=98ba48bb661c4e378ec9a1bd2e16b275]), partition=RANGE (key) PARTITION 1610612730 <= VALUES < 1879048185
I20250112 15:05:54.675179 18391 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 45204e59d67b4abe9ed00311ca03b73d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 15:05:54.675328 18529 tablet_bootstrap.cc:654] T 4e45a9f365b344c2aa0703adea508eeb P aeb32dce70ca494ca326c528e640c9ce: Neither blocks nor log segments found. Creating new log.
I20250112 15:05:54.675954 18529 tablet_bootstrap.cc:492] T 4e45a9f365b344c2aa0703adea508eeb P aeb32dce70ca494ca326c528e640c9ce: No bootstrap required, opened a new log
I20250112 15:05:54.676003 18529 ts_tablet_manager.cc:1397] T 4e45a9f365b344c2aa0703adea508eeb P aeb32dce70ca494ca326c528e640c9ce: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20250112 15:05:54.676143 18529 raft_consensus.cc:357] T 4e45a9f365b344c2aa0703adea508eeb P aeb32dce70ca494ca326c528e640c9ce [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79518508ccd644b28675a5de1e7280e2" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 38531 } } peers { permanent_uuid: "aeb32dce70ca494ca326c528e640c9ce" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 35351 } } peers { permanent_uuid: "90a0b30f5377440b8d72889b9d2f0a70" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 35603 } }
I20250112 15:05:54.676210 18529 raft_consensus.cc:383] T 4e45a9f365b344c2aa0703adea508eeb P aeb32dce70ca494ca326c528e640c9ce [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 15:05:54.676229 18529 raft_consensus.cc:738] T 4e45a9f365b344c2aa0703adea508eeb P aeb32dce70ca494ca326c528e640c9ce [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aeb32dce70ca494ca326c528e640c9ce, State: Initialized, Role: FOLLOWER
I20250112 15:05:54.676278 18529 consensus_queue.cc:260] T 4e45a9f365b344c2aa0703adea508eeb P aeb32dce70ca494ca326c528e640c9ce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79518508ccd644b28675a5de1e7280e2" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 38531 } } peers { permanent_uuid: "aeb32dce70ca494ca326c528e640c9ce" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 35351 } } peers { permanent_uuid: "90a0b30f5377440b8d72889b9d2f0a70" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 35603 } }
I20250112 15:05:54.676424 18529 ts_tablet_manager.cc:1428] T 4e45a9f365b344c2aa0703adea508eeb P aeb32dce70ca494ca326c528e640c9ce: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20250112 15:05:54.676479 18529 tablet_bootstrap.cc:492] T b3b76094dfae41f38c01fd10b38c7a28 P aeb32dce70ca494ca326c528e640c9ce: Bootstrap starting.
I20250112 15:05:54.676913 18529 tablet_bootstrap.cc:654] T b3b76094dfae41f38c01fd10b38c7a28 P aeb32dce70ca494ca326c528e640c9ce: Neither blocks nor log segments found. Creating new log.
I20250112 15:05:54.677505 18529 tablet_bootstrap.cc:492] T b3b76094dfae41f38c01fd10b38c7a28 P aeb32dce70ca494ca326c528e640c9ce: No bootstrap required, opened a new log
I20250112 15:05:54.677533 18529 ts_tablet_manager.cc:1397] T b3b76094dfae41f38c01fd10b38c7a28 P aeb32dce70ca494ca326c528e640c9ce: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20250112 15:05:54.677631 18529 raft_consensus.cc:357] T b3b76094dfae41f38c01fd10b38c7a28 P aeb32dce70ca494ca326c528e640c9ce [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79518508ccd644b28675a5de1e7280e2" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 38531 } } peers { permanent_uuid: "aeb32dce70ca494ca326c528e640c9ce" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 35351 } } peers { permanent_uuid: "90a0b30f5377440b8d72889b9d2f0a70" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 35603 } }
I20250112 15:05:54.677685 18529 raft_consensus.cc:383] T b3b76094dfae41f38c01fd10b38c7a28 P aeb32dce70ca494ca326c528e640c9ce [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 15:05:54.677707 18529 raft_consensus.cc:738] T b3b76094dfae41f38c01fd10b38c7a28 P aeb32dce70ca494ca326c528e640c9ce [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aeb32dce70ca494ca326c528e640c9ce, State: Initialized, Role: FOLLOWER
I20250112 15:05:54.677719 18469 tablet_service.cc:1467] Processing CreateTablet for tablet 2319a98d89eb4234aa86cc3930853fce (DEFAULT_TABLE table=test-workload [id=98ba48bb661c4e378ec9a1bd2e16b275]), partition=RANGE (key) PARTITION 1342177275 <= VALUES < 1610612730
I20250112 15:05:54.677742 18529 consensus_queue.cc:260] T b3b76094dfae41f38c01fd10b38c7a28 P aeb32dce70ca494ca326c528e640c9ce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79518508ccd644b28675a5de1e7280e2" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 38531 } } peers { permanent_uuid: "aeb32dce70ca494ca326c528e640c9ce" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 35351 } } peers { permanent_uuid: "90a0b30f5377440b8d72889b9d2f0a70" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 35603 } }
I20250112 15:05:54.677814 18469 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2319a98d89eb4234aa86cc3930853fce. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 15:05:54.677832 18529 ts_tablet_manager.cc:1428] T b3b76094dfae41f38c01fd10b38c7a28 P aeb32dce70ca494ca326c528e640c9ce: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20250112 15:05:54.677878 18529 tablet_bootstrap.cc:492] T 2319a98d89eb4234aa86cc3930853fce P aeb32dce70ca494ca326c528e640c9ce: Bootstrap starting.
I20250112 15:05:54.678244 18529 tablet_bootstrap.cc:654] T 2319a98d89eb4234aa86cc3930853fce P aeb32dce70ca494ca326c528e640c9ce: Neither blocks nor log segments found. Creating new log.
I20250112 15:05:54.678896 18529 tablet_bootstrap.cc:492] T 2319a98d89eb4234aa86cc3930853fce P aeb32dce70ca494ca326c528e640c9ce: No bootstrap required, opened a new log
I20250112 15:05:54.678949 18529 ts_tablet_manager.cc:1397] T 2319a98d89eb4234aa86cc3930853fce P aeb32dce70ca494ca326c528e640c9ce: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20250112 15:05:54.678982 18470 tablet_service.cc:1467] Processing CreateTablet for tablet 45204e59d67b4abe9ed00311ca03b73d (DEFAULT_TABLE table=test-workload [id=98ba48bb661c4e378ec9a1bd2e16b275]), partition=RANGE (key) PARTITION 1610612730 <= VALUES < 1879048185
I20250112 15:05:54.679069 18470 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 45204e59d67b4abe9ed00311ca03b73d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 15:05:54.679076 18529 raft_consensus.cc:357] T 2319a98d89eb4234aa86cc3930853fce P aeb32dce70ca494ca326c528e640c9ce [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79518508ccd644b28675a5de1e7280e2" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 38531 } } peers { permanent_uuid: "aeb32dce70ca494ca326c528e640c9ce" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 35351 } } peers { permanent_uuid: "90a0b30f5377440b8d72889b9d2f0a70" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 35603 } }
I20250112 15:05:54.679158 18529 raft_consensus.cc:383] T 2319a98d89eb4234aa86cc3930853fce P aeb32dce70ca494ca326c528e640c9ce [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 15:05:54.679183 18529 raft_consensus.cc:738] T 2319a98d89eb4234aa86cc3930853fce P aeb32dce70ca494ca326c528e640c9ce [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aeb32dce70ca494ca326c528e640c9ce, State: Initialized, Role: FOLLOWER
I20250112 15:05:54.679226 18529 consensus_queue.cc:260] T 2319a98d89eb4234aa86cc3930853fce P aeb32dce70ca494ca326c528e640c9ce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79518508ccd644b28675a5de1e7280e2" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 38531 } } peers { permanent_uuid: "aeb32dce70ca494ca326c528e640c9ce" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 35351 } } peers { permanent_uuid: "90a0b30f5377440b8d72889b9d2f0a70" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 35603 } }
I20250112 15:05:54.679298 18529 ts_tablet_manager.cc:1428] T 2319a98d89eb4234aa86cc3930853fce P aeb32dce70ca494ca326c528e640c9ce: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20250112 15:05:54.679342 18529 tablet_bootstrap.cc:492] T 9032f9edb5a746729802355e7d212c95 P aeb32dce70ca494ca326c528e640c9ce: Bootstrap starting.
I20250112 15:05:54.679719 18529 tablet_bootstrap.cc:654] T 9032f9edb5a746729802355e7d212c95 P aeb32dce70ca494ca326c528e640c9ce: Neither blocks nor log segments found. Creating new log.
I20250112 15:05:54.680287 18529 tablet_bootstrap.cc:492] T 9032f9edb5a746729802355e7d212c95 P aeb32dce70ca494ca326c528e640c9ce: No bootstrap required, opened a new log
I20250112 15:05:54.680334 18529 ts_tablet_manager.cc:1397] T 9032f9edb5a746729802355e7d212c95 P aeb32dce70ca494ca326c528e640c9ce: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20250112 15:05:54.680467 18529 raft_consensus.cc:357] T 9032f9edb5a746729802355e7d212c95 P aeb32dce70ca494ca326c528e640c9ce [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79518508ccd644b28675a5de1e7280e2" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 38531 } } peers { permanent_uuid: "aeb32dce70ca494ca326c528e640c9ce" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 35351 } } peers { permanent_uuid: "90a0b30f5377440b8d72889b9d2f0a70" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 35603 } }
I20250112 15:05:54.680522 18530 tablet_bootstrap.cc:492] T e311741568d04cca81089b8bad66b805 P 90a0b30f5377440b8d72889b9d2f0a70: Bootstrap starting.
I20250112 15:05:54.680531 18529 raft_consensus.cc:383] T 9032f9edb5a746729802355e7d212c95 P aeb32dce70ca494ca326c528e640c9ce [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 15:05:54.680625 18529 raft_consensus.cc:738] T 9032f9edb5a746729802355e7d212c95 P aeb32dce70ca494ca326c528e640c9ce [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aeb32dce70ca494ca326c528e640c9ce, State: Initialized, Role: FOLLOWER
I20250112 15:05:54.680636 18528 tablet_bootstrap.cc:492] T cf6c409b42d842cfa25a995f95544e5e P 79518508ccd644b28675a5de1e7280e2: No bootstrap required, opened a new log
I20250112 15:05:54.680686 18528 ts_tablet_manager.cc:1397] T cf6c409b42d842cfa25a995f95544e5e P 79518508ccd644b28675a5de1e7280e2: Time spent bootstrapping tablet: real 0.010s user 0.001s sys 0.000s
I20250112 15:05:54.680673 18529 consensus_queue.cc:260] T 9032f9edb5a746729802355e7d212c95 P aeb32dce70ca494ca326c528e640c9ce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79518508ccd644b28675a5de1e7280e2" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 38531 } } peers { permanent_uuid: "aeb32dce70ca494ca326c528e640c9ce" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 35351 } } peers { permanent_uuid: "90a0b30f5377440b8d72889b9d2f0a70" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 35603 } }
I20250112 15:05:54.680774 18529 ts_tablet_manager.cc:1428] T 9032f9edb5a746729802355e7d212c95 P aeb32dce70ca494ca326c528e640c9ce: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20250112 15:05:54.680850 18529 tablet_bootstrap.cc:492] T ded9df74a71c4d059719cb5401e11822 P aeb32dce70ca494ca326c528e640c9ce: Bootstrap starting.
I20250112 15:05:54.680869 18528 raft_consensus.cc:357] T cf6c409b42d842cfa25a995f95544e5e P 79518508ccd644b28675a5de1e7280e2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79518508ccd644b28675a5de1e7280e2" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 38531 } } peers { permanent_uuid: "aeb32dce70ca494ca326c528e640c9ce" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 35351 } } peers { permanent_uuid: "90a0b30f5377440b8d72889b9d2f0a70" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 35603 } }
I20250112 15:05:54.680927 18528 raft_consensus.cc:383] T cf6c409b42d842cfa25a995f95544e5e P 79518508ccd644b28675a5de1e7280e2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 15:05:54.680950 18528 raft_consensus.cc:738] T cf6c409b42d842cfa25a995f95544e5e P 79518508ccd644b28675a5de1e7280e2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 79518508ccd644b28675a5de1e7280e2, State: Initialized, Role: FOLLOWER
I20250112 15:05:54.680991 18528 consensus_queue.cc:260] T cf6c409b42d842cfa25a995f95544e5e P 79518508ccd644b28675a5de1e7280e2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79518508ccd644b28675a5de1e7280e2" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 38531 } } peers { permanent_uuid: "aeb32dce70ca494ca326c528e640c9ce" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 35351 } } peers { permanent_uuid: "90a0b30f5377440b8d72889b9d2f0a70" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 35603 } }
I20250112 15:05:54.681016 18530 tablet_bootstrap.cc:654] T e311741568d04cca81089b8bad66b805 P 90a0b30f5377440b8d72889b9d2f0a70: Neither blocks nor log segments found. Creating new log.
I20250112 15:05:54.681082 18528 ts_tablet_manager.cc:1428] T cf6c409b42d842cfa25a995f95544e5e P 79518508ccd644b28675a5de1e7280e2: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20250112 15:05:54.681164 18528 tablet_bootstrap.cc:492] T 45204e59d67b4abe9ed00311ca03b73d P 79518508ccd644b28675a5de1e7280e2: Bootstrap starting.
I20250112 15:05:54.681223 18529 tablet_bootstrap.cc:654] T ded9df74a71c4d059719cb5401e11822 P aeb32dce70ca494ca326c528e640c9ce: Neither blocks nor log segments found. Creating new log.
I20250112 15:05:54.681560 18528 tablet_bootstrap.cc:654] T 45204e59d67b4abe9ed00311ca03b73d P 79518508ccd644b28675a5de1e7280e2: Neither blocks nor log segments found. Creating new log.
I20250112 15:05:54.681569 18530 tablet_bootstrap.cc:492] T e311741568d04cca81089b8bad66b805 P 90a0b30f5377440b8d72889b9d2f0a70: No bootstrap required, opened a new log
I20250112 15:05:54.681635 18530 ts_tablet_manager.cc:1397] T e311741568d04cca81089b8bad66b805 P 90a0b30f5377440b8d72889b9d2f0a70: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20250112 15:05:54.681777 18530 raft_consensus.cc:357] T e311741568d04cca81089b8bad66b805 P 90a0b30f5377440b8d72889b9d2f0a70 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90a0b30f5377440b8d72889b9d2f0a70" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 35603 } } peers { permanent_uuid: "79518508ccd644b28675a5de1e7280e2" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 38531 } } peers { permanent_uuid: "aeb32dce70ca494ca326c528e640c9ce" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 35351 } }
I20250112 15:05:54.681851 18530 raft_consensus.cc:383] T e311741568d04cca81089b8bad66b805 P 90a0b30f5377440b8d72889b9d2f0a70 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 15:05:54.681885 18530 raft_consensus.cc:738] T e311741568d04cca81089b8bad66b805 P 90a0b30f5377440b8d72889b9d2f0a70 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 90a0b30f5377440b8d72889b9d2f0a70, State: Initialized, Role: FOLLOWER
I20250112 15:05:54.681927 18530 consensus_queue.cc:260] T e311741568d04cca81089b8bad66b805 P 90a0b30f5377440b8d72889b9d2f0a70 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90a0b30f5377440b8d72889b9d2f0a70" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 35603 } } peers { permanent_uuid: "79518508ccd644b28675a5de1e7280e2" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 38531 } } peers { permanent_uuid: "aeb32dce70ca494ca326c528e640c9ce" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 35351 } }
I20250112 15:05:54.682089 18530 ts_tablet_manager.cc:1428] T e311741568d04cca81089b8bad66b805 P 90a0b30f5377440b8d72889b9d2f0a70: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20250112 15:05:54.682148 18530 tablet_bootstrap.cc:492] T 9032f9edb5a746729802355e7d212c95 P 90a0b30f5377440b8d72889b9d2f0a70: Bootstrap starting.
I20250112 15:05:54.682525 18530 tablet_bootstrap.cc:654] T 9032f9edb5a746729802355e7d212c95 P 90a0b30f5377440b8d72889b9d2f0a70: Neither blocks nor log segments found. Creating new log.
I20250112 15:05:54.683022 18530 tablet_bootstrap.cc:492] T 9032f9edb5a746729802355e7d212c95 P 90a0b30f5377440b8d72889b9d2f0a70: No bootstrap required, opened a new log
I20250112 15:05:54.683070 18528 tablet_bootstrap.cc:492] T 45204e59d67b4abe9ed00311ca03b73d P 79518508ccd644b28675a5de1e7280e2: No bootstrap required, opened a new log
I20250112 15:05:54.683099 18530 ts_tablet_manager.cc:1397] T 9032f9edb5a746729802355e7d212c95 P 90a0b30f5377440b8d72889b9d2f0a70: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20250112 15:05:54.683149 18528 ts_tablet_manager.cc:1397] T 45204e59d67b4abe9ed00311ca03b73d P 79518508ccd644b28675a5de1e7280e2: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20250112 15:05:54.683151 18529 tablet_bootstrap.cc:492] T ded9df74a71c4d059719cb5401e11822 P aeb32dce70ca494ca326c528e640c9ce: No bootstrap required, opened a new log
I20250112 15:05:54.683209 18529 ts_tablet_manager.cc:1397] T ded9df74a71c4d059719cb5401e11822 P aeb32dce70ca494ca326c528e640c9ce: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20250112 15:05:54.683279 18530 raft_consensus.cc:357] T 9032f9edb5a746729802355e7d212c95 P 90a0b30f5377440b8d72889b9d2f0a70 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79518508ccd644b28675a5de1e7280e2" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 38531 } } peers { permanent_uuid: "aeb32dce70ca494ca326c528e640c9ce" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 35351 } } peers { permanent_uuid: "90a0b30f5377440b8d72889b9d2f0a70" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 35603 } }
I20250112 15:05:54.683311 18528 raft_consensus.cc:357] T 45204e59d67b4abe9ed00311ca03b73d P 79518508ccd644b28675a5de1e7280e2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79518508ccd644b28675a5de1e7280e2" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 38531 } } peers { permanent_uuid: "aeb32dce70ca494ca326c528e640c9ce" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 35351 } } peers { permanent_uuid: "90a0b30f5377440b8d72889b9d2f0a70" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 35603 } }
I20250112 15:05:54.683339 18529 raft_consensus.cc:357] T ded9df74a71c4d059719cb5401e11822 P aeb32dce70ca494ca326c528e640c9ce [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79518508ccd644b28675a5de1e7280e2" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 38531 } } peers { permanent_uuid: "aeb32dce70ca494ca326c528e640c9ce" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 35351 } } peers { permanent_uuid: "90a0b30f5377440b8d72889b9d2f0a70" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 35603 } }
I20250112 15:05:54.683344 18530 raft_consensus.cc:383] T 9032f9edb5a746729802355e7d212c95 P 90a0b30f5377440b8d72889b9d2f0a70 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 15:05:54.683411 18529 raft_consensus.cc:383] T ded9df74a71c4d059719cb5401e11822 P aeb32dce70ca494ca326c528e640c9ce [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 15:05:54.683369 18528 raft_consensus.cc:383] T 45204e59d67b4abe9ed00311ca03b73d P 79518508ccd644b28675a5de1e7280e2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 15:05:54.683446 18529 raft_consensus.cc:738] T ded9df74a71c4d059719cb5401e11822 P aeb32dce70ca494ca326c528e640c9ce [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aeb32dce70ca494ca326c528e640c9ce, State: Initialized, Role: FOLLOWER
I20250112 15:05:54.683418 18530 raft_consensus.cc:738] T 9032f9edb5a746729802355e7d212c95 P 90a0b30f5377440b8d72889b9d2f0a70 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 90a0b30f5377440b8d72889b9d2f0a70, State: Initialized, Role: FOLLOWER
I20250112 15:05:54.683473 18528 raft_consensus.cc:738] T 45204e59d67b4abe9ed00311ca03b73d P 79518508ccd644b28675a5de1e7280e2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 79518508ccd644b28675a5de1e7280e2, State: Initialized, Role: FOLLOWER
I20250112 15:05:54.683559 18529 consensus_queue.cc:260] T ded9df74a71c4d059719cb5401e11822 P aeb32dce70ca494ca326c528e640c9ce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79518508ccd644b28675a5de1e7280e2" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 38531 } } peers { permanent_uuid: "aeb32dce70ca494ca326c528e640c9ce" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 35351 } } peers { permanent_uuid: "90a0b30f5377440b8d72889b9d2f0a70" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 35603 } }
I20250112 15:05:54.683562 18530 consensus_queue.cc:260] T 9032f9edb5a746729802355e7d212c95 P 90a0b30f5377440b8d72889b9d2f0a70 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79518508ccd644b28675a5de1e7280e2" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 38531 } } peers { permanent_uuid: "aeb32dce70ca494ca326c528e640c9ce" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 35351 } } peers { permanent_uuid: "90a0b30f5377440b8d72889b9d2f0a70" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 35603 } }
I20250112 15:05:54.683600 18528 consensus_queue.cc:260] T 45204e59d67b4abe9ed00311ca03b73d P 79518508ccd644b28675a5de1e7280e2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79518508ccd644b28675a5de1e7280e2" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 38531 } } peers { permanent_uuid: "aeb32dce70ca494ca326c528e640c9ce" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 35351 } } peers { permanent_uuid: "90a0b30f5377440b8d72889b9d2f0a70" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 35603 } }
I20250112 15:05:54.683658 18529 ts_tablet_manager.cc:1428] T ded9df74a71c4d059719cb5401e11822 P aeb32dce70ca494ca326c528e640c9ce: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20250112 15:05:54.683705 18530 ts_tablet_manager.cc:1428] T 9032f9edb5a746729802355e7d212c95 P 90a0b30f5377440b8d72889b9d2f0a70: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20250112 15:05:54.683719 18528 ts_tablet_manager.cc:1428] T 45204e59d67b4abe9ed00311ca03b73d P 79518508ccd644b28675a5de1e7280e2: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20250112 15:05:54.683761 18530 tablet_bootstrap.cc:492] T cf6c409b42d842cfa25a995f95544e5e P 90a0b30f5377440b8d72889b9d2f0a70: Bootstrap starting.
I20250112 15:05:54.683722 18529 tablet_bootstrap.cc:492] T 45204e59d67b4abe9ed00311ca03b73d P aeb32dce70ca494ca326c528e640c9ce: Bootstrap starting.
I20250112 15:05:54.683785 18528 tablet_bootstrap.cc:492] T 4e45a9f365b344c2aa0703adea508eeb P 79518508ccd644b28675a5de1e7280e2: Bootstrap starting.
I20250112 15:05:54.684217 18530 tablet_bootstrap.cc:654] T cf6c409b42d842cfa25a995f95544e5e P 90a0b30f5377440b8d72889b9d2f0a70: Neither blocks nor log segments found. Creating new log.
I20250112 15:05:54.684260 18529 tablet_bootstrap.cc:654] T 45204e59d67b4abe9ed00311ca03b73d P aeb32dce70ca494ca326c528e640c9ce: Neither blocks nor log segments found. Creating new log.
I20250112 15:05:54.684260 18528 tablet_bootstrap.cc:654] T 4e45a9f365b344c2aa0703adea508eeb P 79518508ccd644b28675a5de1e7280e2: Neither blocks nor log segments found. Creating new log.
I20250112 15:05:54.684831 18530 tablet_bootstrap.cc:492] T cf6c409b42d842cfa25a995f95544e5e P 90a0b30f5377440b8d72889b9d2f0a70: No bootstrap required, opened a new log
I20250112 15:05:54.684877 18530 ts_tablet_manager.cc:1397] T cf6c409b42d842cfa25a995f95544e5e P 90a0b30f5377440b8d72889b9d2f0a70: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20250112 15:05:54.685015 18530 raft_consensus.cc:357] T cf6c409b42d842cfa25a995f95544e5e P 90a0b30f5377440b8d72889b9d2f0a70 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79518508ccd644b28675a5de1e7280e2" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 38531 } } peers { permanent_uuid: "aeb32dce70ca494ca326c528e640c9ce" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 35351 } } peers { permanent_uuid: "90a0b30f5377440b8d72889b9d2f0a70" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 35603 } }
I20250112 15:05:54.685079 18530 raft_consensus.cc:383] T cf6c409b42d842cfa25a995f95544e5e P 90a0b30f5377440b8d72889b9d2f0a70 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 15:05:54.685101 18530 raft_consensus.cc:738] T cf6c409b42d842cfa25a995f95544e5e P 90a0b30f5377440b8d72889b9d2f0a70 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 90a0b30f5377440b8d72889b9d2f0a70, State: Initialized, Role: FOLLOWER
I20250112 15:05:54.685142 18530 consensus_queue.cc:260] T cf6c409b42d842cfa25a995f95544e5e P 90a0b30f5377440b8d72889b9d2f0a70 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79518508ccd644b28675a5de1e7280e2" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 38531 } } peers { permanent_uuid: "aeb32dce70ca494ca326c528e640c9ce" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 35351 } } peers { permanent_uuid: "90a0b30f5377440b8d72889b9d2f0a70" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 35603 } }
I20250112 15:05:54.685300 18530 ts_tablet_manager.cc:1428] T cf6c409b42d842cfa25a995f95544e5e P 90a0b30f5377440b8d72889b9d2f0a70: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20250112 15:05:54.685379 18530 tablet_bootstrap.cc:492] T ded9df74a71c4d059719cb5401e11822 P 90a0b30f5377440b8d72889b9d2f0a70: Bootstrap starting.
I20250112 15:05:54.685748 18530 tablet_bootstrap.cc:654] T ded9df74a71c4d059719cb5401e11822 P 90a0b30f5377440b8d72889b9d2f0a70: Neither blocks nor log segments found. Creating new log.
I20250112 15:05:54.686254 18530 tablet_bootstrap.cc:492] T ded9df74a71c4d059719cb5401e11822 P 90a0b30f5377440b8d72889b9d2f0a70: No bootstrap required, opened a new log
I20250112 15:05:54.686260 18529 tablet_bootstrap.cc:492] T 45204e59d67b4abe9ed00311ca03b73d P aeb32dce70ca494ca326c528e640c9ce: No bootstrap required, opened a new log
I20250112 15:05:54.686321 18530 ts_tablet_manager.cc:1397] T ded9df74a71c4d059719cb5401e11822 P 90a0b30f5377440b8d72889b9d2f0a70: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20250112 15:05:54.686322 18529 ts_tablet_manager.cc:1397] T 45204e59d67b4abe9ed00311ca03b73d P aeb32dce70ca494ca326c528e640c9ce: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.001s
I20250112 15:05:54.686347 18528 tablet_bootstrap.cc:492] T 4e45a9f365b344c2aa0703adea508eeb P 79518508ccd644b28675a5de1e7280e2: No bootstrap required, opened a new log
I20250112 15:05:54.686400 18528 ts_tablet_manager.cc:1397] T 4e45a9f365b344c2aa0703adea508eeb P 79518508ccd644b28675a5de1e7280e2: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.001s
I20250112 15:05:54.686478 18529 raft_consensus.cc:357] T 45204e59d67b4abe9ed00311ca03b73d P aeb32dce70ca494ca326c528e640c9ce [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79518508ccd644b28675a5de1e7280e2" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 38531 } } peers { permanent_uuid: "aeb32dce70ca494ca326c528e640c9ce" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 35351 } } peers { permanent_uuid: "90a0b30f5377440b8d72889b9d2f0a70" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 35603 } }
I20250112 15:05:54.686478 18530 raft_consensus.cc:357] T ded9df74a71c4d059719cb5401e11822 P 90a0b30f5377440b8d72889b9d2f0a70 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79518508ccd644b28675a5de1e7280e2" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 38531 } } peers { permanent_uuid: "aeb32dce70ca494ca326c528e640c9ce" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 35351 } } peers { permanent_uuid: "90a0b30f5377440b8d72889b9d2f0a70" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 35603 } }
I20250112 15:05:54.686571 18530 raft_consensus.cc:383] T ded9df74a71c4d059719cb5401e11822 P 90a0b30f5377440b8d72889b9d2f0a70 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 15:05:54.686511 18528 raft_consensus.cc:357] T 4e45a9f365b344c2aa0703adea508eeb P 79518508ccd644b28675a5de1e7280e2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79518508ccd644b28675a5de1e7280e2" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 38531 } } peers { permanent_uuid: "aeb32dce70ca494ca326c528e640c9ce" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 35351 } } peers { permanent_uuid: "90a0b30f5377440b8d72889b9d2f0a70" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 35603 } }
I20250112 15:05:54.686620 18529 raft_consensus.cc:383] T 45204e59d67b4abe9ed00311ca03b73d P aeb32dce70ca494ca326c528e640c9ce [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 15:05:54.686612 18528 raft_consensus.cc:383] T 4e45a9f365b344c2aa0703adea508eeb P 79518508ccd644b28675a5de1e7280e2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 15:05:54.686604 18530 raft_consensus.cc:738] T ded9df74a71c4d059719cb5401e11822 P 90a0b30f5377440b8d72889b9d2f0a70 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 90a0b30f5377440b8d72889b9d2f0a70, State: Initialized, Role: FOLLOWER
I20250112 15:05:54.686661 18528 raft_consensus.cc:738] T 4e45a9f365b344c2aa0703adea508eeb P 79518508ccd644b28675a5de1e7280e2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 79518508ccd644b28675a5de1e7280e2, State: Initialized, Role: FOLLOWER
I20250112 15:05:54.686664 18529 raft_consensus.cc:738] T 45204e59d67b4abe9ed00311ca03b73d P aeb32dce70ca494ca326c528e640c9ce [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aeb32dce70ca494ca326c528e640c9ce, State: Initialized, Role: FOLLOWER
I20250112 15:05:54.686700 18530 consensus_queue.cc:260] T ded9df74a71c4d059719cb5401e11822 P 90a0b30f5377440b8d72889b9d2f0a70 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79518508ccd644b28675a5de1e7280e2" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 38531 } } peers { permanent_uuid: "aeb32dce70ca494ca326c528e640c9ce" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 35351 } } peers { permanent_uuid: "90a0b30f5377440b8d72889b9d2f0a70" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 35603 } }
I20250112 15:05:54.686766 18529 consensus_queue.cc:260] T 45204e59d67b4abe9ed00311ca03b73d P aeb32dce70ca494ca326c528e640c9ce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79518508ccd644b28675a5de1e7280e2" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 38531 } } peers { permanent_uuid: "aeb32dce70ca494ca326c528e640c9ce" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 35351 } } peers { permanent_uuid: "90a0b30f5377440b8d72889b9d2f0a70" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 35603 } }
I20250112 15:05:54.686713 18528 consensus_queue.cc:260] T 4e45a9f365b344c2aa0703adea508eeb P 79518508ccd644b28675a5de1e7280e2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79518508ccd644b28675a5de1e7280e2" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 38531 } } peers { permanent_uuid: "aeb32dce70ca494ca326c528e640c9ce" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 35351 } } peers { permanent_uuid: "90a0b30f5377440b8d72889b9d2f0a70" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 35603 } }
I20250112 15:05:54.686827 18530 ts_tablet_manager.cc:1428] T ded9df74a71c4d059719cb5401e11822 P 90a0b30f5377440b8d72889b9d2f0a70: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20250112 15:05:54.686862 18529 ts_tablet_manager.cc:1428] T 45204e59d67b4abe9ed00311ca03b73d P aeb32dce70ca494ca326c528e640c9ce: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20250112 15:05:54.686890 18530 tablet_bootstrap.cc:492] T b3b76094dfae41f38c01fd10b38c7a28 P 90a0b30f5377440b8d72889b9d2f0a70: Bootstrap starting.
I20250112 15:05:54.686879 18528 ts_tablet_manager.cc:1428] T 4e45a9f365b344c2aa0703adea508eeb P 79518508ccd644b28675a5de1e7280e2: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20250112 15:05:54.686956 18528 tablet_bootstrap.cc:492] T ded9df74a71c4d059719cb5401e11822 P 79518508ccd644b28675a5de1e7280e2: Bootstrap starting.
I20250112 15:05:54.687299 18530 tablet_bootstrap.cc:654] T b3b76094dfae41f38c01fd10b38c7a28 P 90a0b30f5377440b8d72889b9d2f0a70: Neither blocks nor log segments found. Creating new log.
I20250112 15:05:54.687343 18528 tablet_bootstrap.cc:654] T ded9df74a71c4d059719cb5401e11822 P 79518508ccd644b28675a5de1e7280e2: Neither blocks nor log segments found. Creating new log.
I20250112 15:05:54.687824 18530 tablet_bootstrap.cc:492] T b3b76094dfae41f38c01fd10b38c7a28 P 90a0b30f5377440b8d72889b9d2f0a70: No bootstrap required, opened a new log
I20250112 15:05:54.687865 18530 ts_tablet_manager.cc:1397] T b3b76094dfae41f38c01fd10b38c7a28 P 90a0b30f5377440b8d72889b9d2f0a70: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20250112 15:05:54.687928 18528 tablet_bootstrap.cc:492] T ded9df74a71c4d059719cb5401e11822 P 79518508ccd644b28675a5de1e7280e2: No bootstrap required, opened a new log
I20250112 15:05:54.687973 18528 ts_tablet_manager.cc:1397] T ded9df74a71c4d059719cb5401e11822 P 79518508ccd644b28675a5de1e7280e2: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20250112 15:05:54.688001 18530 raft_consensus.cc:357] T b3b76094dfae41f38c01fd10b38c7a28 P 90a0b30f5377440b8d72889b9d2f0a70 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79518508ccd644b28675a5de1e7280e2" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 38531 } } peers { permanent_uuid: "aeb32dce70ca494ca326c528e640c9ce" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 35351 } } peers { permanent_uuid: "90a0b30f5377440b8d72889b9d2f0a70" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 35603 } }
I20250112 15:05:54.688055 18530 raft_consensus.cc:383] T b3b76094dfae41f38c01fd10b38c7a28 P 90a0b30f5377440b8d72889b9d2f0a70 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 15:05:54.688081 18530 raft_consensus.cc:738] T b3b76094dfae41f38c01fd10b38c7a28 P 90a0b30f5377440b8d72889b9d2f0a70 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 90a0b30f5377440b8d72889b9d2f0a70, State: Initialized, Role: FOLLOWER
I20250112 15:05:54.688110 18528 raft_consensus.cc:357] T ded9df74a71c4d059719cb5401e11822 P 79518508ccd644b28675a5de1e7280e2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79518508ccd644b28675a5de1e7280e2" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 38531 } } peers { permanent_uuid: "aeb32dce70ca494ca326c528e640c9ce" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 35351 } } peers { permanent_uuid: "90a0b30f5377440b8d72889b9d2f0a70" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 35603 } }
I20250112 15:05:54.688123 18530 consensus_queue.cc:260] T b3b76094dfae41f38c01fd10b38c7a28 P 90a0b30f5377440b8d72889b9d2f0a70 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79518508ccd644b28675a5de1e7280e2" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 38531 } } peers { permanent_uuid: "aeb32dce70ca494ca326c528e640c9ce" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 35351 } } peers { permanent_uuid: "90a0b30f5377440b8d72889b9d2f0a70" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 35603 } }
I20250112 15:05:54.688170 18528 raft_consensus.cc:383] T ded9df74a71c4d059719cb5401e11822 P 79518508ccd644b28675a5de1e7280e2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 15:05:54.688195 18528 raft_consensus.cc:738] T ded9df74a71c4d059719cb5401e11822 P 79518508ccd644b28675a5de1e7280e2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 79518508ccd644b28675a5de1e7280e2, State: Initialized, Role: FOLLOWER
I20250112 15:05:54.688216 18530 ts_tablet_manager.cc:1428] T b3b76094dfae41f38c01fd10b38c7a28 P 90a0b30f5377440b8d72889b9d2f0a70: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20250112 15:05:54.688230 18528 consensus_queue.cc:260] T ded9df74a71c4d059719cb5401e11822 P 79518508ccd644b28675a5de1e7280e2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79518508ccd644b28675a5de1e7280e2" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 38531 } } peers { permanent_uuid: "aeb32dce70ca494ca326c528e640c9ce" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 35351 } } peers { permanent_uuid: "90a0b30f5377440b8d72889b9d2f0a70" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 35603 } }
I20250112 15:05:54.688268 18530 tablet_bootstrap.cc:492] T 4e45a9f365b344c2aa0703adea508eeb P 90a0b30f5377440b8d72889b9d2f0a70: Bootstrap starting.
I20250112 15:05:54.688310 18528 ts_tablet_manager.cc:1428] T ded9df74a71c4d059719cb5401e11822 P 79518508ccd644b28675a5de1e7280e2: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20250112 15:05:54.688365 18528 tablet_bootstrap.cc:492] T 2319a98d89eb4234aa86cc3930853fce P 79518508ccd644b28675a5de1e7280e2: Bootstrap starting.
I20250112 15:05:54.688671 18530 tablet_bootstrap.cc:654] T 4e45a9f365b344c2aa0703adea508eeb P 90a0b30f5377440b8d72889b9d2f0a70: Neither blocks nor log segments found. Creating new log.
I20250112 15:05:54.688755 18528 tablet_bootstrap.cc:654] T 2319a98d89eb4234aa86cc3930853fce P 79518508ccd644b28675a5de1e7280e2: Neither blocks nor log segments found. Creating new log.
I20250112 15:05:54.689180 18530 tablet_bootstrap.cc:492] T 4e45a9f365b344c2aa0703adea508eeb P 90a0b30f5377440b8d72889b9d2f0a70: No bootstrap required, opened a new log
I20250112 15:05:54.689224 18530 ts_tablet_manager.cc:1397] T 4e45a9f365b344c2aa0703adea508eeb P 90a0b30f5377440b8d72889b9d2f0a70: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20250112 15:05:54.689287 18528 tablet_bootstrap.cc:492] T 2319a98d89eb4234aa86cc3930853fce P 79518508ccd644b28675a5de1e7280e2: No bootstrap required, opened a new log
I20250112 15:05:54.689337 18528 ts_tablet_manager.cc:1397] T 2319a98d89eb4234aa86cc3930853fce P 79518508ccd644b28675a5de1e7280e2: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.000s
I20250112 15:05:54.689381 18530 raft_consensus.cc:357] T 4e45a9f365b344c2aa0703adea508eeb P 90a0b30f5377440b8d72889b9d2f0a70 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79518508ccd644b28675a5de1e7280e2" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 38531 } } peers { permanent_uuid: "aeb32dce70ca494ca326c528e640c9ce" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 35351 } } peers { permanent_uuid: "90a0b30f5377440b8d72889b9d2f0a70" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 35603 } }
I20250112 15:05:54.689441 18530 raft_consensus.cc:383] T 4e45a9f365b344c2aa0703adea508eeb P 90a0b30f5377440b8d72889b9d2f0a70 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 15:05:54.689466 18530 raft_consensus.cc:738] T 4e45a9f365b344c2aa0703adea508eeb P 90a0b30f5377440b8d72889b9d2f0a70 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 90a0b30f5377440b8d72889b9d2f0a70, State: Initialized, Role: FOLLOWER
I20250112 15:05:54.689486 18528 raft_consensus.cc:357] T 2319a98d89eb4234aa86cc3930853fce P 79518508ccd644b28675a5de1e7280e2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79518508ccd644b28675a5de1e7280e2" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 38531 } } peers { permanent_uuid: "aeb32dce70ca494ca326c528e640c9ce" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 35351 } } peers { permanent_uuid: "90a0b30f5377440b8d72889b9d2f0a70" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 35603 } }
I20250112 15:05:54.689510 18530 consensus_queue.cc:260] T 4e45a9f365b344c2aa0703adea508eeb P 90a0b30f5377440b8d72889b9d2f0a70 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79518508ccd644b28675a5de1e7280e2" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 38531 } } peers { permanent_uuid: "aeb32dce70ca494ca326c528e640c9ce" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 35351 } } peers { permanent_uuid: "90a0b30f5377440b8d72889b9d2f0a70" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 35603 } }
I20250112 15:05:54.689563 18528 raft_consensus.cc:383] T 2319a98d89eb4234aa86cc3930853fce P 79518508ccd644b28675a5de1e7280e2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 15:05:54.689591 18528 raft_consensus.cc:738] T 2319a98d89eb4234aa86cc3930853fce P 79518508ccd644b28675a5de1e7280e2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 79518508ccd644b28675a5de1e7280e2, State: Initialized, Role: FOLLOWER
I20250112 15:05:54.689610 18530 ts_tablet_manager.cc:1428] T 4e45a9f365b344c2aa0703adea508eeb P 90a0b30f5377440b8d72889b9d2f0a70: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20250112 15:05:54.689657 18530 tablet_bootstrap.cc:492] T 2319a98d89eb4234aa86cc3930853fce P 90a0b30f5377440b8d72889b9d2f0a70: Bootstrap starting.
I20250112 15:05:54.689643 18528 consensus_queue.cc:260] T 2319a98d89eb4234aa86cc3930853fce P 79518508ccd644b28675a5de1e7280e2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79518508ccd644b28675a5de1e7280e2" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 38531 } } peers { permanent_uuid: "aeb32dce70ca494ca326c528e640c9ce" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 35351 } } peers { permanent_uuid: "90a0b30f5377440b8d72889b9d2f0a70" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 35603 } }
I20250112 15:05:54.689772 18528 ts_tablet_manager.cc:1428] T 2319a98d89eb4234aa86cc3930853fce P 79518508ccd644b28675a5de1e7280e2: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20250112 15:05:54.689831 18528 tablet_bootstrap.cc:492] T 9032f9edb5a746729802355e7d212c95 P 79518508ccd644b28675a5de1e7280e2: Bootstrap starting.
I20250112 15:05:54.690073 18530 tablet_bootstrap.cc:654] T 2319a98d89eb4234aa86cc3930853fce P 90a0b30f5377440b8d72889b9d2f0a70: Neither blocks nor log segments found. Creating new log.
I20250112 15:05:54.690241 18528 tablet_bootstrap.cc:654] T 9032f9edb5a746729802355e7d212c95 P 79518508ccd644b28675a5de1e7280e2: Neither blocks nor log segments found. Creating new log.
I20250112 15:05:54.690810 18528 tablet_bootstrap.cc:492] T 9032f9edb5a746729802355e7d212c95 P 79518508ccd644b28675a5de1e7280e2: No bootstrap required, opened a new log
I20250112 15:05:54.690835 18530 tablet_bootstrap.cc:492] T 2319a98d89eb4234aa86cc3930853fce P 90a0b30f5377440b8d72889b9d2f0a70: No bootstrap required, opened a new log
I20250112 15:05:54.690850 18528 ts_tablet_manager.cc:1397] T 9032f9edb5a746729802355e7d212c95 P 79518508ccd644b28675a5de1e7280e2: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.000s
I20250112 15:05:54.690912 18530 ts_tablet_manager.cc:1397] T 2319a98d89eb4234aa86cc3930853fce P 90a0b30f5377440b8d72889b9d2f0a70: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20250112 15:05:54.691030 18528 raft_consensus.cc:357] T 9032f9edb5a746729802355e7d212c95 P 79518508ccd644b28675a5de1e7280e2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79518508ccd644b28675a5de1e7280e2" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 38531 } } peers { permanent_uuid: "aeb32dce70ca494ca326c528e640c9ce" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 35351 } } peers { permanent_uuid: "90a0b30f5377440b8d72889b9d2f0a70" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 35603 } }
I20250112 15:05:54.691044 18530 raft_consensus.cc:357] T 2319a98d89eb4234aa86cc3930853fce P 90a0b30f5377440b8d72889b9d2f0a70 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79518508ccd644b28675a5de1e7280e2" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 38531 } } peers { permanent_uuid: "aeb32dce70ca494ca326c528e640c9ce" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 35351 } } peers { permanent_uuid: "90a0b30f5377440b8d72889b9d2f0a70" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 35603 } }
I20250112 15:05:54.691076 18528 raft_consensus.cc:383] T 9032f9edb5a746729802355e7d212c95 P 79518508ccd644b28675a5de1e7280e2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 15:05:54.691085 18530 raft_consensus.cc:383] T 2319a98d89eb4234aa86cc3930853fce P 90a0b30f5377440b8d72889b9d2f0a70 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 15:05:54.691088 18528 raft_consensus.cc:738] T 9032f9edb5a746729802355e7d212c95 P 79518508ccd644b28675a5de1e7280e2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 79518508ccd644b28675a5de1e7280e2, State: Initialized, Role: FOLLOWER
I20250112 15:05:54.691123 18530 raft_consensus.cc:738] T 2319a98d89eb4234aa86cc3930853fce P 90a0b30f5377440b8d72889b9d2f0a70 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 90a0b30f5377440b8d72889b9d2f0a70, State: Initialized, Role: FOLLOWER
I20250112 15:05:54.691155 18528 consensus_queue.cc:260] T 9032f9edb5a746729802355e7d212c95 P 79518508ccd644b28675a5de1e7280e2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79518508ccd644b28675a5de1e7280e2" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 38531 } } peers { permanent_uuid: "aeb32dce70ca494ca326c528e640c9ce" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 35351 } } peers { permanent_uuid: "90a0b30f5377440b8d72889b9d2f0a70" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 35603 } }
I20250112 15:05:54.691172 18530 consensus_queue.cc:260] T 2319a98d89eb4234aa86cc3930853fce P 90a0b30f5377440b8d72889b9d2f0a70 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79518508ccd644b28675a5de1e7280e2" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 38531 } } peers { permanent_uuid: "aeb32dce70ca494ca326c528e640c9ce" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 35351 } } peers { permanent_uuid: "90a0b30f5377440b8d72889b9d2f0a70" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 35603 } }
I20250112 15:05:54.691242 18528 ts_tablet_manager.cc:1428] T 9032f9edb5a746729802355e7d212c95 P 79518508ccd644b28675a5de1e7280e2: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20250112 15:05:54.691286 18528 tablet_bootstrap.cc:492] T b3b76094dfae41f38c01fd10b38c7a28 P 79518508ccd644b28675a5de1e7280e2: Bootstrap starting.
I20250112 15:05:54.691331 18530 ts_tablet_manager.cc:1428] T 2319a98d89eb4234aa86cc3930853fce P 90a0b30f5377440b8d72889b9d2f0a70: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20250112 15:05:54.691377 18530 tablet_bootstrap.cc:492] T 45204e59d67b4abe9ed00311ca03b73d P 90a0b30f5377440b8d72889b9d2f0a70: Bootstrap starting.
I20250112 15:05:54.691658 18528 tablet_bootstrap.cc:654] T b3b76094dfae41f38c01fd10b38c7a28 P 79518508ccd644b28675a5de1e7280e2: Neither blocks nor log segments found. Creating new log.
I20250112 15:05:54.691745 18530 tablet_bootstrap.cc:654] T 45204e59d67b4abe9ed00311ca03b73d P 90a0b30f5377440b8d72889b9d2f0a70: Neither blocks nor log segments found. Creating new log.
I20250112 15:05:54.692286 18528 tablet_bootstrap.cc:492] T b3b76094dfae41f38c01fd10b38c7a28 P 79518508ccd644b28675a5de1e7280e2: No bootstrap required, opened a new log
I20250112 15:05:54.692327 18528 ts_tablet_manager.cc:1397] T b3b76094dfae41f38c01fd10b38c7a28 P 79518508ccd644b28675a5de1e7280e2: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.000s
I20250112 15:05:54.692354 18530 tablet_bootstrap.cc:492] T 45204e59d67b4abe9ed00311ca03b73d P 90a0b30f5377440b8d72889b9d2f0a70: No bootstrap required, opened a new log
I20250112 15:05:54.692404 18530 ts_tablet_manager.cc:1397] T 45204e59d67b4abe9ed00311ca03b73d P 90a0b30f5377440b8d72889b9d2f0a70: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20250112 15:05:54.692539 18530 raft_consensus.cc:357] T 45204e59d67b4abe9ed00311ca03b73d P 90a0b30f5377440b8d72889b9d2f0a70 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79518508ccd644b28675a5de1e7280e2" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 38531 } } peers { permanent_uuid: "aeb32dce70ca494ca326c528e640c9ce" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 35351 } } peers { permanent_uuid: "90a0b30f5377440b8d72889b9d2f0a70" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 35603 } }
I20250112 15:05:54.692621 18530 raft_consensus.cc:383] T 45204e59d67b4abe9ed00311ca03b73d P 90a0b30f5377440b8d72889b9d2f0a70 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 15:05:54.692644 18530 raft_consensus.cc:738] T 45204e59d67b4abe9ed00311ca03b73d P 90a0b30f5377440b8d72889b9d2f0a70 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 90a0b30f5377440b8d72889b9d2f0a70, State: Initialized, Role: FOLLOWER
I20250112 15:05:54.692638 18528 raft_consensus.cc:357] T b3b76094dfae41f38c01fd10b38c7a28 P 79518508ccd644b28675a5de1e7280e2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79518508ccd644b28675a5de1e7280e2" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 38531 } } peers { permanent_uuid: "aeb32dce70ca494ca326c528e640c9ce" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 35351 } } peers { permanent_uuid: "90a0b30f5377440b8d72889b9d2f0a70" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 35603 } }
I20250112 15:05:54.692718 18528 raft_consensus.cc:383] T b3b76094dfae41f38c01fd10b38c7a28 P 79518508ccd644b28675a5de1e7280e2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 15:05:54.692727 18530 consensus_queue.cc:260] T 45204e59d67b4abe9ed00311ca03b73d P 90a0b30f5377440b8d72889b9d2f0a70 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79518508ccd644b28675a5de1e7280e2" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 38531 } } peers { permanent_uuid: "aeb32dce70ca494ca326c528e640c9ce" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 35351 } } peers { permanent_uuid: "90a0b30f5377440b8d72889b9d2f0a70" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 35603 } }
I20250112 15:05:54.692776 18528 raft_consensus.cc:738] T b3b76094dfae41f38c01fd10b38c7a28 P 79518508ccd644b28675a5de1e7280e2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 79518508ccd644b28675a5de1e7280e2, State: Initialized, Role: FOLLOWER
I20250112 15:05:54.692844 18528 consensus_queue.cc:260] T b3b76094dfae41f38c01fd10b38c7a28 P 79518508ccd644b28675a5de1e7280e2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79518508ccd644b28675a5de1e7280e2" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 38531 } } peers { permanent_uuid: "aeb32dce70ca494ca326c528e640c9ce" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 35351 } } peers { permanent_uuid: "90a0b30f5377440b8d72889b9d2f0a70" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 35603 } }
I20250112 15:05:54.692859 18530 ts_tablet_manager.cc:1428] T 45204e59d67b4abe9ed00311ca03b73d P 90a0b30f5377440b8d72889b9d2f0a70: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20250112 15:05:54.692939 18528 ts_tablet_manager.cc:1428] T b3b76094dfae41f38c01fd10b38c7a28 P 79518508ccd644b28675a5de1e7280e2: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20250112 15:05:54.700985 18533 raft_consensus.cc:491] T ded9df74a71c4d059719cb5401e11822 P aeb32dce70ca494ca326c528e640c9ce [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 15:05:54.701040 18533 raft_consensus.cc:513] T ded9df74a71c4d059719cb5401e11822 P aeb32dce70ca494ca326c528e640c9ce [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79518508ccd644b28675a5de1e7280e2" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 38531 } } peers { permanent_uuid: "aeb32dce70ca494ca326c528e640c9ce" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 35351 } } peers { permanent_uuid: "90a0b30f5377440b8d72889b9d2f0a70" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 35603 } }
I20250112 15:05:54.701203 18533 leader_election.cc:290] T ded9df74a71c4d059719cb5401e11822 P aeb32dce70ca494ca326c528e640c9ce [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 79518508ccd644b28675a5de1e7280e2 (127.17.60.193:38531), 90a0b30f5377440b8d72889b9d2f0a70 (127.17.60.195:35603)
I20250112 15:05:54.703653 18331 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ded9df74a71c4d059719cb5401e11822" candidate_uuid: "aeb32dce70ca494ca326c528e640c9ce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "79518508ccd644b28675a5de1e7280e2" is_pre_election: true
I20250112 15:05:54.703728 18331 raft_consensus.cc:2463] T ded9df74a71c4d059719cb5401e11822 P 79518508ccd644b28675a5de1e7280e2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate aeb32dce70ca494ca326c528e640c9ce in term 0.
I20250112 15:05:54.703873 18482 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ded9df74a71c4d059719cb5401e11822" candidate_uuid: "aeb32dce70ca494ca326c528e640c9ce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "90a0b30f5377440b8d72889b9d2f0a70" is_pre_election: true
I20250112 15:05:54.703888 18370 leader_election.cc:304] T ded9df74a71c4d059719cb5401e11822 P aeb32dce70ca494ca326c528e640c9ce [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 79518508ccd644b28675a5de1e7280e2, aeb32dce70ca494ca326c528e640c9ce; no voters:
I20250112 15:05:54.703953 18482 raft_consensus.cc:2463] T ded9df74a71c4d059719cb5401e11822 P 90a0b30f5377440b8d72889b9d2f0a70 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate aeb32dce70ca494ca326c528e640c9ce in term 0.
I20250112 15:05:54.704012 18533 raft_consensus.cc:2798] T ded9df74a71c4d059719cb5401e11822 P aeb32dce70ca494ca326c528e640c9ce [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250112 15:05:54.704057 18533 raft_consensus.cc:491] T ded9df74a71c4d059719cb5401e11822 P aeb32dce70ca494ca326c528e640c9ce [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250112 15:05:54.704087 18533 raft_consensus.cc:3054] T ded9df74a71c4d059719cb5401e11822 P aeb32dce70ca494ca326c528e640c9ce [term 0 FOLLOWER]: Advancing to term 1
I20250112 15:05:54.704675 18533 raft_consensus.cc:513] T ded9df74a71c4d059719cb5401e11822 P aeb32dce70ca494ca326c528e640c9ce [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79518508ccd644b28675a5de1e7280e2" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 38531 } } peers { permanent_uuid: "aeb32dce70ca494ca326c528e640c9ce" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 35351 } } peers { permanent_uuid: "90a0b30f5377440b8d72889b9d2f0a70" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 35603 } }
I20250112 15:05:54.704775 18533 leader_election.cc:290] T ded9df74a71c4d059719cb5401e11822 P aeb32dce70ca494ca326c528e640c9ce [CANDIDATE]: Term 1 election: Requested vote from peers 79518508ccd644b28675a5de1e7280e2 (127.17.60.193:38531), 90a0b30f5377440b8d72889b9d2f0a70 (127.17.60.195:35603)
I20250112 15:05:54.704924 18331 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ded9df74a71c4d059719cb5401e11822" candidate_uuid: "aeb32dce70ca494ca326c528e640c9ce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "79518508ccd644b28675a5de1e7280e2"
I20250112 15:05:54.704965 18482 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ded9df74a71c4d059719cb5401e11822" candidate_uuid: "aeb32dce70ca494ca326c528e640c9ce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "90a0b30f5377440b8d72889b9d2f0a70"
I20250112 15:05:54.704974 18331 raft_consensus.cc:3054] T ded9df74a71c4d059719cb5401e11822 P 79518508ccd644b28675a5de1e7280e2 [term 0 FOLLOWER]: Advancing to term 1
I20250112 15:05:54.705078 18482 raft_consensus.cc:3054] T ded9df74a71c4d059719cb5401e11822 P 90a0b30f5377440b8d72889b9d2f0a70 [term 0 FOLLOWER]: Advancing to term 1
I20250112 15:05:54.705610 18331 raft_consensus.cc:2463] T ded9df74a71c4d059719cb5401e11822 P 79518508ccd644b28675a5de1e7280e2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate aeb32dce70ca494ca326c528e640c9ce in term 1.
I20250112 15:05:54.705658 18482 raft_consensus.cc:2463] T ded9df74a71c4d059719cb5401e11822 P 90a0b30f5377440b8d72889b9d2f0a70 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate aeb32dce70ca494ca326c528e640c9ce in term 1.
I20250112 15:05:54.705773 18370 leader_election.cc:304] T ded9df74a71c4d059719cb5401e11822 P aeb32dce70ca494ca326c528e640c9ce [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 79518508ccd644b28675a5de1e7280e2, aeb32dce70ca494ca326c528e640c9ce; no voters:
I20250112 15:05:54.705868 18533 raft_consensus.cc:2798] T ded9df74a71c4d059719cb5401e11822 P aeb32dce70ca494ca326c528e640c9ce [term 1 FOLLOWER]: Leader election won for term 1
I20250112 15:05:54.705996 18533 raft_consensus.cc:695] T ded9df74a71c4d059719cb5401e11822 P aeb32dce70ca494ca326c528e640c9ce [term 1 LEADER]: Becoming Leader. State: Replica: aeb32dce70ca494ca326c528e640c9ce, State: Running, Role: LEADER
I20250112 15:05:54.706059 18533 consensus_queue.cc:237] T ded9df74a71c4d059719cb5401e11822 P aeb32dce70ca494ca326c528e640c9ce [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79518508ccd644b28675a5de1e7280e2" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 38531 } } peers { permanent_uuid: "aeb32dce70ca494ca326c528e640c9ce" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 35351 } } peers { permanent_uuid: "90a0b30f5377440b8d72889b9d2f0a70" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 35603 } }
I20250112 15:05:54.706308 18544 raft_consensus.cc:491] T 2319a98d89eb4234aa86cc3930853fce P 90a0b30f5377440b8d72889b9d2f0a70 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 15:05:54.706363 18544 raft_consensus.cc:513] T 2319a98d89eb4234aa86cc3930853fce P 90a0b30f5377440b8d72889b9d2f0a70 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79518508ccd644b28675a5de1e7280e2" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 38531 } } peers { permanent_uuid: "aeb32dce70ca494ca326c528e640c9ce" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 35351 } } peers { permanent_uuid: "90a0b30f5377440b8d72889b9d2f0a70" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 35603 } }
I20250112 15:05:54.706507 18544 leader_election.cc:290] T 2319a98d89eb4234aa86cc3930853fce P 90a0b30f5377440b8d72889b9d2f0a70 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 79518508ccd644b28675a5de1e7280e2 (127.17.60.193:38531), aeb32dce70ca494ca326c528e640c9ce (127.17.60.194:35351)
I20250112 15:05:54.706513 18226 catalog_manager.cc:5526] T ded9df74a71c4d059719cb5401e11822 P aeb32dce70ca494ca326c528e640c9ce reported cstate change: term changed from 0 to 1, leader changed from <none> to aeb32dce70ca494ca326c528e640c9ce (127.17.60.194). New cstate: current_term: 1 leader_uuid: "aeb32dce70ca494ca326c528e640c9ce" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79518508ccd644b28675a5de1e7280e2" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 38531 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "aeb32dce70ca494ca326c528e640c9ce" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 35351 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "90a0b30f5377440b8d72889b9d2f0a70" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 35603 } health_report { overall_health: UNKNOWN } } }
I20250112 15:05:54.709559 18331 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2319a98d89eb4234aa86cc3930853fce" candidate_uuid: "90a0b30f5377440b8d72889b9d2f0a70" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "79518508ccd644b28675a5de1e7280e2" is_pre_election: true
I20250112 15:05:54.709619 18331 raft_consensus.cc:2463] T 2319a98d89eb4234aa86cc3930853fce P 79518508ccd644b28675a5de1e7280e2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 90a0b30f5377440b8d72889b9d2f0a70 in term 0.
I20250112 15:05:54.709625 18407 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2319a98d89eb4234aa86cc3930853fce" candidate_uuid: "90a0b30f5377440b8d72889b9d2f0a70" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aeb32dce70ca494ca326c528e640c9ce" is_pre_election: true
I20250112 15:05:54.709709 18407 raft_consensus.cc:2463] T 2319a98d89eb4234aa86cc3930853fce P aeb32dce70ca494ca326c528e640c9ce [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 90a0b30f5377440b8d72889b9d2f0a70 in term 0.
I20250112 15:05:54.709774 18445 leader_election.cc:304] T 2319a98d89eb4234aa86cc3930853fce P 90a0b30f5377440b8d72889b9d2f0a70 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 79518508ccd644b28675a5de1e7280e2, 90a0b30f5377440b8d72889b9d2f0a70; no voters:
I20250112 15:05:54.709862 18544 raft_consensus.cc:2798] T 2319a98d89eb4234aa86cc3930853fce P 90a0b30f5377440b8d72889b9d2f0a70 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250112 15:05:54.709901 18544 raft_consensus.cc:491] T 2319a98d89eb4234aa86cc3930853fce P 90a0b30f5377440b8d72889b9d2f0a70 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250112 15:05:54.709928 18544 raft_consensus.cc:3054] T 2319a98d89eb4234aa86cc3930853fce P 90a0b30f5377440b8d72889b9d2f0a70 [term 0 FOLLOWER]: Advancing to term 1
I20250112 15:05:54.710389 18544 raft_consensus.cc:513] T 2319a98d89eb4234aa86cc3930853fce P 90a0b30f5377440b8d72889b9d2f0a70 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79518508ccd644b28675a5de1e7280e2" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 38531 } } peers { permanent_uuid: "aeb32dce70ca494ca326c528e640c9ce" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 35351 } } peers { permanent_uuid: "90a0b30f5377440b8d72889b9d2f0a70" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 35603 } }
I20250112 15:05:54.710498 18544 leader_election.cc:290] T 2319a98d89eb4234aa86cc3930853fce P 90a0b30f5377440b8d72889b9d2f0a70 [CANDIDATE]: Term 1 election: Requested vote from peers 79518508ccd644b28675a5de1e7280e2 (127.17.60.193:38531), aeb32dce70ca494ca326c528e640c9ce (127.17.60.194:35351)
I20250112 15:05:54.710642 18407 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2319a98d89eb4234aa86cc3930853fce" candidate_uuid: "90a0b30f5377440b8d72889b9d2f0a70" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aeb32dce70ca494ca326c528e640c9ce"
I20250112 15:05:54.710681 18331 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2319a98d89eb4234aa86cc3930853fce" candidate_uuid: "90a0b30f5377440b8d72889b9d2f0a70" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "79518508ccd644b28675a5de1e7280e2"
I20250112 15:05:54.710688 18407 raft_consensus.cc:3054] T 2319a98d89eb4234aa86cc3930853fce P aeb32dce70ca494ca326c528e640c9ce [term 0 FOLLOWER]: Advancing to term 1
I20250112 15:05:54.710752 18331 raft_consensus.cc:3054] T 2319a98d89eb4234aa86cc3930853fce P 79518508ccd644b28675a5de1e7280e2 [term 0 FOLLOWER]: Advancing to term 1
I20250112 15:05:54.711223 18331 raft_consensus.cc:2463] T 2319a98d89eb4234aa86cc3930853fce P 79518508ccd644b28675a5de1e7280e2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 90a0b30f5377440b8d72889b9d2f0a70 in term 1.
I20250112 15:05:54.711239 18407 raft_consensus.cc:2463] T 2319a98d89eb4234aa86cc3930853fce P aeb32dce70ca494ca326c528e640c9ce [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 90a0b30f5377440b8d72889b9d2f0a70 in term 1.
I20250112 15:05:54.711387 18445 leader_election.cc:304] T 2319a98d89eb4234aa86cc3930853fce P 90a0b30f5377440b8d72889b9d2f0a70 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 79518508ccd644b28675a5de1e7280e2, 90a0b30f5377440b8d72889b9d2f0a70; no voters:
I20250112 15:05:54.711468 18544 raft_consensus.cc:2798] T 2319a98d89eb4234aa86cc3930853fce P 90a0b30f5377440b8d72889b9d2f0a70 [term 1 FOLLOWER]: Leader election won for term 1
I20250112 15:05:54.711511 18544 raft_consensus.cc:695] T 2319a98d89eb4234aa86cc3930853fce P 90a0b30f5377440b8d72889b9d2f0a70 [term 1 LEADER]: Becoming Leader. State: Replica: 90a0b30f5377440b8d72889b9d2f0a70, State: Running, Role: LEADER
I20250112 15:05:54.711555 18544 consensus_queue.cc:237] T 2319a98d89eb4234aa86cc3930853fce P 90a0b30f5377440b8d72889b9d2f0a70 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79518508ccd644b28675a5de1e7280e2" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 38531 } } peers { permanent_uuid: "aeb32dce70ca494ca326c528e640c9ce" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 35351 } } peers { permanent_uuid: "90a0b30f5377440b8d72889b9d2f0a70" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 35603 } }
I20250112 15:05:54.712100 18226 catalog_manager.cc:5526] T 2319a98d89eb4234aa86cc3930853fce P 90a0b30f5377440b8d72889b9d2f0a70 reported cstate change: term changed from 0 to 1, leader changed from <none> to 90a0b30f5377440b8d72889b9d2f0a70 (127.17.60.195). New cstate: current_term: 1 leader_uuid: "90a0b30f5377440b8d72889b9d2f0a70" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79518508ccd644b28675a5de1e7280e2" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 38531 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "aeb32dce70ca494ca326c528e640c9ce" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 35351 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "90a0b30f5377440b8d72889b9d2f0a70" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 35603 } health_report { overall_health: HEALTHY } } }
I20250112 15:05:54.720911 18544 raft_consensus.cc:491] T b3b76094dfae41f38c01fd10b38c7a28 P 90a0b30f5377440b8d72889b9d2f0a70 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 15:05:54.720971 18544 raft_consensus.cc:513] T b3b76094dfae41f38c01fd10b38c7a28 P 90a0b30f5377440b8d72889b9d2f0a70 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79518508ccd644b28675a5de1e7280e2" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 38531 } } peers { permanent_uuid: "aeb32dce70ca494ca326c528e640c9ce" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 35351 } } peers { permanent_uuid: "90a0b30f5377440b8d72889b9d2f0a70" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 35603 } }
I20250112 15:05:54.721091 18544 leader_election.cc:290] T b3b76094dfae41f38c01fd10b38c7a28 P 90a0b30f5377440b8d72889b9d2f0a70 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 79518508ccd644b28675a5de1e7280e2 (127.17.60.193:38531), aeb32dce70ca494ca326c528e640c9ce (127.17.60.194:35351)
I20250112 15:05:54.721220 18331 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3b76094dfae41f38c01fd10b38c7a28" candidate_uuid: "90a0b30f5377440b8d72889b9d2f0a70" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "79518508ccd644b28675a5de1e7280e2" is_pre_election: true
I20250112 15:05:54.721233 18407 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3b76094dfae41f38c01fd10b38c7a28" candidate_uuid: "90a0b30f5377440b8d72889b9d2f0a70" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aeb32dce70ca494ca326c528e640c9ce" is_pre_election: true
I20250112 15:05:54.721283 18331 raft_consensus.cc:2463] T b3b76094dfae41f38c01fd10b38c7a28 P 79518508ccd644b28675a5de1e7280e2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 90a0b30f5377440b8d72889b9d2f0a70 in term 0.
I20250112 15:05:54.721298 18407 raft_consensus.cc:2463] T b3b76094dfae41f38c01fd10b38c7a28 P aeb32dce70ca494ca326c528e640c9ce [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 90a0b30f5377440b8d72889b9d2f0a70 in term 0.
I20250112 15:05:54.721386 18445 leader_election.cc:304] T b3b76094dfae41f38c01fd10b38c7a28 P 90a0b30f5377440b8d72889b9d2f0a70 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 79518508ccd644b28675a5de1e7280e2, 90a0b30f5377440b8d72889b9d2f0a70; no voters:
I20250112 15:05:54.721477 18544 raft_consensus.cc:2798] T b3b76094dfae41f38c01fd10b38c7a28 P 90a0b30f5377440b8d72889b9d2f0a70 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250112 15:05:54.721510 18544 raft_consensus.cc:491] T b3b76094dfae41f38c01fd10b38c7a28 P 90a0b30f5377440b8d72889b9d2f0a70 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250112 15:05:54.721522 18544 raft_consensus.cc:3054] T b3b76094dfae41f38c01fd10b38c7a28 P 90a0b30f5377440b8d72889b9d2f0a70 [term 0 FOLLOWER]: Advancing to term 1
I20250112 15:05:54.722013 18544 raft_consensus.cc:513] T b3b76094dfae41f38c01fd10b38c7a28 P 90a0b30f5377440b8d72889b9d2f0a70 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79518508ccd644b28675a5de1e7280e2" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 38531 } } peers { permanent_uuid: "aeb32dce70ca494ca326c528e640c9ce" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 35351 } } peers { permanent_uuid: "90a0b30f5377440b8d72889b9d2f0a70" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 35603 } }
I20250112 15:05:54.722147 18544 leader_election.cc:290] T b3b76094dfae41f38c01fd10b38c7a28 P 90a0b30f5377440b8d72889b9d2f0a70 [CANDIDATE]: Term 1 election: Requested vote from peers 79518508ccd644b28675a5de1e7280e2 (127.17.60.193:38531), aeb32dce70ca494ca326c528e640c9ce (127.17.60.194:35351)
I20250112 15:05:54.722283 18407 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3b76094dfae41f38c01fd10b38c7a28" candidate_uuid: "90a0b30f5377440b8d72889b9d2f0a70" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aeb32dce70ca494ca326c528e640c9ce"
I20250112 15:05:54.722332 18407 raft_consensus.cc:3054] T b3b76094dfae41f38c01fd10b38c7a28 P aeb32dce70ca494ca326c528e640c9ce [term 0 FOLLOWER]: Advancing to term 1
I20250112 15:05:54.722286 18331 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3b76094dfae41f38c01fd10b38c7a28" candidate_uuid: "90a0b30f5377440b8d72889b9d2f0a70" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "79518508ccd644b28675a5de1e7280e2"
I20250112 15:05:54.722366 18331 raft_consensus.cc:3054] T b3b76094dfae41f38c01fd10b38c7a28 P 79518508ccd644b28675a5de1e7280e2 [term 0 FOLLOWER]: Advancing to term 1
I20250112 15:05:54.722797 18407 raft_consensus.cc:2463] T b3b76094dfae41f38c01fd10b38c7a28 P aeb32dce70ca494ca326c528e640c9ce [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 90a0b30f5377440b8d72889b9d2f0a70 in term 1.
I20250112 15:05:54.722854 18331 raft_consensus.cc:2463] T b3b76094dfae41f38c01fd10b38c7a28 P 79518508ccd644b28675a5de1e7280e2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 90a0b30f5377440b8d72889b9d2f0a70 in term 1.
I20250112 15:05:54.722951 18446 leader_election.cc:304] T b3b76094dfae41f38c01fd10b38c7a28 P 90a0b30f5377440b8d72889b9d2f0a70 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 90a0b30f5377440b8d72889b9d2f0a70, aeb32dce70ca494ca326c528e640c9ce; no voters:
I20250112 15:05:54.723067 18544 raft_consensus.cc:2798] T b3b76094dfae41f38c01fd10b38c7a28 P 90a0b30f5377440b8d72889b9d2f0a70 [term 1 FOLLOWER]: Leader election won for term 1
I20250112 15:05:54.723117 18544 raft_consensus.cc:695] T b3b76094dfae41f38c01fd10b38c7a28 P 90a0b30f5377440b8d72889b9d2f0a70 [term 1 LEADER]: Becoming Leader. State: Replica: 90a0b30f5377440b8d72889b9d2f0a70, State: Running, Role: LEADER
I20250112 15:05:54.723166 18544 consensus_queue.cc:237] T b3b76094dfae41f38c01fd10b38c7a28 P 90a0b30f5377440b8d72889b9d2f0a70 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79518508ccd644b28675a5de1e7280e2" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 38531 } } peers { permanent_uuid: "aeb32dce70ca494ca326c528e640c9ce" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 35351 } } peers { permanent_uuid: "90a0b30f5377440b8d72889b9d2f0a70" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 35603 } }
I20250112 15:05:54.723629 18226 catalog_manager.cc:5526] T b3b76094dfae41f38c01fd10b38c7a28 P 90a0b30f5377440b8d72889b9d2f0a70 reported cstate change: term changed from 0 to 1, leader changed from <none> to 90a0b30f5377440b8d72889b9d2f0a70 (127.17.60.195). New cstate: current_term: 1 leader_uuid: "90a0b30f5377440b8d72889b9d2f0a70" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79518508ccd644b28675a5de1e7280e2" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 38531 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "aeb32dce70ca494ca326c528e640c9ce" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 35351 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "90a0b30f5377440b8d72889b9d2f0a70" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 35603 } health_report { overall_health: HEALTHY } } }
I20250112 15:05:54.772353 18544 raft_consensus.cc:491] T e311741568d04cca81089b8bad66b805 P 90a0b30f5377440b8d72889b9d2f0a70 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 15:05:54.772454 18544 raft_consensus.cc:513] T e311741568d04cca81089b8bad66b805 P 90a0b30f5377440b8d72889b9d2f0a70 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90a0b30f5377440b8d72889b9d2f0a70" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 35603 } } peers { permanent_uuid: "79518508ccd644b28675a5de1e7280e2" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 38531 } } peers { permanent_uuid: "aeb32dce70ca494ca326c528e640c9ce" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 35351 } }
I20250112 15:05:54.772620 18544 leader_election.cc:290] T e311741568d04cca81089b8bad66b805 P 90a0b30f5377440b8d72889b9d2f0a70 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 79518508ccd644b28675a5de1e7280e2 (127.17.60.193:38531), aeb32dce70ca494ca326c528e640c9ce (127.17.60.194:35351)
I20250112 15:05:54.772874 18331 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e311741568d04cca81089b8bad66b805" candidate_uuid: "90a0b30f5377440b8d72889b9d2f0a70" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "79518508ccd644b28675a5de1e7280e2" is_pre_election: true
I20250112 15:05:54.772930 18407 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e311741568d04cca81089b8bad66b805" candidate_uuid: "90a0b30f5377440b8d72889b9d2f0a70" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aeb32dce70ca494ca326c528e640c9ce" is_pre_election: true
I20250112 15:05:54.772972 18331 raft_consensus.cc:2463] T e311741568d04cca81089b8bad66b805 P 79518508ccd644b28675a5de1e7280e2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 90a0b30f5377440b8d72889b9d2f0a70 in term 0.
I20250112 15:05:54.772989 18407 raft_consensus.cc:2463] T e311741568d04cca81089b8bad66b805 P aeb32dce70ca494ca326c528e640c9ce [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 90a0b30f5377440b8d72889b9d2f0a70 in term 0.
I20250112 15:05:54.773149 18445 leader_election.cc:304] T e311741568d04cca81089b8bad66b805 P 90a0b30f5377440b8d72889b9d2f0a70 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 79518508ccd644b28675a5de1e7280e2, 90a0b30f5377440b8d72889b9d2f0a70; no voters:
I20250112 15:05:54.773226 18544 raft_consensus.cc:2798] T e311741568d04cca81089b8bad66b805 P 90a0b30f5377440b8d72889b9d2f0a70 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250112 15:05:54.773270 18544 raft_consensus.cc:491] T e311741568d04cca81089b8bad66b805 P 90a0b30f5377440b8d72889b9d2f0a70 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250112 15:05:54.773293 18544 raft_consensus.cc:3054] T e311741568d04cca81089b8bad66b805 P 90a0b30f5377440b8d72889b9d2f0a70 [term 0 FOLLOWER]: Advancing to term 1
I20250112 15:05:54.773841 18544 raft_consensus.cc:513] T e311741568d04cca81089b8bad66b805 P 90a0b30f5377440b8d72889b9d2f0a70 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90a0b30f5377440b8d72889b9d2f0a70" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 35603 } } peers { permanent_uuid: "79518508ccd644b28675a5de1e7280e2" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 38531 } } peers { permanent_uuid: "aeb32dce70ca494ca326c528e640c9ce" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 35351 } }
I20250112 15:05:54.773950 18544 leader_election.cc:290] T e311741568d04cca81089b8bad66b805 P 90a0b30f5377440b8d72889b9d2f0a70 [CANDIDATE]: Term 1 election: Requested vote from peers 79518508ccd644b28675a5de1e7280e2 (127.17.60.193:38531), aeb32dce70ca494ca326c528e640c9ce (127.17.60.194:35351)
I20250112 15:05:54.774101 18331 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e311741568d04cca81089b8bad66b805" candidate_uuid: "90a0b30f5377440b8d72889b9d2f0a70" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "79518508ccd644b28675a5de1e7280e2"
I20250112 15:05:54.774101 18407 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e311741568d04cca81089b8bad66b805" candidate_uuid: "90a0b30f5377440b8d72889b9d2f0a70" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aeb32dce70ca494ca326c528e640c9ce"
I20250112 15:05:54.774205 18331 raft_consensus.cc:3054] T e311741568d04cca81089b8bad66b805 P 79518508ccd644b28675a5de1e7280e2 [term 0 FOLLOWER]: Advancing to term 1
I20250112 15:05:54.774215 18407 raft_consensus.cc:3054] T e311741568d04cca81089b8bad66b805 P aeb32dce70ca494ca326c528e640c9ce [term 0 FOLLOWER]: Advancing to term 1
I20250112 15:05:54.774798 18331 raft_consensus.cc:2463] T e311741568d04cca81089b8bad66b805 P 79518508ccd644b28675a5de1e7280e2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 90a0b30f5377440b8d72889b9d2f0a70 in term 1.
I20250112 15:05:54.774813 18407 raft_consensus.cc:2463] T e311741568d04cca81089b8bad66b805 P aeb32dce70ca494ca326c528e640c9ce [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 90a0b30f5377440b8d72889b9d2f0a70 in term 1.
I20250112 15:05:54.774953 18445 leader_election.cc:304] T e311741568d04cca81089b8bad66b805 P 90a0b30f5377440b8d72889b9d2f0a70 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 79518508ccd644b28675a5de1e7280e2, 90a0b30f5377440b8d72889b9d2f0a70; no voters:
I20250112 15:05:54.775031 18544 raft_consensus.cc:2798] T e311741568d04cca81089b8bad66b805 P 90a0b30f5377440b8d72889b9d2f0a70 [term 1 FOLLOWER]: Leader election won for term 1
I20250112 15:05:54.775081 18544 raft_consensus.cc:695] T e311741568d04cca81089b8bad66b805 P 90a0b30f5377440b8d72889b9d2f0a70 [term 1 LEADER]: Becoming Leader. State: Replica: 90a0b30f5377440b8d72889b9d2f0a70, State: Running, Role: LEADER
I20250112 15:05:54.775130 18544 consensus_queue.cc:237] T e311741568d04cca81089b8bad66b805 P 90a0b30f5377440b8d72889b9d2f0a70 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90a0b30f5377440b8d72889b9d2f0a70" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 35603 } } peers { permanent_uuid: "79518508ccd644b28675a5de1e7280e2" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 38531 } } peers { permanent_uuid: "aeb32dce70ca494ca326c528e640c9ce" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 35351 } }
I20250112 15:05:54.775661 18226 catalog_manager.cc:5526] T e311741568d04cca81089b8bad66b805 P 90a0b30f5377440b8d72889b9d2f0a70 reported cstate change: term changed from 0 to 1, leader changed from <none> to 90a0b30f5377440b8d72889b9d2f0a70 (127.17.60.195). New cstate: current_term: 1 leader_uuid: "90a0b30f5377440b8d72889b9d2f0a70" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90a0b30f5377440b8d72889b9d2f0a70" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 35603 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "79518508ccd644b28675a5de1e7280e2" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 38531 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "aeb32dce70ca494ca326c528e640c9ce" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 35351 } health_report { overall_health: UNKNOWN } } }
I20250112 15:05:54.796243 18534 raft_consensus.cc:491] T cf6c409b42d842cfa25a995f95544e5e P 79518508ccd644b28675a5de1e7280e2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 15:05:54.796343 18534 raft_consensus.cc:513] T cf6c409b42d842cfa25a995f95544e5e P 79518508ccd644b28675a5de1e7280e2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79518508ccd644b28675a5de1e7280e2" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 38531 } } peers { permanent_uuid: "aeb32dce70ca494ca326c528e640c9ce" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 35351 } } peers { permanent_uuid: "90a0b30f5377440b8d72889b9d2f0a70" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 35603 } }
I20250112 15:05:54.796515 18534 leader_election.cc:290] T cf6c409b42d842cfa25a995f95544e5e P 79518508ccd644b28675a5de1e7280e2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers aeb32dce70ca494ca326c528e640c9ce (127.17.60.194:35351), 90a0b30f5377440b8d72889b9d2f0a70 (127.17.60.195:35603)
I20250112 15:05:54.799060 18482 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf6c409b42d842cfa25a995f95544e5e" candidate_uuid: "79518508ccd644b28675a5de1e7280e2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "90a0b30f5377440b8d72889b9d2f0a70" is_pre_election: true
I20250112 15:05:54.799160 18482 raft_consensus.cc:2463] T cf6c409b42d842cfa25a995f95544e5e P 90a0b30f5377440b8d72889b9d2f0a70 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 79518508ccd644b28675a5de1e7280e2 in term 0.
I20250112 15:05:54.799134 18407 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf6c409b42d842cfa25a995f95544e5e" candidate_uuid: "79518508ccd644b28675a5de1e7280e2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aeb32dce70ca494ca326c528e640c9ce" is_pre_election: true
I20250112 15:05:54.799233 18407 raft_consensus.cc:2463] T cf6c409b42d842cfa25a995f95544e5e P aeb32dce70ca494ca326c528e640c9ce [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 79518508ccd644b28675a5de1e7280e2 in term 0.
I20250112 15:05:54.799327 18294 leader_election.cc:304] T cf6c409b42d842cfa25a995f95544e5e P 79518508ccd644b28675a5de1e7280e2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 79518508ccd644b28675a5de1e7280e2, 90a0b30f5377440b8d72889b9d2f0a70; no voters:
I20250112 15:05:54.799409 18534 raft_consensus.cc:2798] T cf6c409b42d842cfa25a995f95544e5e P 79518508ccd644b28675a5de1e7280e2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250112 15:05:54.799448 18534 raft_consensus.cc:491] T cf6c409b42d842cfa25a995f95544e5e P 79518508ccd644b28675a5de1e7280e2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250112 15:05:54.799466 18534 raft_consensus.cc:3054] T cf6c409b42d842cfa25a995f95544e5e P 79518508ccd644b28675a5de1e7280e2 [term 0 FOLLOWER]: Advancing to term 1
I20250112 15:05:54.800107 18534 raft_consensus.cc:513] T cf6c409b42d842cfa25a995f95544e5e P 79518508ccd644b28675a5de1e7280e2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79518508ccd644b28675a5de1e7280e2" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 38531 } } peers { permanent_uuid: "aeb32dce70ca494ca326c528e640c9ce" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 35351 } } peers { permanent_uuid: "90a0b30f5377440b8d72889b9d2f0a70" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 35603 } }
I20250112 15:05:54.800222 18534 leader_election.cc:290] T cf6c409b42d842cfa25a995f95544e5e P 79518508ccd644b28675a5de1e7280e2 [CANDIDATE]: Term 1 election: Requested vote from peers aeb32dce70ca494ca326c528e640c9ce (127.17.60.194:35351), 90a0b30f5377440b8d72889b9d2f0a70 (127.17.60.195:35603)
I20250112 15:05:54.800343 18407 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf6c409b42d842cfa25a995f95544e5e" candidate_uuid: "79518508ccd644b28675a5de1e7280e2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aeb32dce70ca494ca326c528e640c9ce"
I20250112 15:05:54.800392 18482 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf6c409b42d842cfa25a995f95544e5e" candidate_uuid: "79518508ccd644b28675a5de1e7280e2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "90a0b30f5377440b8d72889b9d2f0a70"
I20250112 15:05:54.800416 18407 raft_consensus.cc:3054] T cf6c409b42d842cfa25a995f95544e5e P aeb32dce70ca494ca326c528e640c9ce [term 0 FOLLOWER]: Advancing to term 1
I20250112 15:05:54.800434 18482 raft_consensus.cc:3054] T cf6c409b42d842cfa25a995f95544e5e P 90a0b30f5377440b8d72889b9d2f0a70 [term 0 FOLLOWER]: Advancing to term 1
I20250112 15:05:54.801025 18407 raft_consensus.cc:2463] T cf6c409b42d842cfa25a995f95544e5e P aeb32dce70ca494ca326c528e640c9ce [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 79518508ccd644b28675a5de1e7280e2 in term 1.
I20250112 15:05:54.801025 18482 raft_consensus.cc:2463] T cf6c409b42d842cfa25a995f95544e5e P 90a0b30f5377440b8d72889b9d2f0a70 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 79518508ccd644b28675a5de1e7280e2 in term 1.
I20250112 15:05:54.801205 18294 leader_election.cc:304] T cf6c409b42d842cfa25a995f95544e5e P 79518508ccd644b28675a5de1e7280e2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 79518508ccd644b28675a5de1e7280e2, 90a0b30f5377440b8d72889b9d2f0a70; no voters:
I20250112 15:05:54.801314 18534 raft_consensus.cc:2798] T cf6c409b42d842cfa25a995f95544e5e P 79518508ccd644b28675a5de1e7280e2 [term 1 FOLLOWER]: Leader election won for term 1
I20250112 15:05:54.801445 18534 raft_consensus.cc:695] T cf6c409b42d842cfa25a995f95544e5e P 79518508ccd644b28675a5de1e7280e2 [term 1 LEADER]: Becoming Leader. State: Replica: 79518508ccd644b28675a5de1e7280e2, State: Running, Role: LEADER
I20250112 15:05:54.801491 18534 consensus_queue.cc:237] T cf6c409b42d842cfa25a995f95544e5e P 79518508ccd644b28675a5de1e7280e2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79518508ccd644b28675a5de1e7280e2" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 38531 } } peers { permanent_uuid: "aeb32dce70ca494ca326c528e640c9ce" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 35351 } } peers { permanent_uuid: "90a0b30f5377440b8d72889b9d2f0a70" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 35603 } }
I20250112 15:05:54.802019 18226 catalog_manager.cc:5526] T cf6c409b42d842cfa25a995f95544e5e P 79518508ccd644b28675a5de1e7280e2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 79518508ccd644b28675a5de1e7280e2 (127.17.60.193). New cstate: current_term: 1 leader_uuid: "79518508ccd644b28675a5de1e7280e2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79518508ccd644b28675a5de1e7280e2" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 38531 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "aeb32dce70ca494ca326c528e640c9ce" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 35351 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "90a0b30f5377440b8d72889b9d2f0a70" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 35603 } health_report { overall_health: UNKNOWN } } }
W20250112 15:05:54.843050 18358 tablet.cc:2367] T 2319a98d89eb4234aa86cc3930853fce P 79518508ccd644b28675a5de1e7280e2: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250112 15:05:54.869541 18534 raft_consensus.cc:491] T 45204e59d67b4abe9ed00311ca03b73d P 79518508ccd644b28675a5de1e7280e2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 15:05:54.869644 18534 raft_consensus.cc:513] T 45204e59d67b4abe9ed00311ca03b73d P 79518508ccd644b28675a5de1e7280e2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79518508ccd644b28675a5de1e7280e2" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 38531 } } peers { permanent_uuid: "aeb32dce70ca494ca326c528e640c9ce" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 35351 } } peers { permanent_uuid: "90a0b30f5377440b8d72889b9d2f0a70" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 35603 } }
I20250112 15:05:54.869791 18534 leader_election.cc:290] T 45204e59d67b4abe9ed00311ca03b73d P 79518508ccd644b28675a5de1e7280e2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers aeb32dce70ca494ca326c528e640c9ce (127.17.60.194:35351), 90a0b30f5377440b8d72889b9d2f0a70 (127.17.60.195:35603)
I20250112 15:05:54.870015 18407 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "45204e59d67b4abe9ed00311ca03b73d" candidate_uuid: "79518508ccd644b28675a5de1e7280e2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aeb32dce70ca494ca326c528e640c9ce" is_pre_election: true
I20250112 15:05:54.870090 18407 raft_consensus.cc:2463] T 45204e59d67b4abe9ed00311ca03b73d P aeb32dce70ca494ca326c528e640c9ce [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 79518508ccd644b28675a5de1e7280e2 in term 0.
I20250112 15:05:54.870026 18482 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "45204e59d67b4abe9ed00311ca03b73d" candidate_uuid: "79518508ccd644b28675a5de1e7280e2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "90a0b30f5377440b8d72889b9d2f0a70" is_pre_election: true
I20250112 15:05:54.870134 18482 raft_consensus.cc:2463] T 45204e59d67b4abe9ed00311ca03b73d P 90a0b30f5377440b8d72889b9d2f0a70 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 79518508ccd644b28675a5de1e7280e2 in term 0.
I20250112 15:05:54.870256 18295 leader_election.cc:304] T 45204e59d67b4abe9ed00311ca03b73d P 79518508ccd644b28675a5de1e7280e2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 79518508ccd644b28675a5de1e7280e2, aeb32dce70ca494ca326c528e640c9ce; no voters:
I20250112 15:05:54.870343 18534 raft_consensus.cc:2798] T 45204e59d67b4abe9ed00311ca03b73d P 79518508ccd644b28675a5de1e7280e2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250112 15:05:54.870388 18534 raft_consensus.cc:491] T 45204e59d67b4abe9ed00311ca03b73d P 79518508ccd644b28675a5de1e7280e2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250112 15:05:54.870410 18534 raft_consensus.cc:3054] T 45204e59d67b4abe9ed00311ca03b73d P 79518508ccd644b28675a5de1e7280e2 [term 0 FOLLOWER]: Advancing to term 1
I20250112 15:05:54.871012 18534 raft_consensus.cc:513] T 45204e59d67b4abe9ed00311ca03b73d P 79518508ccd644b28675a5de1e7280e2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79518508ccd644b28675a5de1e7280e2" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 38531 } } peers { permanent_uuid: "aeb32dce70ca494ca326c528e640c9ce" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 35351 } } peers { permanent_uuid: "90a0b30f5377440b8d72889b9d2f0a70" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 35603 } }
I20250112 15:05:54.871135 18534 leader_election.cc:290] T 45204e59d67b4abe9ed00311ca03b73d P 79518508ccd644b28675a5de1e7280e2 [CANDIDATE]: Term 1 election: Requested vote from peers aeb32dce70ca494ca326c528e640c9ce (127.17.60.194:35351), 90a0b30f5377440b8d72889b9d2f0a70 (127.17.60.195:35603)
I20250112 15:05:54.871174 18533 raft_consensus.cc:491] T 9032f9edb5a746729802355e7d212c95 P aeb32dce70ca494ca326c528e640c9ce [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 15:05:54.871223 18533 raft_consensus.cc:513] T 9032f9edb5a746729802355e7d212c95 P aeb32dce70ca494ca326c528e640c9ce [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79518508ccd644b28675a5de1e7280e2" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 38531 } } peers { permanent_uuid: "aeb32dce70ca494ca326c528e640c9ce" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 35351 } } peers { permanent_uuid: "90a0b30f5377440b8d72889b9d2f0a70" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 35603 } }
I20250112 15:05:54.871263 18407 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "45204e59d67b4abe9ed00311ca03b73d" candidate_uuid: "79518508ccd644b28675a5de1e7280e2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aeb32dce70ca494ca326c528e640c9ce"
I20250112 15:05:54.871308 18482 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "45204e59d67b4abe9ed00311ca03b73d" candidate_uuid: "79518508ccd644b28675a5de1e7280e2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "90a0b30f5377440b8d72889b9d2f0a70"
I20250112 15:05:54.871379 18482 raft_consensus.cc:3054] T 45204e59d67b4abe9ed00311ca03b73d P 90a0b30f5377440b8d72889b9d2f0a70 [term 0 FOLLOWER]: Advancing to term 1
I20250112 15:05:54.871327 18533 leader_election.cc:290] T 9032f9edb5a746729802355e7d212c95 P aeb32dce70ca494ca326c528e640c9ce [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 79518508ccd644b28675a5de1e7280e2 (127.17.60.193:38531), 90a0b30f5377440b8d72889b9d2f0a70 (127.17.60.195:35603)
I20250112 15:05:54.871455 18331 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9032f9edb5a746729802355e7d212c95" candidate_uuid: "aeb32dce70ca494ca326c528e640c9ce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "79518508ccd644b28675a5de1e7280e2" is_pre_election: true
I20250112 15:05:54.871511 18331 raft_consensus.cc:2463] T 9032f9edb5a746729802355e7d212c95 P 79518508ccd644b28675a5de1e7280e2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate aeb32dce70ca494ca326c528e640c9ce in term 0.
I20250112 15:05:54.871500 18481 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9032f9edb5a746729802355e7d212c95" candidate_uuid: "aeb32dce70ca494ca326c528e640c9ce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "90a0b30f5377440b8d72889b9d2f0a70" is_pre_election: true
I20250112 15:05:54.871322 18407 raft_consensus.cc:3054] T 45204e59d67b4abe9ed00311ca03b73d P aeb32dce70ca494ca326c528e640c9ce [term 0 FOLLOWER]: Advancing to term 1
I20250112 15:05:54.871610 18481 raft_consensus.cc:2463] T 9032f9edb5a746729802355e7d212c95 P 90a0b30f5377440b8d72889b9d2f0a70 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate aeb32dce70ca494ca326c528e640c9ce in term 0.
I20250112 15:05:54.871701 18370 leader_election.cc:304] T 9032f9edb5a746729802355e7d212c95 P aeb32dce70ca494ca326c528e640c9ce [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 79518508ccd644b28675a5de1e7280e2, aeb32dce70ca494ca326c528e640c9ce; no voters:
I20250112 15:05:54.871773 18533 raft_consensus.cc:2798] T 9032f9edb5a746729802355e7d212c95 P aeb32dce70ca494ca326c528e640c9ce [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250112 15:05:54.871799 18533 raft_consensus.cc:491] T 9032f9edb5a746729802355e7d212c95 P aeb32dce70ca494ca326c528e640c9ce [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250112 15:05:54.871824 18533 raft_consensus.cc:3054] T 9032f9edb5a746729802355e7d212c95 P aeb32dce70ca494ca326c528e640c9ce [term 0 FOLLOWER]: Advancing to term 1
I20250112 15:05:54.872032 18482 raft_consensus.cc:2463] T 45204e59d67b4abe9ed00311ca03b73d P 90a0b30f5377440b8d72889b9d2f0a70 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 79518508ccd644b28675a5de1e7280e2 in term 1.
I20250112 15:05:54.872216 18294 leader_election.cc:304] T 45204e59d67b4abe9ed00311ca03b73d P 79518508ccd644b28675a5de1e7280e2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 79518508ccd644b28675a5de1e7280e2, 90a0b30f5377440b8d72889b9d2f0a70; no voters:
I20250112 15:05:54.872223 18407 raft_consensus.cc:2463] T 45204e59d67b4abe9ed00311ca03b73d P aeb32dce70ca494ca326c528e640c9ce [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 79518508ccd644b28675a5de1e7280e2 in term 1.
I20250112 15:05:54.872296 18534 raft_consensus.cc:2798] T 45204e59d67b4abe9ed00311ca03b73d P 79518508ccd644b28675a5de1e7280e2 [term 1 FOLLOWER]: Leader election won for term 1
I20250112 15:05:54.872337 18534 raft_consensus.cc:695] T 45204e59d67b4abe9ed00311ca03b73d P 79518508ccd644b28675a5de1e7280e2 [term 1 LEADER]: Becoming Leader. State: Replica: 79518508ccd644b28675a5de1e7280e2, State: Running, Role: LEADER
I20250112 15:05:54.872386 18534 consensus_queue.cc:237] T 45204e59d67b4abe9ed00311ca03b73d P 79518508ccd644b28675a5de1e7280e2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79518508ccd644b28675a5de1e7280e2" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 38531 } } peers { permanent_uuid: "aeb32dce70ca494ca326c528e640c9ce" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 35351 } } peers { permanent_uuid: "90a0b30f5377440b8d72889b9d2f0a70" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 35603 } }
I20250112 15:05:54.872504 18533 raft_consensus.cc:513] T 9032f9edb5a746729802355e7d212c95 P aeb32dce70ca494ca326c528e640c9ce [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79518508ccd644b28675a5de1e7280e2" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 38531 } } peers { permanent_uuid: "aeb32dce70ca494ca326c528e640c9ce" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 35351 } } peers { permanent_uuid: "90a0b30f5377440b8d72889b9d2f0a70" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 35603 } }
I20250112 15:05:54.872627 18533 leader_election.cc:290] T 9032f9edb5a746729802355e7d212c95 P aeb32dce70ca494ca326c528e640c9ce [CANDIDATE]: Term 1 election: Requested vote from peers 79518508ccd644b28675a5de1e7280e2 (127.17.60.193:38531), 90a0b30f5377440b8d72889b9d2f0a70 (127.17.60.195:35603)
I20250112 15:05:54.872753 18482 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9032f9edb5a746729802355e7d212c95" candidate_uuid: "aeb32dce70ca494ca326c528e640c9ce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "90a0b30f5377440b8d72889b9d2f0a70"
I20250112 15:05:54.872762 18331 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9032f9edb5a746729802355e7d212c95" candidate_uuid: "aeb32dce70ca494ca326c528e640c9ce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "79518508ccd644b28675a5de1e7280e2"
I20250112 15:05:54.872828 18482 raft_consensus.cc:3054] T 9032f9edb5a746729802355e7d212c95 P 90a0b30f5377440b8d72889b9d2f0a70 [term 0 FOLLOWER]: Advancing to term 1
I20250112 15:05:54.872852 18331 raft_consensus.cc:3054] T 9032f9edb5a746729802355e7d212c95 P 79518508ccd644b28675a5de1e7280e2 [term 0 FOLLOWER]: Advancing to term 1
I20250112 15:05:54.872933 18226 catalog_manager.cc:5526] T 45204e59d67b4abe9ed00311ca03b73d P 79518508ccd644b28675a5de1e7280e2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 79518508ccd644b28675a5de1e7280e2 (127.17.60.193). New cstate: current_term: 1 leader_uuid: "79518508ccd644b28675a5de1e7280e2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79518508ccd644b28675a5de1e7280e2" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 38531 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "aeb32dce70ca494ca326c528e640c9ce" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 35351 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "90a0b30f5377440b8d72889b9d2f0a70" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 35603 } health_report { overall_health: UNKNOWN } } }
I20250112 15:05:54.873359 18482 raft_consensus.cc:2463] T 9032f9edb5a746729802355e7d212c95 P 90a0b30f5377440b8d72889b9d2f0a70 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate aeb32dce70ca494ca326c528e640c9ce in term 1.
I20250112 15:05:54.873474 18331 raft_consensus.cc:2463] T 9032f9edb5a746729802355e7d212c95 P 79518508ccd644b28675a5de1e7280e2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate aeb32dce70ca494ca326c528e640c9ce in term 1.
I20250112 15:05:54.873497 18370 leader_election.cc:304] T 9032f9edb5a746729802355e7d212c95 P aeb32dce70ca494ca326c528e640c9ce [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 90a0b30f5377440b8d72889b9d2f0a70, aeb32dce70ca494ca326c528e640c9ce; no voters:
I20250112 15:05:54.873593 18533 raft_consensus.cc:2798] T 9032f9edb5a746729802355e7d212c95 P aeb32dce70ca494ca326c528e640c9ce [term 1 FOLLOWER]: Leader election won for term 1
I20250112 15:05:54.873656 18533 raft_consensus.cc:695] T 9032f9edb5a746729802355e7d212c95 P aeb32dce70ca494ca326c528e640c9ce [term 1 LEADER]: Becoming Leader. State: Replica: aeb32dce70ca494ca326c528e640c9ce, State: Running, Role: LEADER
I20250112 15:05:54.873706 18533 consensus_queue.cc:237] T 9032f9edb5a746729802355e7d212c95 P aeb32dce70ca494ca326c528e640c9ce [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79518508ccd644b28675a5de1e7280e2" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 38531 } } peers { permanent_uuid: "aeb32dce70ca494ca326c528e640c9ce" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 35351 } } peers { permanent_uuid: "90a0b30f5377440b8d72889b9d2f0a70" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 35603 } }
I20250112 15:05:54.874161 18226 catalog_manager.cc:5526] T 9032f9edb5a746729802355e7d212c95 P aeb32dce70ca494ca326c528e640c9ce reported cstate change: term changed from 0 to 1, leader changed from <none> to aeb32dce70ca494ca326c528e640c9ce (127.17.60.194). New cstate: current_term: 1 leader_uuid: "aeb32dce70ca494ca326c528e640c9ce" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79518508ccd644b28675a5de1e7280e2" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 38531 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "aeb32dce70ca494ca326c528e640c9ce" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 35351 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "90a0b30f5377440b8d72889b9d2f0a70" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 35603 } health_report { overall_health: UNKNOWN } } }
I20250112 15:05:54.986418 18534 raft_consensus.cc:491] T 4e45a9f365b344c2aa0703adea508eeb P 79518508ccd644b28675a5de1e7280e2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 15:05:54.986523 18534 raft_consensus.cc:513] T 4e45a9f365b344c2aa0703adea508eeb P 79518508ccd644b28675a5de1e7280e2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79518508ccd644b28675a5de1e7280e2" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 38531 } } peers { permanent_uuid: "aeb32dce70ca494ca326c528e640c9ce" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 35351 } } peers { permanent_uuid: "90a0b30f5377440b8d72889b9d2f0a70" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 35603 } }
I20250112 15:05:54.986658 18534 leader_election.cc:290] T 4e45a9f365b344c2aa0703adea508eeb P 79518508ccd644b28675a5de1e7280e2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers aeb32dce70ca494ca326c528e640c9ce (127.17.60.194:35351), 90a0b30f5377440b8d72889b9d2f0a70 (127.17.60.195:35603)
I20250112 15:05:54.986922 18482 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e45a9f365b344c2aa0703adea508eeb" candidate_uuid: "79518508ccd644b28675a5de1e7280e2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "90a0b30f5377440b8d72889b9d2f0a70" is_pre_election: true
I20250112 15:05:54.987017 18482 raft_consensus.cc:2463] T 4e45a9f365b344c2aa0703adea508eeb P 90a0b30f5377440b8d72889b9d2f0a70 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 79518508ccd644b28675a5de1e7280e2 in term 0.
I20250112 15:05:54.987179 18407 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e45a9f365b344c2aa0703adea508eeb" candidate_uuid: "79518508ccd644b28675a5de1e7280e2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aeb32dce70ca494ca326c528e640c9ce" is_pre_election: true
I20250112 15:05:54.987241 18407 raft_consensus.cc:2463] T 4e45a9f365b344c2aa0703adea508eeb P aeb32dce70ca494ca326c528e640c9ce [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 79518508ccd644b28675a5de1e7280e2 in term 0.
I20250112 15:05:54.987386 18294 leader_election.cc:304] T 4e45a9f365b344c2aa0703adea508eeb P 79518508ccd644b28675a5de1e7280e2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 79518508ccd644b28675a5de1e7280e2, 90a0b30f5377440b8d72889b9d2f0a70; no voters:
I20250112 15:05:54.987485 18534 raft_consensus.cc:2798] T 4e45a9f365b344c2aa0703adea508eeb P 79518508ccd644b28675a5de1e7280e2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250112 15:05:54.987533 18534 raft_consensus.cc:491] T 4e45a9f365b344c2aa0703adea508eeb P 79518508ccd644b28675a5de1e7280e2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250112 15:05:54.987550 18534 raft_consensus.cc:3054] T 4e45a9f365b344c2aa0703adea508eeb P 79518508ccd644b28675a5de1e7280e2 [term 0 FOLLOWER]: Advancing to term 1
I20250112 15:05:54.988159 18534 raft_consensus.cc:513] T 4e45a9f365b344c2aa0703adea508eeb P 79518508ccd644b28675a5de1e7280e2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79518508ccd644b28675a5de1e7280e2" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 38531 } } peers { permanent_uuid: "aeb32dce70ca494ca326c528e640c9ce" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 35351 } } peers { permanent_uuid: "90a0b30f5377440b8d72889b9d2f0a70" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 35603 } }
I20250112 15:05:54.988282 18534 leader_election.cc:290] T 4e45a9f365b344c2aa0703adea508eeb P 79518508ccd644b28675a5de1e7280e2 [CANDIDATE]: Term 1 election: Requested vote from peers aeb32dce70ca494ca326c528e640c9ce (127.17.60.194:35351), 90a0b30f5377440b8d72889b9d2f0a70 (127.17.60.195:35603)
I20250112 15:05:54.988405 18407 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e45a9f365b344c2aa0703adea508eeb" candidate_uuid: "79518508ccd644b28675a5de1e7280e2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aeb32dce70ca494ca326c528e640c9ce"
I20250112 15:05:54.988458 18407 raft_consensus.cc:3054] T 4e45a9f365b344c2aa0703adea508eeb P aeb32dce70ca494ca326c528e640c9ce [term 0 FOLLOWER]: Advancing to term 1
I20250112 15:05:54.988415 18482 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e45a9f365b344c2aa0703adea508eeb" candidate_uuid: "79518508ccd644b28675a5de1e7280e2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "90a0b30f5377440b8d72889b9d2f0a70"
I20250112 15:05:54.988489 18482 raft_consensus.cc:3054] T 4e45a9f365b344c2aa0703adea508eeb P 90a0b30f5377440b8d72889b9d2f0a70 [term 0 FOLLOWER]: Advancing to term 1
I20250112 15:05:54.988940 18407 raft_consensus.cc:2463] T 4e45a9f365b344c2aa0703adea508eeb P aeb32dce70ca494ca326c528e640c9ce [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 79518508ccd644b28675a5de1e7280e2 in term 1.
I20250112 15:05:54.989039 18482 raft_consensus.cc:2463] T 4e45a9f365b344c2aa0703adea508eeb P 90a0b30f5377440b8d72889b9d2f0a70 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 79518508ccd644b28675a5de1e7280e2 in term 1.
I20250112 15:05:54.989105 18295 leader_election.cc:304] T 4e45a9f365b344c2aa0703adea508eeb P 79518508ccd644b28675a5de1e7280e2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 79518508ccd644b28675a5de1e7280e2, aeb32dce70ca494ca326c528e640c9ce; no voters:
I20250112 15:05:54.989184 18534 raft_consensus.cc:2798] T 4e45a9f365b344c2aa0703adea508eeb P 79518508ccd644b28675a5de1e7280e2 [term 1 FOLLOWER]: Leader election won for term 1
I20250112 15:05:54.989229 18534 raft_consensus.cc:695] T 4e45a9f365b344c2aa0703adea508eeb P 79518508ccd644b28675a5de1e7280e2 [term 1 LEADER]: Becoming Leader. State: Replica: 79518508ccd644b28675a5de1e7280e2, State: Running, Role: LEADER
I20250112 15:05:54.989269 18534 consensus_queue.cc:237] T 4e45a9f365b344c2aa0703adea508eeb P 79518508ccd644b28675a5de1e7280e2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79518508ccd644b28675a5de1e7280e2" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 38531 } } peers { permanent_uuid: "aeb32dce70ca494ca326c528e640c9ce" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 35351 } } peers { permanent_uuid: "90a0b30f5377440b8d72889b9d2f0a70" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 35603 } }
I20250112 15:05:54.989835 18226 catalog_manager.cc:5526] T 4e45a9f365b344c2aa0703adea508eeb P 79518508ccd644b28675a5de1e7280e2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 79518508ccd644b28675a5de1e7280e2 (127.17.60.193). New cstate: current_term: 1 leader_uuid: "79518508ccd644b28675a5de1e7280e2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79518508ccd644b28675a5de1e7280e2" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 38531 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "aeb32dce70ca494ca326c528e640c9ce" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 35351 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "90a0b30f5377440b8d72889b9d2f0a70" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 35603 } health_report { overall_health: UNKNOWN } } }
I20250112 15:05:55.024400 17651 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250112 15:05:55.025882 18575 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250112 15:05:55.026609 18578 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250112 15:05:55.027138 18576 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250112 15:05:55.028090 17651 server_base.cc:1034] running on GCE node
I20250112 15:05:55.028214 17651 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250112 15:05:55.028241 17651 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250112 15:05:55.028254 17651 hybrid_clock.cc:648] HybridClock initialized: now 1736694355028254 us; error 0 us; skew 500 ppm
I20250112 15:05:55.029012 17651 webserver.cc:458] Webserver started at http://127.17.60.196:37413/ using document root <none> and password file <none>
I20250112 15:05:55.029093 17651 fs_manager.cc:362] Metadata directory not provided
I20250112 15:05:55.029145 17651 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250112 15:05:55.029202 17651 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250112 15:05:55.029515 17651 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task95dLSD/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1736694351070047-17651-0/minicluster-data/ts-3-root/instance:
uuid: "6bab59d1be384346a2b1354fb677ec01"
format_stamp: "Formatted at 2025-01-12 15:05:55 on dist-test-slave-b4nz"
I20250112 15:05:55.030416 17651 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.001s sys 0.001s
I20250112 15:05:55.030853 18583 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250112 15:05:55.030993 17651 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20250112 15:05:55.031039 17651 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task95dLSD/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1736694351070047-17651-0/minicluster-data/ts-3-root
uuid: "6bab59d1be384346a2b1354fb677ec01"
format_stamp: "Formatted at 2025-01-12 15:05:55 on dist-test-slave-b4nz"
I20250112 15:05:55.031075 17651 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task95dLSD/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1736694351070047-17651-0/minicluster-data/ts-3-root
metadata directory: /tmp/dist-test-task95dLSD/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1736694351070047-17651-0/minicluster-data/ts-3-root
1 data directories: /tmp/dist-test-task95dLSD/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1736694351070047-17651-0/minicluster-data/ts-3-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250112 15:05:55.036932 17651 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250112 15:05:55.037153 17651 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250112 15:05:55.037272 17651 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250112 15:05:55.037460 17651 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250112 15:05:55.037485 17651 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250112 15:05:55.037505 17651 ts_tablet_manager.cc:610] Registered 0 tablets
I20250112 15:05:55.037532 17651 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250112 15:05:55.040812 17651 rpc_server.cc:307] RPC server started. Bound to: 127.17.60.196:39295
I20250112 15:05:55.040863 18646 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.17.60.196:39295 every 8 connection(s)
I20250112 15:05:55.043686 18647 heartbeater.cc:346] Connected to a master server at 127.17.60.254:45251
I20250112 15:05:55.043745 18647 heartbeater.cc:463] Registering TS with master...
I20250112 15:05:55.043859 18647 heartbeater.cc:510] Master 127.17.60.254:45251 requested a full tablet report, sending...
I20250112 15:05:55.044086 18226 ts_manager.cc:194] Registered new tserver with Master: 6bab59d1be384346a2b1354fb677ec01 (127.17.60.196:39295)
I20250112 15:05:55.044437 18226 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:51760
I20250112 15:05:55.192890 18533 consensus_queue.cc:1035] T ded9df74a71c4d059719cb5401e11822 P aeb32dce70ca494ca326c528e640c9ce [LEADER]: Connected to new peer: Peer: permanent_uuid: "90a0b30f5377440b8d72889b9d2f0a70" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 35603 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250112 15:05:55.194320 18652 consensus_queue.cc:1035] T ded9df74a71c4d059719cb5401e11822 P aeb32dce70ca494ca326c528e640c9ce [LEADER]: Connected to new peer: Peer: permanent_uuid: "79518508ccd644b28675a5de1e7280e2" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 38531 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250112 15:05:55.209684 18544 consensus_queue.cc:1035] T e311741568d04cca81089b8bad66b805 P 90a0b30f5377440b8d72889b9d2f0a70 [LEADER]: Connected to new peer: Peer: permanent_uuid: "79518508ccd644b28675a5de1e7280e2" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 38531 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250112 15:05:55.211057 18544 consensus_queue.cc:1035] T e311741568d04cca81089b8bad66b805 P 90a0b30f5377440b8d72889b9d2f0a70 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aeb32dce70ca494ca326c528e640c9ce" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 35351 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250112 15:05:55.242436 18544 consensus_queue.cc:1035] T 2319a98d89eb4234aa86cc3930853fce P 90a0b30f5377440b8d72889b9d2f0a70 [LEADER]: Connected to new peer: Peer: permanent_uuid: "79518508ccd644b28675a5de1e7280e2" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 38531 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250112 15:05:55.243717 18549 consensus_queue.cc:1035] T 2319a98d89eb4234aa86cc3930853fce P 90a0b30f5377440b8d72889b9d2f0a70 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aeb32dce70ca494ca326c528e640c9ce" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 35351 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250112 15:05:55.267524 18549 consensus_queue.cc:1035] T b3b76094dfae41f38c01fd10b38c7a28 P 90a0b30f5377440b8d72889b9d2f0a70 [LEADER]: Connected to new peer: Peer: permanent_uuid: "79518508ccd644b28675a5de1e7280e2" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 38531 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250112 15:05:55.269045 18549 consensus_queue.cc:1035] T b3b76094dfae41f38c01fd10b38c7a28 P 90a0b30f5377440b8d72889b9d2f0a70 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aeb32dce70ca494ca326c528e640c9ce" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 35351 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250112 15:05:55.311257 18534 consensus_queue.cc:1035] T 45204e59d67b4abe9ed00311ca03b73d P 79518508ccd644b28675a5de1e7280e2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "90a0b30f5377440b8d72889b9d2f0a70" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 35603 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250112 15:05:55.311997 18534 consensus_queue.cc:1035] T 45204e59d67b4abe9ed00311ca03b73d P 79518508ccd644b28675a5de1e7280e2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aeb32dce70ca494ca326c528e640c9ce" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 35351 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250112 15:05:55.325136 18561 consensus_queue.cc:1035] T 9032f9edb5a746729802355e7d212c95 P aeb32dce70ca494ca326c528e640c9ce [LEADER]: Connected to new peer: Peer: permanent_uuid: "90a0b30f5377440b8d72889b9d2f0a70" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 35603 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250112 15:05:55.326471 18533 consensus_queue.cc:1035] T 9032f9edb5a746729802355e7d212c95 P aeb32dce70ca494ca326c528e640c9ce [LEADER]: Connected to new peer: Peer: permanent_uuid: "79518508ccd644b28675a5de1e7280e2" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 38531 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250112 15:05:55.341234 18570 consensus_queue.cc:1035] T cf6c409b42d842cfa25a995f95544e5e P 79518508ccd644b28675a5de1e7280e2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "90a0b30f5377440b8d72889b9d2f0a70" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 35603 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250112 15:05:55.342446 18668 consensus_queue.cc:1035] T cf6c409b42d842cfa25a995f95544e5e P 79518508ccd644b28675a5de1e7280e2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aeb32dce70ca494ca326c528e640c9ce" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 35351 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250112 15:05:55.471999 18668 consensus_queue.cc:1035] T 4e45a9f365b344c2aa0703adea508eeb P 79518508ccd644b28675a5de1e7280e2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aeb32dce70ca494ca326c528e640c9ce" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 35351 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250112 15:05:55.473543 18570 consensus_queue.cc:1035] T 4e45a9f365b344c2aa0703adea508eeb P 79518508ccd644b28675a5de1e7280e2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "90a0b30f5377440b8d72889b9d2f0a70" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 35603 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250112 15:05:55.536701 18282 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250112 15:05:55.539721 18331 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "45204e59d67b4abe9ed00311ca03b73d"
dest_uuid: "79518508ccd644b28675a5de1e7280e2"
mode: GRACEFUL
new_leader_uuid: "aeb32dce70ca494ca326c528e640c9ce"
from {username='slave'} at 127.0.0.1:33520
I20250112 15:05:55.539820 18331 raft_consensus.cc:604] T 45204e59d67b4abe9ed00311ca03b73d P 79518508ccd644b28675a5de1e7280e2 [term 1 LEADER]: Received request to transfer leadership to TS aeb32dce70ca494ca326c528e640c9ce
I20250112 15:05:55.542162 18482 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "e311741568d04cca81089b8bad66b805"
dest_uuid: "90a0b30f5377440b8d72889b9d2f0a70"
mode: GRACEFUL
new_leader_uuid: "79518508ccd644b28675a5de1e7280e2"
from {username='slave'} at 127.0.0.1:39604
I20250112 15:05:55.542232 18482 raft_consensus.cc:604] T e311741568d04cca81089b8bad66b805 P 90a0b30f5377440b8d72889b9d2f0a70 [term 1 LEADER]: Received request to transfer leadership to TS 79518508ccd644b28675a5de1e7280e2
I20250112 15:05:55.542405 18282 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 2
I20250112 15:05:55.542500 18282 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20250112 15:05:55.630385 18549 raft_consensus.cc:988] T e311741568d04cca81089b8bad66b805 P 90a0b30f5377440b8d72889b9d2f0a70: : Instructing follower 79518508ccd644b28675a5de1e7280e2 to start an election
I20250112 15:05:55.630471 18549 raft_consensus.cc:1076] T e311741568d04cca81089b8bad66b805 P 90a0b30f5377440b8d72889b9d2f0a70 [term 1 LEADER]: Signalling peer 79518508ccd644b28675a5de1e7280e2 to start an election
I20250112 15:05:55.630661 18331 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "e311741568d04cca81089b8bad66b805"
dest_uuid: "79518508ccd644b28675a5de1e7280e2"
from {username='slave'} at 127.0.0.1:33516
I20250112 15:05:55.630746 18331 raft_consensus.cc:491] T e311741568d04cca81089b8bad66b805 P 79518508ccd644b28675a5de1e7280e2 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20250112 15:05:55.630776 18331 raft_consensus.cc:3054] T e311741568d04cca81089b8bad66b805 P 79518508ccd644b28675a5de1e7280e2 [term 1 FOLLOWER]: Advancing to term 2
I20250112 15:05:55.631364 18331 raft_consensus.cc:513] T e311741568d04cca81089b8bad66b805 P 79518508ccd644b28675a5de1e7280e2 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90a0b30f5377440b8d72889b9d2f0a70" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 35603 } } peers { permanent_uuid: "79518508ccd644b28675a5de1e7280e2" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 38531 } } peers { permanent_uuid: "aeb32dce70ca494ca326c528e640c9ce" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 35351 } }
I20250112 15:05:55.631500 18331 leader_election.cc:290] T e311741568d04cca81089b8bad66b805 P 79518508ccd644b28675a5de1e7280e2 [CANDIDATE]: Term 2 election: Requested vote from peers 90a0b30f5377440b8d72889b9d2f0a70 (127.17.60.195:35603), aeb32dce70ca494ca326c528e640c9ce (127.17.60.194:35351)
I20250112 15:05:55.631680 18482 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e311741568d04cca81089b8bad66b805" candidate_uuid: "79518508ccd644b28675a5de1e7280e2" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "90a0b30f5377440b8d72889b9d2f0a70"
I20250112 15:05:55.631680 18407 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e311741568d04cca81089b8bad66b805" candidate_uuid: "79518508ccd644b28675a5de1e7280e2" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "aeb32dce70ca494ca326c528e640c9ce"
I20250112 15:05:55.631783 18482 raft_consensus.cc:3049] T e311741568d04cca81089b8bad66b805 P 90a0b30f5377440b8d72889b9d2f0a70 [term 1 LEADER]: Stepping down as leader of term 1
I20250112 15:05:55.631800 18407 raft_consensus.cc:3054] T e311741568d04cca81089b8bad66b805 P aeb32dce70ca494ca326c528e640c9ce [term 1 FOLLOWER]: Advancing to term 2
I20250112 15:05:55.631807 18482 raft_consensus.cc:738] T e311741568d04cca81089b8bad66b805 P 90a0b30f5377440b8d72889b9d2f0a70 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 90a0b30f5377440b8d72889b9d2f0a70, State: Running, Role: LEADER
I20250112 15:05:55.631887 18482 consensus_queue.cc:260] T e311741568d04cca81089b8bad66b805 P 90a0b30f5377440b8d72889b9d2f0a70 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90a0b30f5377440b8d72889b9d2f0a70" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 35603 } } peers { permanent_uuid: "79518508ccd644b28675a5de1e7280e2" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 38531 } } peers { permanent_uuid: "aeb32dce70ca494ca326c528e640c9ce" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 35351 } }
I20250112 15:05:55.631999 18482 raft_consensus.cc:3054] T e311741568d04cca81089b8bad66b805 P 90a0b30f5377440b8d72889b9d2f0a70 [term 1 FOLLOWER]: Advancing to term 2
I20250112 15:05:55.632525 18482 raft_consensus.cc:2463] T e311741568d04cca81089b8bad66b805 P 90a0b30f5377440b8d72889b9d2f0a70 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 79518508ccd644b28675a5de1e7280e2 in term 2.
I20250112 15:05:55.632525 18407 raft_consensus.cc:2463] T e311741568d04cca81089b8bad66b805 P aeb32dce70ca494ca326c528e640c9ce [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 79518508ccd644b28675a5de1e7280e2 in term 2.
I20250112 15:05:55.632750 18294 leader_election.cc:304] T e311741568d04cca81089b8bad66b805 P 79518508ccd644b28675a5de1e7280e2 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 79518508ccd644b28675a5de1e7280e2, 90a0b30f5377440b8d72889b9d2f0a70; no voters:
I20250112 15:05:55.632862 18570 raft_consensus.cc:2798] T e311741568d04cca81089b8bad66b805 P 79518508ccd644b28675a5de1e7280e2 [term 2 FOLLOWER]: Leader election won for term 2
I20250112 15:05:55.632928 18570 raft_consensus.cc:695] T e311741568d04cca81089b8bad66b805 P 79518508ccd644b28675a5de1e7280e2 [term 2 LEADER]: Becoming Leader. State: Replica: 79518508ccd644b28675a5de1e7280e2, State: Running, Role: LEADER
I20250112 15:05:55.632974 18570 consensus_queue.cc:237] T e311741568d04cca81089b8bad66b805 P 79518508ccd644b28675a5de1e7280e2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90a0b30f5377440b8d72889b9d2f0a70" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 35603 } } peers { permanent_uuid: "79518508ccd644b28675a5de1e7280e2" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 38531 } } peers { permanent_uuid: "aeb32dce70ca494ca326c528e640c9ce" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 35351 } }
I20250112 15:05:55.633460 18224 catalog_manager.cc:5526] T e311741568d04cca81089b8bad66b805 P 79518508ccd644b28675a5de1e7280e2 reported cstate change: term changed from 1 to 2, leader changed from 90a0b30f5377440b8d72889b9d2f0a70 (127.17.60.195) to 79518508ccd644b28675a5de1e7280e2 (127.17.60.193). New cstate: current_term: 2 leader_uuid: "79518508ccd644b28675a5de1e7280e2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90a0b30f5377440b8d72889b9d2f0a70" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 35603 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "79518508ccd644b28675a5de1e7280e2" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 38531 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "aeb32dce70ca494ca326c528e640c9ce" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 35351 } health_report { overall_health: UNKNOWN } } }
I20250112 15:05:55.775574 18534 raft_consensus.cc:988] T 45204e59d67b4abe9ed00311ca03b73d P 79518508ccd644b28675a5de1e7280e2: : Instructing follower aeb32dce70ca494ca326c528e640c9ce to start an election
I20250112 15:05:55.775660 18534 raft_consensus.cc:1076] T 45204e59d67b4abe9ed00311ca03b73d P 79518508ccd644b28675a5de1e7280e2 [term 1 LEADER]: Signalling peer aeb32dce70ca494ca326c528e640c9ce to start an election
I20250112 15:05:55.775856 18407 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "45204e59d67b4abe9ed00311ca03b73d"
dest_uuid: "aeb32dce70ca494ca326c528e640c9ce"
from {username='slave'} at 127.0.0.1:50780
I20250112 15:05:55.775923 18407 raft_consensus.cc:491] T 45204e59d67b4abe9ed00311ca03b73d P aeb32dce70ca494ca326c528e640c9ce [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20250112 15:05:55.775950 18407 raft_consensus.cc:3054] T 45204e59d67b4abe9ed00311ca03b73d P aeb32dce70ca494ca326c528e640c9ce [term 1 FOLLOWER]: Advancing to term 2
I20250112 15:05:55.776741 18407 raft_consensus.cc:513] T 45204e59d67b4abe9ed00311ca03b73d P aeb32dce70ca494ca326c528e640c9ce [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79518508ccd644b28675a5de1e7280e2" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 38531 } } peers { permanent_uuid: "aeb32dce70ca494ca326c528e640c9ce" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 35351 } } peers { permanent_uuid: "90a0b30f5377440b8d72889b9d2f0a70" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 35603 } }
I20250112 15:05:55.776850 18407 leader_election.cc:290] T 45204e59d67b4abe9ed00311ca03b73d P aeb32dce70ca494ca326c528e640c9ce [CANDIDATE]: Term 2 election: Requested vote from peers 79518508ccd644b28675a5de1e7280e2 (127.17.60.193:38531), 90a0b30f5377440b8d72889b9d2f0a70 (127.17.60.195:35603)
I20250112 15:05:55.777004 18482 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "45204e59d67b4abe9ed00311ca03b73d" candidate_uuid: "aeb32dce70ca494ca326c528e640c9ce" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "90a0b30f5377440b8d72889b9d2f0a70"
I20250112 15:05:55.777004 18331 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "45204e59d67b4abe9ed00311ca03b73d" candidate_uuid: "aeb32dce70ca494ca326c528e640c9ce" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "79518508ccd644b28675a5de1e7280e2"
I20250112 15:05:55.777103 18482 raft_consensus.cc:3054] T 45204e59d67b4abe9ed00311ca03b73d P 90a0b30f5377440b8d72889b9d2f0a70 [term 1 FOLLOWER]: Advancing to term 2
I20250112 15:05:55.777107 18331 raft_consensus.cc:3049] T 45204e59d67b4abe9ed00311ca03b73d P 79518508ccd644b28675a5de1e7280e2 [term 1 LEADER]: Stepping down as leader of term 1
I20250112 15:05:55.777144 18331 raft_consensus.cc:738] T 45204e59d67b4abe9ed00311ca03b73d P 79518508ccd644b28675a5de1e7280e2 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 79518508ccd644b28675a5de1e7280e2, State: Running, Role: LEADER
I20250112 15:05:55.777199 18331 consensus_queue.cc:260] T 45204e59d67b4abe9ed00311ca03b73d P 79518508ccd644b28675a5de1e7280e2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79518508ccd644b28675a5de1e7280e2" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 38531 } } peers { permanent_uuid: "aeb32dce70ca494ca326c528e640c9ce" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 35351 } } peers { permanent_uuid: "90a0b30f5377440b8d72889b9d2f0a70" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 35603 } }
I20250112 15:05:55.777271 18331 raft_consensus.cc:3054] T 45204e59d67b4abe9ed00311ca03b73d P 79518508ccd644b28675a5de1e7280e2 [term 1 FOLLOWER]: Advancing to term 2
I20250112 15:05:55.777699 18482 raft_consensus.cc:2463] T 45204e59d67b4abe9ed00311ca03b73d P 90a0b30f5377440b8d72889b9d2f0a70 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate aeb32dce70ca494ca326c528e640c9ce in term 2.
I20250112 15:05:55.777817 18331 raft_consensus.cc:2463] T 45204e59d67b4abe9ed00311ca03b73d P 79518508ccd644b28675a5de1e7280e2 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate aeb32dce70ca494ca326c528e640c9ce in term 2.
I20250112 15:05:55.777910 18370 leader_election.cc:304] T 45204e59d67b4abe9ed00311ca03b73d P aeb32dce70ca494ca326c528e640c9ce [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 90a0b30f5377440b8d72889b9d2f0a70, aeb32dce70ca494ca326c528e640c9ce; no voters:
I20250112 15:05:55.778021 18533 raft_consensus.cc:2798] T 45204e59d67b4abe9ed00311ca03b73d P aeb32dce70ca494ca326c528e640c9ce [term 2 FOLLOWER]: Leader election won for term 2
I20250112 15:05:55.778072 18533 raft_consensus.cc:695] T 45204e59d67b4abe9ed00311ca03b73d P aeb32dce70ca494ca326c528e640c9ce [term 2 LEADER]: Becoming Leader. State: Replica: aeb32dce70ca494ca326c528e640c9ce, State: Running, Role: LEADER
I20250112 15:05:55.778124 18533 consensus_queue.cc:237] T 45204e59d67b4abe9ed00311ca03b73d P aeb32dce70ca494ca326c528e640c9ce [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79518508ccd644b28675a5de1e7280e2" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 38531 } } peers { permanent_uuid: "aeb32dce70ca494ca326c528e640c9ce" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 35351 } } peers { permanent_uuid: "90a0b30f5377440b8d72889b9d2f0a70" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 35603 } }
I20250112 15:05:55.778498 18224 catalog_manager.cc:5526] T 45204e59d67b4abe9ed00311ca03b73d P aeb32dce70ca494ca326c528e640c9ce reported cstate change: term changed from 1 to 2, leader changed from 79518508ccd644b28675a5de1e7280e2 (127.17.60.193) to aeb32dce70ca494ca326c528e640c9ce (127.17.60.194). New cstate: current_term: 2 leader_uuid: "aeb32dce70ca494ca326c528e640c9ce" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79518508ccd644b28675a5de1e7280e2" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 38531 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "aeb32dce70ca494ca326c528e640c9ce" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 35351 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "90a0b30f5377440b8d72889b9d2f0a70" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 35603 } health_report { overall_health: UNKNOWN } } }
I20250112 15:05:56.044940 18647 heartbeater.cc:502] Master 127.17.60.254:45251 was elected leader, sending a full tablet report...
I20250112 15:05:56.118002 18482 raft_consensus.cc:1270] T e311741568d04cca81089b8bad66b805 P 90a0b30f5377440b8d72889b9d2f0a70 [term 2 FOLLOWER]: Refusing update from remote peer 79518508ccd644b28675a5de1e7280e2: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250112 15:05:56.118288 18668 consensus_queue.cc:1035] T e311741568d04cca81089b8bad66b805 P 79518508ccd644b28675a5de1e7280e2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "90a0b30f5377440b8d72889b9d2f0a70" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 35603 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250112 15:05:56.119273 18407 raft_consensus.cc:1270] T e311741568d04cca81089b8bad66b805 P aeb32dce70ca494ca326c528e640c9ce [term 2 FOLLOWER]: Refusing update from remote peer 79518508ccd644b28675a5de1e7280e2: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250112 15:05:56.119458 18668 consensus_queue.cc:1035] T e311741568d04cca81089b8bad66b805 P 79518508ccd644b28675a5de1e7280e2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aeb32dce70ca494ca326c528e640c9ce" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 35351 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250112 15:05:56.247733 18482 raft_consensus.cc:1270] T 45204e59d67b4abe9ed00311ca03b73d P 90a0b30f5377440b8d72889b9d2f0a70 [term 2 FOLLOWER]: Refusing update from remote peer aeb32dce70ca494ca326c528e640c9ce: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250112 15:05:56.248000 18652 consensus_queue.cc:1035] T 45204e59d67b4abe9ed00311ca03b73d P aeb32dce70ca494ca326c528e640c9ce [LEADER]: Connected to new peer: Peer: permanent_uuid: "90a0b30f5377440b8d72889b9d2f0a70" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 35603 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250112 15:05:56.249030 18331 raft_consensus.cc:1270] T 45204e59d67b4abe9ed00311ca03b73d P 79518508ccd644b28675a5de1e7280e2 [term 2 FOLLOWER]: Refusing update from remote peer aeb32dce70ca494ca326c528e640c9ce: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250112 15:05:56.249256 18533 consensus_queue.cc:1035] T 45204e59d67b4abe9ed00311ca03b73d P aeb32dce70ca494ca326c528e640c9ce [LEADER]: Connected to new peer: Peer: permanent_uuid: "79518508ccd644b28675a5de1e7280e2" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 38531 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250112 15:05:57.542666 18282 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250112 15:05:57.543088 18331 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "4e45a9f365b344c2aa0703adea508eeb"
dest_uuid: "79518508ccd644b28675a5de1e7280e2"
mode: GRACEFUL
new_leader_uuid: "90a0b30f5377440b8d72889b9d2f0a70"
from {username='slave'} at 127.0.0.1:33520
I20250112 15:05:57.543195 18331 raft_consensus.cc:604] T 4e45a9f365b344c2aa0703adea508eeb P 79518508ccd644b28675a5de1e7280e2 [term 1 LEADER]: Received request to transfer leadership to TS 90a0b30f5377440b8d72889b9d2f0a70
I20250112 15:05:57.543426 18282 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 1
I20250112 15:05:57.543489 18282 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20250112 15:05:57.581414 18682 raft_consensus.cc:988] T 4e45a9f365b344c2aa0703adea508eeb P 79518508ccd644b28675a5de1e7280e2: : Instructing follower 90a0b30f5377440b8d72889b9d2f0a70 to start an election
I20250112 15:05:57.581493 18682 raft_consensus.cc:1076] T 4e45a9f365b344c2aa0703adea508eeb P 79518508ccd644b28675a5de1e7280e2 [term 1 LEADER]: Signalling peer 90a0b30f5377440b8d72889b9d2f0a70 to start an election
I20250112 15:05:57.581704 18482 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "4e45a9f365b344c2aa0703adea508eeb"
dest_uuid: "90a0b30f5377440b8d72889b9d2f0a70"
from {username='slave'} at 127.0.0.1:39598
I20250112 15:05:57.581790 18482 raft_consensus.cc:491] T 4e45a9f365b344c2aa0703adea508eeb P 90a0b30f5377440b8d72889b9d2f0a70 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20250112 15:05:57.581820 18482 raft_consensus.cc:3054] T 4e45a9f365b344c2aa0703adea508eeb P 90a0b30f5377440b8d72889b9d2f0a70 [term 1 FOLLOWER]: Advancing to term 2
I20250112 15:05:57.582430 18482 raft_consensus.cc:513] T 4e45a9f365b344c2aa0703adea508eeb P 90a0b30f5377440b8d72889b9d2f0a70 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79518508ccd644b28675a5de1e7280e2" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 38531 } } peers { permanent_uuid: "aeb32dce70ca494ca326c528e640c9ce" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 35351 } } peers { permanent_uuid: "90a0b30f5377440b8d72889b9d2f0a70" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 35603 } }
I20250112 15:05:57.582573 18482 leader_election.cc:290] T 4e45a9f365b344c2aa0703adea508eeb P 90a0b30f5377440b8d72889b9d2f0a70 [CANDIDATE]: Term 2 election: Requested vote from peers 79518508ccd644b28675a5de1e7280e2 (127.17.60.193:38531), aeb32dce70ca494ca326c528e640c9ce (127.17.60.194:35351)
I20250112 15:05:57.582688 18331 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e45a9f365b344c2aa0703adea508eeb" candidate_uuid: "90a0b30f5377440b8d72889b9d2f0a70" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "79518508ccd644b28675a5de1e7280e2"
I20250112 15:05:57.582756 18331 raft_consensus.cc:3049] T 4e45a9f365b344c2aa0703adea508eeb P 79518508ccd644b28675a5de1e7280e2 [term 1 LEADER]: Stepping down as leader of term 1
I20250112 15:05:57.582731 18407 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e45a9f365b344c2aa0703adea508eeb" candidate_uuid: "90a0b30f5377440b8d72889b9d2f0a70" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "aeb32dce70ca494ca326c528e640c9ce"
I20250112 15:05:57.582785 18331 raft_consensus.cc:738] T 4e45a9f365b344c2aa0703adea508eeb P 79518508ccd644b28675a5de1e7280e2 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 79518508ccd644b28675a5de1e7280e2, State: Running, Role: LEADER
I20250112 15:05:57.582804 18407 raft_consensus.cc:3054] T 4e45a9f365b344c2aa0703adea508eeb P aeb32dce70ca494ca326c528e640c9ce [term 1 FOLLOWER]: Advancing to term 2
I20250112 15:05:57.582840 18331 consensus_queue.cc:260] T 4e45a9f365b344c2aa0703adea508eeb P 79518508ccd644b28675a5de1e7280e2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79518508ccd644b28675a5de1e7280e2" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 38531 } } peers { permanent_uuid: "aeb32dce70ca494ca326c528e640c9ce" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 35351 } } peers { permanent_uuid: "90a0b30f5377440b8d72889b9d2f0a70" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 35603 } }
I20250112 15:05:57.582927 18331 raft_consensus.cc:3054] T 4e45a9f365b344c2aa0703adea508eeb P 79518508ccd644b28675a5de1e7280e2 [term 1 FOLLOWER]: Advancing to term 2
I20250112 15:05:57.583361 18407 raft_consensus.cc:2463] T 4e45a9f365b344c2aa0703adea508eeb P aeb32dce70ca494ca326c528e640c9ce [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 90a0b30f5377440b8d72889b9d2f0a70 in term 2.
I20250112 15:05:57.583492 18331 raft_consensus.cc:2463] T 4e45a9f365b344c2aa0703adea508eeb P 79518508ccd644b28675a5de1e7280e2 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 90a0b30f5377440b8d72889b9d2f0a70 in term 2.
I20250112 15:05:57.583566 18446 leader_election.cc:304] T 4e45a9f365b344c2aa0703adea508eeb P 90a0b30f5377440b8d72889b9d2f0a70 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 90a0b30f5377440b8d72889b9d2f0a70, aeb32dce70ca494ca326c528e640c9ce; no voters:
I20250112 15:05:57.583660 18678 raft_consensus.cc:2798] T 4e45a9f365b344c2aa0703adea508eeb P 90a0b30f5377440b8d72889b9d2f0a70 [term 2 FOLLOWER]: Leader election won for term 2
I20250112 15:05:57.583760 18678 raft_consensus.cc:695] T 4e45a9f365b344c2aa0703adea508eeb P 90a0b30f5377440b8d72889b9d2f0a70 [term 2 LEADER]: Becoming Leader. State: Replica: 90a0b30f5377440b8d72889b9d2f0a70, State: Running, Role: LEADER
I20250112 15:05:57.583811 18678 consensus_queue.cc:237] T 4e45a9f365b344c2aa0703adea508eeb P 90a0b30f5377440b8d72889b9d2f0a70 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79518508ccd644b28675a5de1e7280e2" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 38531 } } peers { permanent_uuid: "aeb32dce70ca494ca326c528e640c9ce" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 35351 } } peers { permanent_uuid: "90a0b30f5377440b8d72889b9d2f0a70" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 35603 } }
I20250112 15:05:57.584328 18224 catalog_manager.cc:5526] T 4e45a9f365b344c2aa0703adea508eeb P 90a0b30f5377440b8d72889b9d2f0a70 reported cstate change: term changed from 1 to 2, leader changed from 79518508ccd644b28675a5de1e7280e2 (127.17.60.193) to 90a0b30f5377440b8d72889b9d2f0a70 (127.17.60.195). New cstate: current_term: 2 leader_uuid: "90a0b30f5377440b8d72889b9d2f0a70" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79518508ccd644b28675a5de1e7280e2" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 38531 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "aeb32dce70ca494ca326c528e640c9ce" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 35351 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "90a0b30f5377440b8d72889b9d2f0a70" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 35603 } health_report { overall_health: HEALTHY } } }
I20250112 15:05:58.002019 18407 raft_consensus.cc:1270] T 4e45a9f365b344c2aa0703adea508eeb P aeb32dce70ca494ca326c528e640c9ce [term 2 FOLLOWER]: Refusing update from remote peer 90a0b30f5377440b8d72889b9d2f0a70: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250112 15:05:58.002306 18678 consensus_queue.cc:1035] T 4e45a9f365b344c2aa0703adea508eeb P 90a0b30f5377440b8d72889b9d2f0a70 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aeb32dce70ca494ca326c528e640c9ce" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 35351 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250112 15:05:58.003683 18331 raft_consensus.cc:1270] T 4e45a9f365b344c2aa0703adea508eeb P 79518508ccd644b28675a5de1e7280e2 [term 2 FOLLOWER]: Refusing update from remote peer 90a0b30f5377440b8d72889b9d2f0a70: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250112 15:05:58.003865 18693 consensus_queue.cc:1035] T 4e45a9f365b344c2aa0703adea508eeb P 90a0b30f5377440b8d72889b9d2f0a70 [LEADER]: Connected to new peer: Peer: permanent_uuid: "79518508ccd644b28675a5de1e7280e2" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 38531 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250112 15:05:59.543694 18282 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250112 15:05:59.544205 18482 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "2319a98d89eb4234aa86cc3930853fce"
dest_uuid: "90a0b30f5377440b8d72889b9d2f0a70"
mode: GRACEFUL
new_leader_uuid: "79518508ccd644b28675a5de1e7280e2"
from {username='slave'} at 127.0.0.1:39604
I20250112 15:05:59.544302 18482 raft_consensus.cc:604] T 2319a98d89eb4234aa86cc3930853fce P 90a0b30f5377440b8d72889b9d2f0a70 [term 1 LEADER]: Received request to transfer leadership to TS 79518508ccd644b28675a5de1e7280e2
I20250112 15:05:59.544512 18282 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 1
I20250112 15:05:59.544602 18282 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20250112 15:05:59.844015 18693 raft_consensus.cc:988] T 2319a98d89eb4234aa86cc3930853fce P 90a0b30f5377440b8d72889b9d2f0a70: : Instructing follower 79518508ccd644b28675a5de1e7280e2 to start an election
I20250112 15:05:59.844102 18693 raft_consensus.cc:1076] T 2319a98d89eb4234aa86cc3930853fce P 90a0b30f5377440b8d72889b9d2f0a70 [term 1 LEADER]: Signalling peer 79518508ccd644b28675a5de1e7280e2 to start an election
I20250112 15:05:59.844295 18331 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "2319a98d89eb4234aa86cc3930853fce"
dest_uuid: "79518508ccd644b28675a5de1e7280e2"
from {username='slave'} at 127.0.0.1:33516
I20250112 15:05:59.844383 18331 raft_consensus.cc:491] T 2319a98d89eb4234aa86cc3930853fce P 79518508ccd644b28675a5de1e7280e2 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20250112 15:05:59.844411 18331 raft_consensus.cc:3054] T 2319a98d89eb4234aa86cc3930853fce P 79518508ccd644b28675a5de1e7280e2 [term 1 FOLLOWER]: Advancing to term 2
I20250112 15:05:59.845090 18331 raft_consensus.cc:513] T 2319a98d89eb4234aa86cc3930853fce P 79518508ccd644b28675a5de1e7280e2 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79518508ccd644b28675a5de1e7280e2" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 38531 } } peers { permanent_uuid: "aeb32dce70ca494ca326c528e640c9ce" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 35351 } } peers { permanent_uuid: "90a0b30f5377440b8d72889b9d2f0a70" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 35603 } }
I20250112 15:05:59.845222 18331 leader_election.cc:290] T 2319a98d89eb4234aa86cc3930853fce P 79518508ccd644b28675a5de1e7280e2 [CANDIDATE]: Term 2 election: Requested vote from peers aeb32dce70ca494ca326c528e640c9ce (127.17.60.194:35351), 90a0b30f5377440b8d72889b9d2f0a70 (127.17.60.195:35603)
I20250112 15:05:59.845357 18407 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2319a98d89eb4234aa86cc3930853fce" candidate_uuid: "79518508ccd644b28675a5de1e7280e2" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "aeb32dce70ca494ca326c528e640c9ce"
I20250112 15:05:59.845372 18482 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2319a98d89eb4234aa86cc3930853fce" candidate_uuid: "79518508ccd644b28675a5de1e7280e2" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "90a0b30f5377440b8d72889b9d2f0a70"
I20250112 15:05:59.845446 18407 raft_consensus.cc:3054] T 2319a98d89eb4234aa86cc3930853fce P aeb32dce70ca494ca326c528e640c9ce [term 1 FOLLOWER]: Advancing to term 2
I20250112 15:05:59.845453 18482 raft_consensus.cc:3049] T 2319a98d89eb4234aa86cc3930853fce P 90a0b30f5377440b8d72889b9d2f0a70 [term 1 LEADER]: Stepping down as leader of term 1
I20250112 15:05:59.845490 18482 raft_consensus.cc:738] T 2319a98d89eb4234aa86cc3930853fce P 90a0b30f5377440b8d72889b9d2f0a70 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 90a0b30f5377440b8d72889b9d2f0a70, State: Running, Role: LEADER
I20250112 15:05:59.845546 18482 consensus_queue.cc:260] T 2319a98d89eb4234aa86cc3930853fce P 90a0b30f5377440b8d72889b9d2f0a70 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79518508ccd644b28675a5de1e7280e2" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 38531 } } peers { permanent_uuid: "aeb32dce70ca494ca326c528e640c9ce" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 35351 } } peers { permanent_uuid: "90a0b30f5377440b8d72889b9d2f0a70" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 35603 } }
I20250112 15:05:59.845621 18482 raft_consensus.cc:3054] T 2319a98d89eb4234aa86cc3930853fce P 90a0b30f5377440b8d72889b9d2f0a70 [term 1 FOLLOWER]: Advancing to term 2
I20250112 15:05:59.846128 18407 raft_consensus.cc:2463] T 2319a98d89eb4234aa86cc3930853fce P aeb32dce70ca494ca326c528e640c9ce [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 79518508ccd644b28675a5de1e7280e2 in term 2.
I20250112 15:05:59.846201 18482 raft_consensus.cc:2463] T 2319a98d89eb4234aa86cc3930853fce P 90a0b30f5377440b8d72889b9d2f0a70 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 79518508ccd644b28675a5de1e7280e2 in term 2.
I20250112 15:05:59.846312 18295 leader_election.cc:304] T 2319a98d89eb4234aa86cc3930853fce P 79518508ccd644b28675a5de1e7280e2 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 79518508ccd644b28675a5de1e7280e2, aeb32dce70ca494ca326c528e640c9ce; no voters:
I20250112 15:05:59.846410 18682 raft_consensus.cc:2798] T 2319a98d89eb4234aa86cc3930853fce P 79518508ccd644b28675a5de1e7280e2 [term 2 FOLLOWER]: Leader election won for term 2
I20250112 15:05:59.846511 18682 raft_consensus.cc:695] T 2319a98d89eb4234aa86cc3930853fce P 79518508ccd644b28675a5de1e7280e2 [term 2 LEADER]: Becoming Leader. State: Replica: 79518508ccd644b28675a5de1e7280e2, State: Running, Role: LEADER
I20250112 15:05:59.846565 18682 consensus_queue.cc:237] T 2319a98d89eb4234aa86cc3930853fce P 79518508ccd644b28675a5de1e7280e2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79518508ccd644b28675a5de1e7280e2" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 38531 } } peers { permanent_uuid: "aeb32dce70ca494ca326c528e640c9ce" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 35351 } } peers { permanent_uuid: "90a0b30f5377440b8d72889b9d2f0a70" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 35603 } }
I20250112 15:05:59.847023 18224 catalog_manager.cc:5526] T 2319a98d89eb4234aa86cc3930853fce P 79518508ccd644b28675a5de1e7280e2 reported cstate change: term changed from 1 to 2, leader changed from 90a0b30f5377440b8d72889b9d2f0a70 (127.17.60.195) to 79518508ccd644b28675a5de1e7280e2 (127.17.60.193). New cstate: current_term: 2 leader_uuid: "79518508ccd644b28675a5de1e7280e2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79518508ccd644b28675a5de1e7280e2" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 38531 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "aeb32dce70ca494ca326c528e640c9ce" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 35351 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "90a0b30f5377440b8d72889b9d2f0a70" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 35603 } health_report { overall_health: UNKNOWN } } }
I20250112 15:06:00.348306 18407 raft_consensus.cc:1270] T 2319a98d89eb4234aa86cc3930853fce P aeb32dce70ca494ca326c528e640c9ce [term 2 FOLLOWER]: Refusing update from remote peer 79518508ccd644b28675a5de1e7280e2: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250112 15:06:00.348598 18682 consensus_queue.cc:1035] T 2319a98d89eb4234aa86cc3930853fce P 79518508ccd644b28675a5de1e7280e2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aeb32dce70ca494ca326c528e640c9ce" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 35351 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250112 15:06:00.349431 18482 raft_consensus.cc:1270] T 2319a98d89eb4234aa86cc3930853fce P 90a0b30f5377440b8d72889b9d2f0a70 [term 2 FOLLOWER]: Refusing update from remote peer 79518508ccd644b28675a5de1e7280e2: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250112 15:06:00.349740 18682 consensus_queue.cc:1035] T 2319a98d89eb4234aa86cc3930853fce P 79518508ccd644b28675a5de1e7280e2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "90a0b30f5377440b8d72889b9d2f0a70" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 35603 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250112 15:06:00.545526 18331 consensus_queue.cc:237] T e311741568d04cca81089b8bad66b805 P 79518508ccd644b28675a5de1e7280e2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "90a0b30f5377440b8d72889b9d2f0a70" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 35603 } } peers { permanent_uuid: "79518508ccd644b28675a5de1e7280e2" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 38531 } } peers { permanent_uuid: "aeb32dce70ca494ca326c528e640c9ce" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 35351 } attrs { replace: true } } peers { permanent_uuid: "6bab59d1be384346a2b1354fb677ec01" member_type: NON_VOTER last_known_addr { host: "127.17.60.196" port: 39295 } attrs { promote: true } }
I20250112 15:06:00.546167 18407 raft_consensus.cc:1270] T e311741568d04cca81089b8bad66b805 P aeb32dce70ca494ca326c528e640c9ce [term 2 FOLLOWER]: Refusing update from remote peer 79518508ccd644b28675a5de1e7280e2: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20250112 15:06:00.546171 18482 raft_consensus.cc:1270] T e311741568d04cca81089b8bad66b805 P 90a0b30f5377440b8d72889b9d2f0a70 [term 2 FOLLOWER]: Refusing update from remote peer 79518508ccd644b28675a5de1e7280e2: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20250112 15:06:00.546401 18712 consensus_queue.cc:1035] T e311741568d04cca81089b8bad66b805 P 79518508ccd644b28675a5de1e7280e2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aeb32dce70ca494ca326c528e640c9ce" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 35351 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250112 15:06:00.546538 18682 consensus_queue.cc:1035] T e311741568d04cca81089b8bad66b805 P 79518508ccd644b28675a5de1e7280e2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "90a0b30f5377440b8d72889b9d2f0a70" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 35603 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250112 15:06:00.547217 18682 raft_consensus.cc:2949] T e311741568d04cca81089b8bad66b805 P 79518508ccd644b28675a5de1e7280e2 [term 2 LEADER]: Committing config change with OpId 2.3: config changed from index -1 to 3, NON_VOTER 6bab59d1be384346a2b1354fb677ec01 (127.17.60.196) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "90a0b30f5377440b8d72889b9d2f0a70" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 35603 } } peers { permanent_uuid: "79518508ccd644b28675a5de1e7280e2" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 38531 } } peers { permanent_uuid: "aeb32dce70ca494ca326c528e640c9ce" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 35351 } attrs { replace: true } } peers { permanent_uuid: "6bab59d1be384346a2b1354fb677ec01" member_type: NON_VOTER last_known_addr { host: "127.17.60.196" port: 39295 } attrs { promote: true } } }
I20250112 15:06:00.547298 18407 raft_consensus.cc:2949] T e311741568d04cca81089b8bad66b805 P aeb32dce70ca494ca326c528e640c9ce [term 2 FOLLOWER]: Committing config change with OpId 2.3: config changed from index -1 to 3, NON_VOTER 6bab59d1be384346a2b1354fb677ec01 (127.17.60.196) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "90a0b30f5377440b8d72889b9d2f0a70" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 35603 } } peers { permanent_uuid: "79518508ccd644b28675a5de1e7280e2" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 38531 } } peers { permanent_uuid: "aeb32dce70ca494ca326c528e640c9ce" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 35351 } attrs { replace: true } } peers { permanent_uuid: "6bab59d1be384346a2b1354fb677ec01" member_type: NON_VOTER last_known_addr { host: "127.17.60.196" port: 39295 } attrs { promote: true } } }
I20250112 15:06:00.547818 18482 raft_consensus.cc:2949] T e311741568d04cca81089b8bad66b805 P 90a0b30f5377440b8d72889b9d2f0a70 [term 2 FOLLOWER]: Committing config change with OpId 2.3: config changed from index -1 to 3, NON_VOTER 6bab59d1be384346a2b1354fb677ec01 (127.17.60.196) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "90a0b30f5377440b8d72889b9d2f0a70" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 35603 } } peers { permanent_uuid: "79518508ccd644b28675a5de1e7280e2" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 38531 } } peers { permanent_uuid: "aeb32dce70ca494ca326c528e640c9ce" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 35351 } attrs { replace: true } } peers { permanent_uuid: "6bab59d1be384346a2b1354fb677ec01" member_type: NON_VOTER last_known_addr { host: "127.17.60.196" port: 39295 } attrs { promote: true } } }
I20250112 15:06:00.548408 18224 catalog_manager.cc:5526] T e311741568d04cca81089b8bad66b805 P aeb32dce70ca494ca326c528e640c9ce reported cstate change: config changed from index -1 to 3, NON_VOTER 6bab59d1be384346a2b1354fb677ec01 (127.17.60.196) added. New cstate: current_term: 2 leader_uuid: "79518508ccd644b28675a5de1e7280e2" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "90a0b30f5377440b8d72889b9d2f0a70" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 35603 } } peers { permanent_uuid: "79518508ccd644b28675a5de1e7280e2" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 38531 } } peers { permanent_uuid: "aeb32dce70ca494ca326c528e640c9ce" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 35351 } attrs { replace: true } } peers { permanent_uuid: "6bab59d1be384346a2b1354fb677ec01" member_type: NON_VOTER last_known_addr { host: "127.17.60.196" port: 39295 } attrs { promote: true } } }
W20250112 15:06:00.549839 18297 consensus_peers.cc:487] T e311741568d04cca81089b8bad66b805 P 79518508ccd644b28675a5de1e7280e2 -> Peer 6bab59d1be384346a2b1354fb677ec01 (127.17.60.196:39295): Couldn't send request to peer 6bab59d1be384346a2b1354fb677ec01. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: e311741568d04cca81089b8bad66b805. This is attempt 1: this message will repeat every 5th retry.
I20250112 15:06:00.551401 18407 consensus_queue.cc:237] T ded9df74a71c4d059719cb5401e11822 P aeb32dce70ca494ca326c528e640c9ce [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "79518508ccd644b28675a5de1e7280e2" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 38531 } } peers { permanent_uuid: "aeb32dce70ca494ca326c528e640c9ce" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 35351 } } peers { permanent_uuid: "90a0b30f5377440b8d72889b9d2f0a70" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 35603 } attrs { replace: true } } peers { permanent_uuid: "6bab59d1be384346a2b1354fb677ec01" member_type: NON_VOTER last_known_addr { host: "127.17.60.196" port: 39295 } attrs { promote: true } }
I20250112 15:06:00.552083 18331 raft_consensus.cc:1270] T ded9df74a71c4d059719cb5401e11822 P 79518508ccd644b28675a5de1e7280e2 [term 1 FOLLOWER]: Refusing update from remote peer aeb32dce70ca494ca326c528e640c9ce: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250112 15:06:00.552102 18482 raft_consensus.cc:1270] T ded9df74a71c4d059719cb5401e11822 P 90a0b30f5377440b8d72889b9d2f0a70 [term 1 FOLLOWER]: Refusing update from remote peer aeb32dce70ca494ca326c528e640c9ce: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250112 15:06:00.552268 18533 consensus_queue.cc:1035] T ded9df74a71c4d059719cb5401e11822 P aeb32dce70ca494ca326c528e640c9ce [LEADER]: Connected to new peer: Peer: permanent_uuid: "79518508ccd644b28675a5de1e7280e2" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 38531 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250112 15:06:00.552366 18722 consensus_queue.cc:1035] T ded9df74a71c4d059719cb5401e11822 P aeb32dce70ca494ca326c528e640c9ce [LEADER]: Connected to new peer: Peer: permanent_uuid: "90a0b30f5377440b8d72889b9d2f0a70" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 35603 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250112 15:06:00.553148 18722 raft_consensus.cc:2949] T ded9df74a71c4d059719cb5401e11822 P aeb32dce70ca494ca326c528e640c9ce [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 6bab59d1be384346a2b1354fb677ec01 (127.17.60.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "79518508ccd644b28675a5de1e7280e2" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 38531 } } peers { permanent_uuid: "aeb32dce70ca494ca326c528e640c9ce" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 35351 } } peers { permanent_uuid: "90a0b30f5377440b8d72889b9d2f0a70" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 35603 } attrs { replace: true } } peers { permanent_uuid: "6bab59d1be384346a2b1354fb677ec01" member_type: NON_VOTER last_known_addr { host: "127.17.60.196" port: 39295 } attrs { promote: true } } }
I20250112 15:06:00.553226 18482 raft_consensus.cc:2949] T ded9df74a71c4d059719cb5401e11822 P 90a0b30f5377440b8d72889b9d2f0a70 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 6bab59d1be384346a2b1354fb677ec01 (127.17.60.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "79518508ccd644b28675a5de1e7280e2" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 38531 } } peers { permanent_uuid: "aeb32dce70ca494ca326c528e640c9ce" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 35351 } } peers { permanent_uuid: "90a0b30f5377440b8d72889b9d2f0a70" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 35603 } attrs { replace: true } } peers { permanent_uuid: "6bab59d1be384346a2b1354fb677ec01" member_type: NON_VOTER last_known_addr { host: "127.17.60.196" port: 39295 } attrs { promote: true } } }
I20250112 15:06:00.553288 18331 raft_consensus.cc:2949] T ded9df74a71c4d059719cb5401e11822 P 79518508ccd644b28675a5de1e7280e2 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 6bab59d1be384346a2b1354fb677ec01 (127.17.60.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "79518508ccd644b28675a5de1e7280e2" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 38531 } } peers { permanent_uuid: "aeb32dce70ca494ca326c528e640c9ce" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 35351 } } peers { permanent_uuid: "90a0b30f5377440b8d72889b9d2f0a70" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 35603 } attrs { replace: true } } peers { permanent_uuid: "6bab59d1be384346a2b1354fb677ec01" member_type: NON_VOTER last_known_addr { host: "127.17.60.196" port: 39295 } attrs { promote: true } } }
I20250112 15:06:00.554294 18225 catalog_manager.cc:5526] T ded9df74a71c4d059719cb5401e11822 P 79518508ccd644b28675a5de1e7280e2 reported cstate change: config changed from index -1 to 2, NON_VOTER 6bab59d1be384346a2b1354fb677ec01 (127.17.60.196) added. New cstate: current_term: 1 leader_uuid: "aeb32dce70ca494ca326c528e640c9ce" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "79518508ccd644b28675a5de1e7280e2" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 38531 } } peers { permanent_uuid: "aeb32dce70ca494ca326c528e640c9ce" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 35351 } } peers { permanent_uuid: "90a0b30f5377440b8d72889b9d2f0a70" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 35603 } attrs { replace: true } } peers { permanent_uuid: "6bab59d1be384346a2b1354fb677ec01" member_type: NON_VOTER last_known_addr { host: "127.17.60.196" port: 39295 } attrs { promote: true } } }
W20250112 15:06:00.555229 18373 consensus_peers.cc:487] T ded9df74a71c4d059719cb5401e11822 P aeb32dce70ca494ca326c528e640c9ce -> Peer 6bab59d1be384346a2b1354fb677ec01 (127.17.60.196:39295): Couldn't send request to peer 6bab59d1be384346a2b1354fb677ec01. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: ded9df74a71c4d059719cb5401e11822. This is attempt 1: this message will repeat every 5th retry.
I20250112 15:06:00.555227 18407 consensus_queue.cc:237] T 9032f9edb5a746729802355e7d212c95 P aeb32dce70ca494ca326c528e640c9ce [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "79518508ccd644b28675a5de1e7280e2" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 38531 } attrs { replace: true } } peers { permanent_uuid: "aeb32dce70ca494ca326c528e640c9ce" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 35351 } } peers { permanent_uuid: "90a0b30f5377440b8d72889b9d2f0a70" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 35603 } } peers { permanent_uuid: "6bab59d1be384346a2b1354fb677ec01" member_type: NON_VOTER last_known_addr { host: "127.17.60.196" port: 39295 } attrs { promote: true } }
I20250112 15:06:00.555657 18331 raft_consensus.cc:1270] T 9032f9edb5a746729802355e7d212c95 P 79518508ccd644b28675a5de1e7280e2 [term 1 FOLLOWER]: Refusing update from remote peer aeb32dce70ca494ca326c528e640c9ce: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250112 15:06:00.555675 18482 raft_consensus.cc:1270] T 9032f9edb5a746729802355e7d212c95 P 90a0b30f5377440b8d72889b9d2f0a70 [term 1 FOLLOWER]: Refusing update from remote peer aeb32dce70ca494ca326c528e640c9ce: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250112 15:06:00.555843 18722 consensus_queue.cc:1035] T 9032f9edb5a746729802355e7d212c95 P aeb32dce70ca494ca326c528e640c9ce [LEADER]: Connected to new peer: Peer: permanent_uuid: "79518508ccd644b28675a5de1e7280e2" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 38531 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250112 15:06:00.555934 18726 consensus_queue.cc:1035] T 9032f9edb5a746729802355e7d212c95 P aeb32dce70ca494ca326c528e640c9ce [LEADER]: Connected to new peer: Peer: permanent_uuid: "90a0b30f5377440b8d72889b9d2f0a70" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 35603 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20250112 15:06:00.556015 18373 consensus_peers.cc:487] T 9032f9edb5a746729802355e7d212c95 P aeb32dce70ca494ca326c528e640c9ce -> Peer 6bab59d1be384346a2b1354fb677ec01 (127.17.60.196:39295): Couldn't send request to peer 6bab59d1be384346a2b1354fb677ec01. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 9032f9edb5a746729802355e7d212c95. This is attempt 1: this message will repeat every 5th retry.
I20250112 15:06:00.556533 18533 raft_consensus.cc:2949] T 9032f9edb5a746729802355e7d212c95 P aeb32dce70ca494ca326c528e640c9ce [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 6bab59d1be384346a2b1354fb677ec01 (127.17.60.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "79518508ccd644b28675a5de1e7280e2" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 38531 } attrs { replace: true } } peers { permanent_uuid: "aeb32dce70ca494ca326c528e640c9ce" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 35351 } } peers { permanent_uuid: "90a0b30f5377440b8d72889b9d2f0a70" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 35603 } } peers { permanent_uuid: "6bab59d1be384346a2b1354fb677ec01" member_type: NON_VOTER last_known_addr { host: "127.17.60.196" port: 39295 } attrs { promote: true } } }
I20250112 15:06:00.556668 18331 raft_consensus.cc:2949] T 9032f9edb5a746729802355e7d212c95 P 79518508ccd644b28675a5de1e7280e2 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 6bab59d1be384346a2b1354fb677ec01 (127.17.60.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "79518508ccd644b28675a5de1e7280e2" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 38531 } attrs { replace: true } } peers { permanent_uuid: "aeb32dce70ca494ca326c528e640c9ce" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 35351 } } peers { permanent_uuid: "90a0b30f5377440b8d72889b9d2f0a70" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 35603 } } peers { permanent_uuid: "6bab59d1be384346a2b1354fb677ec01" member_type: NON_VOTER last_known_addr { host: "127.17.60.196" port: 39295 } attrs { promote: true } } }
I20250112 15:06:00.557394 18482 raft_consensus.cc:2949] T 9032f9edb5a746729802355e7d212c95 P 90a0b30f5377440b8d72889b9d2f0a70 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 6bab59d1be384346a2b1354fb677ec01 (127.17.60.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "79518508ccd644b28675a5de1e7280e2" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 38531 } attrs { replace: true } } peers { permanent_uuid: "aeb32dce70ca494ca326c528e640c9ce" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 35351 } } peers { permanent_uuid: "90a0b30f5377440b8d72889b9d2f0a70" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 35603 } } peers { permanent_uuid: "6bab59d1be384346a2b1354fb677ec01" member_type: NON_VOTER last_known_addr { host: "127.17.60.196" port: 39295 } attrs { promote: true } } }
I20250112 15:06:00.557417 18224 catalog_manager.cc:5526] T 9032f9edb5a746729802355e7d212c95 P 79518508ccd644b28675a5de1e7280e2 reported cstate change: config changed from index -1 to 2, NON_VOTER 6bab59d1be384346a2b1354fb677ec01 (127.17.60.196) added. New cstate: current_term: 1 leader_uuid: "aeb32dce70ca494ca326c528e640c9ce" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "79518508ccd644b28675a5de1e7280e2" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 38531 } attrs { replace: true } } peers { permanent_uuid: "aeb32dce70ca494ca326c528e640c9ce" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 35351 } } peers { permanent_uuid: "90a0b30f5377440b8d72889b9d2f0a70" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 35603 } } peers { permanent_uuid: "6bab59d1be384346a2b1354fb677ec01" member_type: NON_VOTER last_known_addr { host: "127.17.60.196" port: 39295 } attrs { promote: true } } }
I20250112 15:06:00.557983 18331 consensus_queue.cc:237] T cf6c409b42d842cfa25a995f95544e5e P 79518508ccd644b28675a5de1e7280e2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "79518508ccd644b28675a5de1e7280e2" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 38531 } } peers { permanent_uuid: "aeb32dce70ca494ca326c528e640c9ce" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 35351 } } peers { permanent_uuid: "90a0b30f5377440b8d72889b9d2f0a70" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 35603 } attrs { replace: true } } peers { permanent_uuid: "6bab59d1be384346a2b1354fb677ec01" member_type: NON_VOTER last_known_addr { host: "127.17.60.196" port: 39295 } attrs { promote: true } }
I20250112 15:06:00.558545 18407 raft_consensus.cc:1270] T cf6c409b42d842cfa25a995f95544e5e P aeb32dce70ca494ca326c528e640c9ce [term 1 FOLLOWER]: Refusing update from remote peer 79518508ccd644b28675a5de1e7280e2: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20250112 15:06:00.558553 18297 consensus_peers.cc:487] T cf6c409b42d842cfa25a995f95544e5e P 79518508ccd644b28675a5de1e7280e2 -> Peer 6bab59d1be384346a2b1354fb677ec01 (127.17.60.196:39295): Couldn't send request to peer 6bab59d1be384346a2b1354fb677ec01. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: cf6c409b42d842cfa25a995f95544e5e. This is attempt 1: this message will repeat every 5th retry.
I20250112 15:06:00.558713 18712 consensus_queue.cc:1035] T cf6c409b42d842cfa25a995f95544e5e P 79518508ccd644b28675a5de1e7280e2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aeb32dce70ca494ca326c528e640c9ce" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 35351 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250112 15:06:00.558954 18482 raft_consensus.cc:1270] T cf6c409b42d842cfa25a995f95544e5e P 90a0b30f5377440b8d72889b9d2f0a70 [term 1 FOLLOWER]: Refusing update from remote peer 79518508ccd644b28675a5de1e7280e2: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250112 15:06:00.559109 18712 consensus_queue.cc:1035] T cf6c409b42d842cfa25a995f95544e5e P 79518508ccd644b28675a5de1e7280e2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "90a0b30f5377440b8d72889b9d2f0a70" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 35603 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250112 15:06:00.559387 18682 raft_consensus.cc:2949] T cf6c409b42d842cfa25a995f95544e5e P 79518508ccd644b28675a5de1e7280e2 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 6bab59d1be384346a2b1354fb677ec01 (127.17.60.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "79518508ccd644b28675a5de1e7280e2" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 38531 } } peers { permanent_uuid: "aeb32dce70ca494ca326c528e640c9ce" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 35351 } } peers { permanent_uuid: "90a0b30f5377440b8d72889b9d2f0a70" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 35603 } attrs { replace: true } } peers { permanent_uuid: "6bab59d1be384346a2b1354fb677ec01" member_type: NON_VOTER last_known_addr { host: "127.17.60.196" port: 39295 } attrs { promote: true } } }
I20250112 15:06:00.559798 18482 raft_consensus.cc:2949] T cf6c409b42d842cfa25a995f95544e5e P 90a0b30f5377440b8d72889b9d2f0a70 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 6bab59d1be384346a2b1354fb677ec01 (127.17.60.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "79518508ccd644b28675a5de1e7280e2" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 38531 } } peers { permanent_uuid: "aeb32dce70ca494ca326c528e640c9ce" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 35351 } } peers { permanent_uuid: "90a0b30f5377440b8d72889b9d2f0a70" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 35603 } attrs { replace: true } } peers { permanent_uuid: "6bab59d1be384346a2b1354fb677ec01" member_type: NON_VOTER last_known_addr { host: "127.17.60.196" port: 39295 } attrs { promote: true } } }
I20250112 15:06:00.560395 18407 raft_consensus.cc:2949] T cf6c409b42d842cfa25a995f95544e5e P aeb32dce70ca494ca326c528e640c9ce [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 6bab59d1be384346a2b1354fb677ec01 (127.17.60.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "79518508ccd644b28675a5de1e7280e2" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 38531 } } peers { permanent_uuid: "aeb32dce70ca494ca326c528e640c9ce" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 35351 } } peers { permanent_uuid: "90a0b30f5377440b8d72889b9d2f0a70" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 35603 } attrs { replace: true } } peers { permanent_uuid: "6bab59d1be384346a2b1354fb677ec01" member_type: NON_VOTER last_known_addr { host: "127.17.60.196" port: 39295 } attrs { promote: true } } }
I20250112 15:06:00.560706 18226 catalog_manager.cc:5526] T cf6c409b42d842cfa25a995f95544e5e P 79518508ccd644b28675a5de1e7280e2 reported cstate change: config changed from index -1 to 2, NON_VOTER 6bab59d1be384346a2b1354fb677ec01 (127.17.60.196) added. New cstate: current_term: 1 leader_uuid: "79518508ccd644b28675a5de1e7280e2" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "79518508ccd644b28675a5de1e7280e2" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 38531 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "aeb32dce70ca494ca326c528e640c9ce" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 35351 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "90a0b30f5377440b8d72889b9d2f0a70" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 35603 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6bab59d1be384346a2b1354fb677ec01" member_type: NON_VOTER last_known_addr { host: "127.17.60.196" port: 39295 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20250112 15:06:00.942749 18733 ts_tablet_manager.cc:927] T ded9df74a71c4d059719cb5401e11822 P 6bab59d1be384346a2b1354fb677ec01: Initiating tablet copy from peer aeb32dce70ca494ca326c528e640c9ce (127.17.60.194:35351)
I20250112 15:06:00.943012 18733 tablet_copy_client.cc:323] T ded9df74a71c4d059719cb5401e11822 P 6bab59d1be384346a2b1354fb677ec01: tablet copy: Beginning tablet copy session from remote peer at address 127.17.60.194:35351
I20250112 15:06:00.946259 18417 tablet_copy_service.cc:140] P aeb32dce70ca494ca326c528e640c9ce: Received BeginTabletCopySession request for tablet ded9df74a71c4d059719cb5401e11822 from peer 6bab59d1be384346a2b1354fb677ec01 ({username='slave'} at 127.0.0.1:53594)
I20250112 15:06:00.946346 18417 tablet_copy_service.cc:161] P aeb32dce70ca494ca326c528e640c9ce: Beginning new tablet copy session on tablet ded9df74a71c4d059719cb5401e11822 from peer 6bab59d1be384346a2b1354fb677ec01 at {username='slave'} at 127.0.0.1:53594: session id = 6bab59d1be384346a2b1354fb677ec01-ded9df74a71c4d059719cb5401e11822
I20250112 15:06:00.947144 18417 tablet_copy_source_session.cc:215] T ded9df74a71c4d059719cb5401e11822 P aeb32dce70ca494ca326c528e640c9ce: Tablet Copy: opened 0 blocks and 1 log segments
I20250112 15:06:00.947618 18733 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ded9df74a71c4d059719cb5401e11822. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 15:06:00.948813 18733 tablet_copy_client.cc:806] T ded9df74a71c4d059719cb5401e11822 P 6bab59d1be384346a2b1354fb677ec01: tablet copy: Starting download of 0 data blocks...
I20250112 15:06:00.948968 18733 tablet_copy_client.cc:670] T ded9df74a71c4d059719cb5401e11822 P 6bab59d1be384346a2b1354fb677ec01: tablet copy: Starting download of 1 WAL segments...
I20250112 15:06:00.949569 18733 tablet_copy_client.cc:538] T ded9df74a71c4d059719cb5401e11822 P 6bab59d1be384346a2b1354fb677ec01: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250112 15:06:00.950197 18733 tablet_bootstrap.cc:492] T ded9df74a71c4d059719cb5401e11822 P 6bab59d1be384346a2b1354fb677ec01: Bootstrap starting.
I20250112 15:06:00.951699 18733 tablet_bootstrap.cc:492] T ded9df74a71c4d059719cb5401e11822 P 6bab59d1be384346a2b1354fb677ec01: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250112 15:06:00.951912 18733 tablet_bootstrap.cc:492] T ded9df74a71c4d059719cb5401e11822 P 6bab59d1be384346a2b1354fb677ec01: Bootstrap complete.
I20250112 15:06:00.952003 18733 ts_tablet_manager.cc:1397] T ded9df74a71c4d059719cb5401e11822 P 6bab59d1be384346a2b1354fb677ec01: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20250112 15:06:00.952289 18733 raft_consensus.cc:357] T ded9df74a71c4d059719cb5401e11822 P 6bab59d1be384346a2b1354fb677ec01 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "79518508ccd644b28675a5de1e7280e2" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 38531 } } peers { permanent_uuid: "aeb32dce70ca494ca326c528e640c9ce" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 35351 } } peers { permanent_uuid: "90a0b30f5377440b8d72889b9d2f0a70" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 35603 } attrs { replace: true } } peers { permanent_uuid: "6bab59d1be384346a2b1354fb677ec01" member_type: NON_VOTER last_known_addr { host: "127.17.60.196" port: 39295 } attrs { promote: true } }
I20250112 15:06:00.952380 18733 raft_consensus.cc:738] T ded9df74a71c4d059719cb5401e11822 P 6bab59d1be384346a2b1354fb677ec01 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 6bab59d1be384346a2b1354fb677ec01, State: Initialized, Role: LEARNER
I20250112 15:06:00.952417 18733 consensus_queue.cc:260] T ded9df74a71c4d059719cb5401e11822 P 6bab59d1be384346a2b1354fb677ec01 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "79518508ccd644b28675a5de1e7280e2" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 38531 } } peers { permanent_uuid: "aeb32dce70ca494ca326c528e640c9ce" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 35351 } } peers { permanent_uuid: "90a0b30f5377440b8d72889b9d2f0a70" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 35603 } attrs { replace: true } } peers { permanent_uuid: "6bab59d1be384346a2b1354fb677ec01" member_type: NON_VOTER last_known_addr { host: "127.17.60.196" port: 39295 } attrs { promote: true } }
I20250112 15:06:00.952592 18733 ts_tablet_manager.cc:1428] T ded9df74a71c4d059719cb5401e11822 P 6bab59d1be384346a2b1354fb677ec01: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20250112 15:06:00.952786 18417 tablet_copy_service.cc:342] P aeb32dce70ca494ca326c528e640c9ce: Request end of tablet copy session 6bab59d1be384346a2b1354fb677ec01-ded9df74a71c4d059719cb5401e11822 received from {username='slave'} at 127.0.0.1:53594
I20250112 15:06:00.952845 18417 tablet_copy_service.cc:434] P aeb32dce70ca494ca326c528e640c9ce: ending tablet copy session 6bab59d1be384346a2b1354fb677ec01-ded9df74a71c4d059719cb5401e11822 on tablet ded9df74a71c4d059719cb5401e11822 with peer 6bab59d1be384346a2b1354fb677ec01
I20250112 15:06:00.975772 18733 ts_tablet_manager.cc:927] T cf6c409b42d842cfa25a995f95544e5e P 6bab59d1be384346a2b1354fb677ec01: Initiating tablet copy from peer 79518508ccd644b28675a5de1e7280e2 (127.17.60.193:38531)
I20250112 15:06:00.975984 18733 tablet_copy_client.cc:323] T cf6c409b42d842cfa25a995f95544e5e P 6bab59d1be384346a2b1354fb677ec01: tablet copy: Beginning tablet copy session from remote peer at address 127.17.60.193:38531
I20250112 15:06:00.978009 18739 ts_tablet_manager.cc:927] T e311741568d04cca81089b8bad66b805 P 6bab59d1be384346a2b1354fb677ec01: Initiating tablet copy from peer 79518508ccd644b28675a5de1e7280e2 (127.17.60.193:38531)
I20250112 15:06:00.978219 18739 tablet_copy_client.cc:323] T e311741568d04cca81089b8bad66b805 P 6bab59d1be384346a2b1354fb677ec01: tablet copy: Beginning tablet copy session from remote peer at address 127.17.60.193:38531
I20250112 15:06:00.979033 18341 tablet_copy_service.cc:140] P 79518508ccd644b28675a5de1e7280e2: Received BeginTabletCopySession request for tablet cf6c409b42d842cfa25a995f95544e5e from peer 6bab59d1be384346a2b1354fb677ec01 ({username='slave'} at 127.0.0.1:41836)
I20250112 15:06:00.979076 18340 tablet_copy_service.cc:140] P 79518508ccd644b28675a5de1e7280e2: Received BeginTabletCopySession request for tablet e311741568d04cca81089b8bad66b805 from peer 6bab59d1be384346a2b1354fb677ec01 ({username='slave'} at 127.0.0.1:41836)
I20250112 15:06:00.979097 18341 tablet_copy_service.cc:161] P 79518508ccd644b28675a5de1e7280e2: Beginning new tablet copy session on tablet cf6c409b42d842cfa25a995f95544e5e from peer 6bab59d1be384346a2b1354fb677ec01 at {username='slave'} at 127.0.0.1:41836: session id = 6bab59d1be384346a2b1354fb677ec01-cf6c409b42d842cfa25a995f95544e5e
I20250112 15:06:00.979233 18340 tablet_copy_service.cc:161] P 79518508ccd644b28675a5de1e7280e2: Beginning new tablet copy session on tablet e311741568d04cca81089b8bad66b805 from peer 6bab59d1be384346a2b1354fb677ec01 at {username='slave'} at 127.0.0.1:41836: session id = 6bab59d1be384346a2b1354fb677ec01-e311741568d04cca81089b8bad66b805
I20250112 15:06:00.979756 18340 tablet_copy_source_session.cc:215] T e311741568d04cca81089b8bad66b805 P 79518508ccd644b28675a5de1e7280e2: Tablet Copy: opened 0 blocks and 1 log segments
I20250112 15:06:00.979756 18341 tablet_copy_source_session.cc:215] T cf6c409b42d842cfa25a995f95544e5e P 79518508ccd644b28675a5de1e7280e2: Tablet Copy: opened 0 blocks and 1 log segments
I20250112 15:06:00.980159 18739 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e311741568d04cca81089b8bad66b805. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 15:06:00.980257 18733 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cf6c409b42d842cfa25a995f95544e5e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 15:06:00.981432 18739 tablet_copy_client.cc:806] T e311741568d04cca81089b8bad66b805 P 6bab59d1be384346a2b1354fb677ec01: tablet copy: Starting download of 0 data blocks...
I20250112 15:06:00.981436 18733 tablet_copy_client.cc:806] T cf6c409b42d842cfa25a995f95544e5e P 6bab59d1be384346a2b1354fb677ec01: tablet copy: Starting download of 0 data blocks...
I20250112 15:06:00.981604 18739 tablet_copy_client.cc:670] T e311741568d04cca81089b8bad66b805 P 6bab59d1be384346a2b1354fb677ec01: tablet copy: Starting download of 1 WAL segments...
I20250112 15:06:00.981654 18733 tablet_copy_client.cc:670] T cf6c409b42d842cfa25a995f95544e5e P 6bab59d1be384346a2b1354fb677ec01: tablet copy: Starting download of 1 WAL segments...
I20250112 15:06:00.982133 18733 tablet_copy_client.cc:538] T cf6c409b42d842cfa25a995f95544e5e P 6bab59d1be384346a2b1354fb677ec01: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250112 15:06:00.982133 18739 tablet_copy_client.cc:538] T e311741568d04cca81089b8bad66b805 P 6bab59d1be384346a2b1354fb677ec01: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250112 15:06:00.982790 18739 tablet_bootstrap.cc:492] T e311741568d04cca81089b8bad66b805 P 6bab59d1be384346a2b1354fb677ec01: Bootstrap starting.
I20250112 15:06:00.982810 18733 tablet_bootstrap.cc:492] T cf6c409b42d842cfa25a995f95544e5e P 6bab59d1be384346a2b1354fb677ec01: Bootstrap starting.
I20250112 15:06:00.984345 18733 tablet_bootstrap.cc:492] T cf6c409b42d842cfa25a995f95544e5e P 6bab59d1be384346a2b1354fb677ec01: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250112 15:06:00.984524 18733 tablet_bootstrap.cc:492] T cf6c409b42d842cfa25a995f95544e5e P 6bab59d1be384346a2b1354fb677ec01: Bootstrap complete.
I20250112 15:06:00.984635 18733 ts_tablet_manager.cc:1397] T cf6c409b42d842cfa25a995f95544e5e P 6bab59d1be384346a2b1354fb677ec01: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20250112 15:06:00.984797 18733 raft_consensus.cc:357] T cf6c409b42d842cfa25a995f95544e5e P 6bab59d1be384346a2b1354fb677ec01 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "79518508ccd644b28675a5de1e7280e2" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 38531 } } peers { permanent_uuid: "aeb32dce70ca494ca326c528e640c9ce" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 35351 } } peers { permanent_uuid: "90a0b30f5377440b8d72889b9d2f0a70" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 35603 } attrs { replace: true } } peers { permanent_uuid: "6bab59d1be384346a2b1354fb677ec01" member_type: NON_VOTER last_known_addr { host: "127.17.60.196" port: 39295 } attrs { promote: true } }
I20250112 15:06:00.984901 18733 raft_consensus.cc:738] T cf6c409b42d842cfa25a995f95544e5e P 6bab59d1be384346a2b1354fb677ec01 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 6bab59d1be384346a2b1354fb677ec01, State: Initialized, Role: LEARNER
I20250112 15:06:00.984946 18733 consensus_queue.cc:260] T cf6c409b42d842cfa25a995f95544e5e P 6bab59d1be384346a2b1354fb677ec01 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "79518508ccd644b28675a5de1e7280e2" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 38531 } } peers { permanent_uuid: "aeb32dce70ca494ca326c528e640c9ce" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 35351 } } peers { permanent_uuid: "90a0b30f5377440b8d72889b9d2f0a70" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 35603 } attrs { replace: true } } peers { permanent_uuid: "6bab59d1be384346a2b1354fb677ec01" member_type: NON_VOTER last_known_addr { host: "127.17.60.196" port: 39295 } attrs { promote: true } }
I20250112 15:06:00.985116 18733 ts_tablet_manager.cc:1428] T cf6c409b42d842cfa25a995f95544e5e P 6bab59d1be384346a2b1354fb677ec01: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20250112 15:06:00.985308 18340 tablet_copy_service.cc:342] P 79518508ccd644b28675a5de1e7280e2: Request end of tablet copy session 6bab59d1be384346a2b1354fb677ec01-cf6c409b42d842cfa25a995f95544e5e received from {username='slave'} at 127.0.0.1:41836
I20250112 15:06:00.985365 18340 tablet_copy_service.cc:434] P 79518508ccd644b28675a5de1e7280e2: ending tablet copy session 6bab59d1be384346a2b1354fb677ec01-cf6c409b42d842cfa25a995f95544e5e on tablet cf6c409b42d842cfa25a995f95544e5e with peer 6bab59d1be384346a2b1354fb677ec01
I20250112 15:06:00.985913 18739 tablet_bootstrap.cc:492] T e311741568d04cca81089b8bad66b805 P 6bab59d1be384346a2b1354fb677ec01: Bootstrap replayed 1/1 log segments. Stats: ops{read=3 overwritten=0 applied=3 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250112 15:06:00.986083 18739 tablet_bootstrap.cc:492] T e311741568d04cca81089b8bad66b805 P 6bab59d1be384346a2b1354fb677ec01: Bootstrap complete.
I20250112 15:06:00.986174 18739 ts_tablet_manager.cc:1397] T e311741568d04cca81089b8bad66b805 P 6bab59d1be384346a2b1354fb677ec01: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20250112 15:06:00.986332 18739 raft_consensus.cc:357] T e311741568d04cca81089b8bad66b805 P 6bab59d1be384346a2b1354fb677ec01 [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "90a0b30f5377440b8d72889b9d2f0a70" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 35603 } } peers { permanent_uuid: "79518508ccd644b28675a5de1e7280e2" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 38531 } } peers { permanent_uuid: "aeb32dce70ca494ca326c528e640c9ce" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 35351 } attrs { replace: true } } peers { permanent_uuid: "6bab59d1be384346a2b1354fb677ec01" member_type: NON_VOTER last_known_addr { host: "127.17.60.196" port: 39295 } attrs { promote: true } }
I20250112 15:06:00.986433 18739 raft_consensus.cc:738] T e311741568d04cca81089b8bad66b805 P 6bab59d1be384346a2b1354fb677ec01 [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: 6bab59d1be384346a2b1354fb677ec01, State: Initialized, Role: LEARNER
I20250112 15:06:00.986521 18739 consensus_queue.cc:260] T e311741568d04cca81089b8bad66b805 P 6bab59d1be384346a2b1354fb677ec01 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 2.3, Last appended by leader: 3, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "90a0b30f5377440b8d72889b9d2f0a70" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 35603 } } peers { permanent_uuid: "79518508ccd644b28675a5de1e7280e2" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 38531 } } peers { permanent_uuid: "aeb32dce70ca494ca326c528e640c9ce" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 35351 } attrs { replace: true } } peers { permanent_uuid: "6bab59d1be384346a2b1354fb677ec01" member_type: NON_VOTER last_known_addr { host: "127.17.60.196" port: 39295 } attrs { promote: true } }
I20250112 15:06:00.986658 18739 ts_tablet_manager.cc:1428] T e311741568d04cca81089b8bad66b805 P 6bab59d1be384346a2b1354fb677ec01: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20250112 15:06:00.986822 18340 tablet_copy_service.cc:342] P 79518508ccd644b28675a5de1e7280e2: Request end of tablet copy session 6bab59d1be384346a2b1354fb677ec01-e311741568d04cca81089b8bad66b805 received from {username='slave'} at 127.0.0.1:41836
I20250112 15:06:00.986868 18340 tablet_copy_service.cc:434] P 79518508ccd644b28675a5de1e7280e2: ending tablet copy session 6bab59d1be384346a2b1354fb677ec01-e311741568d04cca81089b8bad66b805 on tablet e311741568d04cca81089b8bad66b805 with peer 6bab59d1be384346a2b1354fb677ec01
I20250112 15:06:01.004299 18739 ts_tablet_manager.cc:927] T 9032f9edb5a746729802355e7d212c95 P 6bab59d1be384346a2b1354fb677ec01: Initiating tablet copy from peer aeb32dce70ca494ca326c528e640c9ce (127.17.60.194:35351)
I20250112 15:06:01.004509 18739 tablet_copy_client.cc:323] T 9032f9edb5a746729802355e7d212c95 P 6bab59d1be384346a2b1354fb677ec01: tablet copy: Beginning tablet copy session from remote peer at address 127.17.60.194:35351
I20250112 15:06:01.004748 18417 tablet_copy_service.cc:140] P aeb32dce70ca494ca326c528e640c9ce: Received BeginTabletCopySession request for tablet 9032f9edb5a746729802355e7d212c95 from peer 6bab59d1be384346a2b1354fb677ec01 ({username='slave'} at 127.0.0.1:53594)
I20250112 15:06:01.004832 18417 tablet_copy_service.cc:161] P aeb32dce70ca494ca326c528e640c9ce: Beginning new tablet copy session on tablet 9032f9edb5a746729802355e7d212c95 from peer 6bab59d1be384346a2b1354fb677ec01 at {username='slave'} at 127.0.0.1:53594: session id = 6bab59d1be384346a2b1354fb677ec01-9032f9edb5a746729802355e7d212c95
I20250112 15:06:01.005455 18417 tablet_copy_source_session.cc:215] T 9032f9edb5a746729802355e7d212c95 P aeb32dce70ca494ca326c528e640c9ce: Tablet Copy: opened 0 blocks and 1 log segments
I20250112 15:06:01.005741 18739 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9032f9edb5a746729802355e7d212c95. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 15:06:01.006961 18739 tablet_copy_client.cc:806] T 9032f9edb5a746729802355e7d212c95 P 6bab59d1be384346a2b1354fb677ec01: tablet copy: Starting download of 0 data blocks...
I20250112 15:06:01.007074 18739 tablet_copy_client.cc:670] T 9032f9edb5a746729802355e7d212c95 P 6bab59d1be384346a2b1354fb677ec01: tablet copy: Starting download of 1 WAL segments...
I20250112 15:06:01.007622 18739 tablet_copy_client.cc:538] T 9032f9edb5a746729802355e7d212c95 P 6bab59d1be384346a2b1354fb677ec01: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250112 15:06:01.008311 18739 tablet_bootstrap.cc:492] T 9032f9edb5a746729802355e7d212c95 P 6bab59d1be384346a2b1354fb677ec01: Bootstrap starting.
I20250112 15:06:01.010015 18739 tablet_bootstrap.cc:492] T 9032f9edb5a746729802355e7d212c95 P 6bab59d1be384346a2b1354fb677ec01: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250112 15:06:01.010205 18739 tablet_bootstrap.cc:492] T 9032f9edb5a746729802355e7d212c95 P 6bab59d1be384346a2b1354fb677ec01: Bootstrap complete.
I20250112 15:06:01.010309 18739 ts_tablet_manager.cc:1397] T 9032f9edb5a746729802355e7d212c95 P 6bab59d1be384346a2b1354fb677ec01: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.001s
I20250112 15:06:01.010468 18739 raft_consensus.cc:357] T 9032f9edb5a746729802355e7d212c95 P 6bab59d1be384346a2b1354fb677ec01 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "79518508ccd644b28675a5de1e7280e2" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 38531 } attrs { replace: true } } peers { permanent_uuid: "aeb32dce70ca494ca326c528e640c9ce" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 35351 } } peers { permanent_uuid: "90a0b30f5377440b8d72889b9d2f0a70" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 35603 } } peers { permanent_uuid: "6bab59d1be384346a2b1354fb677ec01" member_type: NON_VOTER last_known_addr { host: "127.17.60.196" port: 39295 } attrs { promote: true } }
I20250112 15:06:01.010577 18739 raft_consensus.cc:738] T 9032f9edb5a746729802355e7d212c95 P 6bab59d1be384346a2b1354fb677ec01 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 6bab59d1be384346a2b1354fb677ec01, State: Initialized, Role: LEARNER
I20250112 15:06:01.010638 18739 consensus_queue.cc:260] T 9032f9edb5a746729802355e7d212c95 P 6bab59d1be384346a2b1354fb677ec01 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "79518508ccd644b28675a5de1e7280e2" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 38531 } attrs { replace: true } } peers { permanent_uuid: "aeb32dce70ca494ca326c528e640c9ce" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 35351 } } peers { permanent_uuid: "90a0b30f5377440b8d72889b9d2f0a70" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 35603 } } peers { permanent_uuid: "6bab59d1be384346a2b1354fb677ec01" member_type: NON_VOTER last_known_addr { host: "127.17.60.196" port: 39295 } attrs { promote: true } }
I20250112 15:06:01.010740 18739 ts_tablet_manager.cc:1428] T 9032f9edb5a746729802355e7d212c95 P 6bab59d1be384346a2b1354fb677ec01: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20250112 15:06:01.010916 18417 tablet_copy_service.cc:342] P aeb32dce70ca494ca326c528e640c9ce: Request end of tablet copy session 6bab59d1be384346a2b1354fb677ec01-9032f9edb5a746729802355e7d212c95 received from {username='slave'} at 127.0.0.1:53594
I20250112 15:06:01.010963 18417 tablet_copy_service.cc:434] P aeb32dce70ca494ca326c528e640c9ce: ending tablet copy session 6bab59d1be384346a2b1354fb677ec01-9032f9edb5a746729802355e7d212c95 on tablet 9032f9edb5a746729802355e7d212c95 with peer 6bab59d1be384346a2b1354fb677ec01
I20250112 15:06:01.065788 17651 tablet_server.cc:178] TabletServer@127.17.60.193:0 shutting down...
W20250112 15:06:01.066673 18210 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer [suppressed 1 similar messages]
W20250112 15:06:01.067130 18276 auto_rebalancer.cc:663] Could not move replica: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20250112 15:06:01.067219 18276 auto_rebalancer.cc:263] scheduled replica move failed to complete: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20250112 15:06:01.067685 18276 auto_rebalancer.cc:663] Could not move replica: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20250112 15:06:01.067819 18276 auto_rebalancer.cc:263] scheduled replica move failed to complete: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20250112 15:06:01.069202 17651 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250112 15:06:01.069319 17651 tablet_replica.cc:331] T b3b76094dfae41f38c01fd10b38c7a28 P 79518508ccd644b28675a5de1e7280e2: stopping tablet replica
I20250112 15:06:01.069381 17651 raft_consensus.cc:2238] T b3b76094dfae41f38c01fd10b38c7a28 P 79518508ccd644b28675a5de1e7280e2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250112 15:06:01.069427 17651 raft_consensus.cc:2267] T b3b76094dfae41f38c01fd10b38c7a28 P 79518508ccd644b28675a5de1e7280e2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 15:06:01.069607 17651 tablet_replica.cc:331] T 9032f9edb5a746729802355e7d212c95 P 79518508ccd644b28675a5de1e7280e2: stopping tablet replica
I20250112 15:06:01.069655 17651 raft_consensus.cc:2238] T 9032f9edb5a746729802355e7d212c95 P 79518508ccd644b28675a5de1e7280e2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250112 15:06:01.069756 17651 raft_consensus.cc:2267] T 9032f9edb5a746729802355e7d212c95 P 79518508ccd644b28675a5de1e7280e2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 15:06:01.070008 17651 tablet_replica.cc:331] T e311741568d04cca81089b8bad66b805 P 79518508ccd644b28675a5de1e7280e2: stopping tablet replica
I20250112 15:06:01.070065 17651 raft_consensus.cc:2238] T e311741568d04cca81089b8bad66b805 P 79518508ccd644b28675a5de1e7280e2 [term 2 LEADER]: Raft consensus shutting down.
I20250112 15:06:01.070156 17651 raft_consensus.cc:2267] T e311741568d04cca81089b8bad66b805 P 79518508ccd644b28675a5de1e7280e2 [term 2 FOLLOWER]: Raft consensus is shut down!
I20250112 15:06:01.070348 17651 tablet_replica.cc:331] T 2319a98d89eb4234aa86cc3930853fce P 79518508ccd644b28675a5de1e7280e2: stopping tablet replica
I20250112 15:06:01.070396 17651 raft_consensus.cc:2238] T 2319a98d89eb4234aa86cc3930853fce P 79518508ccd644b28675a5de1e7280e2 [term 2 LEADER]: Raft consensus shutting down.
I20250112 15:06:01.070448 17651 raft_consensus.cc:2267] T 2319a98d89eb4234aa86cc3930853fce P 79518508ccd644b28675a5de1e7280e2 [term 2 FOLLOWER]: Raft consensus is shut down!
I20250112 15:06:01.070638 17651 tablet_replica.cc:331] T 45204e59d67b4abe9ed00311ca03b73d P 79518508ccd644b28675a5de1e7280e2: stopping tablet replica
I20250112 15:06:01.070684 17651 raft_consensus.cc:2238] T 45204e59d67b4abe9ed00311ca03b73d P 79518508ccd644b28675a5de1e7280e2 [term 2 FOLLOWER]: Raft consensus shutting down.
I20250112 15:06:01.070739 17651 raft_consensus.cc:2267] T 45204e59d67b4abe9ed00311ca03b73d P 79518508ccd644b28675a5de1e7280e2 [term 2 FOLLOWER]: Raft consensus is shut down!
I20250112 15:06:01.070866 17651 tablet_replica.cc:331] T cf6c409b42d842cfa25a995f95544e5e P 79518508ccd644b28675a5de1e7280e2: stopping tablet replica
I20250112 15:06:01.070894 17651 raft_consensus.cc:2238] T cf6c409b42d842cfa25a995f95544e5e P 79518508ccd644b28675a5de1e7280e2 [term 1 LEADER]: Raft consensus shutting down.
I20250112 15:06:01.070955 17651 raft_consensus.cc:2267] T cf6c409b42d842cfa25a995f95544e5e P 79518508ccd644b28675a5de1e7280e2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 15:06:01.071164 17651 tablet_replica.cc:331] T 4e45a9f365b344c2aa0703adea508eeb P 79518508ccd644b28675a5de1e7280e2: stopping tablet replica
I20250112 15:06:01.071206 17651 raft_consensus.cc:2238] T 4e45a9f365b344c2aa0703adea508eeb P 79518508ccd644b28675a5de1e7280e2 [term 2 FOLLOWER]: Raft consensus shutting down.
I20250112 15:06:01.071230 17651 raft_consensus.cc:2267] T 4e45a9f365b344c2aa0703adea508eeb P 79518508ccd644b28675a5de1e7280e2 [term 2 FOLLOWER]: Raft consensus is shut down!
I20250112 15:06:01.071326 17651 tablet_replica.cc:331] T ded9df74a71c4d059719cb5401e11822 P 79518508ccd644b28675a5de1e7280e2: stopping tablet replica
I20250112 15:06:01.071353 17651 raft_consensus.cc:2238] T ded9df74a71c4d059719cb5401e11822 P 79518508ccd644b28675a5de1e7280e2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250112 15:06:01.071380 17651 raft_consensus.cc:2267] T ded9df74a71c4d059719cb5401e11822 P 79518508ccd644b28675a5de1e7280e2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 15:06:01.073606 17651 tablet_server.cc:195] TabletServer@127.17.60.193:0 shutdown complete.
I20250112 15:06:01.074975 17651 tablet_server.cc:178] TabletServer@127.17.60.194:0 shutting down...
W20250112 15:06:01.075433 18276 auto_rebalancer.cc:663] Could not move replica: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20250112 15:06:01.075481 18276 auto_rebalancer.cc:263] scheduled replica move failed to complete: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20250112 15:06:01.075856 18276 auto_rebalancer.cc:663] Could not move replica: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20250112 15:06:01.075896 18276 auto_rebalancer.cc:263] scheduled replica move failed to complete: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20250112 15:06:01.078420 17651 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250112 15:06:01.078541 17651 tablet_replica.cc:331] T e311741568d04cca81089b8bad66b805 P aeb32dce70ca494ca326c528e640c9ce: stopping tablet replica
I20250112 15:06:01.078604 17651 raft_consensus.cc:2238] T e311741568d04cca81089b8bad66b805 P aeb32dce70ca494ca326c528e640c9ce [term 2 FOLLOWER]: Raft consensus shutting down.
I20250112 15:06:01.078660 17651 raft_consensus.cc:2267] T e311741568d04cca81089b8bad66b805 P aeb32dce70ca494ca326c528e640c9ce [term 2 FOLLOWER]: Raft consensus is shut down!
I20250112 15:06:01.078891 17651 tablet_replica.cc:331] T ded9df74a71c4d059719cb5401e11822 P aeb32dce70ca494ca326c528e640c9ce: stopping tablet replica
I20250112 15:06:01.078938 17651 raft_consensus.cc:2238] T ded9df74a71c4d059719cb5401e11822 P aeb32dce70ca494ca326c528e640c9ce [term 1 LEADER]: Raft consensus shutting down.
I20250112 15:06:01.079005 17651 raft_consensus.cc:2267] T ded9df74a71c4d059719cb5401e11822 P aeb32dce70ca494ca326c528e640c9ce [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 15:06:01.079175 17651 tablet_replica.cc:331] T cf6c409b42d842cfa25a995f95544e5e P aeb32dce70ca494ca326c528e640c9ce: stopping tablet replica
I20250112 15:06:01.079213 17651 raft_consensus.cc:2238] T cf6c409b42d842cfa25a995f95544e5e P aeb32dce70ca494ca326c528e640c9ce [term 1 FOLLOWER]: Raft consensus shutting down.
I20250112 15:06:01.079255 17651 raft_consensus.cc:2267] T cf6c409b42d842cfa25a995f95544e5e P aeb32dce70ca494ca326c528e640c9ce [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 15:06:01.079411 17651 tablet_replica.cc:331] T 4e45a9f365b344c2aa0703adea508eeb P aeb32dce70ca494ca326c528e640c9ce: stopping tablet replica
I20250112 15:06:01.079450 17651 raft_consensus.cc:2238] T 4e45a9f365b344c2aa0703adea508eeb P aeb32dce70ca494ca326c528e640c9ce [term 2 FOLLOWER]: Raft consensus shutting down.
I20250112 15:06:01.079488 17651 raft_consensus.cc:2267] T 4e45a9f365b344c2aa0703adea508eeb P aeb32dce70ca494ca326c528e640c9ce [term 2 FOLLOWER]: Raft consensus is shut down!
I20250112 15:06:01.079595 17651 tablet_replica.cc:331] T 45204e59d67b4abe9ed00311ca03b73d P aeb32dce70ca494ca326c528e640c9ce: stopping tablet replica
I20250112 15:06:01.079630 17651 raft_consensus.cc:2238] T 45204e59d67b4abe9ed00311ca03b73d P aeb32dce70ca494ca326c528e640c9ce [term 2 LEADER]: Raft consensus shutting down.
I20250112 15:06:01.079679 17651 raft_consensus.cc:2267] T 45204e59d67b4abe9ed00311ca03b73d P aeb32dce70ca494ca326c528e640c9ce [term 2 FOLLOWER]: Raft consensus is shut down!
I20250112 15:06:01.079780 17651 tablet_replica.cc:331] T 2319a98d89eb4234aa86cc3930853fce P aeb32dce70ca494ca326c528e640c9ce: stopping tablet replica
I20250112 15:06:01.079813 17651 raft_consensus.cc:2238] T 2319a98d89eb4234aa86cc3930853fce P aeb32dce70ca494ca326c528e640c9ce [term 2 FOLLOWER]: Raft consensus shutting down.
I20250112 15:06:01.079857 17651 raft_consensus.cc:2267] T 2319a98d89eb4234aa86cc3930853fce P aeb32dce70ca494ca326c528e640c9ce [term 2 FOLLOWER]: Raft consensus is shut down!
I20250112 15:06:01.080018 17651 tablet_replica.cc:331] T b3b76094dfae41f38c01fd10b38c7a28 P aeb32dce70ca494ca326c528e640c9ce: stopping tablet replica
I20250112 15:06:01.080056 17651 raft_consensus.cc:2238] T b3b76094dfae41f38c01fd10b38c7a28 P aeb32dce70ca494ca326c528e640c9ce [term 1 FOLLOWER]: Raft consensus shutting down.
I20250112 15:06:01.080085 17651 raft_consensus.cc:2267] T b3b76094dfae41f38c01fd10b38c7a28 P aeb32dce70ca494ca326c528e640c9ce [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 15:06:01.080200 17651 tablet_replica.cc:331] T 9032f9edb5a746729802355e7d212c95 P aeb32dce70ca494ca326c528e640c9ce: stopping tablet replica
I20250112 15:06:01.080232 17651 raft_consensus.cc:2238] T 9032f9edb5a746729802355e7d212c95 P aeb32dce70ca494ca326c528e640c9ce [term 1 LEADER]: Raft consensus shutting down.
I20250112 15:06:01.080297 17651 raft_consensus.cc:2267] T 9032f9edb5a746729802355e7d212c95 P aeb32dce70ca494ca326c528e640c9ce [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 15:06:01.092087 17651 tablet_server.cc:195] TabletServer@127.17.60.194:0 shutdown complete.
I20250112 15:06:01.093538 17651 tablet_server.cc:178] TabletServer@127.17.60.195:0 shutting down...
I20250112 15:06:01.096894 17651 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250112 15:06:01.096997 17651 tablet_replica.cc:331] T 45204e59d67b4abe9ed00311ca03b73d P 90a0b30f5377440b8d72889b9d2f0a70: stopping tablet replica
I20250112 15:06:01.097062 17651 raft_consensus.cc:2238] T 45204e59d67b4abe9ed00311ca03b73d P 90a0b30f5377440b8d72889b9d2f0a70 [term 2 FOLLOWER]: Raft consensus shutting down.
I20250112 15:06:01.097106 17651 raft_consensus.cc:2267] T 45204e59d67b4abe9ed00311ca03b73d P 90a0b30f5377440b8d72889b9d2f0a70 [term 2 FOLLOWER]: Raft consensus is shut down!
I20250112 15:06:01.097244 17651 tablet_replica.cc:331] T 2319a98d89eb4234aa86cc3930853fce P 90a0b30f5377440b8d72889b9d2f0a70: stopping tablet replica
I20250112 15:06:01.097275 17651 raft_consensus.cc:2238] T 2319a98d89eb4234aa86cc3930853fce P 90a0b30f5377440b8d72889b9d2f0a70 [term 2 FOLLOWER]: Raft consensus shutting down.
I20250112 15:06:01.097297 17651 raft_consensus.cc:2267] T 2319a98d89eb4234aa86cc3930853fce P 90a0b30f5377440b8d72889b9d2f0a70 [term 2 FOLLOWER]: Raft consensus is shut down!
I20250112 15:06:01.097472 17651 tablet_replica.cc:331] T 9032f9edb5a746729802355e7d212c95 P 90a0b30f5377440b8d72889b9d2f0a70: stopping tablet replica
I20250112 15:06:01.097509 17651 raft_consensus.cc:2238] T 9032f9edb5a746729802355e7d212c95 P 90a0b30f5377440b8d72889b9d2f0a70 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250112 15:06:01.097535 17651 raft_consensus.cc:2267] T 9032f9edb5a746729802355e7d212c95 P 90a0b30f5377440b8d72889b9d2f0a70 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 15:06:01.097704 17651 tablet_replica.cc:331] T e311741568d04cca81089b8bad66b805 P 90a0b30f5377440b8d72889b9d2f0a70: stopping tablet replica
I20250112 15:06:01.097739 17651 raft_consensus.cc:2238] T e311741568d04cca81089b8bad66b805 P 90a0b30f5377440b8d72889b9d2f0a70 [term 2 FOLLOWER]: Raft consensus shutting down.
I20250112 15:06:01.097767 17651 raft_consensus.cc:2267] T e311741568d04cca81089b8bad66b805 P 90a0b30f5377440b8d72889b9d2f0a70 [term 2 FOLLOWER]: Raft consensus is shut down!
I20250112 15:06:01.097939 17651 tablet_replica.cc:331] T 4e45a9f365b344c2aa0703adea508eeb P 90a0b30f5377440b8d72889b9d2f0a70: stopping tablet replica
I20250112 15:06:01.097973 17651 raft_consensus.cc:2238] T 4e45a9f365b344c2aa0703adea508eeb P 90a0b30f5377440b8d72889b9d2f0a70 [term 2 LEADER]: Raft consensus shutting down.
I20250112 15:06:01.098016 17651 raft_consensus.cc:2267] T 4e45a9f365b344c2aa0703adea508eeb P 90a0b30f5377440b8d72889b9d2f0a70 [term 2 FOLLOWER]: Raft consensus is shut down!
I20250112 15:06:01.098098 17651 tablet_replica.cc:331] T cf6c409b42d842cfa25a995f95544e5e P 90a0b30f5377440b8d72889b9d2f0a70: stopping tablet replica
I20250112 15:06:01.098122 17651 raft_consensus.cc:2238] T cf6c409b42d842cfa25a995f95544e5e P 90a0b30f5377440b8d72889b9d2f0a70 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250112 15:06:01.098148 17651 raft_consensus.cc:2267] T cf6c409b42d842cfa25a995f95544e5e P 90a0b30f5377440b8d72889b9d2f0a70 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 15:06:01.098313 17651 tablet_replica.cc:331] T ded9df74a71c4d059719cb5401e11822 P 90a0b30f5377440b8d72889b9d2f0a70: stopping tablet replica
I20250112 15:06:01.098349 17651 raft_consensus.cc:2238] T ded9df74a71c4d059719cb5401e11822 P 90a0b30f5377440b8d72889b9d2f0a70 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250112 15:06:01.098377 17651 raft_consensus.cc:2267] T ded9df74a71c4d059719cb5401e11822 P 90a0b30f5377440b8d72889b9d2f0a70 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 15:06:01.098531 17651 tablet_replica.cc:331] T b3b76094dfae41f38c01fd10b38c7a28 P 90a0b30f5377440b8d72889b9d2f0a70: stopping tablet replica
I20250112 15:06:01.098563 17651 raft_consensus.cc:2238] T b3b76094dfae41f38c01fd10b38c7a28 P 90a0b30f5377440b8d72889b9d2f0a70 [term 1 LEADER]: Raft consensus shutting down.
I20250112 15:06:01.098600 17651 raft_consensus.cc:2267] T b3b76094dfae41f38c01fd10b38c7a28 P 90a0b30f5377440b8d72889b9d2f0a70 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 15:06:01.110402 17651 tablet_server.cc:195] TabletServer@127.17.60.195:0 shutdown complete.
I20250112 15:06:01.111505 17651 tablet_server.cc:178] TabletServer@127.17.60.196:0 shutting down...
I20250112 15:06:01.113576 17651 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250112 15:06:01.113938 17651 tablet_replica.cc:331] T 9032f9edb5a746729802355e7d212c95 P 6bab59d1be384346a2b1354fb677ec01: stopping tablet replica
I20250112 15:06:01.113997 17651 raft_consensus.cc:2238] T 9032f9edb5a746729802355e7d212c95 P 6bab59d1be384346a2b1354fb677ec01 [term 1 LEARNER]: Raft consensus shutting down.
I20250112 15:06:01.114070 17651 raft_consensus.cc:2267] T 9032f9edb5a746729802355e7d212c95 P 6bab59d1be384346a2b1354fb677ec01 [term 1 LEARNER]: Raft consensus is shut down!
I20250112 15:06:01.114295 17651 tablet_replica.cc:331] T cf6c409b42d842cfa25a995f95544e5e P 6bab59d1be384346a2b1354fb677ec01: stopping tablet replica
I20250112 15:06:01.114338 17651 raft_consensus.cc:2238] T cf6c409b42d842cfa25a995f95544e5e P 6bab59d1be384346a2b1354fb677ec01 [term 1 LEARNER]: Raft consensus shutting down.
I20250112 15:06:01.114382 17651 raft_consensus.cc:2267] T cf6c409b42d842cfa25a995f95544e5e P 6bab59d1be384346a2b1354fb677ec01 [term 1 LEARNER]: Raft consensus is shut down!
I20250112 15:06:01.114537 17651 tablet_replica.cc:331] T ded9df74a71c4d059719cb5401e11822 P 6bab59d1be384346a2b1354fb677ec01: stopping tablet replica
I20250112 15:06:01.114574 17651 raft_consensus.cc:2238] T ded9df74a71c4d059719cb5401e11822 P 6bab59d1be384346a2b1354fb677ec01 [term 1 LEARNER]: Raft consensus shutting down.
I20250112 15:06:01.114614 17651 raft_consensus.cc:2267] T ded9df74a71c4d059719cb5401e11822 P 6bab59d1be384346a2b1354fb677ec01 [term 1 LEARNER]: Raft consensus is shut down!
I20250112 15:06:01.114764 17651 tablet_replica.cc:331] T e311741568d04cca81089b8bad66b805 P 6bab59d1be384346a2b1354fb677ec01: stopping tablet replica
I20250112 15:06:01.114799 17651 raft_consensus.cc:2238] T e311741568d04cca81089b8bad66b805 P 6bab59d1be384346a2b1354fb677ec01 [term 2 LEARNER]: Raft consensus shutting down.
I20250112 15:06:01.114827 17651 raft_consensus.cc:2267] T e311741568d04cca81089b8bad66b805 P 6bab59d1be384346a2b1354fb677ec01 [term 2 LEARNER]: Raft consensus is shut down!
I20250112 15:06:01.126340 17651 tablet_server.cc:195] TabletServer@127.17.60.196:0 shutdown complete.
I20250112 15:06:01.127308 17651 master.cc:537] Master@127.17.60.254:45251 shutting down...
I20250112 15:06:01.129626 17651 raft_consensus.cc:2238] T 00000000000000000000000000000000 P 5d4bbf880f214236b21b3546f0554eef [term 1 LEADER]: Raft consensus shutting down.
I20250112 15:06:01.129696 17651 raft_consensus.cc:2267] T 00000000000000000000000000000000 P 5d4bbf880f214236b21b3546f0554eef [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 15:06:01.129725 17651 tablet_replica.cc:331] T 00000000000000000000000000000000 P 5d4bbf880f214236b21b3546f0554eef: stopping tablet replica
I20250112 15:06:01.141350 17651 master.cc:559] Master@127.17.60.254:45251 shutdown complete.
[ OK ] AutoRebalancerTest.AutoRebalancingTurnOffAndOn (7648 ms)
[ RUN ] AutoRebalancerTest.NextLeaderResumesAutoRebalancing
I20250112 15:06:01.149703 17651 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.17.60.254:38227,127.17.60.253:46387,127.17.60.252:43895
I20250112 15:06:01.149876 17651 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20250112 15:06:01.152112 17651 server_base.cc:1034] running on GCE node
W20250112 15:06:01.152652 18749 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250112 15:06:01.153182 18747 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250112 15:06:01.153595 18746 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250112 15:06:01.153740 17651 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250112 15:06:01.153797 17651 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250112 15:06:01.153810 17651 hybrid_clock.cc:648] HybridClock initialized: now 1736694361153811 us; error 0 us; skew 500 ppm
I20250112 15:06:01.154422 17651 webserver.cc:458] Webserver started at http://127.17.60.254:41141/ using document root <none> and password file <none>
I20250112 15:06:01.154498 17651 fs_manager.cc:362] Metadata directory not provided
I20250112 15:06:01.154544 17651 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250112 15:06:01.154588 17651 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250112 15:06:01.154865 17651 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task95dLSD/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1736694351070047-17651-0/minicluster-data/master-0-root/instance:
uuid: "4a9e1db91a694b78b174fa300b67fb97"
format_stamp: "Formatted at 2025-01-12 15:06:01 on dist-test-slave-b4nz"
I20250112 15:06:01.155692 17651 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20250112 15:06:01.156116 18754 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250112 15:06:01.156245 17651 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20250112 15:06:01.156283 17651 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task95dLSD/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1736694351070047-17651-0/minicluster-data/master-0-root
uuid: "4a9e1db91a694b78b174fa300b67fb97"
format_stamp: "Formatted at 2025-01-12 15:06:01 on dist-test-slave-b4nz"
I20250112 15:06:01.156320 17651 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task95dLSD/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1736694351070047-17651-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-task95dLSD/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1736694351070047-17651-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-task95dLSD/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1736694351070047-17651-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250112 15:06:01.172888 17651 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250112 15:06:01.173090 17651 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250112 15:06:01.175554 17651 rpc_server.cc:307] RPC server started. Bound to: 127.17.60.254:38227
I20250112 15:06:01.176208 17651 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20250112 15:06:01.176507 18806 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.17.60.254:38227 every 8 connection(s)
I20250112 15:06:01.180797 18807 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 15:06:01.181628 18807 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.17.60.254" port: 38227 } has no permanent_uuid. Determining permanent_uuid...
W20250112 15:06:01.181671 18809 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250112 15:06:01.182157 18812 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250112 15:06:01.182771 17651 server_base.cc:1034] running on GCE node
W20250112 15:06:01.183769 18810 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250112 15:06:01.183902 17651 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250112 15:06:01.183951 17651 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250112 15:06:01.183975 17651 hybrid_clock.cc:648] HybridClock initialized: now 1736694361183975 us; error 0 us; skew 500 ppm
I20250112 15:06:01.184623 17651 webserver.cc:458] Webserver started at http://127.17.60.253:36537/ using document root <none> and password file <none>
I20250112 15:06:01.184713 17651 fs_manager.cc:362] Metadata directory not provided
I20250112 15:06:01.184765 17651 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250112 15:06:01.184810 18807 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.17.60.253" port: 46387 } has no permanent_uuid. Determining permanent_uuid...
I20250112 15:06:01.184813 17651 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250112 15:06:01.185218 17651 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task95dLSD/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1736694351070047-17651-0/minicluster-data/master-1-root/instance:
uuid: "cd21cea7d1214dd2ab14d1e6c471e6ea"
format_stamp: "Formatted at 2025-01-12 15:06:01 on dist-test-slave-b4nz"
W20250112 15:06:01.185424 18807 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.17.60.253:46387: Network error: Client connection negotiation failed: client connection to 127.17.60.253:46387: connect: Connection refused (error 111)
I20250112 15:06:01.186134 17651 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20250112 15:06:01.186605 18820 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250112 15:06:01.186714 17651 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20250112 15:06:01.186756 17651 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task95dLSD/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1736694351070047-17651-0/minicluster-data/master-1-root
uuid: "cd21cea7d1214dd2ab14d1e6c471e6ea"
format_stamp: "Formatted at 2025-01-12 15:06:01 on dist-test-slave-b4nz"
I20250112 15:06:01.186800 17651 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task95dLSD/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1736694351070047-17651-0/minicluster-data/master-1-root
metadata directory: /tmp/dist-test-task95dLSD/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1736694351070047-17651-0/minicluster-data/master-1-root
1 data directories: /tmp/dist-test-task95dLSD/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1736694351070047-17651-0/minicluster-data/master-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250112 15:06:01.195505 17651 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250112 15:06:01.195721 17651 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250112 15:06:01.198222 17651 rpc_server.cc:307] RPC server started. Bound to: 127.17.60.253:46387
I20250112 15:06:01.198271 18872 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.17.60.253:46387 every 8 connection(s)
I20250112 15:06:01.198647 17651 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20250112 15:06:01.198666 18873 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
W20250112 15:06:01.199860 18875 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250112 15:06:01.199838 18873 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.17.60.254" port: 38227 } has no permanent_uuid. Determining permanent_uuid...
W20250112 15:06:01.200397 18878 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250112 15:06:01.201329 17651 server_base.cc:1034] running on GCE node
W20250112 15:06:01.202009 18876 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250112 15:06:01.202193 17651 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250112 15:06:01.202230 17651 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250112 15:06:01.202253 17651 hybrid_clock.cc:648] HybridClock initialized: now 1736694361202253 us; error 0 us; skew 500 ppm
I20250112 15:06:01.202841 17651 webserver.cc:458] Webserver started at http://127.17.60.252:35291/ using document root <none> and password file <none>
I20250112 15:06:01.202919 17651 fs_manager.cc:362] Metadata directory not provided
I20250112 15:06:01.202960 17651 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250112 15:06:01.203003 17651 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250112 15:06:01.203248 17651 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task95dLSD/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1736694351070047-17651-0/minicluster-data/master-2-root/instance:
uuid: "e5fa9b0933e446c9854df3006fce277f"
format_stamp: "Formatted at 2025-01-12 15:06:01 on dist-test-slave-b4nz"
I20250112 15:06:01.204074 17651 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20250112 15:06:01.204272 18873 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.17.60.253" port: 46387 } has no permanent_uuid. Determining permanent_uuid...
I20250112 15:06:01.204629 18884 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250112 15:06:01.204773 17651 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20250112 15:06:01.204828 17651 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task95dLSD/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1736694351070047-17651-0/minicluster-data/master-2-root
uuid: "e5fa9b0933e446c9854df3006fce277f"
format_stamp: "Formatted at 2025-01-12 15:06:01 on dist-test-slave-b4nz"
I20250112 15:06:01.204864 17651 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task95dLSD/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1736694351070047-17651-0/minicluster-data/master-2-root
metadata directory: /tmp/dist-test-task95dLSD/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1736694351070047-17651-0/minicluster-data/master-2-root
1 data directories: /tmp/dist-test-task95dLSD/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1736694351070047-17651-0/minicluster-data/master-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250112 15:06:01.207192 18873 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.17.60.252" port: 43895 } has no permanent_uuid. Determining permanent_uuid...
W20250112 15:06:01.207718 18873 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.17.60.252:43895: Network error: Client connection negotiation failed: client connection to 127.17.60.252:43895: connect: Connection refused (error 111)
I20250112 15:06:01.210531 17651 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250112 15:06:01.210723 17651 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250112 15:06:01.213100 17651 rpc_server.cc:307] RPC server started. Bound to: 127.17.60.252:43895
I20250112 15:06:01.213145 18938 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.17.60.252:43895 every 8 connection(s)
I20250112 15:06:01.213285 17651 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20250112 15:06:01.213373 18939 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 15:06:01.213951 18939 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.17.60.254" port: 38227 } has no permanent_uuid. Determining permanent_uuid...
I20250112 15:06:01.216348 18939 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.17.60.253" port: 46387 } has no permanent_uuid. Determining permanent_uuid...
I20250112 15:06:01.218436 18939 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.17.60.252" port: 43895 } has no permanent_uuid. Determining permanent_uuid...
I20250112 15:06:01.221547 18939 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e5fa9b0933e446c9854df3006fce277f: Bootstrap starting.
I20250112 15:06:01.221850 18939 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e5fa9b0933e446c9854df3006fce277f: Neither blocks nor log segments found. Creating new log.
I20250112 15:06:01.222395 18939 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e5fa9b0933e446c9854df3006fce277f: No bootstrap required, opened a new log
I20250112 15:06:01.222532 18939 raft_consensus.cc:357] T 00000000000000000000000000000000 P e5fa9b0933e446c9854df3006fce277f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a9e1db91a694b78b174fa300b67fb97" member_type: VOTER last_known_addr { host: "127.17.60.254" port: 38227 } } peers { permanent_uuid: "cd21cea7d1214dd2ab14d1e6c471e6ea" member_type: VOTER last_known_addr { host: "127.17.60.253" port: 46387 } } peers { permanent_uuid: "e5fa9b0933e446c9854df3006fce277f" member_type: VOTER last_known_addr { host: "127.17.60.252" port: 43895 } }
I20250112 15:06:01.222584 18939 raft_consensus.cc:383] T 00000000000000000000000000000000 P e5fa9b0933e446c9854df3006fce277f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 15:06:01.222607 18939 raft_consensus.cc:738] T 00000000000000000000000000000000 P e5fa9b0933e446c9854df3006fce277f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e5fa9b0933e446c9854df3006fce277f, State: Initialized, Role: FOLLOWER
I20250112 15:06:01.222663 18939 consensus_queue.cc:260] T 00000000000000000000000000000000 P e5fa9b0933e446c9854df3006fce277f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a9e1db91a694b78b174fa300b67fb97" member_type: VOTER last_known_addr { host: "127.17.60.254" port: 38227 } } peers { permanent_uuid: "cd21cea7d1214dd2ab14d1e6c471e6ea" member_type: VOTER last_known_addr { host: "127.17.60.253" port: 46387 } } peers { permanent_uuid: "e5fa9b0933e446c9854df3006fce277f" member_type: VOTER last_known_addr { host: "127.17.60.252" port: 43895 } }
I20250112 15:06:01.222824 18939 sys_catalog.cc:564] T 00000000000000000000000000000000 P e5fa9b0933e446c9854df3006fce277f [sys.catalog]: configured and running, proceeding with master startup.
I20250112 15:06:01.222833 18944 sys_catalog.cc:455] T 00000000000000000000000000000000 P e5fa9b0933e446c9854df3006fce277f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a9e1db91a694b78b174fa300b67fb97" member_type: VOTER last_known_addr { host: "127.17.60.254" port: 38227 } } peers { permanent_uuid: "cd21cea7d1214dd2ab14d1e6c471e6ea" member_type: VOTER last_known_addr { host: "127.17.60.253" port: 46387 } } peers { permanent_uuid: "e5fa9b0933e446c9854df3006fce277f" member_type: VOTER last_known_addr { host: "127.17.60.252" port: 43895 } } }
I20250112 15:06:01.222905 18944 sys_catalog.cc:458] T 00000000000000000000000000000000 P e5fa9b0933e446c9854df3006fce277f [sys.catalog]: This master's current role is: FOLLOWER
W20250112 15:06:01.223945 18955 catalog_manager.cc:1559] T 00000000000000000000000000000000 P e5fa9b0933e446c9854df3006fce277f: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20250112 15:06:01.223990 18955 catalog_manager.cc:874] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20250112 15:06:01.227560 18807 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.17.60.253" port: 46387 } attempt: 1
I20250112 15:06:01.229972 18807 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.17.60.252" port: 43895 } has no permanent_uuid. Determining permanent_uuid...
I20250112 15:06:01.232824 18807 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4a9e1db91a694b78b174fa300b67fb97: Bootstrap starting.
I20250112 15:06:01.233181 18807 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4a9e1db91a694b78b174fa300b67fb97: Neither blocks nor log segments found. Creating new log.
I20250112 15:06:01.233690 18807 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4a9e1db91a694b78b174fa300b67fb97: No bootstrap required, opened a new log
I20250112 15:06:01.233834 18807 raft_consensus.cc:357] T 00000000000000000000000000000000 P 4a9e1db91a694b78b174fa300b67fb97 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a9e1db91a694b78b174fa300b67fb97" member_type: VOTER last_known_addr { host: "127.17.60.254" port: 38227 } } peers { permanent_uuid: "cd21cea7d1214dd2ab14d1e6c471e6ea" member_type: VOTER last_known_addr { host: "127.17.60.253" port: 46387 } } peers { permanent_uuid: "e5fa9b0933e446c9854df3006fce277f" member_type: VOTER last_known_addr { host: "127.17.60.252" port: 43895 } }
I20250112 15:06:01.233880 18807 raft_consensus.cc:383] T 00000000000000000000000000000000 P 4a9e1db91a694b78b174fa300b67fb97 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 15:06:01.233894 18807 raft_consensus.cc:738] T 00000000000000000000000000000000 P 4a9e1db91a694b78b174fa300b67fb97 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4a9e1db91a694b78b174fa300b67fb97, State: Initialized, Role: FOLLOWER
I20250112 15:06:01.233932 18807 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4a9e1db91a694b78b174fa300b67fb97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a9e1db91a694b78b174fa300b67fb97" member_type: VOTER last_known_addr { host: "127.17.60.254" port: 38227 } } peers { permanent_uuid: "cd21cea7d1214dd2ab14d1e6c471e6ea" member_type: VOTER last_known_addr { host: "127.17.60.253" port: 46387 } } peers { permanent_uuid: "e5fa9b0933e446c9854df3006fce277f" member_type: VOTER last_known_addr { host: "127.17.60.252" port: 43895 } }
I20250112 15:06:01.234098 18807 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4a9e1db91a694b78b174fa300b67fb97 [sys.catalog]: configured and running, proceeding with master startup.
I20250112 15:06:01.234115 18957 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4a9e1db91a694b78b174fa300b67fb97 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a9e1db91a694b78b174fa300b67fb97" member_type: VOTER last_known_addr { host: "127.17.60.254" port: 38227 } } peers { permanent_uuid: "cd21cea7d1214dd2ab14d1e6c471e6ea" member_type: VOTER last_known_addr { host: "127.17.60.253" port: 46387 } } peers { permanent_uuid: "e5fa9b0933e446c9854df3006fce277f" member_type: VOTER last_known_addr { host: "127.17.60.252" port: 43895 } } }
I20250112 15:06:01.234202 18957 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4a9e1db91a694b78b174fa300b67fb97 [sys.catalog]: This master's current role is: FOLLOWER
I20250112 15:06:01.235133 17651 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 1
W20250112 15:06:01.235231 18968 catalog_manager.cc:1559] T 00000000000000000000000000000000 P 4a9e1db91a694b78b174fa300b67fb97: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20250112 15:06:01.235262 18968 catalog_manager.cc:874] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20250112 15:06:01.268889 18873 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.17.60.252" port: 43895 } attempt: 1
I20250112 15:06:01.272469 18873 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cd21cea7d1214dd2ab14d1e6c471e6ea: Bootstrap starting.
I20250112 15:06:01.272928 18873 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cd21cea7d1214dd2ab14d1e6c471e6ea: Neither blocks nor log segments found. Creating new log.
I20250112 15:06:01.273515 18873 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cd21cea7d1214dd2ab14d1e6c471e6ea: No bootstrap required, opened a new log
I20250112 15:06:01.273669 18873 raft_consensus.cc:357] T 00000000000000000000000000000000 P cd21cea7d1214dd2ab14d1e6c471e6ea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a9e1db91a694b78b174fa300b67fb97" member_type: VOTER last_known_addr { host: "127.17.60.254" port: 38227 } } peers { permanent_uuid: "cd21cea7d1214dd2ab14d1e6c471e6ea" member_type: VOTER last_known_addr { host: "127.17.60.253" port: 46387 } } peers { permanent_uuid: "e5fa9b0933e446c9854df3006fce277f" member_type: VOTER last_known_addr { host: "127.17.60.252" port: 43895 } }
I20250112 15:06:01.273731 18873 raft_consensus.cc:383] T 00000000000000000000000000000000 P cd21cea7d1214dd2ab14d1e6c471e6ea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 15:06:01.273773 18873 raft_consensus.cc:738] T 00000000000000000000000000000000 P cd21cea7d1214dd2ab14d1e6c471e6ea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cd21cea7d1214dd2ab14d1e6c471e6ea, State: Initialized, Role: FOLLOWER
I20250112 15:06:01.273823 18873 consensus_queue.cc:260] T 00000000000000000000000000000000 P cd21cea7d1214dd2ab14d1e6c471e6ea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a9e1db91a694b78b174fa300b67fb97" member_type: VOTER last_known_addr { host: "127.17.60.254" port: 38227 } } peers { permanent_uuid: "cd21cea7d1214dd2ab14d1e6c471e6ea" member_type: VOTER last_known_addr { host: "127.17.60.253" port: 46387 } } peers { permanent_uuid: "e5fa9b0933e446c9854df3006fce277f" member_type: VOTER last_known_addr { host: "127.17.60.252" port: 43895 } }
I20250112 15:06:01.274029 18873 sys_catalog.cc:564] T 00000000000000000000000000000000 P cd21cea7d1214dd2ab14d1e6c471e6ea [sys.catalog]: configured and running, proceeding with master startup.
I20250112 15:06:01.274027 18970 sys_catalog.cc:455] T 00000000000000000000000000000000 P cd21cea7d1214dd2ab14d1e6c471e6ea [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a9e1db91a694b78b174fa300b67fb97" member_type: VOTER last_known_addr { host: "127.17.60.254" port: 38227 } } peers { permanent_uuid: "cd21cea7d1214dd2ab14d1e6c471e6ea" member_type: VOTER last_known_addr { host: "127.17.60.253" port: 46387 } } peers { permanent_uuid: "e5fa9b0933e446c9854df3006fce277f" member_type: VOTER last_known_addr { host: "127.17.60.252" port: 43895 } } }
I20250112 15:06:01.274113 18970 sys_catalog.cc:458] T 00000000000000000000000000000000 P cd21cea7d1214dd2ab14d1e6c471e6ea [sys.catalog]: This master's current role is: FOLLOWER
I20250112 15:06:01.275161 17651 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 2
W20250112 15:06:01.275257 18981 catalog_manager.cc:1559] T 00000000000000000000000000000000 P cd21cea7d1214dd2ab14d1e6c471e6ea: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20250112 15:06:01.275310 18981 catalog_manager.cc:874] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20250112 15:06:01.275411 17651 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250112 15:06:01.277151 18983 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250112 15:06:01.277722 18985 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250112 15:06:01.278195 18982 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250112 15:06:01.278888 17651 server_base.cc:1034] running on GCE node
I20250112 15:06:01.279012 17651 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250112 15:06:01.279037 17651 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250112 15:06:01.279050 17651 hybrid_clock.cc:648] HybridClock initialized: now 1736694361279050 us; error 0 us; skew 500 ppm
I20250112 15:06:01.279752 17651 webserver.cc:458] Webserver started at http://127.17.60.193:41723/ using document root <none> and password file <none>
I20250112 15:06:01.279891 17651 fs_manager.cc:362] Metadata directory not provided
I20250112 15:06:01.280040 17651 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250112 15:06:01.280125 17651 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250112 15:06:01.280404 17651 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task95dLSD/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1736694351070047-17651-0/minicluster-data/ts-0-root/instance:
uuid: "21292fc4406740b1a866cf178a2afa1e"
format_stamp: "Formatted at 2025-01-12 15:06:01 on dist-test-slave-b4nz"
I20250112 15:06:01.281296 17651 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20250112 15:06:01.281704 18990 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250112 15:06:01.281841 17651 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20250112 15:06:01.281885 17651 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task95dLSD/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1736694351070047-17651-0/minicluster-data/ts-0-root
uuid: "21292fc4406740b1a866cf178a2afa1e"
format_stamp: "Formatted at 2025-01-12 15:06:01 on dist-test-slave-b4nz"
I20250112 15:06:01.281929 17651 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task95dLSD/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1736694351070047-17651-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-task95dLSD/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1736694351070047-17651-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-task95dLSD/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1736694351070047-17651-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250112 15:06:01.294653 17651 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250112 15:06:01.294925 17651 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250112 15:06:01.295102 17651 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250112 15:06:01.295295 17651 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250112 15:06:01.295322 17651 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250112 15:06:01.295353 17651 ts_tablet_manager.cc:610] Registered 0 tablets
I20250112 15:06:01.295378 17651 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250112 15:06:01.298487 17651 rpc_server.cc:307] RPC server started. Bound to: 127.17.60.193:35621
I20250112 15:06:01.298528 19053 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.17.60.193:35621 every 8 connection(s)
I20250112 15:06:01.298992 17651 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250112 15:06:01.300886 19064 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250112 15:06:01.301441 19061 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250112 15:06:01.302203 17651 server_base.cc:1034] running on GCE node
I20250112 15:06:01.302534 19054 heartbeater.cc:346] Connected to a master server at 127.17.60.252:43895
I20250112 15:06:01.302601 19054 heartbeater.cc:463] Registering TS with master...
I20250112 15:06:01.302711 19054 heartbeater.cc:510] Master 127.17.60.252:43895 requested a full tablet report, sending...
W20250112 15:06:01.302754 19062 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250112 15:06:01.302966 17651 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250112 15:06:01.302999 17651 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250112 15:06:01.303004 18903 ts_manager.cc:194] Registered new tserver with Master: 21292fc4406740b1a866cf178a2afa1e (127.17.60.193:35621)
I20250112 15:06:01.303031 17651 hybrid_clock.cc:648] HybridClock initialized: now 1736694361303031 us; error 0 us; skew 500 ppm
I20250112 15:06:01.303285 19056 heartbeater.cc:346] Connected to a master server at 127.17.60.253:46387
I20250112 15:06:01.303330 19056 heartbeater.cc:463] Registering TS with master...
I20250112 15:06:01.303417 19056 heartbeater.cc:510] Master 127.17.60.253:46387 requested a full tablet report, sending...
I20250112 15:06:01.303656 18837 ts_manager.cc:194] Registered new tserver with Master: 21292fc4406740b1a866cf178a2afa1e (127.17.60.193:35621)
I20250112 15:06:01.303656 19055 heartbeater.cc:346] Connected to a master server at 127.17.60.254:38227
I20250112 15:06:01.303728 19055 heartbeater.cc:463] Registering TS with master...
I20250112 15:06:01.303792 17651 webserver.cc:458] Webserver started at http://127.17.60.194:37361/ using document root <none> and password file <none>
I20250112 15:06:01.303828 19055 heartbeater.cc:510] Master 127.17.60.254:38227 requested a full tablet report, sending...
I20250112 15:06:01.303889 17651 fs_manager.cc:362] Metadata directory not provided
I20250112 15:06:01.303943 17651 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250112 15:06:01.303997 17651 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250112 15:06:01.304075 18771 ts_manager.cc:194] Registered new tserver with Master: 21292fc4406740b1a866cf178a2afa1e (127.17.60.193:35621)
I20250112 15:06:01.304302 17651 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task95dLSD/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1736694351070047-17651-0/minicluster-data/ts-1-root/instance:
uuid: "7064d93ebf764af4b1035d9667296dc8"
format_stamp: "Formatted at 2025-01-12 15:06:01 on dist-test-slave-b4nz"
I20250112 15:06:01.305207 17651 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20250112 15:06:01.305675 19069 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250112 15:06:01.305827 17651 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20250112 15:06:01.305876 17651 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task95dLSD/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1736694351070047-17651-0/minicluster-data/ts-1-root
uuid: "7064d93ebf764af4b1035d9667296dc8"
format_stamp: "Formatted at 2025-01-12 15:06:01 on dist-test-slave-b4nz"
I20250112 15:06:01.305938 17651 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task95dLSD/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1736694351070047-17651-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-task95dLSD/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1736694351070047-17651-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-task95dLSD/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1736694351070047-17651-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250112 15:06:01.349881 17651 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250112 15:06:01.350152 17651 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250112 15:06:01.350317 17651 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250112 15:06:01.350507 17651 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250112 15:06:01.350533 17651 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250112 15:06:01.350553 17651 ts_tablet_manager.cc:610] Registered 0 tablets
I20250112 15:06:01.350579 17651 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250112 15:06:01.353678 17651 rpc_server.cc:307] RPC server started. Bound to: 127.17.60.194:37421
I20250112 15:06:01.353755 19132 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.17.60.194:37421 every 8 connection(s)
I20250112 15:06:01.354195 17651 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250112 15:06:01.355764 19140 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250112 15:06:01.356832 19141 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250112 15:06:01.357254 19143 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250112 15:06:01.357503 17651 server_base.cc:1034] running on GCE node
I20250112 15:06:01.357652 17651 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250112 15:06:01.357709 17651 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250112 15:06:01.357731 19135 heartbeater.cc:346] Connected to a master server at 127.17.60.253:46387
I20250112 15:06:01.357753 19133 heartbeater.cc:346] Connected to a master server at 127.17.60.252:43895
I20250112 15:06:01.357808 19135 heartbeater.cc:463] Registering TS with master...
I20250112 15:06:01.357869 19133 heartbeater.cc:463] Registering TS with master...
I20250112 15:06:01.357967 19135 heartbeater.cc:510] Master 127.17.60.253:46387 requested a full tablet report, sending...
I20250112 15:06:01.357967 19133 heartbeater.cc:510] Master 127.17.60.252:43895 requested a full tablet report, sending...
I20250112 15:06:01.357744 17651 hybrid_clock.cc:648] HybridClock initialized: now 1736694361357744 us; error 0 us; skew 500 ppm
I20250112 15:06:01.358121 19134 heartbeater.cc:346] Connected to a master server at 127.17.60.254:38227
I20250112 15:06:01.358165 19134 heartbeater.cc:463] Registering TS with master...
I20250112 15:06:01.358249 19134 heartbeater.cc:510] Master 127.17.60.254:38227 requested a full tablet report, sending...
I20250112 15:06:01.358276 18837 ts_manager.cc:194] Registered new tserver with Master: 7064d93ebf764af4b1035d9667296dc8 (127.17.60.194:37421)
I20250112 15:06:01.358428 18903 ts_manager.cc:194] Registered new tserver with Master: 7064d93ebf764af4b1035d9667296dc8 (127.17.60.194:37421)
I20250112 15:06:01.358682 18771 ts_manager.cc:194] Registered new tserver with Master: 7064d93ebf764af4b1035d9667296dc8 (127.17.60.194:37421)
I20250112 15:06:01.358835 17651 webserver.cc:458] Webserver started at http://127.17.60.195:43123/ using document root <none> and password file <none>
I20250112 15:06:01.358922 17651 fs_manager.cc:362] Metadata directory not provided
I20250112 15:06:01.358968 17651 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250112 15:06:01.359014 17651 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250112 15:06:01.359274 17651 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task95dLSD/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1736694351070047-17651-0/minicluster-data/ts-2-root/instance:
uuid: "dd83f0dba7dd403887d9546c0aa261d6"
format_stamp: "Formatted at 2025-01-12 15:06:01 on dist-test-slave-b4nz"
I20250112 15:06:01.360065 17651 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20250112 15:06:01.360476 19148 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250112 15:06:01.360615 17651 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20250112 15:06:01.360657 17651 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task95dLSD/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1736694351070047-17651-0/minicluster-data/ts-2-root
uuid: "dd83f0dba7dd403887d9546c0aa261d6"
format_stamp: "Formatted at 2025-01-12 15:06:01 on dist-test-slave-b4nz"
I20250112 15:06:01.360702 17651 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task95dLSD/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1736694351070047-17651-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-task95dLSD/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1736694351070047-17651-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-task95dLSD/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1736694351070047-17651-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250112 15:06:01.385864 17651 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250112 15:06:01.386138 17651 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250112 15:06:01.386317 17651 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250112 15:06:01.386561 17651 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250112 15:06:01.386592 17651 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250112 15:06:01.386628 17651 ts_tablet_manager.cc:610] Registered 0 tablets
I20250112 15:06:01.386658 17651 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250112 15:06:01.389925 17651 rpc_server.cc:307] RPC server started. Bound to: 127.17.60.195:39489
I20250112 15:06:01.389983 19211 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.17.60.195:39489 every 8 connection(s)
I20250112 15:06:01.393612 19214 heartbeater.cc:346] Connected to a master server at 127.17.60.253:46387
I20250112 15:06:01.393682 19214 heartbeater.cc:463] Registering TS with master...
I20250112 15:06:01.393791 19214 heartbeater.cc:510] Master 127.17.60.253:46387 requested a full tablet report, sending...
I20250112 15:06:01.393812 19213 heartbeater.cc:346] Connected to a master server at 127.17.60.254:38227
I20250112 15:06:01.393826 19212 heartbeater.cc:346] Connected to a master server at 127.17.60.252:43895
I20250112 15:06:01.393884 19213 heartbeater.cc:463] Registering TS with master...
I20250112 15:06:01.393886 19212 heartbeater.cc:463] Registering TS with master...
I20250112 15:06:01.394016 19213 heartbeater.cc:510] Master 127.17.60.254:38227 requested a full tablet report, sending...
I20250112 15:06:01.394088 19212 heartbeater.cc:510] Master 127.17.60.252:43895 requested a full tablet report, sending...
I20250112 15:06:01.394127 18837 ts_manager.cc:194] Registered new tserver with Master: dd83f0dba7dd403887d9546c0aa261d6 (127.17.60.195:39489)
I20250112 15:06:01.394273 18771 ts_manager.cc:194] Registered new tserver with Master: dd83f0dba7dd403887d9546c0aa261d6 (127.17.60.195:39489)
I20250112 15:06:01.394433 18903 ts_manager.cc:194] Registered new tserver with Master: dd83f0dba7dd403887d9546c0aa261d6 (127.17.60.195:39489)
I20250112 15:06:01.394634 17651 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.004387991s
I20250112 15:06:01.423658 18944 raft_consensus.cc:491] T 00000000000000000000000000000000 P e5fa9b0933e446c9854df3006fce277f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 15:06:01.423799 18944 raft_consensus.cc:513] T 00000000000000000000000000000000 P e5fa9b0933e446c9854df3006fce277f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a9e1db91a694b78b174fa300b67fb97" member_type: VOTER last_known_addr { host: "127.17.60.254" port: 38227 } } peers { permanent_uuid: "cd21cea7d1214dd2ab14d1e6c471e6ea" member_type: VOTER last_known_addr { host: "127.17.60.253" port: 46387 } } peers { permanent_uuid: "e5fa9b0933e446c9854df3006fce277f" member_type: VOTER last_known_addr { host: "127.17.60.252" port: 43895 } }
I20250112 15:06:01.424008 18944 leader_election.cc:290] T 00000000000000000000000000000000 P e5fa9b0933e446c9854df3006fce277f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4a9e1db91a694b78b174fa300b67fb97 (127.17.60.254:38227), cd21cea7d1214dd2ab14d1e6c471e6ea (127.17.60.253:46387)
I20250112 15:06:01.424254 18781 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "e5fa9b0933e446c9854df3006fce277f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4a9e1db91a694b78b174fa300b67fb97" is_pre_election: true
I20250112 15:06:01.424280 18847 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "e5fa9b0933e446c9854df3006fce277f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cd21cea7d1214dd2ab14d1e6c471e6ea" is_pre_election: true
I20250112 15:06:01.424355 18781 raft_consensus.cc:2463] T 00000000000000000000000000000000 P 4a9e1db91a694b78b174fa300b67fb97 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e5fa9b0933e446c9854df3006fce277f in term 0.
I20250112 15:06:01.424374 18847 raft_consensus.cc:2463] T 00000000000000000000000000000000 P cd21cea7d1214dd2ab14d1e6c471e6ea [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e5fa9b0933e446c9854df3006fce277f in term 0.
I20250112 15:06:01.424525 18887 leader_election.cc:304] T 00000000000000000000000000000000 P e5fa9b0933e446c9854df3006fce277f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a9e1db91a694b78b174fa300b67fb97, e5fa9b0933e446c9854df3006fce277f; no voters:
I20250112 15:06:01.424636 18944 raft_consensus.cc:2798] T 00000000000000000000000000000000 P e5fa9b0933e446c9854df3006fce277f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250112 15:06:01.424678 18944 raft_consensus.cc:491] T 00000000000000000000000000000000 P e5fa9b0933e446c9854df3006fce277f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250112 15:06:01.424698 18944 raft_consensus.cc:3054] T 00000000000000000000000000000000 P e5fa9b0933e446c9854df3006fce277f [term 0 FOLLOWER]: Advancing to term 1
I20250112 15:06:01.425313 18944 raft_consensus.cc:513] T 00000000000000000000000000000000 P e5fa9b0933e446c9854df3006fce277f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a9e1db91a694b78b174fa300b67fb97" member_type: VOTER last_known_addr { host: "127.17.60.254" port: 38227 } } peers { permanent_uuid: "cd21cea7d1214dd2ab14d1e6c471e6ea" member_type: VOTER last_known_addr { host: "127.17.60.253" port: 46387 } } peers { permanent_uuid: "e5fa9b0933e446c9854df3006fce277f" member_type: VOTER last_known_addr { host: "127.17.60.252" port: 43895 } }
I20250112 15:06:01.425415 18944 leader_election.cc:290] T 00000000000000000000000000000000 P e5fa9b0933e446c9854df3006fce277f [CANDIDATE]: Term 1 election: Requested vote from peers 4a9e1db91a694b78b174fa300b67fb97 (127.17.60.254:38227), cd21cea7d1214dd2ab14d1e6c471e6ea (127.17.60.253:46387)
I20250112 15:06:01.425588 18781 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "e5fa9b0933e446c9854df3006fce277f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4a9e1db91a694b78b174fa300b67fb97"
I20250112 15:06:01.425619 18847 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "e5fa9b0933e446c9854df3006fce277f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cd21cea7d1214dd2ab14d1e6c471e6ea"
I20250112 15:06:01.425654 18781 raft_consensus.cc:3054] T 00000000000000000000000000000000 P 4a9e1db91a694b78b174fa300b67fb97 [term 0 FOLLOWER]: Advancing to term 1
I20250112 15:06:01.425671 18847 raft_consensus.cc:3054] T 00000000000000000000000000000000 P cd21cea7d1214dd2ab14d1e6c471e6ea [term 0 FOLLOWER]: Advancing to term 1
I20250112 15:06:01.426304 18847 raft_consensus.cc:2463] T 00000000000000000000000000000000 P cd21cea7d1214dd2ab14d1e6c471e6ea [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e5fa9b0933e446c9854df3006fce277f in term 1.
I20250112 15:06:01.426357 18781 raft_consensus.cc:2463] T 00000000000000000000000000000000 P 4a9e1db91a694b78b174fa300b67fb97 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e5fa9b0933e446c9854df3006fce277f in term 1.
I20250112 15:06:01.426532 18887 leader_election.cc:304] T 00000000000000000000000000000000 P e5fa9b0933e446c9854df3006fce277f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a9e1db91a694b78b174fa300b67fb97, e5fa9b0933e446c9854df3006fce277f; no voters:
I20250112 15:06:01.426642 18944 raft_consensus.cc:2798] T 00000000000000000000000000000000 P e5fa9b0933e446c9854df3006fce277f [term 1 FOLLOWER]: Leader election won for term 1
I20250112 15:06:01.426789 18944 raft_consensus.cc:695] T 00000000000000000000000000000000 P e5fa9b0933e446c9854df3006fce277f [term 1 LEADER]: Becoming Leader. State: Replica: e5fa9b0933e446c9854df3006fce277f, State: Running, Role: LEADER
I20250112 15:06:01.426838 18944 consensus_queue.cc:237] T 00000000000000000000000000000000 P e5fa9b0933e446c9854df3006fce277f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a9e1db91a694b78b174fa300b67fb97" member_type: VOTER last_known_addr { host: "127.17.60.254" port: 38227 } } peers { permanent_uuid: "cd21cea7d1214dd2ab14d1e6c471e6ea" member_type: VOTER last_known_addr { host: "127.17.60.253" port: 46387 } } peers { permanent_uuid: "e5fa9b0933e446c9854df3006fce277f" member_type: VOTER last_known_addr { host: "127.17.60.252" port: 43895 } }
I20250112 15:06:01.427114 19220 sys_catalog.cc:455] T 00000000000000000000000000000000 P e5fa9b0933e446c9854df3006fce277f [sys.catalog]: SysCatalogTable state changed. Reason: New leader e5fa9b0933e446c9854df3006fce277f. Latest consensus state: current_term: 1 leader_uuid: "e5fa9b0933e446c9854df3006fce277f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a9e1db91a694b78b174fa300b67fb97" member_type: VOTER last_known_addr { host: "127.17.60.254" port: 38227 } } peers { permanent_uuid: "cd21cea7d1214dd2ab14d1e6c471e6ea" member_type: VOTER last_known_addr { host: "127.17.60.253" port: 46387 } } peers { permanent_uuid: "e5fa9b0933e446c9854df3006fce277f" member_type: VOTER last_known_addr { host: "127.17.60.252" port: 43895 } } }
I20250112 15:06:01.427220 19220 sys_catalog.cc:458] T 00000000000000000000000000000000 P e5fa9b0933e446c9854df3006fce277f [sys.catalog]: This master's current role is: LEADER
I20250112 15:06:01.427392 19222 catalog_manager.cc:1476] Loading table and tablet metadata into memory...
I20250112 15:06:01.427529 19222 catalog_manager.cc:1485] Initializing Kudu cluster ID...
I20250112 15:06:01.428046 18847 raft_consensus.cc:1270] T 00000000000000000000000000000000 P cd21cea7d1214dd2ab14d1e6c471e6ea [term 1 FOLLOWER]: Refusing update from remote peer e5fa9b0933e446c9854df3006fce277f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250112 15:06:01.428067 18781 raft_consensus.cc:1270] T 00000000000000000000000000000000 P 4a9e1db91a694b78b174fa300b67fb97 [term 1 FOLLOWER]: Refusing update from remote peer e5fa9b0933e446c9854df3006fce277f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250112 15:06:01.428223 19220 consensus_queue.cc:1035] T 00000000000000000000000000000000 P e5fa9b0933e446c9854df3006fce277f [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a9e1db91a694b78b174fa300b67fb97" member_type: VOTER last_known_addr { host: "127.17.60.254" port: 38227 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250112 15:06:01.428294 18944 consensus_queue.cc:1035] T 00000000000000000000000000000000 P e5fa9b0933e446c9854df3006fce277f [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd21cea7d1214dd2ab14d1e6c471e6ea" member_type: VOTER last_known_addr { host: "127.17.60.253" port: 46387 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250112 15:06:01.428795 18970 sys_catalog.cc:455] T 00000000000000000000000000000000 P cd21cea7d1214dd2ab14d1e6c471e6ea [sys.catalog]: SysCatalogTable state changed. Reason: New leader e5fa9b0933e446c9854df3006fce277f. Latest consensus state: current_term: 1 leader_uuid: "e5fa9b0933e446c9854df3006fce277f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a9e1db91a694b78b174fa300b67fb97" member_type: VOTER last_known_addr { host: "127.17.60.254" port: 38227 } } peers { permanent_uuid: "cd21cea7d1214dd2ab14d1e6c471e6ea" member_type: VOTER last_known_addr { host: "127.17.60.253" port: 46387 } } peers { permanent_uuid: "e5fa9b0933e446c9854df3006fce277f" member_type: VOTER last_known_addr { host: "127.17.60.252" port: 43895 } } }
I20250112 15:06:01.428797 18957 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4a9e1db91a694b78b174fa300b67fb97 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e5fa9b0933e446c9854df3006fce277f. Latest consensus state: current_term: 1 leader_uuid: "e5fa9b0933e446c9854df3006fce277f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a9e1db91a694b78b174fa300b67fb97" member_type: VOTER last_known_addr { host: "127.17.60.254" port: 38227 } } peers { permanent_uuid: "cd21cea7d1214dd2ab14d1e6c471e6ea" member_type: VOTER last_known_addr { host: "127.17.60.253" port: 46387 } } peers { permanent_uuid: "e5fa9b0933e446c9854df3006fce277f" member_type: VOTER last_known_addr { host: "127.17.60.252" port: 43895 } } }
I20250112 15:06:01.428929 18970 sys_catalog.cc:458] T 00000000000000000000000000000000 P cd21cea7d1214dd2ab14d1e6c471e6ea [sys.catalog]: This master's current role is: FOLLOWER
I20250112 15:06:01.428936 18957 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4a9e1db91a694b78b174fa300b67fb97 [sys.catalog]: This master's current role is: FOLLOWER
I20250112 15:06:01.429313 19220 sys_catalog.cc:455] T 00000000000000000000000000000000 P e5fa9b0933e446c9854df3006fce277f [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "e5fa9b0933e446c9854df3006fce277f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a9e1db91a694b78b174fa300b67fb97" member_type: VOTER last_known_addr { host: "127.17.60.254" port: 38227 } } peers { permanent_uuid: "cd21cea7d1214dd2ab14d1e6c471e6ea" member_type: VOTER last_known_addr { host: "127.17.60.253" port: 46387 } } peers { permanent_uuid: "e5fa9b0933e446c9854df3006fce277f" member_type: VOTER last_known_addr { host: "127.17.60.252" port: 43895 } } }
I20250112 15:06:01.429366 19220 sys_catalog.cc:458] T 00000000000000000000000000000000 P e5fa9b0933e446c9854df3006fce277f [sys.catalog]: This master's current role is: LEADER
I20250112 15:06:01.429461 19222 catalog_manager.cc:1348] Generated new cluster ID: 7370a9ebaf3c4c2f9db9b7afecc9de6b
I20250112 15:06:01.429498 19222 catalog_manager.cc:1496] Initializing Kudu internal certificate authority...
I20250112 15:06:01.429534 19220 sys_catalog.cc:455] T 00000000000000000000000000000000 P e5fa9b0933e446c9854df3006fce277f [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "e5fa9b0933e446c9854df3006fce277f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a9e1db91a694b78b174fa300b67fb97" member_type: VOTER last_known_addr { host: "127.17.60.254" port: 38227 } } peers { permanent_uuid: "cd21cea7d1214dd2ab14d1e6c471e6ea" member_type: VOTER last_known_addr { host: "127.17.60.253" port: 46387 } } peers { permanent_uuid: "e5fa9b0933e446c9854df3006fce277f" member_type: VOTER last_known_addr { host: "127.17.60.252" port: 43895 } } }
I20250112 15:06:01.429590 19220 sys_catalog.cc:458] T 00000000000000000000000000000000 P e5fa9b0933e446c9854df3006fce277f [sys.catalog]: This master's current role is: LEADER
I20250112 15:06:01.429684 18957 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4a9e1db91a694b78b174fa300b67fb97 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "e5fa9b0933e446c9854df3006fce277f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a9e1db91a694b78b174fa300b67fb97" member_type: VOTER last_known_addr { host: "127.17.60.254" port: 38227 } } peers { permanent_uuid: "cd21cea7d1214dd2ab14d1e6c471e6ea" member_type: VOTER last_known_addr { host: "127.17.60.253" port: 46387 } } peers { permanent_uuid: "e5fa9b0933e446c9854df3006fce277f" member_type: VOTER last_known_addr { host: "127.17.60.252" port: 43895 } } }
I20250112 15:06:01.429752 18957 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4a9e1db91a694b78b174fa300b67fb97 [sys.catalog]: This master's current role is: FOLLOWER
I20250112 15:06:01.429911 18970 sys_catalog.cc:455] T 00000000000000000000000000000000 P cd21cea7d1214dd2ab14d1e6c471e6ea [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "e5fa9b0933e446c9854df3006fce277f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a9e1db91a694b78b174fa300b67fb97" member_type: VOTER last_known_addr { host: "127.17.60.254" port: 38227 } } peers { permanent_uuid: "cd21cea7d1214dd2ab14d1e6c471e6ea" member_type: VOTER last_known_addr { host: "127.17.60.253" port: 46387 } } peers { permanent_uuid: "e5fa9b0933e446c9854df3006fce277f" member_type: VOTER last_known_addr { host: "127.17.60.252" port: 43895 } } }
I20250112 15:06:01.429975 18970 sys_catalog.cc:458] T 00000000000000000000000000000000 P cd21cea7d1214dd2ab14d1e6c471e6ea [sys.catalog]: This master's current role is: FOLLOWER
I20250112 15:06:01.445647 19222 catalog_manager.cc:1371] Generated new certificate authority record
I20250112 15:06:01.446213 19222 catalog_manager.cc:1505] Loading token signing keys...
I20250112 15:06:01.450573 19222 catalog_manager.cc:5899] T 00000000000000000000000000000000 P e5fa9b0933e446c9854df3006fce277f: Generated new TSK 0
I20250112 15:06:01.450655 19222 catalog_manager.cc:1515] Initializing in-progress tserver states...
I20250112 15:06:02.235505 18968 catalog_manager.cc:1260] Loaded cluster ID: 7370a9ebaf3c4c2f9db9b7afecc9de6b
I20250112 15:06:02.235586 18968 catalog_manager.cc:1553] T 00000000000000000000000000000000 P 4a9e1db91a694b78b174fa300b67fb97: loading cluster ID for follower catalog manager: success
I20250112 15:06:02.236258 18968 catalog_manager.cc:1575] T 00000000000000000000000000000000 P 4a9e1db91a694b78b174fa300b67fb97: acquiring CA information for follower catalog manager: success
I20250112 15:06:02.236410 18968 catalog_manager.cc:1603] T 00000000000000000000000000000000 P 4a9e1db91a694b78b174fa300b67fb97: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20250112 15:06:02.275532 18981 catalog_manager.cc:1260] Loaded cluster ID: 7370a9ebaf3c4c2f9db9b7afecc9de6b
I20250112 15:06:02.275610 18981 catalog_manager.cc:1553] T 00000000000000000000000000000000 P cd21cea7d1214dd2ab14d1e6c471e6ea: loading cluster ID for follower catalog manager: success
I20250112 15:06:02.276335 18981 catalog_manager.cc:1575] T 00000000000000000000000000000000 P cd21cea7d1214dd2ab14d1e6c471e6ea: acquiring CA information for follower catalog manager: success
I20250112 15:06:02.276481 18981 catalog_manager.cc:1603] T 00000000000000000000000000000000 P cd21cea7d1214dd2ab14d1e6c471e6ea: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20250112 15:06:02.304518 18903 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:42992
I20250112 15:06:02.359474 18903 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:43006
I20250112 15:06:02.395368 18903 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:43014
I20250112 15:06:03.223964 18954 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20250112 15:06:03.305084 19054 heartbeater.cc:502] Master 127.17.60.252:43895 was elected leader, sending a full tablet report...
I20250112 15:06:03.359999 19133 heartbeater.cc:502] Master 127.17.60.252:43895 was elected leader, sending a full tablet report...
I20250112 15:06:03.395867 19212 heartbeater.cc:502] Master 127.17.60.252:43895 was elected leader, sending a full tablet report...
I20250112 15:06:03.546123 17651 test_util.cc:274] Using random seed: -1432395256
I20250112 15:06:03.552343 18903 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:43026:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 1
split_rows_range_bounds {
rows: "\004\001\000\252\252\252*\004\001\000TUUU""\004\001\000\252\252\252*\004\001\000TUUU"
indirect_data: """"
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250112 15:06:03.558859 19018 tablet_service.cc:1467] Processing CreateTablet for tablet 2c23ff24709f45fc916a27933aae8556 (DEFAULT_TABLE table=test-workload [id=306cc9c07f484249acc9bc5895b8fe2f]), partition=RANGE (key) PARTITION VALUES < 715827882
I20250112 15:06:03.559008 19018 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2c23ff24709f45fc916a27933aae8556. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 15:06:03.559240 19176 tablet_service.cc:1467] Processing CreateTablet for tablet b1f7a675e15a4c649aacc9dcd25ab71a (DEFAULT_TABLE table=test-workload [id=306cc9c07f484249acc9bc5895b8fe2f]), partition=RANGE (key) PARTITION 1431655764 <= VALUES
I20250112 15:06:03.559334 19176 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b1f7a675e15a4c649aacc9dcd25ab71a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 15:06:03.559386 19097 tablet_service.cc:1467] Processing CreateTablet for tablet 3a85221652b848f59fb223080a4300a4 (DEFAULT_TABLE table=test-workload [id=306cc9c07f484249acc9bc5895b8fe2f]), partition=RANGE (key) PARTITION 715827882 <= VALUES < 1431655764
I20250112 15:06:03.559509 19097 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3a85221652b848f59fb223080a4300a4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 15:06:03.560528 19261 tablet_bootstrap.cc:492] T 2c23ff24709f45fc916a27933aae8556 P 21292fc4406740b1a866cf178a2afa1e: Bootstrap starting.
I20250112 15:06:03.560837 19262 tablet_bootstrap.cc:492] T b1f7a675e15a4c649aacc9dcd25ab71a P dd83f0dba7dd403887d9546c0aa261d6: Bootstrap starting.
I20250112 15:06:03.560957 19263 tablet_bootstrap.cc:492] T 3a85221652b848f59fb223080a4300a4 P 7064d93ebf764af4b1035d9667296dc8: Bootstrap starting.
I20250112 15:06:03.561033 19261 tablet_bootstrap.cc:654] T 2c23ff24709f45fc916a27933aae8556 P 21292fc4406740b1a866cf178a2afa1e: Neither blocks nor log segments found. Creating new log.
I20250112 15:06:03.561252 19262 tablet_bootstrap.cc:654] T b1f7a675e15a4c649aacc9dcd25ab71a P dd83f0dba7dd403887d9546c0aa261d6: Neither blocks nor log segments found. Creating new log.
I20250112 15:06:03.561434 19263 tablet_bootstrap.cc:654] T 3a85221652b848f59fb223080a4300a4 P 7064d93ebf764af4b1035d9667296dc8: Neither blocks nor log segments found. Creating new log.
I20250112 15:06:03.561748 19261 tablet_bootstrap.cc:492] T 2c23ff24709f45fc916a27933aae8556 P 21292fc4406740b1a866cf178a2afa1e: No bootstrap required, opened a new log
I20250112 15:06:03.561784 19262 tablet_bootstrap.cc:492] T b1f7a675e15a4c649aacc9dcd25ab71a P dd83f0dba7dd403887d9546c0aa261d6: No bootstrap required, opened a new log
I20250112 15:06:03.561909 19261 ts_tablet_manager.cc:1397] T 2c23ff24709f45fc916a27933aae8556 P 21292fc4406740b1a866cf178a2afa1e: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20250112 15:06:03.561956 19262 ts_tablet_manager.cc:1397] T b1f7a675e15a4c649aacc9dcd25ab71a P dd83f0dba7dd403887d9546c0aa261d6: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20250112 15:06:03.562049 19263 tablet_bootstrap.cc:492] T 3a85221652b848f59fb223080a4300a4 P 7064d93ebf764af4b1035d9667296dc8: No bootstrap required, opened a new log
I20250112 15:06:03.562094 19263 ts_tablet_manager.cc:1397] T 3a85221652b848f59fb223080a4300a4 P 7064d93ebf764af4b1035d9667296dc8: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20250112 15:06:03.562120 19261 raft_consensus.cc:357] T 2c23ff24709f45fc916a27933aae8556 P 21292fc4406740b1a866cf178a2afa1e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "21292fc4406740b1a866cf178a2afa1e" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 35621 } }
I20250112 15:06:03.562155 19262 raft_consensus.cc:357] T b1f7a675e15a4c649aacc9dcd25ab71a P dd83f0dba7dd403887d9546c0aa261d6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dd83f0dba7dd403887d9546c0aa261d6" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 39489 } }
I20250112 15:06:03.562220 19263 raft_consensus.cc:357] T 3a85221652b848f59fb223080a4300a4 P 7064d93ebf764af4b1035d9667296dc8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7064d93ebf764af4b1035d9667296dc8" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 37421 } }
I20250112 15:06:03.562251 19262 raft_consensus.cc:383] T b1f7a675e15a4c649aacc9dcd25ab71a P dd83f0dba7dd403887d9546c0aa261d6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 15:06:03.562192 19261 raft_consensus.cc:383] T 2c23ff24709f45fc916a27933aae8556 P 21292fc4406740b1a866cf178a2afa1e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 15:06:03.562399 19261 raft_consensus.cc:738] T 2c23ff24709f45fc916a27933aae8556 P 21292fc4406740b1a866cf178a2afa1e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 21292fc4406740b1a866cf178a2afa1e, State: Initialized, Role: FOLLOWER
I20250112 15:06:03.562363 19262 raft_consensus.cc:738] T b1f7a675e15a4c649aacc9dcd25ab71a P dd83f0dba7dd403887d9546c0aa261d6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dd83f0dba7dd403887d9546c0aa261d6, State: Initialized, Role: FOLLOWER
I20250112 15:06:03.562310 19263 raft_consensus.cc:383] T 3a85221652b848f59fb223080a4300a4 P 7064d93ebf764af4b1035d9667296dc8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 15:06:03.562474 19261 consensus_queue.cc:260] T 2c23ff24709f45fc916a27933aae8556 P 21292fc4406740b1a866cf178a2afa1e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "21292fc4406740b1a866cf178a2afa1e" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 35621 } }
I20250112 15:06:03.562505 19262 consensus_queue.cc:260] T b1f7a675e15a4c649aacc9dcd25ab71a P dd83f0dba7dd403887d9546c0aa261d6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dd83f0dba7dd403887d9546c0aa261d6" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 39489 } }
I20250112 15:06:03.562531 19263 raft_consensus.cc:738] T 3a85221652b848f59fb223080a4300a4 P 7064d93ebf764af4b1035d9667296dc8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7064d93ebf764af4b1035d9667296dc8, State: Initialized, Role: FOLLOWER
I20250112 15:06:03.562556 19261 raft_consensus.cc:397] T 2c23ff24709f45fc916a27933aae8556 P 21292fc4406740b1a866cf178a2afa1e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250112 15:06:03.562563 19262 raft_consensus.cc:397] T b1f7a675e15a4c649aacc9dcd25ab71a P dd83f0dba7dd403887d9546c0aa261d6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250112 15:06:03.562609 19263 consensus_queue.cc:260] T 3a85221652b848f59fb223080a4300a4 P 7064d93ebf764af4b1035d9667296dc8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7064d93ebf764af4b1035d9667296dc8" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 37421 } }
I20250112 15:06:03.562672 19262 raft_consensus.cc:491] T b1f7a675e15a4c649aacc9dcd25ab71a P dd83f0dba7dd403887d9546c0aa261d6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250112 15:06:03.562631 19261 raft_consensus.cc:491] T 2c23ff24709f45fc916a27933aae8556 P 21292fc4406740b1a866cf178a2afa1e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250112 15:06:03.562712 19261 raft_consensus.cc:3054] T 2c23ff24709f45fc916a27933aae8556 P 21292fc4406740b1a866cf178a2afa1e [term 0 FOLLOWER]: Advancing to term 1
I20250112 15:06:03.562697 19263 raft_consensus.cc:397] T 3a85221652b848f59fb223080a4300a4 P 7064d93ebf764af4b1035d9667296dc8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250112 15:06:03.562701 19262 raft_consensus.cc:3054] T b1f7a675e15a4c649aacc9dcd25ab71a P dd83f0dba7dd403887d9546c0aa261d6 [term 0 FOLLOWER]: Advancing to term 1
I20250112 15:06:03.562810 19263 raft_consensus.cc:491] T 3a85221652b848f59fb223080a4300a4 P 7064d93ebf764af4b1035d9667296dc8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250112 15:06:03.562889 19263 raft_consensus.cc:3054] T 3a85221652b848f59fb223080a4300a4 P 7064d93ebf764af4b1035d9667296dc8 [term 0 FOLLOWER]: Advancing to term 1
I20250112 15:06:03.563467 19261 raft_consensus.cc:513] T 2c23ff24709f45fc916a27933aae8556 P 21292fc4406740b1a866cf178a2afa1e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "21292fc4406740b1a866cf178a2afa1e" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 35621 } }
I20250112 15:06:03.563524 19262 raft_consensus.cc:513] T b1f7a675e15a4c649aacc9dcd25ab71a P dd83f0dba7dd403887d9546c0aa261d6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dd83f0dba7dd403887d9546c0aa261d6" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 39489 } }
I20250112 15:06:03.563547 19263 raft_consensus.cc:513] T 3a85221652b848f59fb223080a4300a4 P 7064d93ebf764af4b1035d9667296dc8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7064d93ebf764af4b1035d9667296dc8" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 37421 } }
I20250112 15:06:03.563547 19261 leader_election.cc:304] T 2c23ff24709f45fc916a27933aae8556 P 21292fc4406740b1a866cf178a2afa1e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 21292fc4406740b1a866cf178a2afa1e; no voters:
I20250112 15:06:03.563663 19263 leader_election.cc:304] T 3a85221652b848f59fb223080a4300a4 P 7064d93ebf764af4b1035d9667296dc8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7064d93ebf764af4b1035d9667296dc8; no voters:
I20250112 15:06:03.563776 19261 leader_election.cc:290] T 2c23ff24709f45fc916a27933aae8556 P 21292fc4406740b1a866cf178a2afa1e [CANDIDATE]: Term 1 election: Requested vote from peers
I20250112 15:06:03.563776 19263 leader_election.cc:290] T 3a85221652b848f59fb223080a4300a4 P 7064d93ebf764af4b1035d9667296dc8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20250112 15:06:03.563890 19267 raft_consensus.cc:2798] T 3a85221652b848f59fb223080a4300a4 P 7064d93ebf764af4b1035d9667296dc8 [term 1 FOLLOWER]: Leader election won for term 1
I20250112 15:06:03.563984 19261 ts_tablet_manager.cc:1428] T 2c23ff24709f45fc916a27933aae8556 P 21292fc4406740b1a866cf178a2afa1e: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20250112 15:06:03.563990 19263 ts_tablet_manager.cc:1428] T 3a85221652b848f59fb223080a4300a4 P 7064d93ebf764af4b1035d9667296dc8: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250112 15:06:03.563809 19268 raft_consensus.cc:2798] T 2c23ff24709f45fc916a27933aae8556 P 21292fc4406740b1a866cf178a2afa1e [term 1 FOLLOWER]: Leader election won for term 1
I20250112 15:06:03.564013 19267 raft_consensus.cc:695] T 3a85221652b848f59fb223080a4300a4 P 7064d93ebf764af4b1035d9667296dc8 [term 1 LEADER]: Becoming Leader. State: Replica: 7064d93ebf764af4b1035d9667296dc8, State: Running, Role: LEADER
I20250112 15:06:03.564283 19267 consensus_queue.cc:237] T 3a85221652b848f59fb223080a4300a4 P 7064d93ebf764af4b1035d9667296dc8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7064d93ebf764af4b1035d9667296dc8" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 37421 } }
I20250112 15:06:03.564157 19268 raft_consensus.cc:695] T 2c23ff24709f45fc916a27933aae8556 P 21292fc4406740b1a866cf178a2afa1e [term 1 LEADER]: Becoming Leader. State: Replica: 21292fc4406740b1a866cf178a2afa1e, State: Running, Role: LEADER
I20250112 15:06:03.563616 19262 leader_election.cc:304] T b1f7a675e15a4c649aacc9dcd25ab71a P dd83f0dba7dd403887d9546c0aa261d6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: dd83f0dba7dd403887d9546c0aa261d6; no voters:
I20250112 15:06:03.564581 19262 leader_election.cc:290] T b1f7a675e15a4c649aacc9dcd25ab71a P dd83f0dba7dd403887d9546c0aa261d6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20250112 15:06:03.564740 19262 ts_tablet_manager.cc:1428] T b1f7a675e15a4c649aacc9dcd25ab71a P dd83f0dba7dd403887d9546c0aa261d6: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20250112 15:06:03.564817 18903 catalog_manager.cc:5526] T 3a85221652b848f59fb223080a4300a4 P 7064d93ebf764af4b1035d9667296dc8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7064d93ebf764af4b1035d9667296dc8 (127.17.60.194). New cstate: current_term: 1 leader_uuid: "7064d93ebf764af4b1035d9667296dc8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7064d93ebf764af4b1035d9667296dc8" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 37421 } health_report { overall_health: HEALTHY } } }
I20250112 15:06:03.564848 19275 raft_consensus.cc:2798] T b1f7a675e15a4c649aacc9dcd25ab71a P dd83f0dba7dd403887d9546c0aa261d6 [term 1 FOLLOWER]: Leader election won for term 1
I20250112 15:06:03.565135 19275 raft_consensus.cc:695] T b1f7a675e15a4c649aacc9dcd25ab71a P dd83f0dba7dd403887d9546c0aa261d6 [term 1 LEADER]: Becoming Leader. State: Replica: dd83f0dba7dd403887d9546c0aa261d6, State: Running, Role: LEADER
I20250112 15:06:03.565248 19268 consensus_queue.cc:237] T 2c23ff24709f45fc916a27933aae8556 P 21292fc4406740b1a866cf178a2afa1e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "21292fc4406740b1a866cf178a2afa1e" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 35621 } }
I20250112 15:06:03.566466 18903 catalog_manager.cc:5526] T 2c23ff24709f45fc916a27933aae8556 P 21292fc4406740b1a866cf178a2afa1e reported cstate change: term changed from 0 to 1, leader changed from <none> to 21292fc4406740b1a866cf178a2afa1e (127.17.60.193). New cstate: current_term: 1 leader_uuid: "21292fc4406740b1a866cf178a2afa1e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "21292fc4406740b1a866cf178a2afa1e" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 35621 } health_report { overall_health: HEALTHY } } }
I20250112 15:06:03.565770 19275 consensus_queue.cc:237] T b1f7a675e15a4c649aacc9dcd25ab71a P dd83f0dba7dd403887d9546c0aa261d6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dd83f0dba7dd403887d9546c0aa261d6" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 39489 } }
I20250112 15:06:03.568468 18903 catalog_manager.cc:5526] T b1f7a675e15a4c649aacc9dcd25ab71a P dd83f0dba7dd403887d9546c0aa261d6 reported cstate change: term changed from 0 to 1, leader changed from <none> to dd83f0dba7dd403887d9546c0aa261d6 (127.17.60.195). New cstate: current_term: 1 leader_uuid: "dd83f0dba7dd403887d9546c0aa261d6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dd83f0dba7dd403887d9546c0aa261d6" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 39489 } health_report { overall_health: HEALTHY } } }
I20250112 15:06:03.571624 17651 master.cc:537] Master@127.17.60.252:43895 shutting down...
I20250112 15:06:03.573681 17651 raft_consensus.cc:2238] T 00000000000000000000000000000000 P e5fa9b0933e446c9854df3006fce277f [term 1 LEADER]: Raft consensus shutting down.
I20250112 15:06:03.573747 17651 raft_consensus.cc:2267] T 00000000000000000000000000000000 P e5fa9b0933e446c9854df3006fce277f [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 15:06:03.573768 17651 tablet_replica.cc:331] T 00000000000000000000000000000000 P e5fa9b0933e446c9854df3006fce277f: stopping tablet replica
I20250112 15:06:03.586380 17651 master.cc:559] Master@127.17.60.252:43895 shutdown complete.
W20250112 15:06:04.567516 19133 heartbeater.cc:643] Failed to heartbeat to 127.17.60.252:43895 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.17.60.252:43895: connect: Connection refused (error 111)
I20250112 15:06:05.069723 19285 raft_consensus.cc:491] T 00000000000000000000000000000000 P cd21cea7d1214dd2ab14d1e6c471e6ea [term 1 FOLLOWER]: Starting pre-election (detected failure of leader e5fa9b0933e446c9854df3006fce277f)
I20250112 15:06:05.069847 19285 raft_consensus.cc:513] T 00000000000000000000000000000000 P cd21cea7d1214dd2ab14d1e6c471e6ea [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a9e1db91a694b78b174fa300b67fb97" member_type: VOTER last_known_addr { host: "127.17.60.254" port: 38227 } } peers { permanent_uuid: "cd21cea7d1214dd2ab14d1e6c471e6ea" member_type: VOTER last_known_addr { host: "127.17.60.253" port: 46387 } } peers { permanent_uuid: "e5fa9b0933e446c9854df3006fce277f" member_type: VOTER last_known_addr { host: "127.17.60.252" port: 43895 } }
I20250112 15:06:05.069917 19286 raft_consensus.cc:491] T 00000000000000000000000000000000 P 4a9e1db91a694b78b174fa300b67fb97 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader e5fa9b0933e446c9854df3006fce277f)
I20250112 15:06:05.069995 19286 raft_consensus.cc:513] T 00000000000000000000000000000000 P 4a9e1db91a694b78b174fa300b67fb97 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a9e1db91a694b78b174fa300b67fb97" member_type: VOTER last_known_addr { host: "127.17.60.254" port: 38227 } } peers { permanent_uuid: "cd21cea7d1214dd2ab14d1e6c471e6ea" member_type: VOTER last_known_addr { host: "127.17.60.253" port: 46387 } } peers { permanent_uuid: "e5fa9b0933e446c9854df3006fce277f" member_type: VOTER last_known_addr { host: "127.17.60.252" port: 43895 } }
I20250112 15:06:05.070037 19285 leader_election.cc:290] T 00000000000000000000000000000000 P cd21cea7d1214dd2ab14d1e6c471e6ea [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4a9e1db91a694b78b174fa300b67fb97 (127.17.60.254:38227), e5fa9b0933e446c9854df3006fce277f (127.17.60.252:43895)
I20250112 15:06:05.070134 19286 leader_election.cc:290] T 00000000000000000000000000000000 P 4a9e1db91a694b78b174fa300b67fb97 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers cd21cea7d1214dd2ab14d1e6c471e6ea (127.17.60.253:46387), e5fa9b0933e446c9854df3006fce277f (127.17.60.252:43895)
I20250112 15:06:05.070309 18847 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4a9e1db91a694b78b174fa300b67fb97" candidate_term: 2 candidate_status { last_received { term: 1 index: 9 } } ignore_live_leader: false dest_uuid: "cd21cea7d1214dd2ab14d1e6c471e6ea" is_pre_election: true
I20250112 15:06:05.070397 18847 raft_consensus.cc:2463] T 00000000000000000000000000000000 P cd21cea7d1214dd2ab14d1e6c471e6ea [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a9e1db91a694b78b174fa300b67fb97 in term 1.
I20250112 15:06:05.070396 18781 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cd21cea7d1214dd2ab14d1e6c471e6ea" candidate_term: 2 candidate_status { last_received { term: 1 index: 9 } } ignore_live_leader: false dest_uuid: "4a9e1db91a694b78b174fa300b67fb97" is_pre_election: true
I20250112 15:06:05.070499 18781 raft_consensus.cc:2463] T 00000000000000000000000000000000 P 4a9e1db91a694b78b174fa300b67fb97 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cd21cea7d1214dd2ab14d1e6c471e6ea in term 1.
I20250112 15:06:05.070622 18756 leader_election.cc:304] T 00000000000000000000000000000000 P 4a9e1db91a694b78b174fa300b67fb97 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a9e1db91a694b78b174fa300b67fb97, cd21cea7d1214dd2ab14d1e6c471e6ea; no voters:
I20250112 15:06:05.070700 18822 leader_election.cc:304] T 00000000000000000000000000000000 P cd21cea7d1214dd2ab14d1e6c471e6ea [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a9e1db91a694b78b174fa300b67fb97, cd21cea7d1214dd2ab14d1e6c471e6ea; no voters:
I20250112 15:06:05.070716 19286 raft_consensus.cc:2798] T 00000000000000000000000000000000 P 4a9e1db91a694b78b174fa300b67fb97 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250112 15:06:05.070782 19286 raft_consensus.cc:491] T 00000000000000000000000000000000 P 4a9e1db91a694b78b174fa300b67fb97 [term 1 FOLLOWER]: Starting leader election (detected failure of leader e5fa9b0933e446c9854df3006fce277f)
I20250112 15:06:05.070823 19286 raft_consensus.cc:3054] T 00000000000000000000000000000000 P 4a9e1db91a694b78b174fa300b67fb97 [term 1 FOLLOWER]: Advancing to term 2
W20250112 15:06:05.070839 18822 leader_election.cc:336] T 00000000000000000000000000000000 P cd21cea7d1214dd2ab14d1e6c471e6ea [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer e5fa9b0933e446c9854df3006fce277f (127.17.60.252:43895): Network error: Client connection negotiation failed: client connection to 127.17.60.252:43895: connect: Connection refused (error 111)
W20250112 15:06:05.070866 18756 leader_election.cc:336] T 00000000000000000000000000000000 P 4a9e1db91a694b78b174fa300b67fb97 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer e5fa9b0933e446c9854df3006fce277f (127.17.60.252:43895): Network error: Client connection negotiation failed: client connection to 127.17.60.252:43895: connect: Connection refused (error 111)
I20250112 15:06:05.070786 19285 raft_consensus.cc:2798] T 00000000000000000000000000000000 P cd21cea7d1214dd2ab14d1e6c471e6ea [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250112 15:06:05.070986 19285 raft_consensus.cc:491] T 00000000000000000000000000000000 P cd21cea7d1214dd2ab14d1e6c471e6ea [term 1 FOLLOWER]: Starting leader election (detected failure of leader e5fa9b0933e446c9854df3006fce277f)
I20250112 15:06:05.071002 19285 raft_consensus.cc:3054] T 00000000000000000000000000000000 P cd21cea7d1214dd2ab14d1e6c471e6ea [term 1 FOLLOWER]: Advancing to term 2
I20250112 15:06:05.071527 19286 raft_consensus.cc:513] T 00000000000000000000000000000000 P 4a9e1db91a694b78b174fa300b67fb97 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a9e1db91a694b78b174fa300b67fb97" member_type: VOTER last_known_addr { host: "127.17.60.254" port: 38227 } } peers { permanent_uuid: "cd21cea7d1214dd2ab14d1e6c471e6ea" member_type: VOTER last_known_addr { host: "127.17.60.253" port: 46387 } } peers { permanent_uuid: "e5fa9b0933e446c9854df3006fce277f" member_type: VOTER last_known_addr { host: "127.17.60.252" port: 43895 } }
I20250112 15:06:05.071590 19285 raft_consensus.cc:513] T 00000000000000000000000000000000 P cd21cea7d1214dd2ab14d1e6c471e6ea [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a9e1db91a694b78b174fa300b67fb97" member_type: VOTER last_known_addr { host: "127.17.60.254" port: 38227 } } peers { permanent_uuid: "cd21cea7d1214dd2ab14d1e6c471e6ea" member_type: VOTER last_known_addr { host: "127.17.60.253" port: 46387 } } peers { permanent_uuid: "e5fa9b0933e446c9854df3006fce277f" member_type: VOTER last_known_addr { host: "127.17.60.252" port: 43895 } }
I20250112 15:06:05.071631 19286 leader_election.cc:290] T 00000000000000000000000000000000 P 4a9e1db91a694b78b174fa300b67fb97 [CANDIDATE]: Term 2 election: Requested vote from peers cd21cea7d1214dd2ab14d1e6c471e6ea (127.17.60.253:46387), e5fa9b0933e446c9854df3006fce277f (127.17.60.252:43895)
I20250112 15:06:05.071699 19285 leader_election.cc:290] T 00000000000000000000000000000000 P cd21cea7d1214dd2ab14d1e6c471e6ea [CANDIDATE]: Term 2 election: Requested vote from peers 4a9e1db91a694b78b174fa300b67fb97 (127.17.60.254:38227), e5fa9b0933e446c9854df3006fce277f (127.17.60.252:43895)
I20250112 15:06:05.071801 18847 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4a9e1db91a694b78b174fa300b67fb97" candidate_term: 2 candidate_status { last_received { term: 1 index: 9 } } ignore_live_leader: false dest_uuid: "cd21cea7d1214dd2ab14d1e6c471e6ea"
I20250112 15:06:05.071874 18781 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cd21cea7d1214dd2ab14d1e6c471e6ea" candidate_term: 2 candidate_status { last_received { term: 1 index: 9 } } ignore_live_leader: false dest_uuid: "4a9e1db91a694b78b174fa300b67fb97"
I20250112 15:06:05.071875 18847 raft_consensus.cc:2388] T 00000000000000000000000000000000 P cd21cea7d1214dd2ab14d1e6c471e6ea [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 4a9e1db91a694b78b174fa300b67fb97 in current term 2: Already voted for candidate cd21cea7d1214dd2ab14d1e6c471e6ea in this term.
I20250112 15:06:05.071966 18781 raft_consensus.cc:2388] T 00000000000000000000000000000000 P 4a9e1db91a694b78b174fa300b67fb97 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate cd21cea7d1214dd2ab14d1e6c471e6ea in current term 2: Already voted for candidate 4a9e1db91a694b78b174fa300b67fb97 in this term.
W20250112 15:06:05.072170 18822 leader_election.cc:336] T 00000000000000000000000000000000 P cd21cea7d1214dd2ab14d1e6c471e6ea [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer e5fa9b0933e446c9854df3006fce277f (127.17.60.252:43895): Network error: Client connection negotiation failed: client connection to 127.17.60.252:43895: connect: Connection refused (error 111)
I20250112 15:06:05.072212 18822 leader_election.cc:304] T 00000000000000000000000000000000 P cd21cea7d1214dd2ab14d1e6c471e6ea [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cd21cea7d1214dd2ab14d1e6c471e6ea; no voters: 4a9e1db91a694b78b174fa300b67fb97, e5fa9b0933e446c9854df3006fce277f
W20250112 15:06:05.072288 18756 leader_election.cc:336] T 00000000000000000000000000000000 P 4a9e1db91a694b78b174fa300b67fb97 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer e5fa9b0933e446c9854df3006fce277f (127.17.60.252:43895): Network error: Client connection negotiation failed: client connection to 127.17.60.252:43895: connect: Connection refused (error 111)
I20250112 15:06:05.072328 18756 leader_election.cc:304] T 00000000000000000000000000000000 P 4a9e1db91a694b78b174fa300b67fb97 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a9e1db91a694b78b174fa300b67fb97; no voters: cd21cea7d1214dd2ab14d1e6c471e6ea, e5fa9b0933e446c9854df3006fce277f
I20250112 15:06:05.072290 19285 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cd21cea7d1214dd2ab14d1e6c471e6ea [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20250112 15:06:05.072396 19286 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4a9e1db91a694b78b174fa300b67fb97 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20250112 15:06:06.613554 19297 raft_consensus.cc:491] T 00000000000000000000000000000000 P 4a9e1db91a694b78b174fa300b67fb97 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 15:06:06.613658 19297 raft_consensus.cc:513] T 00000000000000000000000000000000 P 4a9e1db91a694b78b174fa300b67fb97 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a9e1db91a694b78b174fa300b67fb97" member_type: VOTER last_known_addr { host: "127.17.60.254" port: 38227 } } peers { permanent_uuid: "cd21cea7d1214dd2ab14d1e6c471e6ea" member_type: VOTER last_known_addr { host: "127.17.60.253" port: 46387 } } peers { permanent_uuid: "e5fa9b0933e446c9854df3006fce277f" member_type: VOTER last_known_addr { host: "127.17.60.252" port: 43895 } }
I20250112 15:06:06.613833 19297 leader_election.cc:290] T 00000000000000000000000000000000 P 4a9e1db91a694b78b174fa300b67fb97 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers cd21cea7d1214dd2ab14d1e6c471e6ea (127.17.60.253:46387), e5fa9b0933e446c9854df3006fce277f (127.17.60.252:43895)
I20250112 15:06:06.614063 18847 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4a9e1db91a694b78b174fa300b67fb97" candidate_term: 3 candidate_status { last_received { term: 1 index: 9 } } ignore_live_leader: false dest_uuid: "cd21cea7d1214dd2ab14d1e6c471e6ea" is_pre_election: true
I20250112 15:06:06.614125 18847 raft_consensus.cc:2463] T 00000000000000000000000000000000 P cd21cea7d1214dd2ab14d1e6c471e6ea [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a9e1db91a694b78b174fa300b67fb97 in term 2.
W20250112 15:06:06.614264 18756 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.17.60.252:43895: connect: Connection refused (error 111) [suppressed 9 similar messages]
I20250112 15:06:06.614467 18756 leader_election.cc:304] T 00000000000000000000000000000000 P 4a9e1db91a694b78b174fa300b67fb97 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a9e1db91a694b78b174fa300b67fb97, cd21cea7d1214dd2ab14d1e6c471e6ea; no voters:
I20250112 15:06:06.614563 19297 raft_consensus.cc:2798] T 00000000000000000000000000000000 P 4a9e1db91a694b78b174fa300b67fb97 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250112 15:06:06.614612 19297 raft_consensus.cc:491] T 00000000000000000000000000000000 P 4a9e1db91a694b78b174fa300b67fb97 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250112 15:06:06.614639 19297 raft_consensus.cc:3054] T 00000000000000000000000000000000 P 4a9e1db91a694b78b174fa300b67fb97 [term 2 FOLLOWER]: Advancing to term 3
W20250112 15:06:06.614692 18756 leader_election.cc:336] T 00000000000000000000000000000000 P 4a9e1db91a694b78b174fa300b67fb97 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer e5fa9b0933e446c9854df3006fce277f (127.17.60.252:43895): Network error: Client connection negotiation failed: client connection to 127.17.60.252:43895: connect: Connection refused (error 111)
I20250112 15:06:06.615345 19297 raft_consensus.cc:513] T 00000000000000000000000000000000 P 4a9e1db91a694b78b174fa300b67fb97 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a9e1db91a694b78b174fa300b67fb97" member_type: VOTER last_known_addr { host: "127.17.60.254" port: 38227 } } peers { permanent_uuid: "cd21cea7d1214dd2ab14d1e6c471e6ea" member_type: VOTER last_known_addr { host: "127.17.60.253" port: 46387 } } peers { permanent_uuid: "e5fa9b0933e446c9854df3006fce277f" member_type: VOTER last_known_addr { host: "127.17.60.252" port: 43895 } }
I20250112 15:06:06.615461 19297 leader_election.cc:290] T 00000000000000000000000000000000 P 4a9e1db91a694b78b174fa300b67fb97 [CANDIDATE]: Term 3 election: Requested vote from peers cd21cea7d1214dd2ab14d1e6c471e6ea (127.17.60.253:46387), e5fa9b0933e446c9854df3006fce277f (127.17.60.252:43895)
I20250112 15:06:06.615674 18847 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4a9e1db91a694b78b174fa300b67fb97" candidate_term: 3 candidate_status { last_received { term: 1 index: 9 } } ignore_live_leader: false dest_uuid: "cd21cea7d1214dd2ab14d1e6c471e6ea"
I20250112 15:06:06.615743 18847 raft_consensus.cc:3054] T 00000000000000000000000000000000 P cd21cea7d1214dd2ab14d1e6c471e6ea [term 2 FOLLOWER]: Advancing to term 3
W20250112 15:06:06.615965 18756 leader_election.cc:336] T 00000000000000000000000000000000 P 4a9e1db91a694b78b174fa300b67fb97 [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer e5fa9b0933e446c9854df3006fce277f (127.17.60.252:43895): Network error: Client connection negotiation failed: client connection to 127.17.60.252:43895: connect: Connection refused (error 111)
I20250112 15:06:06.616326 18847 raft_consensus.cc:2463] T 00000000000000000000000000000000 P cd21cea7d1214dd2ab14d1e6c471e6ea [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a9e1db91a694b78b174fa300b67fb97 in term 3.
I20250112 15:06:06.616475 18756 leader_election.cc:304] T 00000000000000000000000000000000 P 4a9e1db91a694b78b174fa300b67fb97 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4a9e1db91a694b78b174fa300b67fb97, cd21cea7d1214dd2ab14d1e6c471e6ea; no voters: e5fa9b0933e446c9854df3006fce277f
I20250112 15:06:06.616581 19297 raft_consensus.cc:2798] T 00000000000000000000000000000000 P 4a9e1db91a694b78b174fa300b67fb97 [term 3 FOLLOWER]: Leader election won for term 3
I20250112 15:06:06.616708 19297 raft_consensus.cc:695] T 00000000000000000000000000000000 P 4a9e1db91a694b78b174fa300b67fb97 [term 3 LEADER]: Becoming Leader. State: Replica: 4a9e1db91a694b78b174fa300b67fb97, State: Running, Role: LEADER
I20250112 15:06:06.616772 19297 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4a9e1db91a694b78b174fa300b67fb97 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 9, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a9e1db91a694b78b174fa300b67fb97" member_type: VOTER last_known_addr { host: "127.17.60.254" port: 38227 } } peers { permanent_uuid: "cd21cea7d1214dd2ab14d1e6c471e6ea" member_type: VOTER last_known_addr { host: "127.17.60.253" port: 46387 } } peers { permanent_uuid: "e5fa9b0933e446c9854df3006fce277f" member_type: VOTER last_known_addr { host: "127.17.60.252" port: 43895 } }
I20250112 15:06:06.617056 19300 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4a9e1db91a694b78b174fa300b67fb97 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4a9e1db91a694b78b174fa300b67fb97. Latest consensus state: current_term: 3 leader_uuid: "4a9e1db91a694b78b174fa300b67fb97" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a9e1db91a694b78b174fa300b67fb97" member_type: VOTER last_known_addr { host: "127.17.60.254" port: 38227 } } peers { permanent_uuid: "cd21cea7d1214dd2ab14d1e6c471e6ea" member_type: VOTER last_known_addr { host: "127.17.60.253" port: 46387 } } peers { permanent_uuid: "e5fa9b0933e446c9854df3006fce277f" member_type: VOTER last_known_addr { host: "127.17.60.252" port: 43895 } } }
I20250112 15:06:06.617151 19300 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4a9e1db91a694b78b174fa300b67fb97 [sys.catalog]: This master's current role is: LEADER
I20250112 15:06:06.617327 19302 catalog_manager.cc:1476] Loading table and tablet metadata into memory...
I20250112 15:06:06.617524 19302 catalog_manager.cc:670] Loaded metadata for table test-workload [id=306cc9c07f484249acc9bc5895b8fe2f]
I20250112 15:06:06.617604 19302 tablet_loader.cc:96] loaded metadata for tablet 2c23ff24709f45fc916a27933aae8556 (table test-workload [id=306cc9c07f484249acc9bc5895b8fe2f])
I20250112 15:06:06.617631 19302 tablet_loader.cc:96] loaded metadata for tablet 3a85221652b848f59fb223080a4300a4 (table test-workload [id=306cc9c07f484249acc9bc5895b8fe2f])
I20250112 15:06:06.617655 19302 tablet_loader.cc:96] loaded metadata for tablet b1f7a675e15a4c649aacc9dcd25ab71a (table test-workload [id=306cc9c07f484249acc9bc5895b8fe2f])
I20250112 15:06:06.617676 19302 catalog_manager.cc:1485] Initializing Kudu cluster ID...
I20250112 15:06:06.617712 19302 catalog_manager.cc:1260] Loaded cluster ID: 7370a9ebaf3c4c2f9db9b7afecc9de6b
I20250112 15:06:06.617730 19302 catalog_manager.cc:1496] Initializing Kudu internal certificate authority...
I20250112 15:06:06.617851 19302 catalog_manager.cc:1505] Loading token signing keys...
I20250112 15:06:06.617894 19302 catalog_manager.cc:5910] T 00000000000000000000000000000000 P 4a9e1db91a694b78b174fa300b67fb97: Loaded TSK: 0
I20250112 15:06:06.617967 19302 catalog_manager.cc:1515] Initializing in-progress tserver states...
W20250112 15:06:07.069423 18756 consensus_peers.cc:487] T 00000000000000000000000000000000 P 4a9e1db91a694b78b174fa300b67fb97 -> Peer e5fa9b0933e446c9854df3006fce277f (127.17.60.252:43895): Couldn't send request to peer e5fa9b0933e446c9854df3006fce277f. Status: Network error: Client connection negotiation failed: client connection to 127.17.60.252:43895: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20250112 15:06:07.177306 18847 raft_consensus.cc:1270] T 00000000000000000000000000000000 P cd21cea7d1214dd2ab14d1e6c471e6ea [term 3 FOLLOWER]: Refusing update from remote peer 4a9e1db91a694b78b174fa300b67fb97: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 3 index: 10. (index mismatch)
I20250112 15:06:07.177606 19300 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 4a9e1db91a694b78b174fa300b67fb97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd21cea7d1214dd2ab14d1e6c471e6ea" member_type: VOTER last_known_addr { host: "127.17.60.253" port: 46387 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20250112 15:06:07.178128 19303 sys_catalog.cc:455] T 00000000000000000000000000000000 P cd21cea7d1214dd2ab14d1e6c471e6ea [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4a9e1db91a694b78b174fa300b67fb97. Latest consensus state: current_term: 3 leader_uuid: "4a9e1db91a694b78b174fa300b67fb97" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a9e1db91a694b78b174fa300b67fb97" member_type: VOTER last_known_addr { host: "127.17.60.254" port: 38227 } } peers { permanent_uuid: "cd21cea7d1214dd2ab14d1e6c471e6ea" member_type: VOTER last_known_addr { host: "127.17.60.253" port: 46387 } } peers { permanent_uuid: "e5fa9b0933e446c9854df3006fce277f" member_type: VOTER last_known_addr { host: "127.17.60.252" port: 43895 } } }
I20250112 15:06:07.178198 19303 sys_catalog.cc:458] T 00000000000000000000000000000000 P cd21cea7d1214dd2ab14d1e6c471e6ea [sys.catalog]: This master's current role is: FOLLOWER
I20250112 15:06:07.178627 19305 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4a9e1db91a694b78b174fa300b67fb97 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 3 leader_uuid: "4a9e1db91a694b78b174fa300b67fb97" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a9e1db91a694b78b174fa300b67fb97" member_type: VOTER last_known_addr { host: "127.17.60.254" port: 38227 } } peers { permanent_uuid: "cd21cea7d1214dd2ab14d1e6c471e6ea" member_type: VOTER last_known_addr { host: "127.17.60.253" port: 46387 } } peers { permanent_uuid: "e5fa9b0933e446c9854df3006fce277f" member_type: VOTER last_known_addr { host: "127.17.60.252" port: 43895 } } }
I20250112 15:06:07.178719 19305 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4a9e1db91a694b78b174fa300b67fb97 [sys.catalog]: This master's current role is: LEADER
I20250112 15:06:07.178881 19303 sys_catalog.cc:455] T 00000000000000000000000000000000 P cd21cea7d1214dd2ab14d1e6c471e6ea [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 3 leader_uuid: "4a9e1db91a694b78b174fa300b67fb97" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a9e1db91a694b78b174fa300b67fb97" member_type: VOTER last_known_addr { host: "127.17.60.254" port: 38227 } } peers { permanent_uuid: "cd21cea7d1214dd2ab14d1e6c471e6ea" member_type: VOTER last_known_addr { host: "127.17.60.253" port: 46387 } } peers { permanent_uuid: "e5fa9b0933e446c9854df3006fce277f" member_type: VOTER last_known_addr { host: "127.17.60.252" port: 43895 } } }
I20250112 15:06:07.178957 19303 sys_catalog.cc:458] T 00000000000000000000000000000000 P cd21cea7d1214dd2ab14d1e6c471e6ea [sys.catalog]: This master's current role is: FOLLOWER
I20250112 15:06:07.235520 18967 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250112 15:06:07.235641 18967 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20250112 15:06:07.719929 17651 tablet_server.cc:178] TabletServer@127.17.60.193:0 shutting down...
I20250112 15:06:07.722424 17651 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250112 15:06:07.722558 17651 tablet_replica.cc:331] T 2c23ff24709f45fc916a27933aae8556 P 21292fc4406740b1a866cf178a2afa1e: stopping tablet replica
I20250112 15:06:07.722640 17651 raft_consensus.cc:2238] T 2c23ff24709f45fc916a27933aae8556 P 21292fc4406740b1a866cf178a2afa1e [term 1 LEADER]: Raft consensus shutting down.
I20250112 15:06:07.722687 17651 raft_consensus.cc:2267] T 2c23ff24709f45fc916a27933aae8556 P 21292fc4406740b1a866cf178a2afa1e [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 15:06:07.734238 17651 tablet_server.cc:195] TabletServer@127.17.60.193:0 shutdown complete.
I20250112 15:06:07.735291 17651 tablet_server.cc:178] TabletServer@127.17.60.194:0 shutting down...
I20250112 15:06:07.738220 17651 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250112 15:06:07.738360 17651 tablet_replica.cc:331] T 3a85221652b848f59fb223080a4300a4 P 7064d93ebf764af4b1035d9667296dc8: stopping tablet replica
I20250112 15:06:07.738433 17651 raft_consensus.cc:2238] T 3a85221652b848f59fb223080a4300a4 P 7064d93ebf764af4b1035d9667296dc8 [term 1 LEADER]: Raft consensus shutting down.
I20250112 15:06:07.738476 17651 raft_consensus.cc:2267] T 3a85221652b848f59fb223080a4300a4 P 7064d93ebf764af4b1035d9667296dc8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 15:06:07.749943 17651 tablet_server.cc:195] TabletServer@127.17.60.194:0 shutdown complete.
I20250112 15:06:07.750805 17651 tablet_server.cc:178] TabletServer@127.17.60.195:0 shutting down...
I20250112 15:06:07.753849 17651 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250112 15:06:07.754005 17651 tablet_replica.cc:331] T b1f7a675e15a4c649aacc9dcd25ab71a P dd83f0dba7dd403887d9546c0aa261d6: stopping tablet replica
I20250112 15:06:07.754065 17651 raft_consensus.cc:2238] T b1f7a675e15a4c649aacc9dcd25ab71a P dd83f0dba7dd403887d9546c0aa261d6 [term 1 LEADER]: Raft consensus shutting down.
I20250112 15:06:07.754101 17651 raft_consensus.cc:2267] T b1f7a675e15a4c649aacc9dcd25ab71a P dd83f0dba7dd403887d9546c0aa261d6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 15:06:07.765681 17651 tablet_server.cc:195] TabletServer@127.17.60.195:0 shutdown complete.
I20250112 15:06:07.766556 17651 master.cc:537] Master@127.17.60.254:38227 shutting down...
I20250112 15:06:07.768687 17651 raft_consensus.cc:2238] T 00000000000000000000000000000000 P 4a9e1db91a694b78b174fa300b67fb97 [term 3 LEADER]: Raft consensus shutting down.
I20250112 15:06:07.768800 17651 raft_consensus.cc:2267] T 00000000000000000000000000000000 P 4a9e1db91a694b78b174fa300b67fb97 [term 3 FOLLOWER]: Raft consensus is shut down!
I20250112 15:06:07.768841 17651 tablet_replica.cc:331] T 00000000000000000000000000000000 P 4a9e1db91a694b78b174fa300b67fb97: stopping tablet replica
I20250112 15:06:07.780669 17651 master.cc:559] Master@127.17.60.254:38227 shutdown complete.
I20250112 15:06:07.781796 17651 master.cc:537] Master@127.17.60.253:46387 shutting down...
I20250112 15:06:07.783790 17651 raft_consensus.cc:2238] T 00000000000000000000000000000000 P cd21cea7d1214dd2ab14d1e6c471e6ea [term 3 FOLLOWER]: Raft consensus shutting down.
I20250112 15:06:07.783864 17651 raft_consensus.cc:2267] T 00000000000000000000000000000000 P cd21cea7d1214dd2ab14d1e6c471e6ea [term 3 FOLLOWER]: Raft consensus is shut down!
I20250112 15:06:07.783883 17651 tablet_replica.cc:331] T 00000000000000000000000000000000 P cd21cea7d1214dd2ab14d1e6c471e6ea: stopping tablet replica
I20250112 15:06:07.795589 17651 master.cc:559] Master@127.17.60.253:46387 shutdown complete.
[ OK ] AutoRebalancerTest.NextLeaderResumesAutoRebalancing (6651 ms)
[ RUN ] AutoRebalancerTest.MovesScheduledIfAddTserver
I20250112 15:06:07.801141 17651 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.17.60.254:40313
I20250112 15:06:07.801316 17651 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250112 15:06:07.803493 19312 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250112 15:06:07.804150 19313 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250112 15:06:07.804412 17651 server_base.cc:1034] running on GCE node
W20250112 15:06:07.805264 19315 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250112 15:06:07.805385 17651 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250112 15:06:07.805418 17651 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250112 15:06:07.805430 17651 hybrid_clock.cc:648] HybridClock initialized: now 1736694367805431 us; error 0 us; skew 500 ppm
I20250112 15:06:07.813337 17651 webserver.cc:458] Webserver started at http://127.17.60.254:35521/ using document root <none> and password file <none>
I20250112 15:06:07.813429 17651 fs_manager.cc:362] Metadata directory not provided
I20250112 15:06:07.813462 17651 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250112 15:06:07.813498 17651 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250112 15:06:07.813752 17651 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task95dLSD/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1736694351070047-17651-0/minicluster-data/master-0-root/instance:
uuid: "2f88df4aab60498494c42bdc47544b41"
format_stamp: "Formatted at 2025-01-12 15:06:07 on dist-test-slave-b4nz"
I20250112 15:06:07.814652 17651 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20250112 15:06:07.815088 19320 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250112 15:06:07.815249 17651 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20250112 15:06:07.815289 17651 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task95dLSD/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1736694351070047-17651-0/minicluster-data/master-0-root
uuid: "2f88df4aab60498494c42bdc47544b41"
format_stamp: "Formatted at 2025-01-12 15:06:07 on dist-test-slave-b4nz"
I20250112 15:06:07.815333 17651 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task95dLSD/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1736694351070047-17651-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-task95dLSD/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1736694351070047-17651-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-task95dLSD/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1736694351070047-17651-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250112 15:06:07.821295 17651 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250112 15:06:07.821477 17651 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250112 15:06:07.823829 17651 rpc_server.cc:307] RPC server started. Bound to: 127.17.60.254:40313
I20250112 15:06:07.826597 19372 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.17.60.254:40313 every 8 connection(s)
I20250112 15:06:07.826756 19373 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 15:06:07.827760 19373 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2f88df4aab60498494c42bdc47544b41: Bootstrap starting.
I20250112 15:06:07.828069 19373 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2f88df4aab60498494c42bdc47544b41: Neither blocks nor log segments found. Creating new log.
I20250112 15:06:07.828625 19373 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2f88df4aab60498494c42bdc47544b41: No bootstrap required, opened a new log
I20250112 15:06:07.828747 19373 raft_consensus.cc:357] T 00000000000000000000000000000000 P 2f88df4aab60498494c42bdc47544b41 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2f88df4aab60498494c42bdc47544b41" member_type: VOTER }
I20250112 15:06:07.828799 19373 raft_consensus.cc:383] T 00000000000000000000000000000000 P 2f88df4aab60498494c42bdc47544b41 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 15:06:07.828812 19373 raft_consensus.cc:738] T 00000000000000000000000000000000 P 2f88df4aab60498494c42bdc47544b41 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2f88df4aab60498494c42bdc47544b41, State: Initialized, Role: FOLLOWER
I20250112 15:06:07.828853 19373 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2f88df4aab60498494c42bdc47544b41 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2f88df4aab60498494c42bdc47544b41" member_type: VOTER }
I20250112 15:06:07.828889 19373 raft_consensus.cc:397] T 00000000000000000000000000000000 P 2f88df4aab60498494c42bdc47544b41 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250112 15:06:07.828912 19373 raft_consensus.cc:491] T 00000000000000000000000000000000 P 2f88df4aab60498494c42bdc47544b41 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250112 15:06:07.828939 19373 raft_consensus.cc:3054] T 00000000000000000000000000000000 P 2f88df4aab60498494c42bdc47544b41 [term 0 FOLLOWER]: Advancing to term 1
I20250112 15:06:07.829397 19373 raft_consensus.cc:513] T 00000000000000000000000000000000 P 2f88df4aab60498494c42bdc47544b41 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2f88df4aab60498494c42bdc47544b41" member_type: VOTER }
I20250112 15:06:07.829449 19373 leader_election.cc:304] T 00000000000000000000000000000000 P 2f88df4aab60498494c42bdc47544b41 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2f88df4aab60498494c42bdc47544b41; no voters:
I20250112 15:06:07.829553 19373 leader_election.cc:290] T 00000000000000000000000000000000 P 2f88df4aab60498494c42bdc47544b41 [CANDIDATE]: Term 1 election: Requested vote from peers
I20250112 15:06:07.829613 19376 raft_consensus.cc:2798] T 00000000000000000000000000000000 P 2f88df4aab60498494c42bdc47544b41 [term 1 FOLLOWER]: Leader election won for term 1
I20250112 15:06:07.829691 19373 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2f88df4aab60498494c42bdc47544b41 [sys.catalog]: configured and running, proceeding with master startup.
I20250112 15:06:07.829770 19376 raft_consensus.cc:695] T 00000000000000000000000000000000 P 2f88df4aab60498494c42bdc47544b41 [term 1 LEADER]: Becoming Leader. State: Replica: 2f88df4aab60498494c42bdc47544b41, State: Running, Role: LEADER
I20250112 15:06:07.829824 19376 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2f88df4aab60498494c42bdc47544b41 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2f88df4aab60498494c42bdc47544b41" member_type: VOTER }
I20250112 15:06:07.830054 19377 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2f88df4aab60498494c42bdc47544b41 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2f88df4aab60498494c42bdc47544b41" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2f88df4aab60498494c42bdc47544b41" member_type: VOTER } }
I20250112 15:06:07.830078 19378 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2f88df4aab60498494c42bdc47544b41 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2f88df4aab60498494c42bdc47544b41. Latest consensus state: current_term: 1 leader_uuid: "2f88df4aab60498494c42bdc47544b41" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2f88df4aab60498494c42bdc47544b41" member_type: VOTER } }
I20250112 15:06:07.830118 19377 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2f88df4aab60498494c42bdc47544b41 [sys.catalog]: This master's current role is: LEADER
I20250112 15:06:07.830121 19378 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2f88df4aab60498494c42bdc47544b41 [sys.catalog]: This master's current role is: LEADER
I20250112 15:06:07.830333 19382 catalog_manager.cc:1476] Loading table and tablet metadata into memory...
I20250112 15:06:07.830471 19382 catalog_manager.cc:1485] Initializing Kudu cluster ID...
I20250112 15:06:07.830994 17651 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20250112 15:06:07.831012 19382 catalog_manager.cc:1348] Generated new cluster ID: 6607acf25aca4d92b671346a28b22c1a
I20250112 15:06:07.831055 19382 catalog_manager.cc:1496] Initializing Kudu internal certificate authority...
I20250112 15:06:07.844187 19382 catalog_manager.cc:1371] Generated new certificate authority record
I20250112 15:06:07.844641 19382 catalog_manager.cc:1505] Loading token signing keys...
I20250112 15:06:07.858750 19382 catalog_manager.cc:5899] T 00000000000000000000000000000000 P 2f88df4aab60498494c42bdc47544b41: Generated new TSK 0
I20250112 15:06:07.858891 19382 catalog_manager.cc:1515] Initializing in-progress tserver states...
I20250112 15:06:07.862872 17651 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250112 15:06:07.864806 19395 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250112 15:06:07.865070 19394 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250112 15:06:07.865732 19397 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250112 15:06:07.866746 17651 server_base.cc:1034] running on GCE node
I20250112 15:06:07.866847 17651 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250112 15:06:07.866878 17651 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250112 15:06:07.866899 17651 hybrid_clock.cc:648] HybridClock initialized: now 1736694367866899 us; error 0 us; skew 500 ppm
I20250112 15:06:07.867563 17651 webserver.cc:458] Webserver started at http://127.17.60.193:43189/ using document root <none> and password file <none>
I20250112 15:06:07.867658 17651 fs_manager.cc:362] Metadata directory not provided
I20250112 15:06:07.867707 17651 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250112 15:06:07.867763 17651 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250112 15:06:07.868041 17651 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task95dLSD/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1736694351070047-17651-0/minicluster-data/ts-0-root/instance:
uuid: "7bfe4b6f206a49759e8069c99e17dd71"
format_stamp: "Formatted at 2025-01-12 15:06:07 on dist-test-slave-b4nz"
I20250112 15:06:07.868938 17651 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20250112 15:06:07.869376 19403 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250112 15:06:07.869494 17651 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20250112 15:06:07.869535 17651 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task95dLSD/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1736694351070047-17651-0/minicluster-data/ts-0-root
uuid: "7bfe4b6f206a49759e8069c99e17dd71"
format_stamp: "Formatted at 2025-01-12 15:06:07 on dist-test-slave-b4nz"
I20250112 15:06:07.869582 17651 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task95dLSD/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1736694351070047-17651-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-task95dLSD/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1736694351070047-17651-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-task95dLSD/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1736694351070047-17651-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250112 15:06:07.875941 17651 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250112 15:06:07.876165 17651 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250112 15:06:07.876367 17651 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250112 15:06:07.876602 17651 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250112 15:06:07.876646 17651 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250112 15:06:07.876677 17651 ts_tablet_manager.cc:610] Registered 0 tablets
I20250112 15:06:07.876701 17651 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250112 15:06:07.879316 17651 rpc_server.cc:307] RPC server started. Bound to: 127.17.60.193:33177
I20250112 15:06:07.879343 19466 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.17.60.193:33177 every 8 connection(s)
I20250112 15:06:07.879674 17651 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250112 15:06:07.881405 19471 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250112 15:06:07.881624 19472 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250112 15:06:07.882495 19474 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250112 15:06:07.882963 19467 heartbeater.cc:346] Connected to a master server at 127.17.60.254:40313
I20250112 15:06:07.883018 19467 heartbeater.cc:463] Registering TS with master...
I20250112 15:06:07.883116 19467 heartbeater.cc:510] Master 127.17.60.254:40313 requested a full tablet report, sending...
I20250112 15:06:07.883284 17651 server_base.cc:1034] running on GCE node
I20250112 15:06:07.883360 19337 ts_manager.cc:194] Registered new tserver with Master: 7bfe4b6f206a49759e8069c99e17dd71 (127.17.60.193:33177)
I20250112 15:06:07.883409 17651 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250112 15:06:07.883443 17651 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250112 15:06:07.883468 17651 hybrid_clock.cc:648] HybridClock initialized: now 1736694367883469 us; error 0 us; skew 500 ppm
I20250112 15:06:07.884094 17651 webserver.cc:458] Webserver started at http://127.17.60.194:36233/ using document root <none> and password file <none>
I20250112 15:06:07.884099 19337 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:58222
I20250112 15:06:07.884188 17651 fs_manager.cc:362] Metadata directory not provided
I20250112 15:06:07.884289 17651 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250112 15:06:07.884344 17651 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250112 15:06:07.884678 17651 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task95dLSD/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1736694351070047-17651-0/minicluster-data/ts-1-root/instance:
uuid: "af0fcb91bcd440e6939ea80ba1548866"
format_stamp: "Formatted at 2025-01-12 15:06:07 on dist-test-slave-b4nz"
I20250112 15:06:07.885517 17651 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20250112 15:06:07.885962 19479 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250112 15:06:07.886080 17651 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20250112 15:06:07.886139 17651 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task95dLSD/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1736694351070047-17651-0/minicluster-data/ts-1-root
uuid: "af0fcb91bcd440e6939ea80ba1548866"
format_stamp: "Formatted at 2025-01-12 15:06:07 on dist-test-slave-b4nz"
I20250112 15:06:07.886186 17651 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task95dLSD/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1736694351070047-17651-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-task95dLSD/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1736694351070047-17651-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-task95dLSD/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1736694351070047-17651-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250112 15:06:07.914290 17651 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250112 15:06:07.914530 17651 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250112 15:06:07.914727 17651 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250112 15:06:07.914944 17651 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250112 15:06:07.914976 17651 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250112 15:06:07.915011 17651 ts_tablet_manager.cc:610] Registered 0 tablets
I20250112 15:06:07.915036 17651 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250112 15:06:07.917714 17651 rpc_server.cc:307] RPC server started. Bound to: 127.17.60.194:37915
I20250112 15:06:07.917770 19542 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.17.60.194:37915 every 8 connection(s)
I20250112 15:06:07.918077 17651 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250112 15:06:07.919414 19546 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250112 15:06:07.920222 17651 server_base.cc:1034] running on GCE node
W20250112 15:06:07.920827 19547 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250112 15:06:07.920989 19543 heartbeater.cc:346] Connected to a master server at 127.17.60.254:40313
I20250112 15:06:07.921051 19543 heartbeater.cc:463] Registering TS with master...
I20250112 15:06:07.921149 19543 heartbeater.cc:510] Master 127.17.60.254:40313 requested a full tablet report, sending...
I20250112 15:06:07.921380 19337 ts_manager.cc:194] Registered new tserver with Master: af0fcb91bcd440e6939ea80ba1548866 (127.17.60.194:37915)
W20250112 15:06:07.921499 19549 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250112 15:06:07.921690 17651 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250112 15:06:07.921726 17651 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250112 15:06:07.921747 17651 hybrid_clock.cc:648] HybridClock initialized: now 1736694367921747 us; error 0 us; skew 500 ppm
I20250112 15:06:07.921789 19337 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:58230
I20250112 15:06:07.922392 17651 webserver.cc:458] Webserver started at http://127.17.60.195:41485/ using document root <none> and password file <none>
I20250112 15:06:07.922458 17651 fs_manager.cc:362] Metadata directory not provided
I20250112 15:06:07.922511 17651 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250112 15:06:07.922562 17651 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250112 15:06:07.922832 17651 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task95dLSD/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1736694351070047-17651-0/minicluster-data/ts-2-root/instance:
uuid: "016fc71013aa4c6c8bb6e23194529062"
format_stamp: "Formatted at 2025-01-12 15:06:07 on dist-test-slave-b4nz"
I20250112 15:06:07.923645 17651 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20250112 15:06:07.924024 19554 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250112 15:06:07.924131 17651 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20250112 15:06:07.924176 17651 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task95dLSD/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1736694351070047-17651-0/minicluster-data/ts-2-root
uuid: "016fc71013aa4c6c8bb6e23194529062"
format_stamp: "Formatted at 2025-01-12 15:06:07 on dist-test-slave-b4nz"
I20250112 15:06:07.924224 17651 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task95dLSD/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1736694351070047-17651-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-task95dLSD/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1736694351070047-17651-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-task95dLSD/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1736694351070047-17651-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250112 15:06:07.931766 17651 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250112 15:06:07.931958 17651 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250112 15:06:07.932078 17651 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250112 15:06:07.932271 17651 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250112 15:06:07.932298 17651 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250112 15:06:07.932327 17651 ts_tablet_manager.cc:610] Registered 0 tablets
I20250112 15:06:07.932353 17651 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250112 15:06:07.935524 17651 rpc_server.cc:307] RPC server started. Bound to: 127.17.60.195:40923
I20250112 15:06:07.935600 19617 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.17.60.195:40923 every 8 connection(s)
I20250112 15:06:07.938314 19618 heartbeater.cc:346] Connected to a master server at 127.17.60.254:40313
I20250112 15:06:07.938367 19618 heartbeater.cc:463] Registering TS with master...
I20250112 15:06:07.938459 19618 heartbeater.cc:510] Master 127.17.60.254:40313 requested a full tablet report, sending...
I20250112 15:06:07.938668 19337 ts_manager.cc:194] Registered new tserver with Master: 016fc71013aa4c6c8bb6e23194529062 (127.17.60.195:40923)
I20250112 15:06:07.938995 17651 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.003289441s
I20250112 15:06:07.939013 19337 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:58234
I20250112 15:06:08.884722 19467 heartbeater.cc:502] Master 127.17.60.254:40313 was elected leader, sending a full tablet report...
I20250112 15:06:08.922303 19543 heartbeater.cc:502] Master 127.17.60.254:40313 was elected leader, sending a full tablet report...
I20250112 15:06:08.939566 19618 heartbeater.cc:502] Master 127.17.60.254:40313 was elected leader, sending a full tablet report...
I20250112 15:06:08.964203 17651 test_util.cc:274] Using random seed: -1426977177
I20250112 15:06:08.968979 19337 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:58250:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "\004\001\000\377\377\377?""\004\001\000\377\377\377?"
indirect_data: """"
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 15:06:08.969148 19337 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-workload in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 15:06:08.973991 19431 tablet_service.cc:1467] Processing CreateTablet for tablet 32c0a6c93d7f45efa313d196c5f0562c (DEFAULT_TABLE table=test-workload [id=e644f70925e04891b2eabd97597a927a]), partition=RANGE (key) PARTITION VALUES < 1073741823
I20250112 15:06:08.974136 19507 tablet_service.cc:1467] Processing CreateTablet for tablet 32c0a6c93d7f45efa313d196c5f0562c (DEFAULT_TABLE table=test-workload [id=e644f70925e04891b2eabd97597a927a]), partition=RANGE (key) PARTITION VALUES < 1073741823
I20250112 15:06:08.974180 19430 tablet_service.cc:1467] Processing CreateTablet for tablet 06335cb4d4ff431dab85d2b78c33232b (DEFAULT_TABLE table=test-workload [id=e644f70925e04891b2eabd97597a927a]), partition=RANGE (key) PARTITION 1073741823 <= VALUES
I20250112 15:06:08.974231 19507 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 32c0a6c93d7f45efa313d196c5f0562c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 15:06:08.974144 19431 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 32c0a6c93d7f45efa313d196c5f0562c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 15:06:08.974421 19430 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 06335cb4d4ff431dab85d2b78c33232b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 15:06:08.975607 19638 tablet_bootstrap.cc:492] T 06335cb4d4ff431dab85d2b78c33232b P 7bfe4b6f206a49759e8069c99e17dd71: Bootstrap starting.
I20250112 15:06:08.975741 19506 tablet_service.cc:1467] Processing CreateTablet for tablet 06335cb4d4ff431dab85d2b78c33232b (DEFAULT_TABLE table=test-workload [id=e644f70925e04891b2eabd97597a927a]), partition=RANGE (key) PARTITION 1073741823 <= VALUES
I20250112 15:06:08.975755 19639 tablet_bootstrap.cc:492] T 32c0a6c93d7f45efa313d196c5f0562c P af0fcb91bcd440e6939ea80ba1548866: Bootstrap starting.
I20250112 15:06:08.975885 19506 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 06335cb4d4ff431dab85d2b78c33232b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 15:06:08.976015 19638 tablet_bootstrap.cc:654] T 06335cb4d4ff431dab85d2b78c33232b P 7bfe4b6f206a49759e8069c99e17dd71: Neither blocks nor log segments found. Creating new log.
I20250112 15:06:08.976339 19639 tablet_bootstrap.cc:654] T 32c0a6c93d7f45efa313d196c5f0562c P af0fcb91bcd440e6939ea80ba1548866: Neither blocks nor log segments found. Creating new log.
I20250112 15:06:08.976485 19638 tablet_bootstrap.cc:492] T 06335cb4d4ff431dab85d2b78c33232b P 7bfe4b6f206a49759e8069c99e17dd71: No bootstrap required, opened a new log
I20250112 15:06:08.976516 19638 ts_tablet_manager.cc:1397] T 06335cb4d4ff431dab85d2b78c33232b P 7bfe4b6f206a49759e8069c99e17dd71: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20250112 15:06:08.976716 19638 raft_consensus.cc:357] T 06335cb4d4ff431dab85d2b78c33232b P 7bfe4b6f206a49759e8069c99e17dd71 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7bfe4b6f206a49759e8069c99e17dd71" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 33177 } } peers { permanent_uuid: "af0fcb91bcd440e6939ea80ba1548866" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 37915 } } peers { permanent_uuid: "016fc71013aa4c6c8bb6e23194529062" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 40923 } }
I20250112 15:06:08.976897 19638 raft_consensus.cc:383] T 06335cb4d4ff431dab85d2b78c33232b P 7bfe4b6f206a49759e8069c99e17dd71 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 15:06:08.976922 19638 raft_consensus.cc:738] T 06335cb4d4ff431dab85d2b78c33232b P 7bfe4b6f206a49759e8069c99e17dd71 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7bfe4b6f206a49759e8069c99e17dd71, State: Initialized, Role: FOLLOWER
I20250112 15:06:08.976940 19639 tablet_bootstrap.cc:492] T 32c0a6c93d7f45efa313d196c5f0562c P af0fcb91bcd440e6939ea80ba1548866: No bootstrap required, opened a new log
I20250112 15:06:08.976981 19639 ts_tablet_manager.cc:1397] T 32c0a6c93d7f45efa313d196c5f0562c P af0fcb91bcd440e6939ea80ba1548866: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20250112 15:06:08.976965 19638 consensus_queue.cc:260] T 06335cb4d4ff431dab85d2b78c33232b P 7bfe4b6f206a49759e8069c99e17dd71 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7bfe4b6f206a49759e8069c99e17dd71" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 33177 } } peers { permanent_uuid: "af0fcb91bcd440e6939ea80ba1548866" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 37915 } } peers { permanent_uuid: "016fc71013aa4c6c8bb6e23194529062" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 40923 } }
I20250112 15:06:08.977128 19639 raft_consensus.cc:357] T 32c0a6c93d7f45efa313d196c5f0562c P af0fcb91bcd440e6939ea80ba1548866 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7bfe4b6f206a49759e8069c99e17dd71" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 33177 } } peers { permanent_uuid: "016fc71013aa4c6c8bb6e23194529062" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 40923 } } peers { permanent_uuid: "af0fcb91bcd440e6939ea80ba1548866" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 37915 } }
I20250112 15:06:08.977210 19639 raft_consensus.cc:383] T 32c0a6c93d7f45efa313d196c5f0562c P af0fcb91bcd440e6939ea80ba1548866 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 15:06:08.977130 19638 ts_tablet_manager.cc:1428] T 06335cb4d4ff431dab85d2b78c33232b P 7bfe4b6f206a49759e8069c99e17dd71: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20250112 15:06:08.977295 19638 tablet_bootstrap.cc:492] T 32c0a6c93d7f45efa313d196c5f0562c P 7bfe4b6f206a49759e8069c99e17dd71: Bootstrap starting.
I20250112 15:06:08.977241 19639 raft_consensus.cc:738] T 32c0a6c93d7f45efa313d196c5f0562c P af0fcb91bcd440e6939ea80ba1548866 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: af0fcb91bcd440e6939ea80ba1548866, State: Initialized, Role: FOLLOWER
I20250112 15:06:08.977470 19639 consensus_queue.cc:260] T 32c0a6c93d7f45efa313d196c5f0562c P af0fcb91bcd440e6939ea80ba1548866 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7bfe4b6f206a49759e8069c99e17dd71" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 33177 } } peers { permanent_uuid: "016fc71013aa4c6c8bb6e23194529062" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 40923 } } peers { permanent_uuid: "af0fcb91bcd440e6939ea80ba1548866" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 37915 } }
I20250112 15:06:08.977625 19639 ts_tablet_manager.cc:1428] T 32c0a6c93d7f45efa313d196c5f0562c P af0fcb91bcd440e6939ea80ba1548866: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20250112 15:06:08.977684 19639 tablet_bootstrap.cc:492] T 06335cb4d4ff431dab85d2b78c33232b P af0fcb91bcd440e6939ea80ba1548866: Bootstrap starting.
I20250112 15:06:08.977705 19638 tablet_bootstrap.cc:654] T 32c0a6c93d7f45efa313d196c5f0562c P 7bfe4b6f206a49759e8069c99e17dd71: Neither blocks nor log segments found. Creating new log.
I20250112 15:06:08.977713 19582 tablet_service.cc:1467] Processing CreateTablet for tablet 32c0a6c93d7f45efa313d196c5f0562c (DEFAULT_TABLE table=test-workload [id=e644f70925e04891b2eabd97597a927a]), partition=RANGE (key) PARTITION VALUES < 1073741823
I20250112 15:06:08.977820 19582 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 32c0a6c93d7f45efa313d196c5f0562c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 15:06:08.978058 19639 tablet_bootstrap.cc:654] T 06335cb4d4ff431dab85d2b78c33232b P af0fcb91bcd440e6939ea80ba1548866: Neither blocks nor log segments found. Creating new log.
I20250112 15:06:08.978181 19638 tablet_bootstrap.cc:492] T 32c0a6c93d7f45efa313d196c5f0562c P 7bfe4b6f206a49759e8069c99e17dd71: No bootstrap required, opened a new log
I20250112 15:06:08.978210 19638 ts_tablet_manager.cc:1397] T 32c0a6c93d7f45efa313d196c5f0562c P 7bfe4b6f206a49759e8069c99e17dd71: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20250112 15:06:08.978309 19638 raft_consensus.cc:357] T 32c0a6c93d7f45efa313d196c5f0562c P 7bfe4b6f206a49759e8069c99e17dd71 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7bfe4b6f206a49759e8069c99e17dd71" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 33177 } } peers { permanent_uuid: "016fc71013aa4c6c8bb6e23194529062" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 40923 } } peers { permanent_uuid: "af0fcb91bcd440e6939ea80ba1548866" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 37915 } }
I20250112 15:06:08.978353 19638 raft_consensus.cc:383] T 32c0a6c93d7f45efa313d196c5f0562c P 7bfe4b6f206a49759e8069c99e17dd71 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 15:06:08.978374 19638 raft_consensus.cc:738] T 32c0a6c93d7f45efa313d196c5f0562c P 7bfe4b6f206a49759e8069c99e17dd71 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7bfe4b6f206a49759e8069c99e17dd71, State: Initialized, Role: FOLLOWER
I20250112 15:06:08.978431 19638 consensus_queue.cc:260] T 32c0a6c93d7f45efa313d196c5f0562c P 7bfe4b6f206a49759e8069c99e17dd71 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7bfe4b6f206a49759e8069c99e17dd71" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 33177 } } peers { permanent_uuid: "016fc71013aa4c6c8bb6e23194529062" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 40923 } } peers { permanent_uuid: "af0fcb91bcd440e6939ea80ba1548866" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 37915 } }
I20250112 15:06:08.978444 19581 tablet_service.cc:1467] Processing CreateTablet for tablet 06335cb4d4ff431dab85d2b78c33232b (DEFAULT_TABLE table=test-workload [id=e644f70925e04891b2eabd97597a927a]), partition=RANGE (key) PARTITION 1073741823 <= VALUES
I20250112 15:06:08.978513 19638 ts_tablet_manager.cc:1428] T 32c0a6c93d7f45efa313d196c5f0562c P 7bfe4b6f206a49759e8069c99e17dd71: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20250112 15:06:08.978529 19581 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 06335cb4d4ff431dab85d2b78c33232b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 15:06:08.979212 19646 tablet_bootstrap.cc:492] T 32c0a6c93d7f45efa313d196c5f0562c P 016fc71013aa4c6c8bb6e23194529062: Bootstrap starting.
I20250112 15:06:08.979604 19646 tablet_bootstrap.cc:654] T 32c0a6c93d7f45efa313d196c5f0562c P 016fc71013aa4c6c8bb6e23194529062: Neither blocks nor log segments found. Creating new log.
I20250112 15:06:08.980147 19646 tablet_bootstrap.cc:492] T 32c0a6c93d7f45efa313d196c5f0562c P 016fc71013aa4c6c8bb6e23194529062: No bootstrap required, opened a new log
I20250112 15:06:08.980219 19646 ts_tablet_manager.cc:1397] T 32c0a6c93d7f45efa313d196c5f0562c P 016fc71013aa4c6c8bb6e23194529062: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20250112 15:06:08.980286 19639 tablet_bootstrap.cc:492] T 06335cb4d4ff431dab85d2b78c33232b P af0fcb91bcd440e6939ea80ba1548866: No bootstrap required, opened a new log
I20250112 15:06:08.980343 19639 ts_tablet_manager.cc:1397] T 06335cb4d4ff431dab85d2b78c33232b P af0fcb91bcd440e6939ea80ba1548866: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.001s
I20250112 15:06:08.980365 19646 raft_consensus.cc:357] T 32c0a6c93d7f45efa313d196c5f0562c P 016fc71013aa4c6c8bb6e23194529062 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7bfe4b6f206a49759e8069c99e17dd71" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 33177 } } peers { permanent_uuid: "016fc71013aa4c6c8bb6e23194529062" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 40923 } } peers { permanent_uuid: "af0fcb91bcd440e6939ea80ba1548866" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 37915 } }
I20250112 15:06:08.980423 19646 raft_consensus.cc:383] T 32c0a6c93d7f45efa313d196c5f0562c P 016fc71013aa4c6c8bb6e23194529062 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 15:06:08.980448 19646 raft_consensus.cc:738] T 32c0a6c93d7f45efa313d196c5f0562c P 016fc71013aa4c6c8bb6e23194529062 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 016fc71013aa4c6c8bb6e23194529062, State: Initialized, Role: FOLLOWER
I20250112 15:06:08.980479 19639 raft_consensus.cc:357] T 06335cb4d4ff431dab85d2b78c33232b P af0fcb91bcd440e6939ea80ba1548866 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7bfe4b6f206a49759e8069c99e17dd71" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 33177 } } peers { permanent_uuid: "af0fcb91bcd440e6939ea80ba1548866" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 37915 } } peers { permanent_uuid: "016fc71013aa4c6c8bb6e23194529062" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 40923 } }
I20250112 15:06:08.980508 19646 consensus_queue.cc:260] T 32c0a6c93d7f45efa313d196c5f0562c P 016fc71013aa4c6c8bb6e23194529062 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7bfe4b6f206a49759e8069c99e17dd71" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 33177 } } peers { permanent_uuid: "016fc71013aa4c6c8bb6e23194529062" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 40923 } } peers { permanent_uuid: "af0fcb91bcd440e6939ea80ba1548866" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 37915 } }
I20250112 15:06:08.980537 19639 raft_consensus.cc:383] T 06335cb4d4ff431dab85d2b78c33232b P af0fcb91bcd440e6939ea80ba1548866 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 15:06:08.980594 19639 raft_consensus.cc:738] T 06335cb4d4ff431dab85d2b78c33232b P af0fcb91bcd440e6939ea80ba1548866 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: af0fcb91bcd440e6939ea80ba1548866, State: Initialized, Role: FOLLOWER
I20250112 15:06:08.980645 19639 consensus_queue.cc:260] T 06335cb4d4ff431dab85d2b78c33232b P af0fcb91bcd440e6939ea80ba1548866 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7bfe4b6f206a49759e8069c99e17dd71" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 33177 } } peers { permanent_uuid: "af0fcb91bcd440e6939ea80ba1548866" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 37915 } } peers { permanent_uuid: "016fc71013aa4c6c8bb6e23194529062" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 40923 } }
I20250112 15:06:08.980726 19639 ts_tablet_manager.cc:1428] T 06335cb4d4ff431dab85d2b78c33232b P af0fcb91bcd440e6939ea80ba1548866: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20250112 15:06:08.980873 19646 ts_tablet_manager.cc:1428] T 32c0a6c93d7f45efa313d196c5f0562c P 016fc71013aa4c6c8bb6e23194529062: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20250112 15:06:08.980953 19646 tablet_bootstrap.cc:492] T 06335cb4d4ff431dab85d2b78c33232b P 016fc71013aa4c6c8bb6e23194529062: Bootstrap starting.
I20250112 15:06:08.981410 19646 tablet_bootstrap.cc:654] T 06335cb4d4ff431dab85d2b78c33232b P 016fc71013aa4c6c8bb6e23194529062: Neither blocks nor log segments found. Creating new log.
I20250112 15:06:08.981921 19646 tablet_bootstrap.cc:492] T 06335cb4d4ff431dab85d2b78c33232b P 016fc71013aa4c6c8bb6e23194529062: No bootstrap required, opened a new log
I20250112 15:06:08.981956 19646 ts_tablet_manager.cc:1397] T 06335cb4d4ff431dab85d2b78c33232b P 016fc71013aa4c6c8bb6e23194529062: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20250112 15:06:08.982081 19646 raft_consensus.cc:357] T 06335cb4d4ff431dab85d2b78c33232b P 016fc71013aa4c6c8bb6e23194529062 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7bfe4b6f206a49759e8069c99e17dd71" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 33177 } } peers { permanent_uuid: "af0fcb91bcd440e6939ea80ba1548866" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 37915 } } peers { permanent_uuid: "016fc71013aa4c6c8bb6e23194529062" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 40923 } }
I20250112 15:06:08.982133 19646 raft_consensus.cc:383] T 06335cb4d4ff431dab85d2b78c33232b P 016fc71013aa4c6c8bb6e23194529062 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 15:06:08.982146 19646 raft_consensus.cc:738] T 06335cb4d4ff431dab85d2b78c33232b P 016fc71013aa4c6c8bb6e23194529062 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 016fc71013aa4c6c8bb6e23194529062, State: Initialized, Role: FOLLOWER
I20250112 15:06:08.982178 19646 consensus_queue.cc:260] T 06335cb4d4ff431dab85d2b78c33232b P 016fc71013aa4c6c8bb6e23194529062 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7bfe4b6f206a49759e8069c99e17dd71" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 33177 } } peers { permanent_uuid: "af0fcb91bcd440e6939ea80ba1548866" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 37915 } } peers { permanent_uuid: "016fc71013aa4c6c8bb6e23194529062" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 40923 } }
I20250112 15:06:08.982254 19646 ts_tablet_manager.cc:1428] T 06335cb4d4ff431dab85d2b78c33232b P 016fc71013aa4c6c8bb6e23194529062: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20250112 15:06:08.993912 19648 raft_consensus.cc:491] T 06335cb4d4ff431dab85d2b78c33232b P 016fc71013aa4c6c8bb6e23194529062 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 15:06:08.994000 19648 raft_consensus.cc:513] T 06335cb4d4ff431dab85d2b78c33232b P 016fc71013aa4c6c8bb6e23194529062 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7bfe4b6f206a49759e8069c99e17dd71" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 33177 } } peers { permanent_uuid: "af0fcb91bcd440e6939ea80ba1548866" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 37915 } } peers { permanent_uuid: "016fc71013aa4c6c8bb6e23194529062" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 40923 } }
I20250112 15:06:08.994169 19648 leader_election.cc:290] T 06335cb4d4ff431dab85d2b78c33232b P 016fc71013aa4c6c8bb6e23194529062 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7bfe4b6f206a49759e8069c99e17dd71 (127.17.60.193:33177), af0fcb91bcd440e6939ea80ba1548866 (127.17.60.194:37915)
I20250112 15:06:08.996646 19441 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "06335cb4d4ff431dab85d2b78c33232b" candidate_uuid: "016fc71013aa4c6c8bb6e23194529062" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7bfe4b6f206a49759e8069c99e17dd71" is_pre_election: true
I20250112 15:06:08.996649 19517 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "06335cb4d4ff431dab85d2b78c33232b" candidate_uuid: "016fc71013aa4c6c8bb6e23194529062" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af0fcb91bcd440e6939ea80ba1548866" is_pre_election: true
I20250112 15:06:08.996749 19441 raft_consensus.cc:2463] T 06335cb4d4ff431dab85d2b78c33232b P 7bfe4b6f206a49759e8069c99e17dd71 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 016fc71013aa4c6c8bb6e23194529062 in term 0.
I20250112 15:06:08.996766 19517 raft_consensus.cc:2463] T 06335cb4d4ff431dab85d2b78c33232b P af0fcb91bcd440e6939ea80ba1548866 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 016fc71013aa4c6c8bb6e23194529062 in term 0.
I20250112 15:06:08.996886 19558 leader_election.cc:304] T 06335cb4d4ff431dab85d2b78c33232b P 016fc71013aa4c6c8bb6e23194529062 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 016fc71013aa4c6c8bb6e23194529062, 7bfe4b6f206a49759e8069c99e17dd71; no voters:
I20250112 15:06:08.996978 19648 raft_consensus.cc:2798] T 06335cb4d4ff431dab85d2b78c33232b P 016fc71013aa4c6c8bb6e23194529062 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250112 15:06:08.997020 19648 raft_consensus.cc:491] T 06335cb4d4ff431dab85d2b78c33232b P 016fc71013aa4c6c8bb6e23194529062 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250112 15:06:08.997051 19648 raft_consensus.cc:3054] T 06335cb4d4ff431dab85d2b78c33232b P 016fc71013aa4c6c8bb6e23194529062 [term 0 FOLLOWER]: Advancing to term 1
I20250112 15:06:08.997646 19648 raft_consensus.cc:513] T 06335cb4d4ff431dab85d2b78c33232b P 016fc71013aa4c6c8bb6e23194529062 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7bfe4b6f206a49759e8069c99e17dd71" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 33177 } } peers { permanent_uuid: "af0fcb91bcd440e6939ea80ba1548866" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 37915 } } peers { permanent_uuid: "016fc71013aa4c6c8bb6e23194529062" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 40923 } }
I20250112 15:06:08.997766 19648 leader_election.cc:290] T 06335cb4d4ff431dab85d2b78c33232b P 016fc71013aa4c6c8bb6e23194529062 [CANDIDATE]: Term 1 election: Requested vote from peers 7bfe4b6f206a49759e8069c99e17dd71 (127.17.60.193:33177), af0fcb91bcd440e6939ea80ba1548866 (127.17.60.194:37915)
I20250112 15:06:08.997869 19441 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "06335cb4d4ff431dab85d2b78c33232b" candidate_uuid: "016fc71013aa4c6c8bb6e23194529062" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7bfe4b6f206a49759e8069c99e17dd71"
I20250112 15:06:08.997936 19441 raft_consensus.cc:3054] T 06335cb4d4ff431dab85d2b78c33232b P 7bfe4b6f206a49759e8069c99e17dd71 [term 0 FOLLOWER]: Advancing to term 1
I20250112 15:06:08.997903 19517 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "06335cb4d4ff431dab85d2b78c33232b" candidate_uuid: "016fc71013aa4c6c8bb6e23194529062" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af0fcb91bcd440e6939ea80ba1548866"
I20250112 15:06:08.997990 19517 raft_consensus.cc:3054] T 06335cb4d4ff431dab85d2b78c33232b P af0fcb91bcd440e6939ea80ba1548866 [term 0 FOLLOWER]: Advancing to term 1
I20250112 15:06:08.998463 19441 raft_consensus.cc:2463] T 06335cb4d4ff431dab85d2b78c33232b P 7bfe4b6f206a49759e8069c99e17dd71 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 016fc71013aa4c6c8bb6e23194529062 in term 1.
I20250112 15:06:08.998610 19517 raft_consensus.cc:2463] T 06335cb4d4ff431dab85d2b78c33232b P af0fcb91bcd440e6939ea80ba1548866 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 016fc71013aa4c6c8bb6e23194529062 in term 1.
I20250112 15:06:08.998615 19558 leader_election.cc:304] T 06335cb4d4ff431dab85d2b78c33232b P 016fc71013aa4c6c8bb6e23194529062 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 016fc71013aa4c6c8bb6e23194529062, 7bfe4b6f206a49759e8069c99e17dd71; no voters:
I20250112 15:06:08.998694 19648 raft_consensus.cc:2798] T 06335cb4d4ff431dab85d2b78c33232b P 016fc71013aa4c6c8bb6e23194529062 [term 1 FOLLOWER]: Leader election won for term 1
I20250112 15:06:08.998845 19648 raft_consensus.cc:695] T 06335cb4d4ff431dab85d2b78c33232b P 016fc71013aa4c6c8bb6e23194529062 [term 1 LEADER]: Becoming Leader. State: Replica: 016fc71013aa4c6c8bb6e23194529062, State: Running, Role: LEADER
I20250112 15:06:08.998900 19648 consensus_queue.cc:237] T 06335cb4d4ff431dab85d2b78c33232b P 016fc71013aa4c6c8bb6e23194529062 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7bfe4b6f206a49759e8069c99e17dd71" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 33177 } } peers { permanent_uuid: "af0fcb91bcd440e6939ea80ba1548866" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 37915 } } peers { permanent_uuid: "016fc71013aa4c6c8bb6e23194529062" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 40923 } }
I20250112 15:06:08.999380 19337 catalog_manager.cc:5526] T 06335cb4d4ff431dab85d2b78c33232b P 016fc71013aa4c6c8bb6e23194529062 reported cstate change: term changed from 0 to 1, leader changed from <none> to 016fc71013aa4c6c8bb6e23194529062 (127.17.60.195). New cstate: current_term: 1 leader_uuid: "016fc71013aa4c6c8bb6e23194529062" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7bfe4b6f206a49759e8069c99e17dd71" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 33177 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "af0fcb91bcd440e6939ea80ba1548866" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 37915 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "016fc71013aa4c6c8bb6e23194529062" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 40923 } health_report { overall_health: HEALTHY } } }
I20250112 15:06:09.034812 19642 raft_consensus.cc:491] T 32c0a6c93d7f45efa313d196c5f0562c P 7bfe4b6f206a49759e8069c99e17dd71 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 15:06:09.034945 19642 raft_consensus.cc:513] T 32c0a6c93d7f45efa313d196c5f0562c P 7bfe4b6f206a49759e8069c99e17dd71 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7bfe4b6f206a49759e8069c99e17dd71" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 33177 } } peers { permanent_uuid: "016fc71013aa4c6c8bb6e23194529062" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 40923 } } peers { permanent_uuid: "af0fcb91bcd440e6939ea80ba1548866" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 37915 } }
I20250112 15:06:09.035250 19642 leader_election.cc:290] T 32c0a6c93d7f45efa313d196c5f0562c P 7bfe4b6f206a49759e8069c99e17dd71 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 016fc71013aa4c6c8bb6e23194529062 (127.17.60.195:40923), af0fcb91bcd440e6939ea80ba1548866 (127.17.60.194:37915)
I20250112 15:06:09.038149 19592 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "32c0a6c93d7f45efa313d196c5f0562c" candidate_uuid: "7bfe4b6f206a49759e8069c99e17dd71" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "016fc71013aa4c6c8bb6e23194529062" is_pre_election: true
I20250112 15:06:09.038271 19592 raft_consensus.cc:2463] T 32c0a6c93d7f45efa313d196c5f0562c P 016fc71013aa4c6c8bb6e23194529062 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7bfe4b6f206a49759e8069c99e17dd71 in term 0.
I20250112 15:06:09.038414 19407 leader_election.cc:304] T 32c0a6c93d7f45efa313d196c5f0562c P 7bfe4b6f206a49759e8069c99e17dd71 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 016fc71013aa4c6c8bb6e23194529062, 7bfe4b6f206a49759e8069c99e17dd71; no voters:
I20250112 15:06:09.038533 19642 raft_consensus.cc:2798] T 32c0a6c93d7f45efa313d196c5f0562c P 7bfe4b6f206a49759e8069c99e17dd71 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250112 15:06:09.038578 19642 raft_consensus.cc:491] T 32c0a6c93d7f45efa313d196c5f0562c P 7bfe4b6f206a49759e8069c99e17dd71 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250112 15:06:09.038599 19642 raft_consensus.cc:3054] T 32c0a6c93d7f45efa313d196c5f0562c P 7bfe4b6f206a49759e8069c99e17dd71 [term 0 FOLLOWER]: Advancing to term 1
I20250112 15:06:09.038683 19517 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "32c0a6c93d7f45efa313d196c5f0562c" candidate_uuid: "7bfe4b6f206a49759e8069c99e17dd71" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af0fcb91bcd440e6939ea80ba1548866" is_pre_election: true
I20250112 15:06:09.038745 19517 raft_consensus.cc:2463] T 32c0a6c93d7f45efa313d196c5f0562c P af0fcb91bcd440e6939ea80ba1548866 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7bfe4b6f206a49759e8069c99e17dd71 in term 0.
I20250112 15:06:09.039276 19642 raft_consensus.cc:513] T 32c0a6c93d7f45efa313d196c5f0562c P 7bfe4b6f206a49759e8069c99e17dd71 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7bfe4b6f206a49759e8069c99e17dd71" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 33177 } } peers { permanent_uuid: "016fc71013aa4c6c8bb6e23194529062" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 40923 } } peers { permanent_uuid: "af0fcb91bcd440e6939ea80ba1548866" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 37915 } }
I20250112 15:06:09.039386 19642 leader_election.cc:290] T 32c0a6c93d7f45efa313d196c5f0562c P 7bfe4b6f206a49759e8069c99e17dd71 [CANDIDATE]: Term 1 election: Requested vote from peers 016fc71013aa4c6c8bb6e23194529062 (127.17.60.195:40923), af0fcb91bcd440e6939ea80ba1548866 (127.17.60.194:37915)
I20250112 15:06:09.039548 19592 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "32c0a6c93d7f45efa313d196c5f0562c" candidate_uuid: "7bfe4b6f206a49759e8069c99e17dd71" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "016fc71013aa4c6c8bb6e23194529062"
I20250112 15:06:09.039551 19517 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "32c0a6c93d7f45efa313d196c5f0562c" candidate_uuid: "7bfe4b6f206a49759e8069c99e17dd71" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af0fcb91bcd440e6939ea80ba1548866"
I20250112 15:06:09.039646 19592 raft_consensus.cc:3054] T 32c0a6c93d7f45efa313d196c5f0562c P 016fc71013aa4c6c8bb6e23194529062 [term 0 FOLLOWER]: Advancing to term 1
I20250112 15:06:09.039654 19517 raft_consensus.cc:3054] T 32c0a6c93d7f45efa313d196c5f0562c P af0fcb91bcd440e6939ea80ba1548866 [term 0 FOLLOWER]: Advancing to term 1
I20250112 15:06:09.040302 19592 raft_consensus.cc:2463] T 32c0a6c93d7f45efa313d196c5f0562c P 016fc71013aa4c6c8bb6e23194529062 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7bfe4b6f206a49759e8069c99e17dd71 in term 1.
I20250112 15:06:09.040302 19517 raft_consensus.cc:2463] T 32c0a6c93d7f45efa313d196c5f0562c P af0fcb91bcd440e6939ea80ba1548866 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7bfe4b6f206a49759e8069c99e17dd71 in term 1.
I20250112 15:06:09.040513 19405 leader_election.cc:304] T 32c0a6c93d7f45efa313d196c5f0562c P 7bfe4b6f206a49759e8069c99e17dd71 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7bfe4b6f206a49759e8069c99e17dd71, af0fcb91bcd440e6939ea80ba1548866; no voters:
I20250112 15:06:09.040603 19642 raft_consensus.cc:2798] T 32c0a6c93d7f45efa313d196c5f0562c P 7bfe4b6f206a49759e8069c99e17dd71 [term 1 FOLLOWER]: Leader election won for term 1
I20250112 15:06:09.040766 19642 raft_consensus.cc:695] T 32c0a6c93d7f45efa313d196c5f0562c P 7bfe4b6f206a49759e8069c99e17dd71 [term 1 LEADER]: Becoming Leader. State: Replica: 7bfe4b6f206a49759e8069c99e17dd71, State: Running, Role: LEADER
I20250112 15:06:09.040820 19642 consensus_queue.cc:237] T 32c0a6c93d7f45efa313d196c5f0562c P 7bfe4b6f206a49759e8069c99e17dd71 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7bfe4b6f206a49759e8069c99e17dd71" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 33177 } } peers { permanent_uuid: "016fc71013aa4c6c8bb6e23194529062" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 40923 } } peers { permanent_uuid: "af0fcb91bcd440e6939ea80ba1548866" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 37915 } }
I20250112 15:06:09.041324 19337 catalog_manager.cc:5526] T 32c0a6c93d7f45efa313d196c5f0562c P 7bfe4b6f206a49759e8069c99e17dd71 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7bfe4b6f206a49759e8069c99e17dd71 (127.17.60.193). New cstate: current_term: 1 leader_uuid: "7bfe4b6f206a49759e8069c99e17dd71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7bfe4b6f206a49759e8069c99e17dd71" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 33177 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "016fc71013aa4c6c8bb6e23194529062" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 40923 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "af0fcb91bcd440e6939ea80ba1548866" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 37915 } health_report { overall_health: UNKNOWN } } }
I20250112 15:06:09.438750 19648 consensus_queue.cc:1035] T 06335cb4d4ff431dab85d2b78c33232b P 016fc71013aa4c6c8bb6e23194529062 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7bfe4b6f206a49759e8069c99e17dd71" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 33177 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250112 15:06:09.440162 19648 consensus_queue.cc:1035] T 06335cb4d4ff431dab85d2b78c33232b P 016fc71013aa4c6c8bb6e23194529062 [LEADER]: Connected to new peer: Peer: permanent_uuid: "af0fcb91bcd440e6939ea80ba1548866" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 37915 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250112 15:06:09.484493 19642 consensus_queue.cc:1035] T 32c0a6c93d7f45efa313d196c5f0562c P 7bfe4b6f206a49759e8069c99e17dd71 [LEADER]: Connected to new peer: Peer: permanent_uuid: "016fc71013aa4c6c8bb6e23194529062" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 40923 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250112 15:06:09.485724 19664 consensus_queue.cc:1035] T 32c0a6c93d7f45efa313d196c5f0562c P 7bfe4b6f206a49759e8069c99e17dd71 [LEADER]: Connected to new peer: Peer: permanent_uuid: "af0fcb91bcd440e6939ea80ba1548866" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 37915 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250112 15:06:09.831022 19392 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250112 15:06:09.831133 19392 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20250112 15:06:09.831152 19392 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20250112 15:06:11.831317 19392 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250112 15:06:11.831486 19392 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20250112 15:06:11.831527 19392 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20250112 15:06:13.831661 19392 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250112 15:06:13.831774 19392 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20250112 15:06:13.831799 19392 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20250112 15:06:15.831962 19392 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250112 15:06:15.832116 19392 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20250112 15:06:15.832154 19392 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20250112 15:06:17.832293 19392 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250112 15:06:17.832422 19392 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20250112 15:06:17.832456 19392 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20250112 15:06:19.832619 19392 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250112 15:06:19.832744 19392 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20250112 15:06:19.832782 19392 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20250112 15:06:20.095249 17651 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250112 15:06:20.098045 19668 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250112 15:06:20.098819 19671 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250112 15:06:20.099996 19669 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250112 15:06:20.100637 17651 server_base.cc:1034] running on GCE node
I20250112 15:06:20.100777 17651 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250112 15:06:20.100807 17651 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250112 15:06:20.100827 17651 hybrid_clock.cc:648] HybridClock initialized: now 1736694380100827 us; error 0 us; skew 500 ppm
I20250112 15:06:20.101786 17651 webserver.cc:458] Webserver started at http://127.17.60.196:34835/ using document root <none> and password file <none>
I20250112 15:06:20.101893 17651 fs_manager.cc:362] Metadata directory not provided
I20250112 15:06:20.101948 17651 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250112 15:06:20.102001 17651 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250112 15:06:20.102330 17651 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task95dLSD/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1736694351070047-17651-0/minicluster-data/ts-3-root/instance:
uuid: "51ea1f349eda43d0b0f5b76ef7bcd824"
format_stamp: "Formatted at 2025-01-12 15:06:20 on dist-test-slave-b4nz"
I20250112 15:06:20.103210 17651 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20250112 15:06:20.103649 19676 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250112 15:06:20.103773 17651 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20250112 15:06:20.103828 17651 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task95dLSD/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1736694351070047-17651-0/minicluster-data/ts-3-root
uuid: "51ea1f349eda43d0b0f5b76ef7bcd824"
format_stamp: "Formatted at 2025-01-12 15:06:20 on dist-test-slave-b4nz"
I20250112 15:06:20.103873 17651 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task95dLSD/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1736694351070047-17651-0/minicluster-data/ts-3-root
metadata directory: /tmp/dist-test-task95dLSD/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1736694351070047-17651-0/minicluster-data/ts-3-root
1 data directories: /tmp/dist-test-task95dLSD/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1736694351070047-17651-0/minicluster-data/ts-3-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250112 15:06:20.114554 17651 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250112 15:06:20.114748 17651 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250112 15:06:20.114900 17651 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250112 15:06:20.115101 17651 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250112 15:06:20.115135 17651 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250112 15:06:20.115165 17651 ts_tablet_manager.cc:610] Registered 0 tablets
I20250112 15:06:20.115190 17651 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250112 15:06:20.118445 17651 rpc_server.cc:307] RPC server started. Bound to: 127.17.60.196:35303
I20250112 15:06:20.118487 19739 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.17.60.196:35303 every 8 connection(s)
I20250112 15:06:20.121497 19740 heartbeater.cc:346] Connected to a master server at 127.17.60.254:40313
I20250112 15:06:20.121556 19740 heartbeater.cc:463] Registering TS with master...
I20250112 15:06:20.121656 19740 heartbeater.cc:510] Master 127.17.60.254:40313 requested a full tablet report, sending...
I20250112 15:06:20.121852 19336 ts_manager.cc:194] Registered new tserver with Master: 51ea1f349eda43d0b0f5b76ef7bcd824 (127.17.60.196:35303)
I20250112 15:06:20.122206 19336 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:34664
I20250112 15:06:20.836380 19592 consensus_queue.cc:237] T 06335cb4d4ff431dab85d2b78c33232b P 016fc71013aa4c6c8bb6e23194529062 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "7bfe4b6f206a49759e8069c99e17dd71" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 33177 } } peers { permanent_uuid: "af0fcb91bcd440e6939ea80ba1548866" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 37915 } attrs { replace: true } } peers { permanent_uuid: "016fc71013aa4c6c8bb6e23194529062" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 40923 } } peers { permanent_uuid: "51ea1f349eda43d0b0f5b76ef7bcd824" member_type: NON_VOTER last_known_addr { host: "127.17.60.196" port: 35303 } attrs { promote: true } }
I20250112 15:06:20.837126 19441 raft_consensus.cc:1270] T 06335cb4d4ff431dab85d2b78c33232b P 7bfe4b6f206a49759e8069c99e17dd71 [term 1 FOLLOWER]: Refusing update from remote peer 016fc71013aa4c6c8bb6e23194529062: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250112 15:06:20.837199 19517 raft_consensus.cc:1270] T 06335cb4d4ff431dab85d2b78c33232b P af0fcb91bcd440e6939ea80ba1548866 [term 1 FOLLOWER]: Refusing update from remote peer 016fc71013aa4c6c8bb6e23194529062: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250112 15:06:20.837414 19748 consensus_queue.cc:1035] T 06335cb4d4ff431dab85d2b78c33232b P 016fc71013aa4c6c8bb6e23194529062 [LEADER]: Connected to new peer: Peer: permanent_uuid: "af0fcb91bcd440e6939ea80ba1548866" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 37915 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250112 15:06:20.837586 19747 consensus_queue.cc:1035] T 06335cb4d4ff431dab85d2b78c33232b P 016fc71013aa4c6c8bb6e23194529062 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7bfe4b6f206a49759e8069c99e17dd71" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 33177 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250112 15:06:20.838335 19666 raft_consensus.cc:2949] T 06335cb4d4ff431dab85d2b78c33232b P 016fc71013aa4c6c8bb6e23194529062 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 51ea1f349eda43d0b0f5b76ef7bcd824 (127.17.60.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "7bfe4b6f206a49759e8069c99e17dd71" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 33177 } } peers { permanent_uuid: "af0fcb91bcd440e6939ea80ba1548866" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 37915 } attrs { replace: true } } peers { permanent_uuid: "016fc71013aa4c6c8bb6e23194529062" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 40923 } } peers { permanent_uuid: "51ea1f349eda43d0b0f5b76ef7bcd824" member_type: NON_VOTER last_known_addr { host: "127.17.60.196" port: 35303 } attrs { promote: true } } }
I20250112 15:06:20.838446 19517 raft_consensus.cc:2949] T 06335cb4d4ff431dab85d2b78c33232b P af0fcb91bcd440e6939ea80ba1548866 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 51ea1f349eda43d0b0f5b76ef7bcd824 (127.17.60.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "7bfe4b6f206a49759e8069c99e17dd71" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 33177 } } peers { permanent_uuid: "af0fcb91bcd440e6939ea80ba1548866" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 37915 } attrs { replace: true } } peers { permanent_uuid: "016fc71013aa4c6c8bb6e23194529062" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 40923 } } peers { permanent_uuid: "51ea1f349eda43d0b0f5b76ef7bcd824" member_type: NON_VOTER last_known_addr { host: "127.17.60.196" port: 35303 } attrs { promote: true } } }
I20250112 15:06:20.839525 19336 catalog_manager.cc:5526] T 06335cb4d4ff431dab85d2b78c33232b P 016fc71013aa4c6c8bb6e23194529062 reported cstate change: config changed from index -1 to 2, NON_VOTER 51ea1f349eda43d0b0f5b76ef7bcd824 (127.17.60.196) added. New cstate: current_term: 1 leader_uuid: "016fc71013aa4c6c8bb6e23194529062" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "7bfe4b6f206a49759e8069c99e17dd71" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 33177 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "af0fcb91bcd440e6939ea80ba1548866" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 37915 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "016fc71013aa4c6c8bb6e23194529062" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 40923 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "51ea1f349eda43d0b0f5b76ef7bcd824" member_type: NON_VOTER last_known_addr { host: "127.17.60.196" port: 35303 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20250112 15:06:20.839919 19441 raft_consensus.cc:2949] T 06335cb4d4ff431dab85d2b78c33232b P 7bfe4b6f206a49759e8069c99e17dd71 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 51ea1f349eda43d0b0f5b76ef7bcd824 (127.17.60.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "7bfe4b6f206a49759e8069c99e17dd71" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 33177 } } peers { permanent_uuid: "af0fcb91bcd440e6939ea80ba1548866" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 37915 } attrs { replace: true } } peers { permanent_uuid: "016fc71013aa4c6c8bb6e23194529062" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 40923 } } peers { permanent_uuid: "51ea1f349eda43d0b0f5b76ef7bcd824" member_type: NON_VOTER last_known_addr { host: "127.17.60.196" port: 35303 } attrs { promote: true } } }
W20250112 15:06:20.841018 19556 consensus_peers.cc:487] T 06335cb4d4ff431dab85d2b78c33232b P 016fc71013aa4c6c8bb6e23194529062 -> Peer 51ea1f349eda43d0b0f5b76ef7bcd824 (127.17.60.196:35303): Couldn't send request to peer 51ea1f349eda43d0b0f5b76ef7bcd824. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 06335cb4d4ff431dab85d2b78c33232b. This is attempt 1: this message will repeat every 5th retry.
I20250112 15:06:21.122669 19740 heartbeater.cc:502] Master 127.17.60.254:40313 was elected leader, sending a full tablet report...
I20250112 15:06:21.391999 19758 ts_tablet_manager.cc:927] T 06335cb4d4ff431dab85d2b78c33232b P 51ea1f349eda43d0b0f5b76ef7bcd824: Initiating tablet copy from peer 016fc71013aa4c6c8bb6e23194529062 (127.17.60.195:40923)
I20250112 15:06:21.392208 19758 tablet_copy_client.cc:323] T 06335cb4d4ff431dab85d2b78c33232b P 51ea1f349eda43d0b0f5b76ef7bcd824: tablet copy: Beginning tablet copy session from remote peer at address 127.17.60.195:40923
I20250112 15:06:21.394958 19602 tablet_copy_service.cc:140] P 016fc71013aa4c6c8bb6e23194529062: Received BeginTabletCopySession request for tablet 06335cb4d4ff431dab85d2b78c33232b from peer 51ea1f349eda43d0b0f5b76ef7bcd824 ({username='slave'} at 127.0.0.1:33438)
I20250112 15:06:21.395037 19602 tablet_copy_service.cc:161] P 016fc71013aa4c6c8bb6e23194529062: Beginning new tablet copy session on tablet 06335cb4d4ff431dab85d2b78c33232b from peer 51ea1f349eda43d0b0f5b76ef7bcd824 at {username='slave'} at 127.0.0.1:33438: session id = 51ea1f349eda43d0b0f5b76ef7bcd824-06335cb4d4ff431dab85d2b78c33232b
I20250112 15:06:21.395709 19602 tablet_copy_source_session.cc:215] T 06335cb4d4ff431dab85d2b78c33232b P 016fc71013aa4c6c8bb6e23194529062: Tablet Copy: opened 0 blocks and 1 log segments
I20250112 15:06:21.396104 19758 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 06335cb4d4ff431dab85d2b78c33232b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 15:06:21.397557 19758 tablet_copy_client.cc:806] T 06335cb4d4ff431dab85d2b78c33232b P 51ea1f349eda43d0b0f5b76ef7bcd824: tablet copy: Starting download of 0 data blocks...
I20250112 15:06:21.397682 19758 tablet_copy_client.cc:670] T 06335cb4d4ff431dab85d2b78c33232b P 51ea1f349eda43d0b0f5b76ef7bcd824: tablet copy: Starting download of 1 WAL segments...
I20250112 15:06:21.398154 19758 tablet_copy_client.cc:538] T 06335cb4d4ff431dab85d2b78c33232b P 51ea1f349eda43d0b0f5b76ef7bcd824: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250112 15:06:21.398877 19758 tablet_bootstrap.cc:492] T 06335cb4d4ff431dab85d2b78c33232b P 51ea1f349eda43d0b0f5b76ef7bcd824: Bootstrap starting.
I20250112 15:06:21.400445 19758 tablet_bootstrap.cc:492] T 06335cb4d4ff431dab85d2b78c33232b P 51ea1f349eda43d0b0f5b76ef7bcd824: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250112 15:06:21.400658 19758 tablet_bootstrap.cc:492] T 06335cb4d4ff431dab85d2b78c33232b P 51ea1f349eda43d0b0f5b76ef7bcd824: Bootstrap complete.
I20250112 15:06:21.400755 19758 ts_tablet_manager.cc:1397] T 06335cb4d4ff431dab85d2b78c33232b P 51ea1f349eda43d0b0f5b76ef7bcd824: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20250112 15:06:21.400933 19758 raft_consensus.cc:357] T 06335cb4d4ff431dab85d2b78c33232b P 51ea1f349eda43d0b0f5b76ef7bcd824 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "7bfe4b6f206a49759e8069c99e17dd71" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 33177 } } peers { permanent_uuid: "af0fcb91bcd440e6939ea80ba1548866" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 37915 } attrs { replace: true } } peers { permanent_uuid: "016fc71013aa4c6c8bb6e23194529062" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 40923 } } peers { permanent_uuid: "51ea1f349eda43d0b0f5b76ef7bcd824" member_type: NON_VOTER last_known_addr { host: "127.17.60.196" port: 35303 } attrs { promote: true } }
I20250112 15:06:21.401026 19758 raft_consensus.cc:738] T 06335cb4d4ff431dab85d2b78c33232b P 51ea1f349eda43d0b0f5b76ef7bcd824 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 51ea1f349eda43d0b0f5b76ef7bcd824, State: Initialized, Role: LEARNER
I20250112 15:06:21.401075 19758 consensus_queue.cc:260] T 06335cb4d4ff431dab85d2b78c33232b P 51ea1f349eda43d0b0f5b76ef7bcd824 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "7bfe4b6f206a49759e8069c99e17dd71" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 33177 } } peers { permanent_uuid: "af0fcb91bcd440e6939ea80ba1548866" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 37915 } attrs { replace: true } } peers { permanent_uuid: "016fc71013aa4c6c8bb6e23194529062" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 40923 } } peers { permanent_uuid: "51ea1f349eda43d0b0f5b76ef7bcd824" member_type: NON_VOTER last_known_addr { host: "127.17.60.196" port: 35303 } attrs { promote: true } }
I20250112 15:06:21.401242 19758 ts_tablet_manager.cc:1428] T 06335cb4d4ff431dab85d2b78c33232b P 51ea1f349eda43d0b0f5b76ef7bcd824: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20250112 15:06:21.401418 19602 tablet_copy_service.cc:342] P 016fc71013aa4c6c8bb6e23194529062: Request end of tablet copy session 51ea1f349eda43d0b0f5b76ef7bcd824-06335cb4d4ff431dab85d2b78c33232b received from {username='slave'} at 127.0.0.1:33438
I20250112 15:06:21.401476 19602 tablet_copy_service.cc:434] P 016fc71013aa4c6c8bb6e23194529062: ending tablet copy session 51ea1f349eda43d0b0f5b76ef7bcd824-06335cb4d4ff431dab85d2b78c33232b on tablet 06335cb4d4ff431dab85d2b78c33232b with peer 51ea1f349eda43d0b0f5b76ef7bcd824
I20250112 15:06:21.832903 19392 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250112 15:06:21.833037 19392 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20250112 15:06:21.833079 19392 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20250112 15:06:22.002897 19714 raft_consensus.cc:1212] T 06335cb4d4ff431dab85d2b78c33232b P 51ea1f349eda43d0b0f5b76ef7bcd824 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20250112 15:06:22.617923 19766 raft_consensus.cc:1059] T 06335cb4d4ff431dab85d2b78c33232b P 016fc71013aa4c6c8bb6e23194529062: attempting to promote NON_VOTER 51ea1f349eda43d0b0f5b76ef7bcd824 to VOTER
I20250112 15:06:22.618180 19766 consensus_queue.cc:237] T 06335cb4d4ff431dab85d2b78c33232b P 016fc71013aa4c6c8bb6e23194529062 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "7bfe4b6f206a49759e8069c99e17dd71" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 33177 } } peers { permanent_uuid: "af0fcb91bcd440e6939ea80ba1548866" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 37915 } attrs { replace: true } } peers { permanent_uuid: "016fc71013aa4c6c8bb6e23194529062" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 40923 } } peers { permanent_uuid: "51ea1f349eda43d0b0f5b76ef7bcd824" member_type: VOTER last_known_addr { host: "127.17.60.196" port: 35303 } attrs { promote: false } }
I20250112 15:06:22.618660 19714 raft_consensus.cc:1270] T 06335cb4d4ff431dab85d2b78c33232b P 51ea1f349eda43d0b0f5b76ef7bcd824 [term 1 LEARNER]: Refusing update from remote peer 016fc71013aa4c6c8bb6e23194529062: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250112 15:06:22.618665 19441 raft_consensus.cc:1270] T 06335cb4d4ff431dab85d2b78c33232b P 7bfe4b6f206a49759e8069c99e17dd71 [term 1 FOLLOWER]: Refusing update from remote peer 016fc71013aa4c6c8bb6e23194529062: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250112 15:06:22.618665 19517 raft_consensus.cc:1270] T 06335cb4d4ff431dab85d2b78c33232b P af0fcb91bcd440e6939ea80ba1548866 [term 1 FOLLOWER]: Refusing update from remote peer 016fc71013aa4c6c8bb6e23194529062: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250112 15:06:22.618886 19768 consensus_queue.cc:1035] T 06335cb4d4ff431dab85d2b78c33232b P 016fc71013aa4c6c8bb6e23194529062 [LEADER]: Connected to new peer: Peer: permanent_uuid: "51ea1f349eda43d0b0f5b76ef7bcd824" member_type: VOTER last_known_addr { host: "127.17.60.196" port: 35303 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250112 15:06:22.619040 19766 consensus_queue.cc:1035] T 06335cb4d4ff431dab85d2b78c33232b P 016fc71013aa4c6c8bb6e23194529062 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7bfe4b6f206a49759e8069c99e17dd71" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 33177 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250112 15:06:22.619174 19764 consensus_queue.cc:1035] T 06335cb4d4ff431dab85d2b78c33232b P 016fc71013aa4c6c8bb6e23194529062 [LEADER]: Connected to new peer: Peer: permanent_uuid: "af0fcb91bcd440e6939ea80ba1548866" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 37915 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250112 15:06:22.619946 19768 raft_consensus.cc:2949] T 06335cb4d4ff431dab85d2b78c33232b P 016fc71013aa4c6c8bb6e23194529062 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 51ea1f349eda43d0b0f5b76ef7bcd824 (127.17.60.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "7bfe4b6f206a49759e8069c99e17dd71" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 33177 } } peers { permanent_uuid: "af0fcb91bcd440e6939ea80ba1548866" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 37915 } attrs { replace: true } } peers { permanent_uuid: "016fc71013aa4c6c8bb6e23194529062" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 40923 } } peers { permanent_uuid: "51ea1f349eda43d0b0f5b76ef7bcd824" member_type: VOTER last_known_addr { host: "127.17.60.196" port: 35303 } attrs { promote: false } } }
I20250112 15:06:22.620061 19714 raft_consensus.cc:2949] T 06335cb4d4ff431dab85d2b78c33232b P 51ea1f349eda43d0b0f5b76ef7bcd824 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 51ea1f349eda43d0b0f5b76ef7bcd824 (127.17.60.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "7bfe4b6f206a49759e8069c99e17dd71" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 33177 } } peers { permanent_uuid: "af0fcb91bcd440e6939ea80ba1548866" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 37915 } attrs { replace: true } } peers { permanent_uuid: "016fc71013aa4c6c8bb6e23194529062" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 40923 } } peers { permanent_uuid: "51ea1f349eda43d0b0f5b76ef7bcd824" member_type: VOTER last_known_addr { host: "127.17.60.196" port: 35303 } attrs { promote: false } } }
I20250112 15:06:22.620071 19517 raft_consensus.cc:2949] T 06335cb4d4ff431dab85d2b78c33232b P af0fcb91bcd440e6939ea80ba1548866 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 51ea1f349eda43d0b0f5b76ef7bcd824 (127.17.60.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "7bfe4b6f206a49759e8069c99e17dd71" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 33177 } } peers { permanent_uuid: "af0fcb91bcd440e6939ea80ba1548866" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 37915 } attrs { replace: true } } peers { permanent_uuid: "016fc71013aa4c6c8bb6e23194529062" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 40923 } } peers { permanent_uuid: "51ea1f349eda43d0b0f5b76ef7bcd824" member_type: VOTER last_known_addr { host: "127.17.60.196" port: 35303 } attrs { promote: false } } }
I20250112 15:06:22.621220 19441 raft_consensus.cc:2949] T 06335cb4d4ff431dab85d2b78c33232b P 7bfe4b6f206a49759e8069c99e17dd71 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 51ea1f349eda43d0b0f5b76ef7bcd824 (127.17.60.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "7bfe4b6f206a49759e8069c99e17dd71" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 33177 } } peers { permanent_uuid: "af0fcb91bcd440e6939ea80ba1548866" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 37915 } attrs { replace: true } } peers { permanent_uuid: "016fc71013aa4c6c8bb6e23194529062" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 40923 } } peers { permanent_uuid: "51ea1f349eda43d0b0f5b76ef7bcd824" member_type: VOTER last_known_addr { host: "127.17.60.196" port: 35303 } attrs { promote: false } } }
I20250112 15:06:22.621352 19336 catalog_manager.cc:5526] T 06335cb4d4ff431dab85d2b78c33232b P 51ea1f349eda43d0b0f5b76ef7bcd824 reported cstate change: config changed from index 2 to 3, 51ea1f349eda43d0b0f5b76ef7bcd824 (127.17.60.196) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "016fc71013aa4c6c8bb6e23194529062" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "7bfe4b6f206a49759e8069c99e17dd71" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 33177 } } peers { permanent_uuid: "af0fcb91bcd440e6939ea80ba1548866" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 37915 } attrs { replace: true } } peers { permanent_uuid: "016fc71013aa4c6c8bb6e23194529062" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 40923 } } peers { permanent_uuid: "51ea1f349eda43d0b0f5b76ef7bcd824" member_type: VOTER last_known_addr { host: "127.17.60.196" port: 35303 } attrs { promote: false } } }
I20250112 15:06:22.622895 19592 consensus_queue.cc:237] T 06335cb4d4ff431dab85d2b78c33232b P 016fc71013aa4c6c8bb6e23194529062 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7bfe4b6f206a49759e8069c99e17dd71" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 33177 } } peers { permanent_uuid: "016fc71013aa4c6c8bb6e23194529062" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 40923 } } peers { permanent_uuid: "51ea1f349eda43d0b0f5b76ef7bcd824" member_type: VOTER last_known_addr { host: "127.17.60.196" port: 35303 } attrs { promote: false } }
I20250112 15:06:22.623214 19441 raft_consensus.cc:1270] T 06335cb4d4ff431dab85d2b78c33232b P 7bfe4b6f206a49759e8069c99e17dd71 [term 1 FOLLOWER]: Refusing update from remote peer 016fc71013aa4c6c8bb6e23194529062: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20250112 15:06:22.623229 19714 raft_consensus.cc:1270] T 06335cb4d4ff431dab85d2b78c33232b P 51ea1f349eda43d0b0f5b76ef7bcd824 [term 1 FOLLOWER]: Refusing update from remote peer 016fc71013aa4c6c8bb6e23194529062: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20250112 15:06:22.623447 19766 consensus_queue.cc:1035] T 06335cb4d4ff431dab85d2b78c33232b P 016fc71013aa4c6c8bb6e23194529062 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7bfe4b6f206a49759e8069c99e17dd71" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 33177 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250112 15:06:22.623569 19766 consensus_queue.cc:1035] T 06335cb4d4ff431dab85d2b78c33232b P 016fc71013aa4c6c8bb6e23194529062 [LEADER]: Connected to new peer: Peer: permanent_uuid: "51ea1f349eda43d0b0f5b76ef7bcd824" member_type: VOTER last_known_addr { host: "127.17.60.196" port: 35303 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250112 15:06:22.623952 19766 raft_consensus.cc:2949] T 06335cb4d4ff431dab85d2b78c33232b P 016fc71013aa4c6c8bb6e23194529062 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER af0fcb91bcd440e6939ea80ba1548866 (127.17.60.194) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7bfe4b6f206a49759e8069c99e17dd71" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 33177 } } peers { permanent_uuid: "016fc71013aa4c6c8bb6e23194529062" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 40923 } } peers { permanent_uuid: "51ea1f349eda43d0b0f5b76ef7bcd824" member_type: VOTER last_known_addr { host: "127.17.60.196" port: 35303 } attrs { promote: false } } }
I20250112 15:06:22.624063 19441 raft_consensus.cc:2949] T 06335cb4d4ff431dab85d2b78c33232b P 7bfe4b6f206a49759e8069c99e17dd71 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER af0fcb91bcd440e6939ea80ba1548866 (127.17.60.194) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7bfe4b6f206a49759e8069c99e17dd71" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 33177 } } peers { permanent_uuid: "016fc71013aa4c6c8bb6e23194529062" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 40923 } } peers { permanent_uuid: "51ea1f349eda43d0b0f5b76ef7bcd824" member_type: VOTER last_known_addr { host: "127.17.60.196" port: 35303 } attrs { promote: false } } }
I20250112 15:06:22.624197 19714 raft_consensus.cc:2949] T 06335cb4d4ff431dab85d2b78c33232b P 51ea1f349eda43d0b0f5b76ef7bcd824 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER af0fcb91bcd440e6939ea80ba1548866 (127.17.60.194) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7bfe4b6f206a49759e8069c99e17dd71" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 33177 } } peers { permanent_uuid: "016fc71013aa4c6c8bb6e23194529062" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 40923 } } peers { permanent_uuid: "51ea1f349eda43d0b0f5b76ef7bcd824" member_type: VOTER last_known_addr { host: "127.17.60.196" port: 35303 } attrs { promote: false } } }
I20250112 15:06:22.624977 19324 catalog_manager.cc:5039] ChangeConfig:REMOVE_PEER RPC for tablet 06335cb4d4ff431dab85d2b78c33232b with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250112 15:06:22.625036 19337 catalog_manager.cc:5526] T 06335cb4d4ff431dab85d2b78c33232b P 7bfe4b6f206a49759e8069c99e17dd71 reported cstate change: config changed from index 3 to 4, VOTER af0fcb91bcd440e6939ea80ba1548866 (127.17.60.194) evicted. New cstate: current_term: 1 leader_uuid: "016fc71013aa4c6c8bb6e23194529062" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7bfe4b6f206a49759e8069c99e17dd71" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 33177 } } peers { permanent_uuid: "016fc71013aa4c6c8bb6e23194529062" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 40923 } } peers { permanent_uuid: "51ea1f349eda43d0b0f5b76ef7bcd824" member_type: VOTER last_known_addr { host: "127.17.60.196" port: 35303 } attrs { promote: false } } }
I20250112 15:06:22.625828 19506 tablet_service.cc:1514] Processing DeleteTablet for tablet 06335cb4d4ff431dab85d2b78c33232b with delete_type TABLET_DATA_TOMBSTONED (TS af0fcb91bcd440e6939ea80ba1548866 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:60066
I20250112 15:06:22.626039 19778 tablet_replica.cc:331] T 06335cb4d4ff431dab85d2b78c33232b P af0fcb91bcd440e6939ea80ba1548866: stopping tablet replica
I20250112 15:06:22.626097 19778 raft_consensus.cc:2238] T 06335cb4d4ff431dab85d2b78c33232b P af0fcb91bcd440e6939ea80ba1548866 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250112 15:06:22.626138 19778 raft_consensus.cc:2267] T 06335cb4d4ff431dab85d2b78c33232b P af0fcb91bcd440e6939ea80ba1548866 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 15:06:22.626353 19778 ts_tablet_manager.cc:1905] T 06335cb4d4ff431dab85d2b78c33232b P af0fcb91bcd440e6939ea80ba1548866: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250112 15:06:22.627460 19778 ts_tablet_manager.cc:1918] T 06335cb4d4ff431dab85d2b78c33232b P af0fcb91bcd440e6939ea80ba1548866: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20250112 15:06:22.627511 19778 log.cc:1198] T 06335cb4d4ff431dab85d2b78c33232b P af0fcb91bcd440e6939ea80ba1548866: Deleting WAL directory at /tmp/dist-test-task95dLSD/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1736694351070047-17651-0/minicluster-data/ts-1-root/wals/06335cb4d4ff431dab85d2b78c33232b
I20250112 15:06:22.627799 19322 catalog_manager.cc:4872] TS af0fcb91bcd440e6939ea80ba1548866 (127.17.60.194:37915): tablet 06335cb4d4ff431dab85d2b78c33232b (table test-workload [id=e644f70925e04891b2eabd97597a927a]) successfully deleted
I20250112 15:06:23.833274 19392 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250112 15:06:23.833418 19392 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20250112 15:06:23.833460 19392 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20250112 15:06:25.833633 19392 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250112 15:06:25.833766 19392 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20250112 15:06:25.833802 19392 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20250112 15:06:27.833979 19392 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250112 15:06:27.834117 19392 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20250112 15:06:27.834141 19392 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20250112 15:06:29.834295 19392 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250112 15:06:29.834434 19392 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20250112 15:06:29.834460 19392 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20250112 15:06:31.834646 19392 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250112 15:06:31.834801 19392 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20250112 15:06:31.834818 19392 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20250112 15:06:33.834966 19392 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250112 15:06:33.835096 19392 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20250112 15:06:33.835114 19392 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20250112 15:06:34.456923 17651 tablet_server.cc:178] TabletServer@127.17.60.193:0 shutting down...
I20250112 15:06:34.460213 17651 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250112 15:06:34.460335 17651 tablet_replica.cc:331] T 32c0a6c93d7f45efa313d196c5f0562c P 7bfe4b6f206a49759e8069c99e17dd71: stopping tablet replica
I20250112 15:06:34.460407 17651 raft_consensus.cc:2238] T 32c0a6c93d7f45efa313d196c5f0562c P 7bfe4b6f206a49759e8069c99e17dd71 [term 1 LEADER]: Raft consensus shutting down.
I20250112 15:06:34.460472 17651 raft_consensus.cc:2267] T 32c0a6c93d7f45efa313d196c5f0562c P 7bfe4b6f206a49759e8069c99e17dd71 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 15:06:34.460677 17651 tablet_replica.cc:331] T 06335cb4d4ff431dab85d2b78c33232b P 7bfe4b6f206a49759e8069c99e17dd71: stopping tablet replica
I20250112 15:06:34.460717 17651 raft_consensus.cc:2238] T 06335cb4d4ff431dab85d2b78c33232b P 7bfe4b6f206a49759e8069c99e17dd71 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250112 15:06:34.460767 17651 raft_consensus.cc:2267] T 06335cb4d4ff431dab85d2b78c33232b P 7bfe4b6f206a49759e8069c99e17dd71 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 15:06:34.473405 17651 tablet_server.cc:195] TabletServer@127.17.60.193:0 shutdown complete.
I20250112 15:06:34.474561 17651 tablet_server.cc:178] TabletServer@127.17.60.194:0 shutting down...
I20250112 15:06:34.477869 17651 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250112 15:06:34.478009 17651 tablet_replica.cc:331] T 32c0a6c93d7f45efa313d196c5f0562c P af0fcb91bcd440e6939ea80ba1548866: stopping tablet replica
I20250112 15:06:34.478050 17651 raft_consensus.cc:2238] T 32c0a6c93d7f45efa313d196c5f0562c P af0fcb91bcd440e6939ea80ba1548866 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250112 15:06:34.478096 17651 raft_consensus.cc:2267] T 32c0a6c93d7f45efa313d196c5f0562c P af0fcb91bcd440e6939ea80ba1548866 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 15:06:34.489722 17651 tablet_server.cc:195] TabletServer@127.17.60.194:0 shutdown complete.
I20250112 15:06:34.490615 17651 tablet_server.cc:178] TabletServer@127.17.60.195:0 shutting down...
I20250112 15:06:34.493774 17651 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250112 15:06:34.493901 17651 tablet_replica.cc:331] T 06335cb4d4ff431dab85d2b78c33232b P 016fc71013aa4c6c8bb6e23194529062: stopping tablet replica
I20250112 15:06:34.493986 17651 raft_consensus.cc:2238] T 06335cb4d4ff431dab85d2b78c33232b P 016fc71013aa4c6c8bb6e23194529062 [term 1 LEADER]: Raft consensus shutting down.
I20250112 15:06:34.494047 17651 raft_consensus.cc:2267] T 06335cb4d4ff431dab85d2b78c33232b P 016fc71013aa4c6c8bb6e23194529062 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 15:06:34.494182 17651 tablet_replica.cc:331] T 32c0a6c93d7f45efa313d196c5f0562c P 016fc71013aa4c6c8bb6e23194529062: stopping tablet replica
I20250112 15:06:34.494210 17651 raft_consensus.cc:2238] T 32c0a6c93d7f45efa313d196c5f0562c P 016fc71013aa4c6c8bb6e23194529062 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250112 15:06:34.494227 17651 raft_consensus.cc:2267] T 32c0a6c93d7f45efa313d196c5f0562c P 016fc71013aa4c6c8bb6e23194529062 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 15:06:34.505650 17651 tablet_server.cc:195] TabletServer@127.17.60.195:0 shutdown complete.
I20250112 15:06:34.506481 17651 tablet_server.cc:178] TabletServer@127.17.60.196:0 shutting down...
I20250112 15:06:34.509553 17651 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250112 15:06:34.509665 17651 tablet_replica.cc:331] T 06335cb4d4ff431dab85d2b78c33232b P 51ea1f349eda43d0b0f5b76ef7bcd824: stopping tablet replica
I20250112 15:06:34.509728 17651 raft_consensus.cc:2238] T 06335cb4d4ff431dab85d2b78c33232b P 51ea1f349eda43d0b0f5b76ef7bcd824 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250112 15:06:34.509778 17651 raft_consensus.cc:2267] T 06335cb4d4ff431dab85d2b78c33232b P 51ea1f349eda43d0b0f5b76ef7bcd824 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 15:06:34.521171 17651 tablet_server.cc:195] TabletServer@127.17.60.196:0 shutdown complete.
I20250112 15:06:34.521973 17651 master.cc:537] Master@127.17.60.254:40313 shutting down...
I20250112 15:06:34.524454 17651 raft_consensus.cc:2238] T 00000000000000000000000000000000 P 2f88df4aab60498494c42bdc47544b41 [term 1 LEADER]: Raft consensus shutting down.
I20250112 15:06:34.524523 17651 raft_consensus.cc:2267] T 00000000000000000000000000000000 P 2f88df4aab60498494c42bdc47544b41 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 15:06:34.524542 17651 tablet_replica.cc:331] T 00000000000000000000000000000000 P 2f88df4aab60498494c42bdc47544b41: stopping tablet replica
I20250112 15:06:34.536217 17651 master.cc:559] Master@127.17.60.254:40313 shutdown complete.
[ OK ] AutoRebalancerTest.MovesScheduledIfAddTserver (26740 ms)
[ RUN ] AutoRebalancerTest.NoReplicaMovesIfNoTservers
I20250112 15:06:34.541332 17651 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.17.60.254:37177
I20250112 15:06:34.541451 17651 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250112 15:06:34.543509 19781 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250112 15:06:34.544334 17651 server_base.cc:1034] running on GCE node
W20250112 15:06:34.544991 19784 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250112 15:06:34.546033 19782 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250112 15:06:34.546152 17651 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250112 15:06:34.546188 17651 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250112 15:06:34.546200 17651 hybrid_clock.cc:648] HybridClock initialized: now 1736694394546201 us; error 0 us; skew 500 ppm
I20250112 15:06:34.554140 17651 webserver.cc:458] Webserver started at http://127.17.60.254:41519/ using document root <none> and password file <none>
I20250112 15:06:34.554227 17651 fs_manager.cc:362] Metadata directory not provided
I20250112 15:06:34.554261 17651 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250112 15:06:34.554294 17651 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250112 15:06:34.554528 17651 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task95dLSD/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTservers.1736694351070047-17651-0/minicluster-data/master-0-root/instance:
uuid: "6e716b590c474d79ba0a4fae84acc587"
format_stamp: "Formatted at 2025-01-12 15:06:34 on dist-test-slave-b4nz"
I20250112 15:06:34.555315 17651 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20250112 15:06:34.555689 19789 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250112 15:06:34.555806 17651 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20250112 15:06:34.555853 17651 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task95dLSD/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTservers.1736694351070047-17651-0/minicluster-data/master-0-root
uuid: "6e716b590c474d79ba0a4fae84acc587"
format_stamp: "Formatted at 2025-01-12 15:06:34 on dist-test-slave-b4nz"
I20250112 15:06:34.555891 17651 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task95dLSD/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTservers.1736694351070047-17651-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-task95dLSD/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTservers.1736694351070047-17651-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-task95dLSD/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTservers.1736694351070047-17651-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250112 15:06:34.566833 17651 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250112 15:06:34.567016 17651 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250112 15:06:34.569545 17651 rpc_server.cc:307] RPC server started. Bound to: 127.17.60.254:37177
I20250112 15:06:34.571894 19841 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.17.60.254:37177 every 8 connection(s)
I20250112 15:06:34.572024 19842 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 15:06:34.573060 19842 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6e716b590c474d79ba0a4fae84acc587: Bootstrap starting.
I20250112 15:06:34.573351 19842 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6e716b590c474d79ba0a4fae84acc587: Neither blocks nor log segments found. Creating new log.
I20250112 15:06:34.573791 19842 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6e716b590c474d79ba0a4fae84acc587: No bootstrap required, opened a new log
I20250112 15:06:34.573905 19842 raft_consensus.cc:357] T 00000000000000000000000000000000 P 6e716b590c474d79ba0a4fae84acc587 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6e716b590c474d79ba0a4fae84acc587" member_type: VOTER }
I20250112 15:06:34.573971 19842 raft_consensus.cc:383] T 00000000000000000000000000000000 P 6e716b590c474d79ba0a4fae84acc587 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 15:06:34.573999 19842 raft_consensus.cc:738] T 00000000000000000000000000000000 P 6e716b590c474d79ba0a4fae84acc587 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6e716b590c474d79ba0a4fae84acc587, State: Initialized, Role: FOLLOWER
I20250112 15:06:34.574054 19842 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6e716b590c474d79ba0a4fae84acc587 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6e716b590c474d79ba0a4fae84acc587" member_type: VOTER }
I20250112 15:06:34.574097 19842 raft_consensus.cc:397] T 00000000000000000000000000000000 P 6e716b590c474d79ba0a4fae84acc587 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250112 15:06:34.574121 19842 raft_consensus.cc:491] T 00000000000000000000000000000000 P 6e716b590c474d79ba0a4fae84acc587 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250112 15:06:34.574151 19842 raft_consensus.cc:3054] T 00000000000000000000000000000000 P 6e716b590c474d79ba0a4fae84acc587 [term 0 FOLLOWER]: Advancing to term 1
I20250112 15:06:34.574587 19842 raft_consensus.cc:513] T 00000000000000000000000000000000 P 6e716b590c474d79ba0a4fae84acc587 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6e716b590c474d79ba0a4fae84acc587" member_type: VOTER }
I20250112 15:06:34.574638 19842 leader_election.cc:304] T 00000000000000000000000000000000 P 6e716b590c474d79ba0a4fae84acc587 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6e716b590c474d79ba0a4fae84acc587; no voters:
I20250112 15:06:34.574730 19842 leader_election.cc:290] T 00000000000000000000000000000000 P 6e716b590c474d79ba0a4fae84acc587 [CANDIDATE]: Term 1 election: Requested vote from peers
I20250112 15:06:34.574774 19845 raft_consensus.cc:2798] T 00000000000000000000000000000000 P 6e716b590c474d79ba0a4fae84acc587 [term 1 FOLLOWER]: Leader election won for term 1
I20250112 15:06:34.574868 19842 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6e716b590c474d79ba0a4fae84acc587 [sys.catalog]: configured and running, proceeding with master startup.
I20250112 15:06:34.574949 19845 raft_consensus.cc:695] T 00000000000000000000000000000000 P 6e716b590c474d79ba0a4fae84acc587 [term 1 LEADER]: Becoming Leader. State: Replica: 6e716b590c474d79ba0a4fae84acc587, State: Running, Role: LEADER
I20250112 15:06:34.575008 19845 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6e716b590c474d79ba0a4fae84acc587 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6e716b590c474d79ba0a4fae84acc587" member_type: VOTER }
I20250112 15:06:34.575220 19846 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6e716b590c474d79ba0a4fae84acc587 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6e716b590c474d79ba0a4fae84acc587" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6e716b590c474d79ba0a4fae84acc587" member_type: VOTER } }
I20250112 15:06:34.575243 19847 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6e716b590c474d79ba0a4fae84acc587 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6e716b590c474d79ba0a4fae84acc587. Latest consensus state: current_term: 1 leader_uuid: "6e716b590c474d79ba0a4fae84acc587" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6e716b590c474d79ba0a4fae84acc587" member_type: VOTER } }
I20250112 15:06:34.575296 19846 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6e716b590c474d79ba0a4fae84acc587 [sys.catalog]: This master's current role is: LEADER
I20250112 15:06:34.575300 19847 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6e716b590c474d79ba0a4fae84acc587 [sys.catalog]: This master's current role is: LEADER
I20250112 15:06:34.575547 19851 catalog_manager.cc:1476] Loading table and tablet metadata into memory...
I20250112 15:06:34.575726 19851 catalog_manager.cc:1485] Initializing Kudu cluster ID...
I20250112 15:06:34.576126 17651 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20250112 15:06:34.576256 19851 catalog_manager.cc:1348] Generated new cluster ID: cfa6239fcc974d028c56a55f174e6601
I20250112 15:06:34.576295 19851 catalog_manager.cc:1496] Initializing Kudu internal certificate authority...
I20250112 15:06:34.580780 19851 catalog_manager.cc:1371] Generated new certificate authority record
I20250112 15:06:34.581135 19851 catalog_manager.cc:1505] Loading token signing keys...
I20250112 15:06:34.586005 19851 catalog_manager.cc:5899] T 00000000000000000000000000000000 P 6e716b590c474d79ba0a4fae84acc587: Generated new TSK 0
I20250112 15:06:34.586073 19851 catalog_manager.cc:1515] Initializing in-progress tserver states...
I20250112 15:06:34.591605 17651 internal_mini_cluster.cc:371] 0 TS(s) registered with all masters after 4.0699e-05s
I20250112 15:06:46.667747 17651 master.cc:537] Master@127.17.60.254:37177 shutting down...
I20250112 15:06:46.670184 17651 raft_consensus.cc:2238] T 00000000000000000000000000000000 P 6e716b590c474d79ba0a4fae84acc587 [term 1 LEADER]: Raft consensus shutting down.
I20250112 15:06:46.670275 17651 raft_consensus.cc:2267] T 00000000000000000000000000000000 P 6e716b590c474d79ba0a4fae84acc587 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 15:06:46.670296 17651 tablet_replica.cc:331] T 00000000000000000000000000000000 P 6e716b590c474d79ba0a4fae84acc587: stopping tablet replica
I20250112 15:06:46.682756 17651 master.cc:559] Master@127.17.60.254:37177 shutdown complete.
[ OK ] AutoRebalancerTest.NoReplicaMovesIfNoTservers (12144 ms)
[ RUN ] AutoRebalancerTest.NoReplicaMovesIfNoTablets
I20250112 15:06:46.685460 17651 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.17.60.254:39079
I20250112 15:06:46.685606 17651 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250112 15:06:46.687626 19865 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250112 15:06:46.688210 19867 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250112 15:06:46.688683 19864 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250112 15:06:46.689353 17651 server_base.cc:1034] running on GCE node
I20250112 15:06:46.689435 17651 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250112 15:06:46.689460 17651 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250112 15:06:46.689471 17651 hybrid_clock.cc:648] HybridClock initialized: now 1736694406689472 us; error 0 us; skew 500 ppm
I20250112 15:06:46.694788 17651 webserver.cc:458] Webserver started at http://127.17.60.254:33959/ using document root <none> and password file <none>
I20250112 15:06:46.694875 17651 fs_manager.cc:362] Metadata directory not provided
I20250112 15:06:46.694911 17651 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250112 15:06:46.694942 17651 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250112 15:06:46.695188 17651 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task95dLSD/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1736694351070047-17651-0/minicluster-data/master-0-root/instance:
uuid: "f556cbdb44744d629db3edc53b3b4819"
format_stamp: "Formatted at 2025-01-12 15:06:46 on dist-test-slave-b4nz"
I20250112 15:06:46.696004 17651 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20250112 15:06:46.696434 19872 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250112 15:06:46.696554 17651 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20250112 15:06:46.696610 17651 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task95dLSD/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1736694351070047-17651-0/minicluster-data/master-0-root
uuid: "f556cbdb44744d629db3edc53b3b4819"
format_stamp: "Formatted at 2025-01-12 15:06:46 on dist-test-slave-b4nz"
I20250112 15:06:46.696651 17651 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task95dLSD/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1736694351070047-17651-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-task95dLSD/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1736694351070047-17651-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-task95dLSD/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1736694351070047-17651-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250112 15:06:46.701835 17651 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250112 15:06:46.702004 17651 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250112 15:06:46.705173 17651 rpc_server.cc:307] RPC server started. Bound to: 127.17.60.254:39079
I20250112 15:06:46.705703 19924 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.17.60.254:39079 every 8 connection(s)
I20250112 15:06:46.705832 19925 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 15:06:46.706760 19925 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f556cbdb44744d629db3edc53b3b4819: Bootstrap starting.
I20250112 15:06:46.707041 19925 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f556cbdb44744d629db3edc53b3b4819: Neither blocks nor log segments found. Creating new log.
I20250112 15:06:46.707469 19925 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f556cbdb44744d629db3edc53b3b4819: No bootstrap required, opened a new log
I20250112 15:06:46.707564 19925 raft_consensus.cc:357] T 00000000000000000000000000000000 P f556cbdb44744d629db3edc53b3b4819 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f556cbdb44744d629db3edc53b3b4819" member_type: VOTER }
I20250112 15:06:46.707610 19925 raft_consensus.cc:383] T 00000000000000000000000000000000 P f556cbdb44744d629db3edc53b3b4819 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 15:06:46.707623 19925 raft_consensus.cc:738] T 00000000000000000000000000000000 P f556cbdb44744d629db3edc53b3b4819 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f556cbdb44744d629db3edc53b3b4819, State: Initialized, Role: FOLLOWER
I20250112 15:06:46.707657 19925 consensus_queue.cc:260] T 00000000000000000000000000000000 P f556cbdb44744d629db3edc53b3b4819 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f556cbdb44744d629db3edc53b3b4819" member_type: VOTER }
I20250112 15:06:46.707698 19925 raft_consensus.cc:397] T 00000000000000000000000000000000 P f556cbdb44744d629db3edc53b3b4819 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250112 15:06:46.707719 19925 raft_consensus.cc:491] T 00000000000000000000000000000000 P f556cbdb44744d629db3edc53b3b4819 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250112 15:06:46.707738 19925 raft_consensus.cc:3054] T 00000000000000000000000000000000 P f556cbdb44744d629db3edc53b3b4819 [term 0 FOLLOWER]: Advancing to term 1
I20250112 15:06:46.708158 19925 raft_consensus.cc:513] T 00000000000000000000000000000000 P f556cbdb44744d629db3edc53b3b4819 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f556cbdb44744d629db3edc53b3b4819" member_type: VOTER }
I20250112 15:06:46.708222 19925 leader_election.cc:304] T 00000000000000000000000000000000 P f556cbdb44744d629db3edc53b3b4819 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f556cbdb44744d629db3edc53b3b4819; no voters:
I20250112 15:06:46.708307 19925 leader_election.cc:290] T 00000000000000000000000000000000 P f556cbdb44744d629db3edc53b3b4819 [CANDIDATE]: Term 1 election: Requested vote from peers
I20250112 15:06:46.708372 19928 raft_consensus.cc:2798] T 00000000000000000000000000000000 P f556cbdb44744d629db3edc53b3b4819 [term 1 FOLLOWER]: Leader election won for term 1
I20250112 15:06:46.708431 19925 sys_catalog.cc:564] T 00000000000000000000000000000000 P f556cbdb44744d629db3edc53b3b4819 [sys.catalog]: configured and running, proceeding with master startup.
I20250112 15:06:46.708501 19928 raft_consensus.cc:695] T 00000000000000000000000000000000 P f556cbdb44744d629db3edc53b3b4819 [term 1 LEADER]: Becoming Leader. State: Replica: f556cbdb44744d629db3edc53b3b4819, State: Running, Role: LEADER
I20250112 15:06:46.708546 19928 consensus_queue.cc:237] T 00000000000000000000000000000000 P f556cbdb44744d629db3edc53b3b4819 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f556cbdb44744d629db3edc53b3b4819" member_type: VOTER }
I20250112 15:06:46.708827 19929 sys_catalog.cc:455] T 00000000000000000000000000000000 P f556cbdb44744d629db3edc53b3b4819 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f556cbdb44744d629db3edc53b3b4819" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f556cbdb44744d629db3edc53b3b4819" member_type: VOTER } }
I20250112 15:06:46.708844 19930 sys_catalog.cc:455] T 00000000000000000000000000000000 P f556cbdb44744d629db3edc53b3b4819 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f556cbdb44744d629db3edc53b3b4819. Latest consensus state: current_term: 1 leader_uuid: "f556cbdb44744d629db3edc53b3b4819" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f556cbdb44744d629db3edc53b3b4819" member_type: VOTER } }
I20250112 15:06:46.708935 19929 sys_catalog.cc:458] T 00000000000000000000000000000000 P f556cbdb44744d629db3edc53b3b4819 [sys.catalog]: This master's current role is: LEADER
I20250112 15:06:46.709151 19930 sys_catalog.cc:458] T 00000000000000000000000000000000 P f556cbdb44744d629db3edc53b3b4819 [sys.catalog]: This master's current role is: LEADER
I20250112 15:06:46.709345 19938 catalog_manager.cc:1476] Loading table and tablet metadata into memory...
I20250112 15:06:46.709481 19938 catalog_manager.cc:1485] Initializing Kudu cluster ID...
I20250112 15:06:46.709749 17651 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20250112 15:06:46.710054 19938 catalog_manager.cc:1348] Generated new cluster ID: cfd7bb3607394ed5b979723237ca4c73
I20250112 15:06:46.710095 19938 catalog_manager.cc:1496] Initializing Kudu internal certificate authority...
I20250112 15:06:46.722846 19938 catalog_manager.cc:1371] Generated new certificate authority record
I20250112 15:06:46.723217 19938 catalog_manager.cc:1505] Loading token signing keys...
I20250112 15:06:46.726543 19938 catalog_manager.cc:5899] T 00000000000000000000000000000000 P f556cbdb44744d629db3edc53b3b4819: Generated new TSK 0
I20250112 15:06:46.726624 19938 catalog_manager.cc:1515] Initializing in-progress tserver states...
I20250112 15:06:46.741555 17651 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250112 15:06:46.743520 19947 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250112 15:06:46.744596 19949 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250112 15:06:46.744819 19946 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250112 15:06:46.745638 17651 server_base.cc:1034] running on GCE node
I20250112 15:06:46.745731 17651 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250112 15:06:46.745757 17651 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250112 15:06:46.745779 17651 hybrid_clock.cc:648] HybridClock initialized: now 1736694406745778 us; error 0 us; skew 500 ppm
I20250112 15:06:46.746423 17651 webserver.cc:458] Webserver started at http://127.17.60.193:35767/ using document root <none> and password file <none>
I20250112 15:06:46.746510 17651 fs_manager.cc:362] Metadata directory not provided
I20250112 15:06:46.746565 17651 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250112 15:06:46.746623 17651 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250112 15:06:46.746907 17651 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task95dLSD/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1736694351070047-17651-0/minicluster-data/ts-0-root/instance:
uuid: "46a2da94ffb7463bbe0617b80bade982"
format_stamp: "Formatted at 2025-01-12 15:06:46 on dist-test-slave-b4nz"
I20250112 15:06:46.747742 17651 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20250112 15:06:46.748171 19954 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250112 15:06:46.748296 17651 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20250112 15:06:46.748342 17651 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task95dLSD/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1736694351070047-17651-0/minicluster-data/ts-0-root
uuid: "46a2da94ffb7463bbe0617b80bade982"
format_stamp: "Formatted at 2025-01-12 15:06:46 on dist-test-slave-b4nz"
I20250112 15:06:46.748391 17651 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task95dLSD/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1736694351070047-17651-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-task95dLSD/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1736694351070047-17651-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-task95dLSD/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1736694351070047-17651-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250112 15:06:46.765539 17651 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250112 15:06:46.765774 17651 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250112 15:06:46.765920 17651 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250112 15:06:46.766139 17651 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250112 15:06:46.766166 17651 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250112 15:06:46.766201 17651 ts_tablet_manager.cc:610] Registered 0 tablets
I20250112 15:06:46.766230 17651 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250112 15:06:46.768913 17651 rpc_server.cc:307] RPC server started. Bound to: 127.17.60.193:44927
I20250112 15:06:46.768983 20017 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.17.60.193:44927 every 8 connection(s)
I20250112 15:06:46.769301 17651 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250112 15:06:46.771001 20025 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250112 15:06:46.771423 20022 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250112 15:06:46.772190 20023 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250112 15:06:46.772789 20018 heartbeater.cc:346] Connected to a master server at 127.17.60.254:39079
I20250112 15:06:46.772809 17651 server_base.cc:1034] running on GCE node
I20250112 15:06:46.772868 20018 heartbeater.cc:463] Registering TS with master...
I20250112 15:06:46.772943 17651 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250112 15:06:46.772974 17651 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250112 15:06:46.772991 20018 heartbeater.cc:510] Master 127.17.60.254:39079 requested a full tablet report, sending...
I20250112 15:06:46.773000 17651 hybrid_clock.cc:648] HybridClock initialized: now 1736694406773001 us; error 0 us; skew 500 ppm
I20250112 15:06:46.773241 19889 ts_manager.cc:194] Registered new tserver with Master: 46a2da94ffb7463bbe0617b80bade982 (127.17.60.193:44927)
I20250112 15:06:46.773679 17651 webserver.cc:458] Webserver started at http://127.17.60.194:42655/ using document root <none> and password file <none>
I20250112 15:06:46.773773 17651 fs_manager.cc:362] Metadata directory not provided
I20250112 15:06:46.773814 17651 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250112 15:06:46.773854 17651 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250112 15:06:46.774011 19889 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:51794
I20250112 15:06:46.774173 17651 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task95dLSD/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1736694351070047-17651-0/minicluster-data/ts-1-root/instance:
uuid: "71e5651c4e92461a968e92966e1db48a"
format_stamp: "Formatted at 2025-01-12 15:06:46 on dist-test-slave-b4nz"
I20250112 15:06:46.775022 17651 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20250112 15:06:46.775458 20030 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250112 15:06:46.775579 17651 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20250112 15:06:46.775619 17651 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task95dLSD/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1736694351070047-17651-0/minicluster-data/ts-1-root
uuid: "71e5651c4e92461a968e92966e1db48a"
format_stamp: "Formatted at 2025-01-12 15:06:46 on dist-test-slave-b4nz"
I20250112 15:06:46.775664 17651 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task95dLSD/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1736694351070047-17651-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-task95dLSD/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1736694351070047-17651-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-task95dLSD/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1736694351070047-17651-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250112 15:06:46.802626 17651 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250112 15:06:46.802886 17651 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250112 15:06:46.803040 17651 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250112 15:06:46.803252 17651 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250112 15:06:46.803282 17651 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250112 15:06:46.803313 17651 ts_tablet_manager.cc:610] Registered 0 tablets
I20250112 15:06:46.803344 17651 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250112 15:06:46.806161 17651 rpc_server.cc:307] RPC server started. Bound to: 127.17.60.194:42635
I20250112 15:06:46.806216 20093 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.17.60.194:42635 every 8 connection(s)
I20250112 15:06:46.806529 17651 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250112 15:06:46.807979 20097 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250112 15:06:46.808815 20098 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250112 15:06:46.809593 20100 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250112 15:06:46.809723 20094 heartbeater.cc:346] Connected to a master server at 127.17.60.254:39079
I20250112 15:06:46.809783 20094 heartbeater.cc:463] Registering TS with master...
I20250112 15:06:46.809899 20094 heartbeater.cc:510] Master 127.17.60.254:39079 requested a full tablet report, sending...
I20250112 15:06:46.810138 19889 ts_manager.cc:194] Registered new tserver with Master: 71e5651c4e92461a968e92966e1db48a (127.17.60.194:42635)
I20250112 15:06:46.810403 17651 server_base.cc:1034] running on GCE node
I20250112 15:06:46.810518 17651 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250112 15:06:46.810544 17651 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250112 15:06:46.810565 17651 hybrid_clock.cc:648] HybridClock initialized: now 1736694406810566 us; error 0 us; skew 500 ppm
I20250112 15:06:46.810694 19889 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:51800
I20250112 15:06:46.811261 17651 webserver.cc:458] Webserver started at http://127.17.60.195:44351/ using document root <none> and password file <none>
I20250112 15:06:46.811353 17651 fs_manager.cc:362] Metadata directory not provided
I20250112 15:06:46.811400 17651 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250112 15:06:46.811452 17651 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250112 15:06:46.811710 17651 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task95dLSD/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1736694351070047-17651-0/minicluster-data/ts-2-root/instance:
uuid: "cfb00f7e0c6f4bd4a305dc9b4dfa15e6"
format_stamp: "Formatted at 2025-01-12 15:06:46 on dist-test-slave-b4nz"
I20250112 15:06:46.812631 17651 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20250112 15:06:46.813104 20105 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250112 15:06:46.813239 17651 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20250112 15:06:46.813282 17651 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task95dLSD/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1736694351070047-17651-0/minicluster-data/ts-2-root
uuid: "cfb00f7e0c6f4bd4a305dc9b4dfa15e6"
format_stamp: "Formatted at 2025-01-12 15:06:46 on dist-test-slave-b4nz"
I20250112 15:06:46.813339 17651 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task95dLSD/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1736694351070047-17651-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-task95dLSD/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1736694351070047-17651-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-task95dLSD/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1736694351070047-17651-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250112 15:06:46.826314 17651 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250112 15:06:46.826551 17651 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250112 15:06:46.826706 17651 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250112 15:06:46.826892 17651 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250112 15:06:46.826921 17651 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250112 15:06:46.826953 17651 ts_tablet_manager.cc:610] Registered 0 tablets
I20250112 15:06:46.826979 17651 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250112 15:06:46.830188 17651 rpc_server.cc:307] RPC server started. Bound to: 127.17.60.195:44321
I20250112 15:06:46.830267 20168 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.17.60.195:44321 every 8 connection(s)
I20250112 15:06:46.833211 20169 heartbeater.cc:346] Connected to a master server at 127.17.60.254:39079
I20250112 15:06:46.833277 20169 heartbeater.cc:463] Registering TS with master...
I20250112 15:06:46.833382 20169 heartbeater.cc:510] Master 127.17.60.254:39079 requested a full tablet report, sending...
I20250112 15:06:46.833617 19889 ts_manager.cc:194] Registered new tserver with Master: cfb00f7e0c6f4bd4a305dc9b4dfa15e6 (127.17.60.195:44321)
I20250112 15:06:46.833716 17651 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.003329635s
I20250112 15:06:46.833993 19889 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:51810
I20250112 15:06:47.774544 20018 heartbeater.cc:502] Master 127.17.60.254:39079 was elected leader, sending a full tablet report...
I20250112 15:06:47.811244 20094 heartbeater.cc:502] Master 127.17.60.254:39079 was elected leader, sending a full tablet report...
I20250112 15:06:47.834484 20169 heartbeater.cc:502] Master 127.17.60.254:39079 was elected leader, sending a full tablet report...
I20250112 15:06:48.709826 19943 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20250112 15:06:50.710052 19943 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20250112 15:06:52.710251 19943 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20250112 15:06:54.710458 19943 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20250112 15:06:56.710664 19943 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20250112 15:06:58.710875 19943 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20250112 15:06:58.911190 17651 tablet_server.cc:178] TabletServer@127.17.60.193:0 shutting down...
I20250112 15:06:58.914359 17651 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250112 15:06:58.925806 17651 tablet_server.cc:195] TabletServer@127.17.60.193:0 shutdown complete.
I20250112 15:06:58.926673 17651 tablet_server.cc:178] TabletServer@127.17.60.194:0 shutting down...
I20250112 15:06:58.929623 17651 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250112 15:06:58.941131 17651 tablet_server.cc:195] TabletServer@127.17.60.194:0 shutdown complete.
I20250112 15:06:58.941888 17651 tablet_server.cc:178] TabletServer@127.17.60.195:0 shutting down...
I20250112 15:06:58.944491 17651 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250112 15:06:58.956010 17651 tablet_server.cc:195] TabletServer@127.17.60.195:0 shutdown complete.
I20250112 15:06:58.956851 17651 master.cc:537] Master@127.17.60.254:39079 shutting down...
I20250112 15:06:58.960609 17651 raft_consensus.cc:2238] T 00000000000000000000000000000000 P f556cbdb44744d629db3edc53b3b4819 [term 1 LEADER]: Raft consensus shutting down.
I20250112 15:06:58.960695 17651 raft_consensus.cc:2267] T 00000000000000000000000000000000 P f556cbdb44744d629db3edc53b3b4819 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 15:06:58.960762 17651 tablet_replica.cc:331] T 00000000000000000000000000000000 P f556cbdb44744d629db3edc53b3b4819: stopping tablet replica
I20250112 15:06:58.973341 17651 master.cc:559] Master@127.17.60.254:39079 shutdown complete.
[ OK ] AutoRebalancerTest.NoReplicaMovesIfNoTablets (12291 ms)
[ RUN ] AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne
I20250112 15:06:58.976702 17651 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.17.60.254:37135
I20250112 15:06:58.976877 17651 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20250112 15:06:58.979110 17651 server_base.cc:1034] running on GCE node
W20250112 15:06:58.979411 20173 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250112 15:06:58.980005 20176 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250112 15:06:58.981069 20174 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250112 15:06:58.981259 17651 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250112 15:06:58.981293 17651 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250112 15:06:58.981305 17651 hybrid_clock.cc:648] HybridClock initialized: now 1736694418981305 us; error 0 us; skew 500 ppm
I20250112 15:06:58.981973 17651 webserver.cc:458] Webserver started at http://127.17.60.254:46517/ using document root <none> and password file <none>
I20250112 15:06:58.982069 17651 fs_manager.cc:362] Metadata directory not provided
I20250112 15:06:58.982117 17651 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250112 15:06:58.982172 17651 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250112 15:06:58.982448 17651 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task95dLSD/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1736694351070047-17651-0/minicluster-data/master-0-root/instance:
uuid: "ee58d62a60fa44fc80db045024b87e6a"
format_stamp: "Formatted at 2025-01-12 15:06:58 on dist-test-slave-b4nz"
I20250112 15:06:58.983285 17651 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20250112 15:06:58.983717 20181 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250112 15:06:58.983841 17651 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20250112 15:06:58.983898 17651 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task95dLSD/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1736694351070047-17651-0/minicluster-data/master-0-root
uuid: "ee58d62a60fa44fc80db045024b87e6a"
format_stamp: "Formatted at 2025-01-12 15:06:58 on dist-test-slave-b4nz"
I20250112 15:06:58.983954 17651 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task95dLSD/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1736694351070047-17651-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-task95dLSD/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1736694351070047-17651-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-task95dLSD/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1736694351070047-17651-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250112 15:06:58.997893 17651 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250112 15:06:58.998109 17651 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250112 15:06:59.000761 17651 rpc_server.cc:307] RPC server started. Bound to: 127.17.60.254:37135
I20250112 15:06:59.001915 20233 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.17.60.254:37135 every 8 connection(s)
I20250112 15:06:59.002038 20234 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 15:06:59.003000 20234 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ee58d62a60fa44fc80db045024b87e6a: Bootstrap starting.
I20250112 15:06:59.003322 20234 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ee58d62a60fa44fc80db045024b87e6a: Neither blocks nor log segments found. Creating new log.
I20250112 15:06:59.003775 20234 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ee58d62a60fa44fc80db045024b87e6a: No bootstrap required, opened a new log
I20250112 15:06:59.003898 20234 raft_consensus.cc:357] T 00000000000000000000000000000000 P ee58d62a60fa44fc80db045024b87e6a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ee58d62a60fa44fc80db045024b87e6a" member_type: VOTER }
I20250112 15:06:59.003950 20234 raft_consensus.cc:383] T 00000000000000000000000000000000 P ee58d62a60fa44fc80db045024b87e6a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 15:06:59.003963 20234 raft_consensus.cc:738] T 00000000000000000000000000000000 P ee58d62a60fa44fc80db045024b87e6a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ee58d62a60fa44fc80db045024b87e6a, State: Initialized, Role: FOLLOWER
I20250112 15:06:59.004014 20234 consensus_queue.cc:260] T 00000000000000000000000000000000 P ee58d62a60fa44fc80db045024b87e6a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ee58d62a60fa44fc80db045024b87e6a" member_type: VOTER }
I20250112 15:06:59.004060 20234 raft_consensus.cc:397] T 00000000000000000000000000000000 P ee58d62a60fa44fc80db045024b87e6a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250112 15:06:59.004073 20234 raft_consensus.cc:491] T 00000000000000000000000000000000 P ee58d62a60fa44fc80db045024b87e6a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250112 15:06:59.004101 20234 raft_consensus.cc:3054] T 00000000000000000000000000000000 P ee58d62a60fa44fc80db045024b87e6a [term 0 FOLLOWER]: Advancing to term 1
I20250112 15:06:59.004626 20234 raft_consensus.cc:513] T 00000000000000000000000000000000 P ee58d62a60fa44fc80db045024b87e6a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ee58d62a60fa44fc80db045024b87e6a" member_type: VOTER }
I20250112 15:06:59.004683 20234 leader_election.cc:304] T 00000000000000000000000000000000 P ee58d62a60fa44fc80db045024b87e6a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ee58d62a60fa44fc80db045024b87e6a; no voters:
I20250112 15:06:59.004791 20234 leader_election.cc:290] T 00000000000000000000000000000000 P ee58d62a60fa44fc80db045024b87e6a [CANDIDATE]: Term 1 election: Requested vote from peers
I20250112 15:06:59.004841 20237 raft_consensus.cc:2798] T 00000000000000000000000000000000 P ee58d62a60fa44fc80db045024b87e6a [term 1 FOLLOWER]: Leader election won for term 1
I20250112 15:06:59.004928 20234 sys_catalog.cc:564] T 00000000000000000000000000000000 P ee58d62a60fa44fc80db045024b87e6a [sys.catalog]: configured and running, proceeding with master startup.
I20250112 15:06:59.005010 20237 raft_consensus.cc:695] T 00000000000000000000000000000000 P ee58d62a60fa44fc80db045024b87e6a [term 1 LEADER]: Becoming Leader. State: Replica: ee58d62a60fa44fc80db045024b87e6a, State: Running, Role: LEADER
I20250112 15:06:59.005069 20237 consensus_queue.cc:237] T 00000000000000000000000000000000 P ee58d62a60fa44fc80db045024b87e6a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ee58d62a60fa44fc80db045024b87e6a" member_type: VOTER }
I20250112 15:06:59.005308 20239 sys_catalog.cc:455] T 00000000000000000000000000000000 P ee58d62a60fa44fc80db045024b87e6a [sys.catalog]: SysCatalogTable state changed. Reason: New leader ee58d62a60fa44fc80db045024b87e6a. Latest consensus state: current_term: 1 leader_uuid: "ee58d62a60fa44fc80db045024b87e6a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ee58d62a60fa44fc80db045024b87e6a" member_type: VOTER } }
I20250112 15:06:59.005297 20238 sys_catalog.cc:455] T 00000000000000000000000000000000 P ee58d62a60fa44fc80db045024b87e6a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ee58d62a60fa44fc80db045024b87e6a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ee58d62a60fa44fc80db045024b87e6a" member_type: VOTER } }
I20250112 15:06:59.005432 20239 sys_catalog.cc:458] T 00000000000000000000000000000000 P ee58d62a60fa44fc80db045024b87e6a [sys.catalog]: This master's current role is: LEADER
I20250112 15:06:59.005458 20238 sys_catalog.cc:458] T 00000000000000000000000000000000 P ee58d62a60fa44fc80db045024b87e6a [sys.catalog]: This master's current role is: LEADER
I20250112 15:06:59.005642 20245 catalog_manager.cc:1476] Loading table and tablet metadata into memory...
I20250112 15:06:59.005803 20245 catalog_manager.cc:1485] Initializing Kudu cluster ID...
I20250112 15:06:59.006266 20245 catalog_manager.cc:1348] Generated new cluster ID: 728ab2354c7747c8947238890bd0092c
I20250112 15:06:59.006248 17651 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20250112 15:06:59.006333 20245 catalog_manager.cc:1496] Initializing Kudu internal certificate authority...
I20250112 15:06:59.013240 20245 catalog_manager.cc:1371] Generated new certificate authority record
I20250112 15:06:59.013698 20245 catalog_manager.cc:1505] Loading token signing keys...
I20250112 15:06:59.020390 20245 catalog_manager.cc:5899] T 00000000000000000000000000000000 P ee58d62a60fa44fc80db045024b87e6a: Generated new TSK 0
I20250112 15:06:59.020471 20245 catalog_manager.cc:1515] Initializing in-progress tserver states...
I20250112 15:06:59.021899 17651 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250112 15:06:59.023468 20255 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250112 15:06:59.024438 20258 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250112 15:06:59.025207 20256 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250112 15:06:59.025976 17651 server_base.cc:1034] running on GCE node
I20250112 15:06:59.026074 17651 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250112 15:06:59.026104 17651 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250112 15:06:59.026115 17651 hybrid_clock.cc:648] HybridClock initialized: now 1736694419026116 us; error 0 us; skew 500 ppm
I20250112 15:06:59.026736 17651 webserver.cc:458] Webserver started at http://127.17.60.193:33409/ using document root <none> and password file <none>
I20250112 15:06:59.026806 17651 fs_manager.cc:362] Metadata directory not provided
I20250112 15:06:59.026840 17651 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250112 15:06:59.026872 17651 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250112 15:06:59.027109 17651 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task95dLSD/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1736694351070047-17651-0/minicluster-data/ts-0-root/instance:
uuid: "25273da20e0442e9bacf9a876b831b02"
format_stamp: "Formatted at 2025-01-12 15:06:59 on dist-test-slave-b4nz"
I20250112 15:06:59.027869 17651 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20250112 15:06:59.028252 20263 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250112 15:06:59.028374 17651 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20250112 15:06:59.028422 17651 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task95dLSD/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1736694351070047-17651-0/minicluster-data/ts-0-root
uuid: "25273da20e0442e9bacf9a876b831b02"
format_stamp: "Formatted at 2025-01-12 15:06:59 on dist-test-slave-b4nz"
I20250112 15:06:59.028460 17651 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task95dLSD/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1736694351070047-17651-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-task95dLSD/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1736694351070047-17651-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-task95dLSD/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1736694351070047-17651-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250112 15:06:59.047369 17651 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250112 15:06:59.047595 17651 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250112 15:06:59.047727 17651 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250112 15:06:59.047919 17651 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250112 15:06:59.047955 17651 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250112 15:06:59.047978 17651 ts_tablet_manager.cc:610] Registered 0 tablets
I20250112 15:06:59.048005 17651 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250112 15:06:59.050750 17651 rpc_server.cc:307] RPC server started. Bound to: 127.17.60.193:37751
I20250112 15:06:59.050784 20326 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.17.60.193:37751 every 8 connection(s)
I20250112 15:06:59.051116 17651 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250112 15:06:59.052506 20331 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250112 15:06:59.053339 20332 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250112 15:06:59.054252 17651 server_base.cc:1034] running on GCE node
I20250112 15:06:59.054447 20327 heartbeater.cc:346] Connected to a master server at 127.17.60.254:37135
I20250112 15:06:59.054494 20327 heartbeater.cc:463] Registering TS with master...
I20250112 15:06:59.054589 20327 heartbeater.cc:510] Master 127.17.60.254:37135 requested a full tablet report, sending...
I20250112 15:06:59.054821 20198 ts_manager.cc:194] Registered new tserver with Master: 25273da20e0442e9bacf9a876b831b02 (127.17.60.193:37751)
W20250112 15:06:59.054961 20334 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250112 15:06:59.055096 17651 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250112 15:06:59.055140 17651 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250112 15:06:59.055157 17651 hybrid_clock.cc:648] HybridClock initialized: now 1736694419055157 us; error 0 us; skew 500 ppm
I20250112 15:06:59.055581 20198 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:44888
I20250112 15:06:59.055872 17651 webserver.cc:458] Webserver started at http://127.17.60.194:45053/ using document root <none> and password file <none>
I20250112 15:06:59.055959 17651 fs_manager.cc:362] Metadata directory not provided
I20250112 15:06:59.056006 17651 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250112 15:06:59.056058 17651 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250112 15:06:59.056324 17651 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task95dLSD/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1736694351070047-17651-0/minicluster-data/ts-1-root/instance:
uuid: "c1ae22c98d1842b892aed72776bd07d3"
format_stamp: "Formatted at 2025-01-12 15:06:59 on dist-test-slave-b4nz"
I20250112 15:06:59.057152 17651 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20250112 15:06:59.057576 20339 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250112 15:06:59.057705 17651 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20250112 15:06:59.057746 17651 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task95dLSD/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1736694351070047-17651-0/minicluster-data/ts-1-root
uuid: "c1ae22c98d1842b892aed72776bd07d3"
format_stamp: "Formatted at 2025-01-12 15:06:59 on dist-test-slave-b4nz"
I20250112 15:06:59.057792 17651 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task95dLSD/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1736694351070047-17651-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-task95dLSD/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1736694351070047-17651-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-task95dLSD/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1736694351070047-17651-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250112 15:06:59.065515 17651 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250112 15:06:59.065713 17651 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250112 15:06:59.065851 17651 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250112 15:06:59.066015 17651 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250112 15:06:59.066044 17651 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250112 15:06:59.066092 17651 ts_tablet_manager.cc:610] Registered 0 tablets
I20250112 15:06:59.066115 17651 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250112 15:06:59.068972 17651 rpc_server.cc:307] RPC server started. Bound to: 127.17.60.194:43523
I20250112 15:06:59.069020 20402 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.17.60.194:43523 every 8 connection(s)
I20250112 15:06:59.069315 17651 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250112 15:06:59.070642 20406 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250112 15:06:59.071394 20407 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250112 15:06:59.072213 20409 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250112 15:06:59.072273 20403 heartbeater.cc:346] Connected to a master server at 127.17.60.254:37135
I20250112 15:06:59.072353 20403 heartbeater.cc:463] Registering TS with master...
I20250112 15:06:59.072450 20403 heartbeater.cc:510] Master 127.17.60.254:37135 requested a full tablet report, sending...
I20250112 15:06:59.072665 20198 ts_manager.cc:194] Registered new tserver with Master: c1ae22c98d1842b892aed72776bd07d3 (127.17.60.194:43523)
I20250112 15:06:59.072816 17651 server_base.cc:1034] running on GCE node
I20250112 15:06:59.072907 17651 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250112 15:06:59.072935 17651 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250112 15:06:59.072952 17651 hybrid_clock.cc:648] HybridClock initialized: now 1736694419072951 us; error 0 us; skew 500 ppm
I20250112 15:06:59.073072 20198 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:44890
I20250112 15:06:59.073597 17651 webserver.cc:458] Webserver started at http://127.17.60.195:32947/ using document root <none> and password file <none>
I20250112 15:06:59.073685 17651 fs_manager.cc:362] Metadata directory not provided
I20250112 15:06:59.073735 17651 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250112 15:06:59.073781 17651 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250112 15:06:59.074048 17651 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task95dLSD/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1736694351070047-17651-0/minicluster-data/ts-2-root/instance:
uuid: "a6557b390b7f4af4899141316d97a17c"
format_stamp: "Formatted at 2025-01-12 15:06:59 on dist-test-slave-b4nz"
I20250112 15:06:59.074885 17651 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20250112 15:06:59.075284 20414 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250112 15:06:59.075398 17651 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20250112 15:06:59.075436 17651 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task95dLSD/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1736694351070047-17651-0/minicluster-data/ts-2-root
uuid: "a6557b390b7f4af4899141316d97a17c"
format_stamp: "Formatted at 2025-01-12 15:06:59 on dist-test-slave-b4nz"
I20250112 15:06:59.075480 17651 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task95dLSD/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1736694351070047-17651-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-task95dLSD/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1736694351070047-17651-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-task95dLSD/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1736694351070047-17651-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250112 15:06:59.088003 17651 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250112 15:06:59.088294 17651 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250112 15:06:59.088442 17651 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250112 15:06:59.088701 17651 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250112 15:06:59.088732 17651 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250112 15:06:59.088761 17651 ts_tablet_manager.cc:610] Registered 0 tablets
I20250112 15:06:59.088789 17651 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250112 15:06:59.091939 17651 rpc_server.cc:307] RPC server started. Bound to: 127.17.60.195:39411
I20250112 15:06:59.091996 20477 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.17.60.195:39411 every 8 connection(s)
I20250112 15:06:59.092321 17651 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250112 15:06:59.093761 20481 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250112 15:06:59.094388 20482 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250112 15:06:59.095036 20484 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250112 15:06:59.095296 17651 server_base.cc:1034] running on GCE node
I20250112 15:06:59.095425 17651 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
I20250112 15:06:59.095414 20478 heartbeater.cc:346] Connected to a master server at 127.17.60.254:37135
W20250112 15:06:59.095481 17651 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250112 15:06:59.095502 20478 heartbeater.cc:463] Registering TS with master...
I20250112 15:06:59.095504 17651 hybrid_clock.cc:648] HybridClock initialized: now 1736694419095505 us; error 0 us; skew 500 ppm
I20250112 15:06:59.095628 20478 heartbeater.cc:510] Master 127.17.60.254:37135 requested a full tablet report, sending...
I20250112 15:06:59.095836 20198 ts_manager.cc:194] Registered new tserver with Master: a6557b390b7f4af4899141316d97a17c (127.17.60.195:39411)
I20250112 15:06:59.096228 17651 webserver.cc:458] Webserver started at http://127.17.60.196:37147/ using document root <none> and password file <none>
I20250112 15:06:59.096275 20198 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:44892
I20250112 15:06:59.096304 17651 fs_manager.cc:362] Metadata directory not provided
I20250112 15:06:59.096369 17651 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250112 15:06:59.096428 17651 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250112 15:06:59.096755 17651 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task95dLSD/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1736694351070047-17651-0/minicluster-data/ts-3-root/instance:
uuid: "b22628b371a843b8be736c586961d0dd"
format_stamp: "Formatted at 2025-01-12 15:06:59 on dist-test-slave-b4nz"
I20250112 15:06:59.097633 17651 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20250112 15:06:59.098053 20489 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250112 15:06:59.098181 17651 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20250112 15:06:59.098222 17651 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task95dLSD/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1736694351070047-17651-0/minicluster-data/ts-3-root
uuid: "b22628b371a843b8be736c586961d0dd"
format_stamp: "Formatted at 2025-01-12 15:06:59 on dist-test-slave-b4nz"
I20250112 15:06:59.098271 17651 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task95dLSD/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1736694351070047-17651-0/minicluster-data/ts-3-root
metadata directory: /tmp/dist-test-task95dLSD/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1736694351070047-17651-0/minicluster-data/ts-3-root
1 data directories: /tmp/dist-test-task95dLSD/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1736694351070047-17651-0/minicluster-data/ts-3-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250112 15:06:59.109143 17651 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250112 15:06:59.109396 17651 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250112 15:06:59.109542 17651 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250112 15:06:59.109773 17651 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250112 15:06:59.109800 17651 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250112 15:06:59.109835 17651 ts_tablet_manager.cc:610] Registered 0 tablets
I20250112 15:06:59.109863 17651 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250112 15:06:59.113049 17651 rpc_server.cc:307] RPC server started. Bound to: 127.17.60.196:46165
I20250112 15:06:59.113101 20552 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.17.60.196:46165 every 8 connection(s)
I20250112 15:06:59.115983 20553 heartbeater.cc:346] Connected to a master server at 127.17.60.254:37135
I20250112 15:06:59.116046 20553 heartbeater.cc:463] Registering TS with master...
I20250112 15:06:59.116156 20553 heartbeater.cc:510] Master 127.17.60.254:37135 requested a full tablet report, sending...
I20250112 15:06:59.116405 20198 ts_manager.cc:194] Registered new tserver with Master: b22628b371a843b8be736c586961d0dd (127.17.60.196:46165)
I20250112 15:06:59.116542 17651 internal_mini_cluster.cc:371] 4 TS(s) registered with all masters after 0.00335003s
I20250112 15:06:59.116812 20198 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:44896
I20250112 15:07:00.056295 20327 heartbeater.cc:502] Master 127.17.60.254:37135 was elected leader, sending a full tablet report...
I20250112 15:07:00.073877 20403 heartbeater.cc:502] Master 127.17.60.254:37135 was elected leader, sending a full tablet report...
I20250112 15:07:00.096841 20478 heartbeater.cc:502] Master 127.17.60.254:37135 was elected leader, sending a full tablet report...
I20250112 15:07:00.117293 20553 heartbeater.cc:502] Master 127.17.60.254:37135 was elected leader, sending a full tablet report...
I20250112 15:07:00.144842 17651 test_util.cc:274] Using random seed: -1375796536
I20250112 15:07:00.149237 20198 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40246:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250112 15:07:00.154217 20517 tablet_service.cc:1467] Processing CreateTablet for tablet 828b0e3ee4684ecd9c48836ba876ac09 (DEFAULT_TABLE table=test-workload [id=dc7614e5f7ab48e39c2099ac7bbea988]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 15:07:00.154366 20517 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 828b0e3ee4684ecd9c48836ba876ac09. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 15:07:00.154517 20291 tablet_service.cc:1467] Processing CreateTablet for tablet 828b0e3ee4684ecd9c48836ba876ac09 (DEFAULT_TABLE table=test-workload [id=dc7614e5f7ab48e39c2099ac7bbea988]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 15:07:00.154613 20291 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 828b0e3ee4684ecd9c48836ba876ac09. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 15:07:00.155205 20367 tablet_service.cc:1467] Processing CreateTablet for tablet 828b0e3ee4684ecd9c48836ba876ac09 (DEFAULT_TABLE table=test-workload [id=dc7614e5f7ab48e39c2099ac7bbea988]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 15:07:00.155313 20367 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 828b0e3ee4684ecd9c48836ba876ac09. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 15:07:00.155841 20575 tablet_bootstrap.cc:492] T 828b0e3ee4684ecd9c48836ba876ac09 P b22628b371a843b8be736c586961d0dd: Bootstrap starting.
I20250112 15:07:00.156275 20575 tablet_bootstrap.cc:654] T 828b0e3ee4684ecd9c48836ba876ac09 P b22628b371a843b8be736c586961d0dd: Neither blocks nor log segments found. Creating new log.
I20250112 15:07:00.156437 20576 tablet_bootstrap.cc:492] T 828b0e3ee4684ecd9c48836ba876ac09 P c1ae22c98d1842b892aed72776bd07d3: Bootstrap starting.
I20250112 15:07:00.156670 20574 tablet_bootstrap.cc:492] T 828b0e3ee4684ecd9c48836ba876ac09 P 25273da20e0442e9bacf9a876b831b02: Bootstrap starting.
I20250112 15:07:00.156968 20576 tablet_bootstrap.cc:654] T 828b0e3ee4684ecd9c48836ba876ac09 P c1ae22c98d1842b892aed72776bd07d3: Neither blocks nor log segments found. Creating new log.
I20250112 15:07:00.157013 20575 tablet_bootstrap.cc:492] T 828b0e3ee4684ecd9c48836ba876ac09 P b22628b371a843b8be736c586961d0dd: No bootstrap required, opened a new log
I20250112 15:07:00.157088 20575 ts_tablet_manager.cc:1397] T 828b0e3ee4684ecd9c48836ba876ac09 P b22628b371a843b8be736c586961d0dd: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20250112 15:07:00.157120 20574 tablet_bootstrap.cc:654] T 828b0e3ee4684ecd9c48836ba876ac09 P 25273da20e0442e9bacf9a876b831b02: Neither blocks nor log segments found. Creating new log.
I20250112 15:07:00.157289 20575 raft_consensus.cc:357] T 828b0e3ee4684ecd9c48836ba876ac09 P b22628b371a843b8be736c586961d0dd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1ae22c98d1842b892aed72776bd07d3" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 43523 } } peers { permanent_uuid: "b22628b371a843b8be736c586961d0dd" member_type: VOTER last_known_addr { host: "127.17.60.196" port: 46165 } } peers { permanent_uuid: "25273da20e0442e9bacf9a876b831b02" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 37751 } }
I20250112 15:07:00.157397 20575 raft_consensus.cc:383] T 828b0e3ee4684ecd9c48836ba876ac09 P b22628b371a843b8be736c586961d0dd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 15:07:00.157433 20575 raft_consensus.cc:738] T 828b0e3ee4684ecd9c48836ba876ac09 P b22628b371a843b8be736c586961d0dd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b22628b371a843b8be736c586961d0dd, State: Initialized, Role: FOLLOWER
I20250112 15:07:00.157481 20575 consensus_queue.cc:260] T 828b0e3ee4684ecd9c48836ba876ac09 P b22628b371a843b8be736c586961d0dd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1ae22c98d1842b892aed72776bd07d3" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 43523 } } peers { permanent_uuid: "b22628b371a843b8be736c586961d0dd" member_type: VOTER last_known_addr { host: "127.17.60.196" port: 46165 } } peers { permanent_uuid: "25273da20e0442e9bacf9a876b831b02" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 37751 } }
I20250112 15:07:00.157613 20576 tablet_bootstrap.cc:492] T 828b0e3ee4684ecd9c48836ba876ac09 P c1ae22c98d1842b892aed72776bd07d3: No bootstrap required, opened a new log
I20250112 15:07:00.157662 20576 ts_tablet_manager.cc:1397] T 828b0e3ee4684ecd9c48836ba876ac09 P c1ae22c98d1842b892aed72776bd07d3: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20250112 15:07:00.157689 20575 ts_tablet_manager.cc:1428] T 828b0e3ee4684ecd9c48836ba876ac09 P b22628b371a843b8be736c586961d0dd: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20250112 15:07:00.157824 20576 raft_consensus.cc:357] T 828b0e3ee4684ecd9c48836ba876ac09 P c1ae22c98d1842b892aed72776bd07d3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1ae22c98d1842b892aed72776bd07d3" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 43523 } } peers { permanent_uuid: "b22628b371a843b8be736c586961d0dd" member_type: VOTER last_known_addr { host: "127.17.60.196" port: 46165 } } peers { permanent_uuid: "25273da20e0442e9bacf9a876b831b02" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 37751 } }
I20250112 15:07:00.157886 20576 raft_consensus.cc:383] T 828b0e3ee4684ecd9c48836ba876ac09 P c1ae22c98d1842b892aed72776bd07d3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 15:07:00.157907 20576 raft_consensus.cc:738] T 828b0e3ee4684ecd9c48836ba876ac09 P c1ae22c98d1842b892aed72776bd07d3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c1ae22c98d1842b892aed72776bd07d3, State: Initialized, Role: FOLLOWER
I20250112 15:07:00.157946 20576 consensus_queue.cc:260] T 828b0e3ee4684ecd9c48836ba876ac09 P c1ae22c98d1842b892aed72776bd07d3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1ae22c98d1842b892aed72776bd07d3" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 43523 } } peers { permanent_uuid: "b22628b371a843b8be736c586961d0dd" member_type: VOTER last_known_addr { host: "127.17.60.196" port: 46165 } } peers { permanent_uuid: "25273da20e0442e9bacf9a876b831b02" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 37751 } }
I20250112 15:07:00.158072 20576 ts_tablet_manager.cc:1428] T 828b0e3ee4684ecd9c48836ba876ac09 P c1ae22c98d1842b892aed72776bd07d3: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20250112 15:07:00.158380 20574 tablet_bootstrap.cc:492] T 828b0e3ee4684ecd9c48836ba876ac09 P 25273da20e0442e9bacf9a876b831b02: No bootstrap required, opened a new log
I20250112 15:07:00.158427 20574 ts_tablet_manager.cc:1397] T 828b0e3ee4684ecd9c48836ba876ac09 P 25273da20e0442e9bacf9a876b831b02: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20250112 15:07:00.158573 20574 raft_consensus.cc:357] T 828b0e3ee4684ecd9c48836ba876ac09 P 25273da20e0442e9bacf9a876b831b02 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1ae22c98d1842b892aed72776bd07d3" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 43523 } } peers { permanent_uuid: "b22628b371a843b8be736c586961d0dd" member_type: VOTER last_known_addr { host: "127.17.60.196" port: 46165 } } peers { permanent_uuid: "25273da20e0442e9bacf9a876b831b02" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 37751 } }
I20250112 15:07:00.158627 20574 raft_consensus.cc:383] T 828b0e3ee4684ecd9c48836ba876ac09 P 25273da20e0442e9bacf9a876b831b02 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 15:07:00.158639 20574 raft_consensus.cc:738] T 828b0e3ee4684ecd9c48836ba876ac09 P 25273da20e0442e9bacf9a876b831b02 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 25273da20e0442e9bacf9a876b831b02, State: Initialized, Role: FOLLOWER
I20250112 15:07:00.158694 20574 consensus_queue.cc:260] T 828b0e3ee4684ecd9c48836ba876ac09 P 25273da20e0442e9bacf9a876b831b02 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1ae22c98d1842b892aed72776bd07d3" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 43523 } } peers { permanent_uuid: "b22628b371a843b8be736c586961d0dd" member_type: VOTER last_known_addr { host: "127.17.60.196" port: 46165 } } peers { permanent_uuid: "25273da20e0442e9bacf9a876b831b02" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 37751 } }
I20250112 15:07:00.158834 20574 ts_tablet_manager.cc:1428] T 828b0e3ee4684ecd9c48836ba876ac09 P 25273da20e0442e9bacf9a876b831b02: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20250112 15:07:00.223883 20582 raft_consensus.cc:491] T 828b0e3ee4684ecd9c48836ba876ac09 P 25273da20e0442e9bacf9a876b831b02 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 15:07:00.223975 20582 raft_consensus.cc:513] T 828b0e3ee4684ecd9c48836ba876ac09 P 25273da20e0442e9bacf9a876b831b02 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1ae22c98d1842b892aed72776bd07d3" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 43523 } } peers { permanent_uuid: "b22628b371a843b8be736c586961d0dd" member_type: VOTER last_known_addr { host: "127.17.60.196" port: 46165 } } peers { permanent_uuid: "25273da20e0442e9bacf9a876b831b02" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 37751 } }
I20250112 15:07:00.224189 20582 leader_election.cc:290] T 828b0e3ee4684ecd9c48836ba876ac09 P 25273da20e0442e9bacf9a876b831b02 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c1ae22c98d1842b892aed72776bd07d3 (127.17.60.194:43523), b22628b371a843b8be736c586961d0dd (127.17.60.196:46165)
I20250112 15:07:00.226713 20527 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "828b0e3ee4684ecd9c48836ba876ac09" candidate_uuid: "25273da20e0442e9bacf9a876b831b02" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b22628b371a843b8be736c586961d0dd" is_pre_election: true
I20250112 15:07:00.226814 20527 raft_consensus.cc:2463] T 828b0e3ee4684ecd9c48836ba876ac09 P b22628b371a843b8be736c586961d0dd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 25273da20e0442e9bacf9a876b831b02 in term 0.
I20250112 15:07:00.226716 20377 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "828b0e3ee4684ecd9c48836ba876ac09" candidate_uuid: "25273da20e0442e9bacf9a876b831b02" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c1ae22c98d1842b892aed72776bd07d3" is_pre_election: true
I20250112 15:07:00.226859 20377 raft_consensus.cc:2463] T 828b0e3ee4684ecd9c48836ba876ac09 P c1ae22c98d1842b892aed72776bd07d3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 25273da20e0442e9bacf9a876b831b02 in term 0.
I20250112 15:07:00.226956 20266 leader_election.cc:304] T 828b0e3ee4684ecd9c48836ba876ac09 P 25273da20e0442e9bacf9a876b831b02 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 25273da20e0442e9bacf9a876b831b02, b22628b371a843b8be736c586961d0dd; no voters:
I20250112 15:07:00.227054 20582 raft_consensus.cc:2798] T 828b0e3ee4684ecd9c48836ba876ac09 P 25273da20e0442e9bacf9a876b831b02 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250112 15:07:00.227094 20582 raft_consensus.cc:491] T 828b0e3ee4684ecd9c48836ba876ac09 P 25273da20e0442e9bacf9a876b831b02 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250112 15:07:00.227109 20582 raft_consensus.cc:3054] T 828b0e3ee4684ecd9c48836ba876ac09 P 25273da20e0442e9bacf9a876b831b02 [term 0 FOLLOWER]: Advancing to term 1
I20250112 15:07:00.227691 20582 raft_consensus.cc:513] T 828b0e3ee4684ecd9c48836ba876ac09 P 25273da20e0442e9bacf9a876b831b02 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1ae22c98d1842b892aed72776bd07d3" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 43523 } } peers { permanent_uuid: "b22628b371a843b8be736c586961d0dd" member_type: VOTER last_known_addr { host: "127.17.60.196" port: 46165 } } peers { permanent_uuid: "25273da20e0442e9bacf9a876b831b02" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 37751 } }
I20250112 15:07:00.227806 20582 leader_election.cc:290] T 828b0e3ee4684ecd9c48836ba876ac09 P 25273da20e0442e9bacf9a876b831b02 [CANDIDATE]: Term 1 election: Requested vote from peers c1ae22c98d1842b892aed72776bd07d3 (127.17.60.194:43523), b22628b371a843b8be736c586961d0dd (127.17.60.196:46165)
I20250112 15:07:00.227941 20377 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "828b0e3ee4684ecd9c48836ba876ac09" candidate_uuid: "25273da20e0442e9bacf9a876b831b02" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c1ae22c98d1842b892aed72776bd07d3"
I20250112 15:07:00.227958 20527 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "828b0e3ee4684ecd9c48836ba876ac09" candidate_uuid: "25273da20e0442e9bacf9a876b831b02" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b22628b371a843b8be736c586961d0dd"
I20250112 15:07:00.228031 20377 raft_consensus.cc:3054] T 828b0e3ee4684ecd9c48836ba876ac09 P c1ae22c98d1842b892aed72776bd07d3 [term 0 FOLLOWER]: Advancing to term 1
I20250112 15:07:00.228044 20527 raft_consensus.cc:3054] T 828b0e3ee4684ecd9c48836ba876ac09 P b22628b371a843b8be736c586961d0dd [term 0 FOLLOWER]: Advancing to term 1
I20250112 15:07:00.228749 20527 raft_consensus.cc:2463] T 828b0e3ee4684ecd9c48836ba876ac09 P b22628b371a843b8be736c586961d0dd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 25273da20e0442e9bacf9a876b831b02 in term 1.
I20250112 15:07:00.228749 20377 raft_consensus.cc:2463] T 828b0e3ee4684ecd9c48836ba876ac09 P c1ae22c98d1842b892aed72776bd07d3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 25273da20e0442e9bacf9a876b831b02 in term 1.
I20250112 15:07:00.228928 20266 leader_election.cc:304] T 828b0e3ee4684ecd9c48836ba876ac09 P 25273da20e0442e9bacf9a876b831b02 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 25273da20e0442e9bacf9a876b831b02, b22628b371a843b8be736c586961d0dd; no voters:
I20250112 15:07:00.229038 20582 raft_consensus.cc:2798] T 828b0e3ee4684ecd9c48836ba876ac09 P 25273da20e0442e9bacf9a876b831b02 [term 1 FOLLOWER]: Leader election won for term 1
I20250112 15:07:00.229194 20582 raft_consensus.cc:695] T 828b0e3ee4684ecd9c48836ba876ac09 P 25273da20e0442e9bacf9a876b831b02 [term 1 LEADER]: Becoming Leader. State: Replica: 25273da20e0442e9bacf9a876b831b02, State: Running, Role: LEADER
I20250112 15:07:00.229249 20582 consensus_queue.cc:237] T 828b0e3ee4684ecd9c48836ba876ac09 P 25273da20e0442e9bacf9a876b831b02 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1ae22c98d1842b892aed72776bd07d3" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 43523 } } peers { permanent_uuid: "b22628b371a843b8be736c586961d0dd" member_type: VOTER last_known_addr { host: "127.17.60.196" port: 46165 } } peers { permanent_uuid: "25273da20e0442e9bacf9a876b831b02" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 37751 } }
I20250112 15:07:00.229779 20198 catalog_manager.cc:5526] T 828b0e3ee4684ecd9c48836ba876ac09 P 25273da20e0442e9bacf9a876b831b02 reported cstate change: term changed from 0 to 1, leader changed from <none> to 25273da20e0442e9bacf9a876b831b02 (127.17.60.193). New cstate: current_term: 1 leader_uuid: "25273da20e0442e9bacf9a876b831b02" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1ae22c98d1842b892aed72776bd07d3" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 43523 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b22628b371a843b8be736c586961d0dd" member_type: VOTER last_known_addr { host: "127.17.60.196" port: 46165 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "25273da20e0442e9bacf9a876b831b02" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 37751 } health_report { overall_health: HEALTHY } } }
W20250112 15:07:00.302001 20328 tablet.cc:2367] T 828b0e3ee4684ecd9c48836ba876ac09 P 25273da20e0442e9bacf9a876b831b02: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250112 15:07:00.769096 20587 consensus_queue.cc:1035] T 828b0e3ee4684ecd9c48836ba876ac09 P 25273da20e0442e9bacf9a876b831b02 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b22628b371a843b8be736c586961d0dd" member_type: VOTER last_known_addr { host: "127.17.60.196" port: 46165 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250112 15:07:00.770670 20587 consensus_queue.cc:1035] T 828b0e3ee4684ecd9c48836ba876ac09 P 25273da20e0442e9bacf9a876b831b02 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c1ae22c98d1842b892aed72776bd07d3" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 43523 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250112 15:07:01.006296 20252 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250112 15:07:01.006407 20252 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20250112 15:07:01.006424 20252 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20250112 15:07:03.006582 20252 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250112 15:07:03.006721 20252 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20250112 15:07:03.006757 20252 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20250112 15:07:05.006901 20252 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250112 15:07:05.007032 20252 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20250112 15:07:05.007052 20252 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20250112 15:07:07.007223 20252 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250112 15:07:07.007378 20252 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20250112 15:07:07.007400 20252 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20250112 15:07:09.007548 20252 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250112 15:07:09.007678 20252 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20250112 15:07:09.007701 20252 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20250112 15:07:11.007862 20252 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250112 15:07:11.008023 20252 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20250112 15:07:11.008046 20252 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20250112 15:07:11.296236 17651 tablet_server.cc:178] TabletServer@127.17.60.193:0 shutting down...
I20250112 15:07:11.298878 17651 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250112 15:07:11.299039 17651 tablet_replica.cc:331] T 828b0e3ee4684ecd9c48836ba876ac09 P 25273da20e0442e9bacf9a876b831b02: stopping tablet replica
I20250112 15:07:11.299119 17651 raft_consensus.cc:2238] T 828b0e3ee4684ecd9c48836ba876ac09 P 25273da20e0442e9bacf9a876b831b02 [term 1 LEADER]: Raft consensus shutting down.
I20250112 15:07:11.299193 17651 raft_consensus.cc:2267] T 828b0e3ee4684ecd9c48836ba876ac09 P 25273da20e0442e9bacf9a876b831b02 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 15:07:11.311753 17651 tablet_server.cc:195] TabletServer@127.17.60.193:0 shutdown complete.
I20250112 15:07:11.312731 17651 tablet_server.cc:178] TabletServer@127.17.60.194:0 shutting down...
I20250112 15:07:11.315549 17651 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250112 15:07:11.315655 17651 tablet_replica.cc:331] T 828b0e3ee4684ecd9c48836ba876ac09 P c1ae22c98d1842b892aed72776bd07d3: stopping tablet replica
I20250112 15:07:11.315722 17651 raft_consensus.cc:2238] T 828b0e3ee4684ecd9c48836ba876ac09 P c1ae22c98d1842b892aed72776bd07d3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250112 15:07:11.315800 17651 raft_consensus.cc:2267] T 828b0e3ee4684ecd9c48836ba876ac09 P c1ae22c98d1842b892aed72776bd07d3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 15:07:11.327409 17651 tablet_server.cc:195] TabletServer@127.17.60.194:0 shutdown complete.
I20250112 15:07:11.328310 17651 tablet_server.cc:178] TabletServer@127.17.60.195:0 shutting down...
I20250112 15:07:11.331470 17651 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250112 15:07:11.342837 17651 tablet_server.cc:195] TabletServer@127.17.60.195:0 shutdown complete.
I20250112 15:07:11.343504 17651 tablet_server.cc:178] TabletServer@127.17.60.196:0 shutting down...
I20250112 15:07:11.346909 17651 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250112 15:07:11.347090 17651 tablet_replica.cc:331] T 828b0e3ee4684ecd9c48836ba876ac09 P b22628b371a843b8be736c586961d0dd: stopping tablet replica
I20250112 15:07:11.347142 17651 raft_consensus.cc:2238] T 828b0e3ee4684ecd9c48836ba876ac09 P b22628b371a843b8be736c586961d0dd [term 1 FOLLOWER]: Raft consensus shutting down.
I20250112 15:07:11.347190 17651 raft_consensus.cc:2267] T 828b0e3ee4684ecd9c48836ba876ac09 P b22628b371a843b8be736c586961d0dd [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 15:07:11.358740 17651 tablet_server.cc:195] TabletServer@127.17.60.196:0 shutdown complete.
I20250112 15:07:11.359596 17651 master.cc:537] Master@127.17.60.254:37135 shutting down...
I20250112 15:07:11.362241 17651 raft_consensus.cc:2238] T 00000000000000000000000000000000 P ee58d62a60fa44fc80db045024b87e6a [term 1 LEADER]: Raft consensus shutting down.
I20250112 15:07:11.362314 17651 raft_consensus.cc:2267] T 00000000000000000000000000000000 P ee58d62a60fa44fc80db045024b87e6a [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 15:07:11.362355 17651 tablet_replica.cc:331] T 00000000000000000000000000000000 P ee58d62a60fa44fc80db045024b87e6a: stopping tablet replica
I20250112 15:07:11.373950 17651 master.cc:559] Master@127.17.60.254:37135 shutdown complete.
[ OK ] AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne (12401 ms)
[ RUN ] AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy
I20250112 15:07:11.378422 17651 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.17.60.254:39221
I20250112 15:07:11.378568 17651 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250112 15:07:11.380766 20601 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250112 15:07:11.381830 17651 server_base.cc:1034] running on GCE node
W20250112 15:07:11.382076 20598 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250112 15:07:11.382607 20599 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250112 15:07:11.382735 17651 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250112 15:07:11.382777 17651 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250112 15:07:11.382797 17651 hybrid_clock.cc:648] HybridClock initialized: now 1736694431382796 us; error 0 us; skew 500 ppm
I20250112 15:07:11.383440 17651 webserver.cc:458] Webserver started at http://127.17.60.254:40691/ using document root <none> and password file <none>
I20250112 15:07:11.383523 17651 fs_manager.cc:362] Metadata directory not provided
I20250112 15:07:11.383569 17651 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250112 15:07:11.383620 17651 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250112 15:07:11.383880 17651 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task95dLSD/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1736694351070047-17651-0/minicluster-data/master-0-root/instance:
uuid: "b3f2173c64b24bada17b7788fe45b77c"
format_stamp: "Formatted at 2025-01-12 15:07:11 on dist-test-slave-b4nz"
I20250112 15:07:11.384729 17651 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20250112 15:07:11.385152 20606 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250112 15:07:11.385283 17651 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20250112 15:07:11.385324 17651 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task95dLSD/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1736694351070047-17651-0/minicluster-data/master-0-root
uuid: "b3f2173c64b24bada17b7788fe45b77c"
format_stamp: "Formatted at 2025-01-12 15:07:11 on dist-test-slave-b4nz"
I20250112 15:07:11.385373 17651 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task95dLSD/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1736694351070047-17651-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-task95dLSD/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1736694351070047-17651-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-task95dLSD/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1736694351070047-17651-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250112 15:07:11.413502 17651 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250112 15:07:11.413715 17651 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250112 15:07:11.416319 17651 rpc_server.cc:307] RPC server started. Bound to: 127.17.60.254:39221
I20250112 15:07:11.418212 20658 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.17.60.254:39221 every 8 connection(s)
I20250112 15:07:11.418375 20659 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 15:07:11.419404 20659 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b3f2173c64b24bada17b7788fe45b77c: Bootstrap starting.
I20250112 15:07:11.419727 20659 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b3f2173c64b24bada17b7788fe45b77c: Neither blocks nor log segments found. Creating new log.
I20250112 15:07:11.420202 20659 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b3f2173c64b24bada17b7788fe45b77c: No bootstrap required, opened a new log
I20250112 15:07:11.420325 20659 raft_consensus.cc:357] T 00000000000000000000000000000000 P b3f2173c64b24bada17b7788fe45b77c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b3f2173c64b24bada17b7788fe45b77c" member_type: VOTER }
I20250112 15:07:11.420374 20659 raft_consensus.cc:383] T 00000000000000000000000000000000 P b3f2173c64b24bada17b7788fe45b77c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 15:07:11.420398 20659 raft_consensus.cc:738] T 00000000000000000000000000000000 P b3f2173c64b24bada17b7788fe45b77c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b3f2173c64b24bada17b7788fe45b77c, State: Initialized, Role: FOLLOWER
I20250112 15:07:11.420454 20659 consensus_queue.cc:260] T 00000000000000000000000000000000 P b3f2173c64b24bada17b7788fe45b77c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b3f2173c64b24bada17b7788fe45b77c" member_type: VOTER }
I20250112 15:07:11.420493 20659 raft_consensus.cc:397] T 00000000000000000000000000000000 P b3f2173c64b24bada17b7788fe45b77c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250112 15:07:11.420521 20659 raft_consensus.cc:491] T 00000000000000000000000000000000 P b3f2173c64b24bada17b7788fe45b77c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250112 15:07:11.420547 20659 raft_consensus.cc:3054] T 00000000000000000000000000000000 P b3f2173c64b24bada17b7788fe45b77c [term 0 FOLLOWER]: Advancing to term 1
I20250112 15:07:11.421017 20659 raft_consensus.cc:513] T 00000000000000000000000000000000 P b3f2173c64b24bada17b7788fe45b77c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b3f2173c64b24bada17b7788fe45b77c" member_type: VOTER }
I20250112 15:07:11.421072 20659 leader_election.cc:304] T 00000000000000000000000000000000 P b3f2173c64b24bada17b7788fe45b77c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b3f2173c64b24bada17b7788fe45b77c; no voters:
I20250112 15:07:11.421173 20659 leader_election.cc:290] T 00000000000000000000000000000000 P b3f2173c64b24bada17b7788fe45b77c [CANDIDATE]: Term 1 election: Requested vote from peers
I20250112 15:07:11.421224 20662 raft_consensus.cc:2798] T 00000000000000000000000000000000 P b3f2173c64b24bada17b7788fe45b77c [term 1 FOLLOWER]: Leader election won for term 1
I20250112 15:07:11.421329 20659 sys_catalog.cc:564] T 00000000000000000000000000000000 P b3f2173c64b24bada17b7788fe45b77c [sys.catalog]: configured and running, proceeding with master startup.
I20250112 15:07:11.421408 20662 raft_consensus.cc:695] T 00000000000000000000000000000000 P b3f2173c64b24bada17b7788fe45b77c [term 1 LEADER]: Becoming Leader. State: Replica: b3f2173c64b24bada17b7788fe45b77c, State: Running, Role: LEADER
I20250112 15:07:11.421484 20662 consensus_queue.cc:237] T 00000000000000000000000000000000 P b3f2173c64b24bada17b7788fe45b77c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b3f2173c64b24bada17b7788fe45b77c" member_type: VOTER }
I20250112 15:07:11.421692 20663 sys_catalog.cc:455] T 00000000000000000000000000000000 P b3f2173c64b24bada17b7788fe45b77c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b3f2173c64b24bada17b7788fe45b77c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b3f2173c64b24bada17b7788fe45b77c" member_type: VOTER } }
I20250112 15:07:11.421696 20664 sys_catalog.cc:455] T 00000000000000000000000000000000 P b3f2173c64b24bada17b7788fe45b77c [sys.catalog]: SysCatalogTable state changed. Reason: New leader b3f2173c64b24bada17b7788fe45b77c. Latest consensus state: current_term: 1 leader_uuid: "b3f2173c64b24bada17b7788fe45b77c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b3f2173c64b24bada17b7788fe45b77c" member_type: VOTER } }
I20250112 15:07:11.421762 20663 sys_catalog.cc:458] T 00000000000000000000000000000000 P b3f2173c64b24bada17b7788fe45b77c [sys.catalog]: This master's current role is: LEADER
I20250112 15:07:11.421765 20664 sys_catalog.cc:458] T 00000000000000000000000000000000 P b3f2173c64b24bada17b7788fe45b77c [sys.catalog]: This master's current role is: LEADER
I20250112 15:07:11.421986 20668 catalog_manager.cc:1476] Loading table and tablet metadata into memory...
I20250112 15:07:11.422153 20668 catalog_manager.cc:1485] Initializing Kudu cluster ID...
I20250112 15:07:11.422652 17651 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20250112 15:07:11.422731 20668 catalog_manager.cc:1348] Generated new cluster ID: 3a30305ea7bd45759c0cdfbc4c522ad1
I20250112 15:07:11.422777 20668 catalog_manager.cc:1496] Initializing Kudu internal certificate authority...
I20250112 15:07:11.439538 20668 catalog_manager.cc:1371] Generated new certificate authority record
I20250112 15:07:11.439946 20668 catalog_manager.cc:1505] Loading token signing keys...
I20250112 15:07:11.449064 20668 catalog_manager.cc:5899] T 00000000000000000000000000000000 P b3f2173c64b24bada17b7788fe45b77c: Generated new TSK 0
I20250112 15:07:11.449152 20668 catalog_manager.cc:1515] Initializing in-progress tserver states...
I20250112 15:07:11.454517 17651 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250112 15:07:11.456039 20680 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250112 15:07:11.456893 20681 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250112 15:07:11.457908 20683 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250112 15:07:11.458765 17651 server_base.cc:1034] running on GCE node
I20250112 15:07:11.458859 17651 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250112 15:07:11.458885 17651 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250112 15:07:11.458905 17651 hybrid_clock.cc:648] HybridClock initialized: now 1736694431458906 us; error 0 us; skew 500 ppm
I20250112 15:07:11.459575 17651 webserver.cc:458] Webserver started at http://127.17.60.193:37217/ using document root <none> and password file <none>
I20250112 15:07:11.459662 17651 fs_manager.cc:362] Metadata directory not provided
I20250112 15:07:11.459705 17651 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250112 15:07:11.459756 17651 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250112 15:07:11.460033 17651 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task95dLSD/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1736694351070047-17651-0/minicluster-data/ts-0-root/instance:
uuid: "d4830a9a55864449b2c7ec66d577afdc"
format_stamp: "Formatted at 2025-01-12 15:07:11 on dist-test-slave-b4nz"
I20250112 15:07:11.460914 17651 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20250112 15:07:11.461349 20688 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250112 15:07:11.461515 17651 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20250112 15:07:11.461560 17651 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task95dLSD/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1736694351070047-17651-0/minicluster-data/ts-0-root
uuid: "d4830a9a55864449b2c7ec66d577afdc"
format_stamp: "Formatted at 2025-01-12 15:07:11 on dist-test-slave-b4nz"
I20250112 15:07:11.461609 17651 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task95dLSD/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1736694351070047-17651-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-task95dLSD/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1736694351070047-17651-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-task95dLSD/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1736694351070047-17651-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250112 15:07:11.477310 17651 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250112 15:07:11.477530 17651 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250112 15:07:11.477674 17651 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250112 15:07:11.477854 17651 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250112 15:07:11.477882 17651 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250112 15:07:11.477913 17651 ts_tablet_manager.cc:610] Registered 0 tablets
I20250112 15:07:11.477949 17651 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250112 15:07:11.480552 17651 rpc_server.cc:307] RPC server started. Bound to: 127.17.60.193:35133
I20250112 15:07:11.480612 20751 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.17.60.193:35133 every 8 connection(s)
I20250112 15:07:11.480917 17651 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250112 15:07:11.482367 20756 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250112 15:07:11.483207 20757 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250112 15:07:11.484109 20759 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250112 15:07:11.484329 20752 heartbeater.cc:346] Connected to a master server at 127.17.60.254:39221
I20250112 15:07:11.484381 17651 server_base.cc:1034] running on GCE node
I20250112 15:07:11.484390 20752 heartbeater.cc:463] Registering TS with master...
I20250112 15:07:11.484491 17651 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250112 15:07:11.484512 17651 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250112 15:07:11.484544 17651 hybrid_clock.cc:648] HybridClock initialized: now 1736694431484544 us; error 0 us; skew 500 ppm
I20250112 15:07:11.484544 20752 heartbeater.cc:510] Master 127.17.60.254:39221 requested a full tablet report, sending...
I20250112 15:07:11.484869 20623 ts_manager.cc:194] Registered new tserver with Master: d4830a9a55864449b2c7ec66d577afdc (127.17.60.193:35133)
I20250112 15:07:11.485249 17651 webserver.cc:458] Webserver started at http://127.17.60.194:41529/ using document root <none> and password file <none>
I20250112 15:07:11.485322 17651 fs_manager.cc:362] Metadata directory not provided
I20250112 15:07:11.485355 17651 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250112 15:07:11.485400 17651 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250112 15:07:11.485563 20623 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:54210
I20250112 15:07:11.485673 17651 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task95dLSD/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1736694351070047-17651-0/minicluster-data/ts-1-root/instance:
uuid: "08f2c47737324d57be0949b15e1d7f04"
format_stamp: "Formatted at 2025-01-12 15:07:11 on dist-test-slave-b4nz"
I20250112 15:07:11.486641 17651 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20250112 15:07:11.487042 20764 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250112 15:07:11.487136 17651 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20250112 15:07:11.487172 17651 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task95dLSD/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1736694351070047-17651-0/minicluster-data/ts-1-root
uuid: "08f2c47737324d57be0949b15e1d7f04"
format_stamp: "Formatted at 2025-01-12 15:07:11 on dist-test-slave-b4nz"
I20250112 15:07:11.487206 17651 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task95dLSD/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1736694351070047-17651-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-task95dLSD/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1736694351070047-17651-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-task95dLSD/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1736694351070047-17651-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250112 15:07:11.491324 17651 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250112 15:07:11.491516 17651 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250112 15:07:11.491621 17651 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250112 15:07:11.491760 17651 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250112 15:07:11.491782 17651 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250112 15:07:11.491806 17651 ts_tablet_manager.cc:610] Registered 0 tablets
I20250112 15:07:11.491823 17651 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250112 15:07:11.494421 17651 rpc_server.cc:307] RPC server started. Bound to: 127.17.60.194:36509
I20250112 15:07:11.494473 20827 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.17.60.194:36509 every 8 connection(s)
I20250112 15:07:11.494736 17651 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250112 15:07:11.495960 20831 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250112 15:07:11.496707 20834 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250112 15:07:11.497426 20832 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250112 15:07:11.497640 20828 heartbeater.cc:346] Connected to a master server at 127.17.60.254:39221
I20250112 15:07:11.497692 20828 heartbeater.cc:463] Registering TS with master...
I20250112 15:07:11.497787 20828 heartbeater.cc:510] Master 127.17.60.254:39221 requested a full tablet report, sending...
I20250112 15:07:11.497990 20623 ts_manager.cc:194] Registered new tserver with Master: 08f2c47737324d57be0949b15e1d7f04 (127.17.60.194:36509)
I20250112 15:07:11.498140 17651 server_base.cc:1034] running on GCE node
I20250112 15:07:11.498257 17651 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250112 15:07:11.498288 17651 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250112 15:07:11.498307 17651 hybrid_clock.cc:648] HybridClock initialized: now 1736694431498308 us; error 0 us; skew 500 ppm
I20250112 15:07:11.498433 20623 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:54212
I20250112 15:07:11.498987 17651 webserver.cc:458] Webserver started at http://127.17.60.195:39421/ using document root <none> and password file <none>
I20250112 15:07:11.499080 17651 fs_manager.cc:362] Metadata directory not provided
I20250112 15:07:11.499132 17651 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250112 15:07:11.499181 17651 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250112 15:07:11.499454 17651 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task95dLSD/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1736694351070047-17651-0/minicluster-data/ts-2-root/instance:
uuid: "2021b61b34df4d1db1d3269848a30b34"
format_stamp: "Formatted at 2025-01-12 15:07:11 on dist-test-slave-b4nz"
I20250112 15:07:11.500309 17651 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20250112 15:07:11.500748 20839 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250112 15:07:11.500869 17651 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20250112 15:07:11.500910 17651 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task95dLSD/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1736694351070047-17651-0/minicluster-data/ts-2-root
uuid: "2021b61b34df4d1db1d3269848a30b34"
format_stamp: "Formatted at 2025-01-12 15:07:11 on dist-test-slave-b4nz"
I20250112 15:07:11.500954 17651 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task95dLSD/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1736694351070047-17651-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-task95dLSD/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1736694351070047-17651-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-task95dLSD/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1736694351070047-17651-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250112 15:07:11.510116 17651 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250112 15:07:11.510324 17651 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250112 15:07:11.510459 17651 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250112 15:07:11.510632 17651 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250112 15:07:11.510658 17651 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250112 15:07:11.510686 17651 ts_tablet_manager.cc:610] Registered 0 tablets
I20250112 15:07:11.510710 17651 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250112 15:07:11.513895 17651 rpc_server.cc:307] RPC server started. Bound to: 127.17.60.195:43653
I20250112 15:07:11.513952 20902 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.17.60.195:43653 every 8 connection(s)
I20250112 15:07:11.516850 20903 heartbeater.cc:346] Connected to a master server at 127.17.60.254:39221
I20250112 15:07:11.516901 20903 heartbeater.cc:463] Registering TS with master...
I20250112 15:07:11.517001 20903 heartbeater.cc:510] Master 127.17.60.254:39221 requested a full tablet report, sending...
I20250112 15:07:11.517206 20623 ts_manager.cc:194] Registered new tserver with Master: 2021b61b34df4d1db1d3269848a30b34 (127.17.60.195:43653)
I20250112 15:07:11.517371 17651 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.003303195s
I20250112 15:07:11.517534 20623 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:54214
I20250112 15:07:12.486260 20752 heartbeater.cc:502] Master 127.17.60.254:39221 was elected leader, sending a full tablet report...
I20250112 15:07:12.499229 20828 heartbeater.cc:502] Master 127.17.60.254:39221 was elected leader, sending a full tablet report...
I20250112 15:07:12.518082 20903 heartbeater.cc:502] Master 127.17.60.254:39221 was elected leader, sending a full tablet report...
I20250112 15:07:12.545642 17651 test_util.cc:274] Using random seed: -1363395737
I20250112 15:07:12.550626 20623 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54234:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 15:07:12.550763 20623 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-workload in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 15:07:12.555537 20716 tablet_service.cc:1467] Processing CreateTablet for tablet 5b6b4427200a48f1b0ac51e5574c520b (DEFAULT_TABLE table=test-workload [id=de06745df7d54728af100b0f00ee53e1]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 15:07:12.555687 20716 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5b6b4427200a48f1b0ac51e5574c520b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 15:07:12.556186 20867 tablet_service.cc:1467] Processing CreateTablet for tablet 5b6b4427200a48f1b0ac51e5574c520b (DEFAULT_TABLE table=test-workload [id=de06745df7d54728af100b0f00ee53e1]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 15:07:12.556308 20867 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5b6b4427200a48f1b0ac51e5574c520b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 15:07:12.556751 20792 tablet_service.cc:1467] Processing CreateTablet for tablet 5b6b4427200a48f1b0ac51e5574c520b (DEFAULT_TABLE table=test-workload [id=de06745df7d54728af100b0f00ee53e1]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 15:07:12.556870 20792 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5b6b4427200a48f1b0ac51e5574c520b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 15:07:12.557195 20924 tablet_bootstrap.cc:492] T 5b6b4427200a48f1b0ac51e5574c520b P d4830a9a55864449b2c7ec66d577afdc: Bootstrap starting.
I20250112 15:07:12.557672 20925 tablet_bootstrap.cc:492] T 5b6b4427200a48f1b0ac51e5574c520b P 2021b61b34df4d1db1d3269848a30b34: Bootstrap starting.
I20250112 15:07:12.557703 20924 tablet_bootstrap.cc:654] T 5b6b4427200a48f1b0ac51e5574c520b P d4830a9a55864449b2c7ec66d577afdc: Neither blocks nor log segments found. Creating new log.
I20250112 15:07:12.558097 20925 tablet_bootstrap.cc:654] T 5b6b4427200a48f1b0ac51e5574c520b P 2021b61b34df4d1db1d3269848a30b34: Neither blocks nor log segments found. Creating new log.
I20250112 15:07:12.558354 20927 tablet_bootstrap.cc:492] T 5b6b4427200a48f1b0ac51e5574c520b P 08f2c47737324d57be0949b15e1d7f04: Bootstrap starting.
I20250112 15:07:12.558475 20924 tablet_bootstrap.cc:492] T 5b6b4427200a48f1b0ac51e5574c520b P d4830a9a55864449b2c7ec66d577afdc: No bootstrap required, opened a new log
I20250112 15:07:12.558569 20924 ts_tablet_manager.cc:1397] T 5b6b4427200a48f1b0ac51e5574c520b P d4830a9a55864449b2c7ec66d577afdc: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20250112 15:07:12.558759 20924 raft_consensus.cc:357] T 5b6b4427200a48f1b0ac51e5574c520b P d4830a9a55864449b2c7ec66d577afdc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4830a9a55864449b2c7ec66d577afdc" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 35133 } } peers { permanent_uuid: "2021b61b34df4d1db1d3269848a30b34" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 43653 } } peers { permanent_uuid: "08f2c47737324d57be0949b15e1d7f04" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 36509 } }
I20250112 15:07:12.558818 20924 raft_consensus.cc:383] T 5b6b4427200a48f1b0ac51e5574c520b P d4830a9a55864449b2c7ec66d577afdc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 15:07:12.558823 20925 tablet_bootstrap.cc:492] T 5b6b4427200a48f1b0ac51e5574c520b P 2021b61b34df4d1db1d3269848a30b34: No bootstrap required, opened a new log
I20250112 15:07:12.558876 20927 tablet_bootstrap.cc:654] T 5b6b4427200a48f1b0ac51e5574c520b P 08f2c47737324d57be0949b15e1d7f04: Neither blocks nor log segments found. Creating new log.
I20250112 15:07:12.558930 20925 ts_tablet_manager.cc:1397] T 5b6b4427200a48f1b0ac51e5574c520b P 2021b61b34df4d1db1d3269848a30b34: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20250112 15:07:12.558851 20924 raft_consensus.cc:738] T 5b6b4427200a48f1b0ac51e5574c520b P d4830a9a55864449b2c7ec66d577afdc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d4830a9a55864449b2c7ec66d577afdc, State: Initialized, Role: FOLLOWER
I20250112 15:07:12.559070 20924 consensus_queue.cc:260] T 5b6b4427200a48f1b0ac51e5574c520b P d4830a9a55864449b2c7ec66d577afdc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4830a9a55864449b2c7ec66d577afdc" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 35133 } } peers { permanent_uuid: "2021b61b34df4d1db1d3269848a30b34" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 43653 } } peers { permanent_uuid: "08f2c47737324d57be0949b15e1d7f04" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 36509 } }
I20250112 15:07:12.559113 20925 raft_consensus.cc:357] T 5b6b4427200a48f1b0ac51e5574c520b P 2021b61b34df4d1db1d3269848a30b34 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4830a9a55864449b2c7ec66d577afdc" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 35133 } } peers { permanent_uuid: "2021b61b34df4d1db1d3269848a30b34" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 43653 } } peers { permanent_uuid: "08f2c47737324d57be0949b15e1d7f04" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 36509 } }
I20250112 15:07:12.559201 20925 raft_consensus.cc:383] T 5b6b4427200a48f1b0ac51e5574c520b P 2021b61b34df4d1db1d3269848a30b34 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 15:07:12.559227 20925 raft_consensus.cc:738] T 5b6b4427200a48f1b0ac51e5574c520b P 2021b61b34df4d1db1d3269848a30b34 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2021b61b34df4d1db1d3269848a30b34, State: Initialized, Role: FOLLOWER
I20250112 15:07:12.559274 20925 consensus_queue.cc:260] T 5b6b4427200a48f1b0ac51e5574c520b P 2021b61b34df4d1db1d3269848a30b34 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4830a9a55864449b2c7ec66d577afdc" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 35133 } } peers { permanent_uuid: "2021b61b34df4d1db1d3269848a30b34" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 43653 } } peers { permanent_uuid: "08f2c47737324d57be0949b15e1d7f04" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 36509 } }
I20250112 15:07:12.559273 20924 ts_tablet_manager.cc:1428] T 5b6b4427200a48f1b0ac51e5574c520b P d4830a9a55864449b2c7ec66d577afdc: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20250112 15:07:12.559428 20925 ts_tablet_manager.cc:1428] T 5b6b4427200a48f1b0ac51e5574c520b P 2021b61b34df4d1db1d3269848a30b34: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20250112 15:07:12.559934 20927 tablet_bootstrap.cc:492] T 5b6b4427200a48f1b0ac51e5574c520b P 08f2c47737324d57be0949b15e1d7f04: No bootstrap required, opened a new log
I20250112 15:07:12.559998 20927 ts_tablet_manager.cc:1397] T 5b6b4427200a48f1b0ac51e5574c520b P 08f2c47737324d57be0949b15e1d7f04: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20250112 15:07:12.560184 20927 raft_consensus.cc:357] T 5b6b4427200a48f1b0ac51e5574c520b P 08f2c47737324d57be0949b15e1d7f04 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4830a9a55864449b2c7ec66d577afdc" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 35133 } } peers { permanent_uuid: "2021b61b34df4d1db1d3269848a30b34" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 43653 } } peers { permanent_uuid: "08f2c47737324d57be0949b15e1d7f04" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 36509 } }
I20250112 15:07:12.560243 20927 raft_consensus.cc:383] T 5b6b4427200a48f1b0ac51e5574c520b P 08f2c47737324d57be0949b15e1d7f04 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 15:07:12.560256 20927 raft_consensus.cc:738] T 5b6b4427200a48f1b0ac51e5574c520b P 08f2c47737324d57be0949b15e1d7f04 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 08f2c47737324d57be0949b15e1d7f04, State: Initialized, Role: FOLLOWER
I20250112 15:07:12.560310 20927 consensus_queue.cc:260] T 5b6b4427200a48f1b0ac51e5574c520b P 08f2c47737324d57be0949b15e1d7f04 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4830a9a55864449b2c7ec66d577afdc" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 35133 } } peers { permanent_uuid: "2021b61b34df4d1db1d3269848a30b34" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 43653 } } peers { permanent_uuid: "08f2c47737324d57be0949b15e1d7f04" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 36509 } }
I20250112 15:07:12.560459 20927 ts_tablet_manager.cc:1428] T 5b6b4427200a48f1b0ac51e5574c520b P 08f2c47737324d57be0949b15e1d7f04: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20250112 15:07:12.854030 20931 raft_consensus.cc:491] T 5b6b4427200a48f1b0ac51e5574c520b P 2021b61b34df4d1db1d3269848a30b34 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 15:07:12.854180 20931 raft_consensus.cc:513] T 5b6b4427200a48f1b0ac51e5574c520b P 2021b61b34df4d1db1d3269848a30b34 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4830a9a55864449b2c7ec66d577afdc" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 35133 } } peers { permanent_uuid: "2021b61b34df4d1db1d3269848a30b34" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 43653 } } peers { permanent_uuid: "08f2c47737324d57be0949b15e1d7f04" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 36509 } }
I20250112 15:07:12.854425 20931 leader_election.cc:290] T 5b6b4427200a48f1b0ac51e5574c520b P 2021b61b34df4d1db1d3269848a30b34 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d4830a9a55864449b2c7ec66d577afdc (127.17.60.193:35133), 08f2c47737324d57be0949b15e1d7f04 (127.17.60.194:36509)
I20250112 15:07:12.858054 20726 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b6b4427200a48f1b0ac51e5574c520b" candidate_uuid: "2021b61b34df4d1db1d3269848a30b34" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4830a9a55864449b2c7ec66d577afdc" is_pre_election: true
I20250112 15:07:12.858054 20802 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b6b4427200a48f1b0ac51e5574c520b" candidate_uuid: "2021b61b34df4d1db1d3269848a30b34" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "08f2c47737324d57be0949b15e1d7f04" is_pre_election: true
I20250112 15:07:12.858196 20726 raft_consensus.cc:2463] T 5b6b4427200a48f1b0ac51e5574c520b P d4830a9a55864449b2c7ec66d577afdc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2021b61b34df4d1db1d3269848a30b34 in term 0.
I20250112 15:07:12.858204 20802 raft_consensus.cc:2463] T 5b6b4427200a48f1b0ac51e5574c520b P 08f2c47737324d57be0949b15e1d7f04 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2021b61b34df4d1db1d3269848a30b34 in term 0.
I20250112 15:07:12.858369 20840 leader_election.cc:304] T 5b6b4427200a48f1b0ac51e5574c520b P 2021b61b34df4d1db1d3269848a30b34 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 08f2c47737324d57be0949b15e1d7f04, 2021b61b34df4d1db1d3269848a30b34; no voters:
I20250112 15:07:12.858450 20931 raft_consensus.cc:2798] T 5b6b4427200a48f1b0ac51e5574c520b P 2021b61b34df4d1db1d3269848a30b34 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250112 15:07:12.858494 20931 raft_consensus.cc:491] T 5b6b4427200a48f1b0ac51e5574c520b P 2021b61b34df4d1db1d3269848a30b34 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250112 15:07:12.858520 20931 raft_consensus.cc:3054] T 5b6b4427200a48f1b0ac51e5574c520b P 2021b61b34df4d1db1d3269848a30b34 [term 0 FOLLOWER]: Advancing to term 1
I20250112 15:07:12.859150 20931 raft_consensus.cc:513] T 5b6b4427200a48f1b0ac51e5574c520b P 2021b61b34df4d1db1d3269848a30b34 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4830a9a55864449b2c7ec66d577afdc" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 35133 } } peers { permanent_uuid: "2021b61b34df4d1db1d3269848a30b34" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 43653 } } peers { permanent_uuid: "08f2c47737324d57be0949b15e1d7f04" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 36509 } }
I20250112 15:07:12.859263 20931 leader_election.cc:290] T 5b6b4427200a48f1b0ac51e5574c520b P 2021b61b34df4d1db1d3269848a30b34 [CANDIDATE]: Term 1 election: Requested vote from peers d4830a9a55864449b2c7ec66d577afdc (127.17.60.193:35133), 08f2c47737324d57be0949b15e1d7f04 (127.17.60.194:36509)
I20250112 15:07:12.859402 20726 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b6b4427200a48f1b0ac51e5574c520b" candidate_uuid: "2021b61b34df4d1db1d3269848a30b34" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4830a9a55864449b2c7ec66d577afdc"
I20250112 15:07:12.859468 20726 raft_consensus.cc:3054] T 5b6b4427200a48f1b0ac51e5574c520b P d4830a9a55864449b2c7ec66d577afdc [term 0 FOLLOWER]: Advancing to term 1
I20250112 15:07:12.859411 20802 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b6b4427200a48f1b0ac51e5574c520b" candidate_uuid: "2021b61b34df4d1db1d3269848a30b34" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "08f2c47737324d57be0949b15e1d7f04"
I20250112 15:07:12.859556 20802 raft_consensus.cc:3054] T 5b6b4427200a48f1b0ac51e5574c520b P 08f2c47737324d57be0949b15e1d7f04 [term 0 FOLLOWER]: Advancing to term 1
I20250112 15:07:12.860155 20726 raft_consensus.cc:2463] T 5b6b4427200a48f1b0ac51e5574c520b P d4830a9a55864449b2c7ec66d577afdc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2021b61b34df4d1db1d3269848a30b34 in term 1.
I20250112 15:07:12.860270 20802 raft_consensus.cc:2463] T 5b6b4427200a48f1b0ac51e5574c520b P 08f2c47737324d57be0949b15e1d7f04 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2021b61b34df4d1db1d3269848a30b34 in term 1.
I20250112 15:07:12.860364 20843 leader_election.cc:304] T 5b6b4427200a48f1b0ac51e5574c520b P 2021b61b34df4d1db1d3269848a30b34 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2021b61b34df4d1db1d3269848a30b34, d4830a9a55864449b2c7ec66d577afdc; no voters:
I20250112 15:07:12.860478 20931 raft_consensus.cc:2798] T 5b6b4427200a48f1b0ac51e5574c520b P 2021b61b34df4d1db1d3269848a30b34 [term 1 FOLLOWER]: Leader election won for term 1
I20250112 15:07:12.860638 20931 raft_consensus.cc:695] T 5b6b4427200a48f1b0ac51e5574c520b P 2021b61b34df4d1db1d3269848a30b34 [term 1 LEADER]: Becoming Leader. State: Replica: 2021b61b34df4d1db1d3269848a30b34, State: Running, Role: LEADER
I20250112 15:07:12.860684 20931 consensus_queue.cc:237] T 5b6b4427200a48f1b0ac51e5574c520b P 2021b61b34df4d1db1d3269848a30b34 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4830a9a55864449b2c7ec66d577afdc" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 35133 } } peers { permanent_uuid: "2021b61b34df4d1db1d3269848a30b34" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 43653 } } peers { permanent_uuid: "08f2c47737324d57be0949b15e1d7f04" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 36509 } }
I20250112 15:07:12.861238 20623 catalog_manager.cc:5526] T 5b6b4427200a48f1b0ac51e5574c520b P 2021b61b34df4d1db1d3269848a30b34 reported cstate change: term changed from 0 to 1, leader changed from <none> to 2021b61b34df4d1db1d3269848a30b34 (127.17.60.195). New cstate: current_term: 1 leader_uuid: "2021b61b34df4d1db1d3269848a30b34" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4830a9a55864449b2c7ec66d577afdc" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 35133 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2021b61b34df4d1db1d3269848a30b34" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 43653 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "08f2c47737324d57be0949b15e1d7f04" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 36509 } health_report { overall_health: UNKNOWN } } }
I20250112 15:07:13.422523 20672 placement_policy_util.cc:407] tablet 5b6b4427200a48f1b0ac51e5574c520b: detected majority of replicas (2 of 3) at location L0
I20250112 15:07:13.422722 20678 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250112 15:07:13.422847 20678 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20250112 15:07:13.422879 20678 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20250112 15:07:13.474951 20937 consensus_queue.cc:1035] T 5b6b4427200a48f1b0ac51e5574c520b P 2021b61b34df4d1db1d3269848a30b34 [LEADER]: Connected to new peer: Peer: permanent_uuid: "08f2c47737324d57be0949b15e1d7f04" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 36509 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250112 15:07:13.476671 20940 consensus_queue.cc:1035] T 5b6b4427200a48f1b0ac51e5574c520b P 2021b61b34df4d1db1d3269848a30b34 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d4830a9a55864449b2c7ec66d577afdc" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 35133 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250112 15:07:14.423048 20672 placement_policy_util.cc:407] tablet 5b6b4427200a48f1b0ac51e5574c520b: detected majority of replicas (2 of 3) at location L0
I20250112 15:07:15.423040 20678 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250112 15:07:15.423184 20678 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20250112 15:07:15.423233 20678 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20250112 15:07:15.423393 20672 placement_policy_util.cc:407] tablet 5b6b4427200a48f1b0ac51e5574c520b: detected majority of replicas (2 of 3) at location L0
I20250112 15:07:16.423697 20672 placement_policy_util.cc:407] tablet 5b6b4427200a48f1b0ac51e5574c520b: detected majority of replicas (2 of 3) at location L0
I20250112 15:07:17.423382 20678 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250112 15:07:17.423520 20678 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20250112 15:07:17.423542 20678 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20250112 15:07:17.424013 20672 placement_policy_util.cc:407] tablet 5b6b4427200a48f1b0ac51e5574c520b: detected majority of replicas (2 of 3) at location L0
I20250112 15:07:18.424302 20672 placement_policy_util.cc:407] tablet 5b6b4427200a48f1b0ac51e5574c520b: detected majority of replicas (2 of 3) at location L0
I20250112 15:07:19.423682 20678 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250112 15:07:19.423806 20678 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20250112 15:07:19.423831 20678 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20250112 15:07:19.424616 20672 placement_policy_util.cc:407] tablet 5b6b4427200a48f1b0ac51e5574c520b: detected majority of replicas (2 of 3) at location L0
I20250112 15:07:20.424924 20672 placement_policy_util.cc:407] tablet 5b6b4427200a48f1b0ac51e5574c520b: detected majority of replicas (2 of 3) at location L0
I20250112 15:07:21.423997 20678 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250112 15:07:21.424136 20678 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20250112 15:07:21.424163 20678 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20250112 15:07:21.425231 20672 placement_policy_util.cc:407] tablet 5b6b4427200a48f1b0ac51e5574c520b: detected majority of replicas (2 of 3) at location L0
I20250112 15:07:22.425652 20672 placement_policy_util.cc:407] tablet 5b6b4427200a48f1b0ac51e5574c520b: detected majority of replicas (2 of 3) at location L0
I20250112 15:07:23.424394 20678 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250112 15:07:23.424527 20678 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20250112 15:07:23.424585 20678 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20250112 15:07:23.425938 20672 placement_policy_util.cc:407] tablet 5b6b4427200a48f1b0ac51e5574c520b: detected majority of replicas (2 of 3) at location L0
I20250112 15:07:23.958228 17651 tablet_server.cc:178] TabletServer@127.17.60.193:0 shutting down...
I20250112 15:07:23.960943 17651 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250112 15:07:23.961079 17651 tablet_replica.cc:331] T 5b6b4427200a48f1b0ac51e5574c520b P d4830a9a55864449b2c7ec66d577afdc: stopping tablet replica
I20250112 15:07:23.961131 17651 raft_consensus.cc:2238] T 5b6b4427200a48f1b0ac51e5574c520b P d4830a9a55864449b2c7ec66d577afdc [term 1 FOLLOWER]: Raft consensus shutting down.
I20250112 15:07:23.961181 17651 raft_consensus.cc:2267] T 5b6b4427200a48f1b0ac51e5574c520b P d4830a9a55864449b2c7ec66d577afdc [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 15:07:23.972970 17651 tablet_server.cc:195] TabletServer@127.17.60.193:0 shutdown complete.
I20250112 15:07:23.973810 17651 tablet_server.cc:178] TabletServer@127.17.60.194:0 shutting down...
I20250112 15:07:23.976845 17651 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250112 15:07:23.977001 17651 tablet_replica.cc:331] T 5b6b4427200a48f1b0ac51e5574c520b P 08f2c47737324d57be0949b15e1d7f04: stopping tablet replica
I20250112 15:07:23.977077 17651 raft_consensus.cc:2238] T 5b6b4427200a48f1b0ac51e5574c520b P 08f2c47737324d57be0949b15e1d7f04 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250112 15:07:23.977126 17651 raft_consensus.cc:2267] T 5b6b4427200a48f1b0ac51e5574c520b P 08f2c47737324d57be0949b15e1d7f04 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 15:07:23.988771 17651 tablet_server.cc:195] TabletServer@127.17.60.194:0 shutdown complete.
I20250112 15:07:23.989661 17651 tablet_server.cc:178] TabletServer@127.17.60.195:0 shutting down...
I20250112 15:07:23.992684 17651 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250112 15:07:23.992871 17651 tablet_replica.cc:331] T 5b6b4427200a48f1b0ac51e5574c520b P 2021b61b34df4d1db1d3269848a30b34: stopping tablet replica
I20250112 15:07:23.992919 17651 raft_consensus.cc:2238] T 5b6b4427200a48f1b0ac51e5574c520b P 2021b61b34df4d1db1d3269848a30b34 [term 1 LEADER]: Raft consensus shutting down.
I20250112 15:07:23.992992 17651 raft_consensus.cc:2267] T 5b6b4427200a48f1b0ac51e5574c520b P 2021b61b34df4d1db1d3269848a30b34 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 15:07:24.004774 17651 tablet_server.cc:195] TabletServer@127.17.60.195:0 shutdown complete.
I20250112 15:07:24.005596 17651 master.cc:537] Master@127.17.60.254:39221 shutting down...
I20250112 15:07:24.007856 17651 raft_consensus.cc:2238] T 00000000000000000000000000000000 P b3f2173c64b24bada17b7788fe45b77c [term 1 LEADER]: Raft consensus shutting down.
I20250112 15:07:24.007915 17651 raft_consensus.cc:2267] T 00000000000000000000000000000000 P b3f2173c64b24bada17b7788fe45b77c [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 15:07:24.007938 17651 tablet_replica.cc:331] T 00000000000000000000000000000000 P b3f2173c64b24bada17b7788fe45b77c: stopping tablet replica
I20250112 15:07:24.019968 17651 master.cc:559] Master@127.17.60.254:39221 shutdown complete.
[ OK ] AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy (12645 ms)
[ RUN ] AutoRebalancerTest.TestMaxMovesPerServer
I20250112 15:07:24.024272 17651 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.17.60.254:42121
I20250112 15:07:24.024427 17651 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250112 15:07:24.025728 20947 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250112 15:07:24.026494 20950 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250112 15:07:24.027218 20948 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250112 15:07:24.028331 17651 server_base.cc:1034] running on GCE node
I20250112 15:07:24.028421 17651 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250112 15:07:24.028453 17651 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250112 15:07:24.028470 17651 hybrid_clock.cc:648] HybridClock initialized: now 1736694444028469 us; error 0 us; skew 500 ppm
I20250112 15:07:24.033588 17651 webserver.cc:458] Webserver started at http://127.17.60.254:42787/ using document root <none> and password file <none>
I20250112 15:07:24.033668 17651 fs_manager.cc:362] Metadata directory not provided
I20250112 15:07:24.033721 17651 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250112 15:07:24.033767 17651 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250112 15:07:24.034076 17651 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task95dLSD/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1736694351070047-17651-0/minicluster-data/master-0-root/instance:
uuid: "99889f1e9af44c26ae27b2aa189394be"
format_stamp: "Formatted at 2025-01-12 15:07:24 on dist-test-slave-b4nz"
I20250112 15:07:24.034956 17651 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20250112 15:07:24.035396 20955 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250112 15:07:24.035543 17651 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20250112 15:07:24.035589 17651 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task95dLSD/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1736694351070047-17651-0/minicluster-data/master-0-root
uuid: "99889f1e9af44c26ae27b2aa189394be"
format_stamp: "Formatted at 2025-01-12 15:07:24 on dist-test-slave-b4nz"
I20250112 15:07:24.035626 17651 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task95dLSD/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1736694351070047-17651-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-task95dLSD/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1736694351070047-17651-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-task95dLSD/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1736694351070047-17651-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250112 15:07:24.057749 17651 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250112 15:07:24.057953 17651 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250112 15:07:24.060295 17651 rpc_server.cc:307] RPC server started. Bound to: 127.17.60.254:42121
I20250112 15:07:24.061848 21007 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.17.60.254:42121 every 8 connection(s)
I20250112 15:07:24.062031 21008 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 15:07:24.063088 21008 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 99889f1e9af44c26ae27b2aa189394be: Bootstrap starting.
I20250112 15:07:24.063405 21008 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 99889f1e9af44c26ae27b2aa189394be: Neither blocks nor log segments found. Creating new log.
I20250112 15:07:24.063866 21008 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 99889f1e9af44c26ae27b2aa189394be: No bootstrap required, opened a new log
I20250112 15:07:24.064005 21008 raft_consensus.cc:357] T 00000000000000000000000000000000 P 99889f1e9af44c26ae27b2aa189394be [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "99889f1e9af44c26ae27b2aa189394be" member_type: VOTER }
I20250112 15:07:24.064054 21008 raft_consensus.cc:383] T 00000000000000000000000000000000 P 99889f1e9af44c26ae27b2aa189394be [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 15:07:24.064075 21008 raft_consensus.cc:738] T 00000000000000000000000000000000 P 99889f1e9af44c26ae27b2aa189394be [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 99889f1e9af44c26ae27b2aa189394be, State: Initialized, Role: FOLLOWER
I20250112 15:07:24.064124 21008 consensus_queue.cc:260] T 00000000000000000000000000000000 P 99889f1e9af44c26ae27b2aa189394be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "99889f1e9af44c26ae27b2aa189394be" member_type: VOTER }
I20250112 15:07:24.064155 21008 raft_consensus.cc:397] T 00000000000000000000000000000000 P 99889f1e9af44c26ae27b2aa189394be [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250112 15:07:24.064177 21008 raft_consensus.cc:491] T 00000000000000000000000000000000 P 99889f1e9af44c26ae27b2aa189394be [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250112 15:07:24.064204 21008 raft_consensus.cc:3054] T 00000000000000000000000000000000 P 99889f1e9af44c26ae27b2aa189394be [term 0 FOLLOWER]: Advancing to term 1
I20250112 15:07:24.064666 21008 raft_consensus.cc:513] T 00000000000000000000000000000000 P 99889f1e9af44c26ae27b2aa189394be [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "99889f1e9af44c26ae27b2aa189394be" member_type: VOTER }
I20250112 15:07:24.064718 21008 leader_election.cc:304] T 00000000000000000000000000000000 P 99889f1e9af44c26ae27b2aa189394be [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 99889f1e9af44c26ae27b2aa189394be; no voters:
I20250112 15:07:24.064812 21008 leader_election.cc:290] T 00000000000000000000000000000000 P 99889f1e9af44c26ae27b2aa189394be [CANDIDATE]: Term 1 election: Requested vote from peers
I20250112 15:07:24.064860 21011 raft_consensus.cc:2798] T 00000000000000000000000000000000 P 99889f1e9af44c26ae27b2aa189394be [term 1 FOLLOWER]: Leader election won for term 1
I20250112 15:07:24.064941 21008 sys_catalog.cc:564] T 00000000000000000000000000000000 P 99889f1e9af44c26ae27b2aa189394be [sys.catalog]: configured and running, proceeding with master startup.
I20250112 15:07:24.064993 21011 raft_consensus.cc:695] T 00000000000000000000000000000000 P 99889f1e9af44c26ae27b2aa189394be [term 1 LEADER]: Becoming Leader. State: Replica: 99889f1e9af44c26ae27b2aa189394be, State: Running, Role: LEADER
I20250112 15:07:24.065053 21011 consensus_queue.cc:237] T 00000000000000000000000000000000 P 99889f1e9af44c26ae27b2aa189394be [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "99889f1e9af44c26ae27b2aa189394be" member_type: VOTER }
I20250112 15:07:24.065243 21012 sys_catalog.cc:455] T 00000000000000000000000000000000 P 99889f1e9af44c26ae27b2aa189394be [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "99889f1e9af44c26ae27b2aa189394be" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "99889f1e9af44c26ae27b2aa189394be" member_type: VOTER } }
I20250112 15:07:24.065258 21013 sys_catalog.cc:455] T 00000000000000000000000000000000 P 99889f1e9af44c26ae27b2aa189394be [sys.catalog]: SysCatalogTable state changed. Reason: New leader 99889f1e9af44c26ae27b2aa189394be. Latest consensus state: current_term: 1 leader_uuid: "99889f1e9af44c26ae27b2aa189394be" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "99889f1e9af44c26ae27b2aa189394be" member_type: VOTER } }
I20250112 15:07:24.065315 21012 sys_catalog.cc:458] T 00000000000000000000000000000000 P 99889f1e9af44c26ae27b2aa189394be [sys.catalog]: This master's current role is: LEADER
I20250112 15:07:24.065332 21013 sys_catalog.cc:458] T 00000000000000000000000000000000 P 99889f1e9af44c26ae27b2aa189394be [sys.catalog]: This master's current role is: LEADER
I20250112 15:07:24.065613 21016 catalog_manager.cc:1476] Loading table and tablet metadata into memory...
I20250112 15:07:24.065742 21016 catalog_manager.cc:1485] Initializing Kudu cluster ID...
I20250112 15:07:24.066140 17651 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20250112 15:07:24.066301 21016 catalog_manager.cc:1348] Generated new cluster ID: 2b543f922f0343eaaba7ac4cfbc43531
I20250112 15:07:24.066334 21016 catalog_manager.cc:1496] Initializing Kudu internal certificate authority...
I20250112 15:07:24.077525 21016 catalog_manager.cc:1371] Generated new certificate authority record
I20250112 15:07:24.077930 21016 catalog_manager.cc:1505] Loading token signing keys...
I20250112 15:07:24.084645 21016 catalog_manager.cc:5899] T 00000000000000000000000000000000 P 99889f1e9af44c26ae27b2aa189394be: Generated new TSK 0
I20250112 15:07:24.084717 21016 catalog_manager.cc:1515] Initializing in-progress tserver states...
I20250112 15:07:24.097944 17651 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250112 15:07:24.100006 21032 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250112 15:07:24.100303 21030 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250112 15:07:24.101125 21029 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250112 15:07:24.101933 17651 server_base.cc:1034] running on GCE node
I20250112 15:07:24.102031 17651 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250112 15:07:24.102063 17651 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250112 15:07:24.102080 17651 hybrid_clock.cc:648] HybridClock initialized: now 1736694444102080 us; error 0 us; skew 500 ppm
I20250112 15:07:24.102713 17651 webserver.cc:458] Webserver started at http://127.17.60.193:44811/ using document root <none> and password file <none>
I20250112 15:07:24.102820 17651 fs_manager.cc:362] Metadata directory not provided
I20250112 15:07:24.102872 17651 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250112 15:07:24.102926 17651 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250112 15:07:24.103217 17651 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task95dLSD/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1736694351070047-17651-0/minicluster-data/ts-0-root/instance:
uuid: "2537bbbd1a9947ae9facd87567e3ccb9"
format_stamp: "Formatted at 2025-01-12 15:07:24 on dist-test-slave-b4nz"
I20250112 15:07:24.104101 17651 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20250112 15:07:24.104496 21037 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250112 15:07:24.104637 17651 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20250112 15:07:24.104682 17651 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task95dLSD/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1736694351070047-17651-0/minicluster-data/ts-0-root
uuid: "2537bbbd1a9947ae9facd87567e3ccb9"
format_stamp: "Formatted at 2025-01-12 15:07:24 on dist-test-slave-b4nz"
I20250112 15:07:24.104717 17651 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task95dLSD/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1736694351070047-17651-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-task95dLSD/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1736694351070047-17651-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-task95dLSD/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1736694351070047-17651-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250112 15:07:24.133715 17651 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250112 15:07:24.133934 17651 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250112 15:07:24.134084 17651 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250112 15:07:24.134279 17651 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250112 15:07:24.134308 17651 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250112 15:07:24.134335 17651 ts_tablet_manager.cc:610] Registered 0 tablets
I20250112 15:07:24.134361 17651 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250112 15:07:24.136950 17651 rpc_server.cc:307] RPC server started. Bound to: 127.17.60.193:33173
I20250112 15:07:24.137048 21100 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.17.60.193:33173 every 8 connection(s)
I20250112 15:07:24.137337 17651 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250112 15:07:24.139103 21106 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250112 15:07:24.139458 21105 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250112 15:07:24.140336 17651 server_base.cc:1034] running on GCE node
I20250112 15:07:24.140640 21101 heartbeater.cc:346] Connected to a master server at 127.17.60.254:42121
I20250112 15:07:24.140697 21101 heartbeater.cc:463] Registering TS with master...
I20250112 15:07:24.140797 21101 heartbeater.cc:510] Master 127.17.60.254:42121 requested a full tablet report, sending...
W20250112 15:07:24.140960 21108 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250112 15:07:24.141081 20972 ts_manager.cc:194] Registered new tserver with Master: 2537bbbd1a9947ae9facd87567e3ccb9 (127.17.60.193:33173)
I20250112 15:07:24.141093 17651 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250112 15:07:24.141142 17651 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250112 15:07:24.141168 17651 hybrid_clock.cc:648] HybridClock initialized: now 1736694444141168 us; error 0 us; skew 500 ppm
I20250112 15:07:24.141590 20972 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:39606
I20250112 15:07:24.141834 17651 webserver.cc:458] Webserver started at http://127.17.60.194:36869/ using document root <none> and password file <none>
I20250112 15:07:24.141954 17651 fs_manager.cc:362] Metadata directory not provided
I20250112 15:07:24.141993 17651 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250112 15:07:24.142041 17651 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250112 15:07:24.142297 17651 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task95dLSD/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1736694351070047-17651-0/minicluster-data/ts-1-root/instance:
uuid: "c561ec3374b94b95be0c155d80ff0bff"
format_stamp: "Formatted at 2025-01-12 15:07:24 on dist-test-slave-b4nz"
I20250112 15:07:24.143133 17651 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20250112 15:07:24.143551 21113 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250112 15:07:24.143676 17651 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20250112 15:07:24.143724 17651 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task95dLSD/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1736694351070047-17651-0/minicluster-data/ts-1-root
uuid: "c561ec3374b94b95be0c155d80ff0bff"
format_stamp: "Formatted at 2025-01-12 15:07:24 on dist-test-slave-b4nz"
I20250112 15:07:24.143760 17651 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task95dLSD/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1736694351070047-17651-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-task95dLSD/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1736694351070047-17651-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-task95dLSD/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1736694351070047-17651-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250112 15:07:24.150878 17651 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250112 15:07:24.151067 17651 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250112 15:07:24.151173 17651 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250112 15:07:24.151332 17651 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250112 15:07:24.151356 17651 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250112 15:07:24.151373 17651 ts_tablet_manager.cc:610] Registered 0 tablets
I20250112 15:07:24.151404 17651 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250112 15:07:24.154137 17651 rpc_server.cc:307] RPC server started. Bound to: 127.17.60.194:33627
I20250112 15:07:24.154196 21176 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.17.60.194:33627 every 8 connection(s)
I20250112 15:07:24.154477 17651 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250112 15:07:24.155680 21180 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250112 15:07:24.156157 21181 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250112 15:07:24.157090 21183 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250112 15:07:24.157297 21177 heartbeater.cc:346] Connected to a master server at 127.17.60.254:42121
I20250112 15:07:24.157357 21177 heartbeater.cc:463] Registering TS with master...
I20250112 15:07:24.157456 21177 heartbeater.cc:510] Master 127.17.60.254:42121 requested a full tablet report, sending...
I20250112 15:07:24.157683 20972 ts_manager.cc:194] Registered new tserver with Master: c561ec3374b94b95be0c155d80ff0bff (127.17.60.194:33627)
I20250112 15:07:24.157966 17651 server_base.cc:1034] running on GCE node
I20250112 15:07:24.158075 17651 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250112 15:07:24.158106 17651 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250112 15:07:24.158123 17651 hybrid_clock.cc:648] HybridClock initialized: now 1736694444158124 us; error 0 us; skew 500 ppm
I20250112 15:07:24.158133 20972 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:39610
I20250112 15:07:24.158809 17651 webserver.cc:458] Webserver started at http://127.17.60.195:33715/ using document root <none> and password file <none>
I20250112 15:07:24.158874 17651 fs_manager.cc:362] Metadata directory not provided
I20250112 15:07:24.158924 17651 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250112 15:07:24.158957 17651 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250112 15:07:24.159193 17651 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task95dLSD/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1736694351070047-17651-0/minicluster-data/ts-2-root/instance:
uuid: "3710b74ed2044ce5995533df32e67351"
format_stamp: "Formatted at 2025-01-12 15:07:24 on dist-test-slave-b4nz"
I20250112 15:07:24.159989 17651 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20250112 15:07:24.160391 21188 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250112 15:07:24.160506 17651 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20250112 15:07:24.160552 17651 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task95dLSD/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1736694351070047-17651-0/minicluster-data/ts-2-root
uuid: "3710b74ed2044ce5995533df32e67351"
format_stamp: "Formatted at 2025-01-12 15:07:24 on dist-test-slave-b4nz"
I20250112 15:07:24.160619 17651 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task95dLSD/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1736694351070047-17651-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-task95dLSD/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1736694351070047-17651-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-task95dLSD/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1736694351070047-17651-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250112 15:07:24.169835 17651 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250112 15:07:24.170054 17651 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250112 15:07:24.170186 17651 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250112 15:07:24.170348 17651 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250112 15:07:24.170384 17651 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250112 15:07:24.170403 17651 ts_tablet_manager.cc:610] Registered 0 tablets
I20250112 15:07:24.170418 17651 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250112 15:07:24.173463 17651 rpc_server.cc:307] RPC server started. Bound to: 127.17.60.195:38589
I20250112 15:07:24.173525 21251 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.17.60.195:38589 every 8 connection(s)
I20250112 15:07:24.176456 21252 heartbeater.cc:346] Connected to a master server at 127.17.60.254:42121
I20250112 15:07:24.176515 21252 heartbeater.cc:463] Registering TS with master...
I20250112 15:07:24.176635 21252 heartbeater.cc:510] Master 127.17.60.254:42121 requested a full tablet report, sending...
I20250112 15:07:24.176828 20972 ts_manager.cc:194] Registered new tserver with Master: 3710b74ed2044ce5995533df32e67351 (127.17.60.195:38589)
I20250112 15:07:24.176916 17651 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.003282516s
I20250112 15:07:24.177166 20972 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:39618
I20250112 15:07:25.142117 21101 heartbeater.cc:502] Master 127.17.60.254:42121 was elected leader, sending a full tablet report...
I20250112 15:07:25.158581 21177 heartbeater.cc:502] Master 127.17.60.254:42121 was elected leader, sending a full tablet report...
I20250112 15:07:25.177629 21252 heartbeater.cc:502] Master 127.17.60.254:42121 was elected leader, sending a full tablet report...
I20250112 15:07:25.201838 17651 test_util.cc:274] Using random seed: -1350739541
I20250112 15:07:25.206933 20972 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39628:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "\004\001\000\252\252\252\n\004\001\000TUU\025\004\001\000\376\377\377\037\004\001\000\250\252\252*\004\001\000RUU5\004\001\000\374\377\377?\004\001\000\246\252\252J\004\001\000PUUU\004\001\000\372\377\377_\004\001\000\244\252\252j\004\001\000NUUu""\004\001\000\252\252\252\n\004\001\000TUU\025\004\001\000\376\377\377\037\004\001\000\250\252\252*\004\001\000RUU5\004\001\000\374\377\377?\004\001\000\246\252\252J\004\001\000PUUU\004\001\000\372\377\377_\004\001\000\244\252\252j\004\001\000NUUu"
indirect_data: """"
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 15:07:25.207104 20972 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-workload in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 15:07:25.212821 21141 tablet_service.cc:1467] Processing CreateTablet for tablet 02c33d457b6b47138dd712850819b619 (DEFAULT_TABLE table=test-workload [id=cdaf6d3a65a54ea08e8e2c0ecdfc9de7]), partition=RANGE (key) PARTITION VALUES < 178956970
I20250112 15:07:25.212925 21134 tablet_service.cc:1467] Processing CreateTablet for tablet bfaeae584ec04772b2d6fcd088022117 (DEFAULT_TABLE table=test-workload [id=cdaf6d3a65a54ea08e8e2c0ecdfc9de7]), partition=RANGE (key) PARTITION 1252698790 <= VALUES < 1431655760
I20250112 15:07:25.213021 21141 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 02c33d457b6b47138dd712850819b619. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 15:07:25.213052 21213 tablet_service.cc:1467] Processing CreateTablet for tablet ef90b704208140a3983ffe6105939319 (DEFAULT_TABLE table=test-workload [id=cdaf6d3a65a54ea08e8e2c0ecdfc9de7]), partition=RANGE (key) PARTITION 536870910 <= VALUES < 715827880
I20250112 15:07:25.213078 21216 tablet_service.cc:1467] Processing CreateTablet for tablet 02c33d457b6b47138dd712850819b619 (DEFAULT_TABLE table=test-workload [id=cdaf6d3a65a54ea08e8e2c0ecdfc9de7]), partition=RANGE (key) PARTITION VALUES < 178956970
I20250112 15:07:25.213157 21213 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ef90b704208140a3983ffe6105939319. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 15:07:25.213236 21216 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 02c33d457b6b47138dd712850819b619. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 15:07:25.214520 21216 tablet_service.cc:1467] Processing CreateTablet for tablet 16f16fcd5cf347e5853189dc4da55c7f (DEFAULT_TABLE table=test-workload [id=cdaf6d3a65a54ea08e8e2c0ecdfc9de7]), partition=RANGE (key) PARTITION 1789569700 <= VALUES < 1968526670
I20250112 15:07:25.214550 21213 tablet_service.cc:1467] Processing CreateTablet for tablet 57e47b07fdcf434e96ca5dd0354e93b2 (DEFAULT_TABLE table=test-workload [id=cdaf6d3a65a54ea08e8e2c0ecdfc9de7]), partition=RANGE (key) PARTITION 1968526670 <= VALUES
I20250112 15:07:25.214609 21216 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 16f16fcd5cf347e5853189dc4da55c7f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 15:07:25.214596 21140 tablet_service.cc:1467] Processing CreateTablet for tablet 1eb2b7e6edd34a7b83f9b918d3cdf2ed (DEFAULT_TABLE table=test-workload [id=cdaf6d3a65a54ea08e8e2c0ecdfc9de7]), partition=RANGE (key) PARTITION 178956970 <= VALUES < 357913940
I20250112 15:07:25.212862 21138 tablet_service.cc:1467] Processing CreateTablet for tablet ef90b704208140a3983ffe6105939319 (DEFAULT_TABLE table=test-workload [id=cdaf6d3a65a54ea08e8e2c0ecdfc9de7]), partition=RANGE (key) PARTITION 536870910 <= VALUES < 715827880
I20250112 15:07:25.214713 21137 tablet_service.cc:1467] Processing CreateTablet for tablet 5bcec60d3b974a9bb24aedfc738e1594 (DEFAULT_TABLE table=test-workload [id=cdaf6d3a65a54ea08e8e2c0ecdfc9de7]), partition=RANGE (key) PARTITION 715827880 <= VALUES < 894784850
I20250112 15:07:25.214722 21140 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1eb2b7e6edd34a7b83f9b918d3cdf2ed. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 15:07:25.214769 21213 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 57e47b07fdcf434e96ca5dd0354e93b2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 15:07:25.214790 21137 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5bcec60d3b974a9bb24aedfc738e1594. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 15:07:25.213052 21208 tablet_service.cc:1467] Processing CreateTablet for tablet 69c23627ab854000b360030c384dc966 (DEFAULT_TABLE table=test-workload [id=cdaf6d3a65a54ea08e8e2c0ecdfc9de7]), partition=RANGE (key) PARTITION 1431655760 <= VALUES < 1610612730
I20250112 15:07:25.215816 21208 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 69c23627ab854000b360030c384dc966. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 15:07:25.215863 21134 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bfaeae584ec04772b2d6fcd088022117. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 15:07:25.215924 21136 tablet_service.cc:1467] Processing CreateTablet for tablet 8c7dbaf285a04da9b4eec92a29ab7d6c (DEFAULT_TABLE table=test-workload [id=cdaf6d3a65a54ea08e8e2c0ecdfc9de7]), partition=RANGE (key) PARTITION 894784850 <= VALUES < 1073741820
I20250112 15:07:25.216022 21136 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8c7dbaf285a04da9b4eec92a29ab7d6c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 15:07:25.216655 21138 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ef90b704208140a3983ffe6105939319. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 15:07:25.217229 21215 tablet_service.cc:1467] Processing CreateTablet for tablet 1eb2b7e6edd34a7b83f9b918d3cdf2ed (DEFAULT_TABLE table=test-workload [id=cdaf6d3a65a54ea08e8e2c0ecdfc9de7]), partition=RANGE (key) PARTITION 178956970 <= VALUES < 357913940
I20250112 15:07:25.217294 21215 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1eb2b7e6edd34a7b83f9b918d3cdf2ed. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 15:07:25.217708 21273 tablet_bootstrap.cc:492] T 02c33d457b6b47138dd712850819b619 P c561ec3374b94b95be0c155d80ff0bff: Bootstrap starting.
I20250112 15:07:25.218042 21135 tablet_service.cc:1467] Processing CreateTablet for tablet 9601bcda24ed42e7b93ab9740f1e271e (DEFAULT_TABLE table=test-workload [id=cdaf6d3a65a54ea08e8e2c0ecdfc9de7]), partition=RANGE (key) PARTITION 1073741820 <= VALUES < 1252698790
I20250112 15:07:25.218052 21273 tablet_bootstrap.cc:654] T 02c33d457b6b47138dd712850819b619 P c561ec3374b94b95be0c155d80ff0bff: Neither blocks nor log segments found. Creating new log.
I20250112 15:07:25.218106 21272 tablet_bootstrap.cc:492] T ef90b704208140a3983ffe6105939319 P 3710b74ed2044ce5995533df32e67351: Bootstrap starting.
I20250112 15:07:25.218245 21214 tablet_service.cc:1467] Processing CreateTablet for tablet 80cc505a04394c53a18c5beeded4655d (DEFAULT_TABLE table=test-workload [id=cdaf6d3a65a54ea08e8e2c0ecdfc9de7]), partition=RANGE (key) PARTITION 357913940 <= VALUES < 536870910
I20250112 15:07:25.218322 21214 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 80cc505a04394c53a18c5beeded4655d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 15:07:25.218556 21272 tablet_bootstrap.cc:654] T ef90b704208140a3983ffe6105939319 P 3710b74ed2044ce5995533df32e67351: Neither blocks nor log segments found. Creating new log.
I20250112 15:07:25.218750 21273 tablet_bootstrap.cc:492] T 02c33d457b6b47138dd712850819b619 P c561ec3374b94b95be0c155d80ff0bff: No bootstrap required, opened a new log
I20250112 15:07:25.218799 21273 ts_tablet_manager.cc:1397] T 02c33d457b6b47138dd712850819b619 P c561ec3374b94b95be0c155d80ff0bff: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20250112 15:07:25.218984 21273 raft_consensus.cc:357] T 02c33d457b6b47138dd712850819b619 P c561ec3374b94b95be0c155d80ff0bff [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3710b74ed2044ce5995533df32e67351" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 38589 } } peers { permanent_uuid: "2537bbbd1a9947ae9facd87567e3ccb9" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 33173 } } peers { permanent_uuid: "c561ec3374b94b95be0c155d80ff0bff" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 33627 } }
I20250112 15:07:25.219049 21273 raft_consensus.cc:383] T 02c33d457b6b47138dd712850819b619 P c561ec3374b94b95be0c155d80ff0bff [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 15:07:25.219069 21273 raft_consensus.cc:738] T 02c33d457b6b47138dd712850819b619 P c561ec3374b94b95be0c155d80ff0bff [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c561ec3374b94b95be0c155d80ff0bff, State: Initialized, Role: FOLLOWER
I20250112 15:07:25.219080 21272 tablet_bootstrap.cc:492] T ef90b704208140a3983ffe6105939319 P 3710b74ed2044ce5995533df32e67351: No bootstrap required, opened a new log
I20250112 15:07:25.219118 21272 ts_tablet_manager.cc:1397] T ef90b704208140a3983ffe6105939319 P 3710b74ed2044ce5995533df32e67351: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20250112 15:07:25.219126 21273 consensus_queue.cc:260] T 02c33d457b6b47138dd712850819b619 P c561ec3374b94b95be0c155d80ff0bff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3710b74ed2044ce5995533df32e67351" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 38589 } } peers { permanent_uuid: "2537bbbd1a9947ae9facd87567e3ccb9" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 33173 } } peers { permanent_uuid: "c561ec3374b94b95be0c155d80ff0bff" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 33627 } }
I20250112 15:07:25.219254 21272 raft_consensus.cc:357] T ef90b704208140a3983ffe6105939319 P 3710b74ed2044ce5995533df32e67351 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2537bbbd1a9947ae9facd87567e3ccb9" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 33173 } } peers { permanent_uuid: "c561ec3374b94b95be0c155d80ff0bff" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 33627 } } peers { permanent_uuid: "3710b74ed2044ce5995533df32e67351" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 38589 } }
I20250112 15:07:25.219291 21273 ts_tablet_manager.cc:1428] T 02c33d457b6b47138dd712850819b619 P c561ec3374b94b95be0c155d80ff0bff: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20250112 15:07:25.219336 21272 raft_consensus.cc:383] T ef90b704208140a3983ffe6105939319 P 3710b74ed2044ce5995533df32e67351 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 15:07:25.219357 21272 raft_consensus.cc:738] T ef90b704208140a3983ffe6105939319 P 3710b74ed2044ce5995533df32e67351 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3710b74ed2044ce5995533df32e67351, State: Initialized, Role: FOLLOWER
I20250112 15:07:25.219372 21273 tablet_bootstrap.cc:492] T 5bcec60d3b974a9bb24aedfc738e1594 P c561ec3374b94b95be0c155d80ff0bff: Bootstrap starting.
I20250112 15:07:25.219412 21272 consensus_queue.cc:260] T ef90b704208140a3983ffe6105939319 P 3710b74ed2044ce5995533df32e67351 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2537bbbd1a9947ae9facd87567e3ccb9" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 33173 } } peers { permanent_uuid: "c561ec3374b94b95be0c155d80ff0bff" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 33627 } } peers { permanent_uuid: "3710b74ed2044ce5995533df32e67351" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 38589 } }
I20250112 15:07:25.219527 21133 tablet_service.cc:1467] Processing CreateTablet for tablet 69c23627ab854000b360030c384dc966 (DEFAULT_TABLE table=test-workload [id=cdaf6d3a65a54ea08e8e2c0ecdfc9de7]), partition=RANGE (key) PARTITION 1431655760 <= VALUES < 1610612730
I20250112 15:07:25.219566 21272 ts_tablet_manager.cc:1428] T ef90b704208140a3983ffe6105939319 P 3710b74ed2044ce5995533df32e67351: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20250112 15:07:25.219623 21272 tablet_bootstrap.cc:492] T 02c33d457b6b47138dd712850819b619 P 3710b74ed2044ce5995533df32e67351: Bootstrap starting.
I20250112 15:07:25.219801 21132 tablet_service.cc:1467] Processing CreateTablet for tablet 9d4bbba799324d7fbbac082520c65686 (DEFAULT_TABLE table=test-workload [id=cdaf6d3a65a54ea08e8e2c0ecdfc9de7]), partition=RANGE (key) PARTITION 1610612730 <= VALUES < 1789569700
I20250112 15:07:25.219941 21211 tablet_service.cc:1467] Processing CreateTablet for tablet 8c7dbaf285a04da9b4eec92a29ab7d6c (DEFAULT_TABLE table=test-workload [id=cdaf6d3a65a54ea08e8e2c0ecdfc9de7]), partition=RANGE (key) PARTITION 894784850 <= VALUES < 1073741820
I20250112 15:07:25.220016 21211 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8c7dbaf285a04da9b4eec92a29ab7d6c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 15:07:25.220098 21272 tablet_bootstrap.cc:654] T 02c33d457b6b47138dd712850819b619 P 3710b74ed2044ce5995533df32e67351: Neither blocks nor log segments found. Creating new log.
I20250112 15:07:25.220678 21272 tablet_bootstrap.cc:492] T 02c33d457b6b47138dd712850819b619 P 3710b74ed2044ce5995533df32e67351: No bootstrap required, opened a new log
I20250112 15:07:25.220715 21272 ts_tablet_manager.cc:1397] T 02c33d457b6b47138dd712850819b619 P 3710b74ed2044ce5995533df32e67351: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20250112 15:07:25.220862 21272 raft_consensus.cc:357] T 02c33d457b6b47138dd712850819b619 P 3710b74ed2044ce5995533df32e67351 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3710b74ed2044ce5995533df32e67351" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 38589 } } peers { permanent_uuid: "2537bbbd1a9947ae9facd87567e3ccb9" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 33173 } } peers { permanent_uuid: "c561ec3374b94b95be0c155d80ff0bff" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 33627 } }
I20250112 15:07:25.220924 21272 raft_consensus.cc:383] T 02c33d457b6b47138dd712850819b619 P 3710b74ed2044ce5995533df32e67351 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 15:07:25.220942 21272 raft_consensus.cc:738] T 02c33d457b6b47138dd712850819b619 P 3710b74ed2044ce5995533df32e67351 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3710b74ed2044ce5995533df32e67351, State: Initialized, Role: FOLLOWER
I20250112 15:07:25.220999 21272 consensus_queue.cc:260] T 02c33d457b6b47138dd712850819b619 P 3710b74ed2044ce5995533df32e67351 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3710b74ed2044ce5995533df32e67351" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 38589 } } peers { permanent_uuid: "2537bbbd1a9947ae9facd87567e3ccb9" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 33173 } } peers { permanent_uuid: "c561ec3374b94b95be0c155d80ff0bff" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 33627 } }
I20250112 15:07:25.221071 21272 ts_tablet_manager.cc:1428] T 02c33d457b6b47138dd712850819b619 P 3710b74ed2044ce5995533df32e67351: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20250112 15:07:25.221117 21272 tablet_bootstrap.cc:492] T 16f16fcd5cf347e5853189dc4da55c7f P 3710b74ed2044ce5995533df32e67351: Bootstrap starting.
I20250112 15:07:25.221369 21210 tablet_service.cc:1467] Processing CreateTablet for tablet 9601bcda24ed42e7b93ab9740f1e271e (DEFAULT_TABLE table=test-workload [id=cdaf6d3a65a54ea08e8e2c0ecdfc9de7]), partition=RANGE (key) PARTITION 1073741820 <= VALUES < 1252698790
I20250112 15:07:25.221463 21210 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9601bcda24ed42e7b93ab9740f1e271e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 15:07:25.221504 21272 tablet_bootstrap.cc:654] T 16f16fcd5cf347e5853189dc4da55c7f P 3710b74ed2044ce5995533df32e67351: Neither blocks nor log segments found. Creating new log.
I20250112 15:07:25.222051 21272 tablet_bootstrap.cc:492] T 16f16fcd5cf347e5853189dc4da55c7f P 3710b74ed2044ce5995533df32e67351: No bootstrap required, opened a new log
I20250112 15:07:25.222107 21272 ts_tablet_manager.cc:1397] T 16f16fcd5cf347e5853189dc4da55c7f P 3710b74ed2044ce5995533df32e67351: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20250112 15:07:25.222255 21272 raft_consensus.cc:357] T 16f16fcd5cf347e5853189dc4da55c7f P 3710b74ed2044ce5995533df32e67351 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2537bbbd1a9947ae9facd87567e3ccb9" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 33173 } } peers { permanent_uuid: "c561ec3374b94b95be0c155d80ff0bff" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 33627 } } peers { permanent_uuid: "3710b74ed2044ce5995533df32e67351" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 38589 } }
I20250112 15:07:25.222316 21272 raft_consensus.cc:383] T 16f16fcd5cf347e5853189dc4da55c7f P 3710b74ed2044ce5995533df32e67351 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 15:07:25.222340 21272 raft_consensus.cc:738] T 16f16fcd5cf347e5853189dc4da55c7f P 3710b74ed2044ce5995533df32e67351 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3710b74ed2044ce5995533df32e67351, State: Initialized, Role: FOLLOWER
I20250112 15:07:25.222400 21272 consensus_queue.cc:260] T 16f16fcd5cf347e5853189dc4da55c7f P 3710b74ed2044ce5995533df32e67351 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2537bbbd1a9947ae9facd87567e3ccb9" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 33173 } } peers { permanent_uuid: "c561ec3374b94b95be0c155d80ff0bff" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 33627 } } peers { permanent_uuid: "3710b74ed2044ce5995533df32e67351" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 38589 } }
I20250112 15:07:25.222492 21272 ts_tablet_manager.cc:1428] T 16f16fcd5cf347e5853189dc4da55c7f P 3710b74ed2044ce5995533df32e67351: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20250112 15:07:25.222548 21272 tablet_bootstrap.cc:492] T 57e47b07fdcf434e96ca5dd0354e93b2 P 3710b74ed2044ce5995533df32e67351: Bootstrap starting.
I20250112 15:07:25.222573 21209 tablet_service.cc:1467] Processing CreateTablet for tablet bfaeae584ec04772b2d6fcd088022117 (DEFAULT_TABLE table=test-workload [id=cdaf6d3a65a54ea08e8e2c0ecdfc9de7]), partition=RANGE (key) PARTITION 1252698790 <= VALUES < 1431655760
I20250112 15:07:25.222666 21209 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bfaeae584ec04772b2d6fcd088022117. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 15:07:25.223022 21272 tablet_bootstrap.cc:654] T 57e47b07fdcf434e96ca5dd0354e93b2 P 3710b74ed2044ce5995533df32e67351: Neither blocks nor log segments found. Creating new log.
I20250112 15:07:25.223603 21272 tablet_bootstrap.cc:492] T 57e47b07fdcf434e96ca5dd0354e93b2 P 3710b74ed2044ce5995533df32e67351: No bootstrap required, opened a new log
I20250112 15:07:25.223654 21272 ts_tablet_manager.cc:1397] T 57e47b07fdcf434e96ca5dd0354e93b2 P 3710b74ed2044ce5995533df32e67351: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20250112 15:07:25.223789 21272 raft_consensus.cc:357] T 57e47b07fdcf434e96ca5dd0354e93b2 P 3710b74ed2044ce5995533df32e67351 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2537bbbd1a9947ae9facd87567e3ccb9" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 33173 } } peers { permanent_uuid: "c561ec3374b94b95be0c155d80ff0bff" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 33627 } } peers { permanent_uuid: "3710b74ed2044ce5995533df32e67351" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 38589 } }
I20250112 15:07:25.223850 21272 raft_consensus.cc:383] T 57e47b07fdcf434e96ca5dd0354e93b2 P 3710b74ed2044ce5995533df32e67351 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 15:07:25.223837 21207 tablet_service.cc:1467] Processing CreateTablet for tablet 9d4bbba799324d7fbbac082520c65686 (DEFAULT_TABLE table=test-workload [id=cdaf6d3a65a54ea08e8e2c0ecdfc9de7]), partition=RANGE (key) PARTITION 1610612730 <= VALUES < 1789569700
I20250112 15:07:25.223942 21065 tablet_service.cc:1467] Processing CreateTablet for tablet 02c33d457b6b47138dd712850819b619 (DEFAULT_TABLE table=test-workload [id=cdaf6d3a65a54ea08e8e2c0ecdfc9de7]), partition=RANGE (key) PARTITION VALUES < 178956970
I20250112 15:07:25.223991 21207 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9d4bbba799324d7fbbac082520c65686. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 15:07:25.224035 21065 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 02c33d457b6b47138dd712850819b619. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 15:07:25.225147 21281 tablet_bootstrap.cc:492] T 02c33d457b6b47138dd712850819b619 P 2537bbbd1a9947ae9facd87567e3ccb9: Bootstrap starting.
I20250112 15:07:25.225299 21272 raft_consensus.cc:738] T 57e47b07fdcf434e96ca5dd0354e93b2 P 3710b74ed2044ce5995533df32e67351 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3710b74ed2044ce5995533df32e67351, State: Initialized, Role: FOLLOWER
I20250112 15:07:25.214661 21139 tablet_service.cc:1467] Processing CreateTablet for tablet 80cc505a04394c53a18c5beeded4655d (DEFAULT_TABLE table=test-workload [id=cdaf6d3a65a54ea08e8e2c0ecdfc9de7]), partition=RANGE (key) PARTITION 357913940 <= VALUES < 536870910
I20250112 15:07:25.225522 21272 consensus_queue.cc:260] T 57e47b07fdcf434e96ca5dd0354e93b2 P 3710b74ed2044ce5995533df32e67351 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2537bbbd1a9947ae9facd87567e3ccb9" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 33173 } } peers { permanent_uuid: "c561ec3374b94b95be0c155d80ff0bff" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 33627 } } peers { permanent_uuid: "3710b74ed2044ce5995533df32e67351" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 38589 } }
I20250112 15:07:25.225633 21272 ts_tablet_manager.cc:1428] T 57e47b07fdcf434e96ca5dd0354e93b2 P 3710b74ed2044ce5995533df32e67351: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20250112 15:07:25.225701 21272 tablet_bootstrap.cc:492] T 69c23627ab854000b360030c384dc966 P 3710b74ed2044ce5995533df32e67351: Bootstrap starting.
I20250112 15:07:25.225765 21061 tablet_service.cc:1467] Processing CreateTablet for tablet 8c7dbaf285a04da9b4eec92a29ab7d6c (DEFAULT_TABLE table=test-workload [id=cdaf6d3a65a54ea08e8e2c0ecdfc9de7]), partition=RANGE (key) PARTITION 894784850 <= VALUES < 1073741820
I20250112 15:07:25.225793 21281 tablet_bootstrap.cc:654] T 02c33d457b6b47138dd712850819b619 P 2537bbbd1a9947ae9facd87567e3ccb9: Neither blocks nor log segments found. Creating new log.
I20250112 15:07:25.225862 21061 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8c7dbaf285a04da9b4eec92a29ab7d6c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 15:07:25.225914 21064 tablet_service.cc:1467] Processing CreateTablet for tablet 80cc505a04394c53a18c5beeded4655d (DEFAULT_TABLE table=test-workload [id=cdaf6d3a65a54ea08e8e2c0ecdfc9de7]), partition=RANGE (key) PARTITION 357913940 <= VALUES < 536870910
I20250112 15:07:25.226003 21064 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 80cc505a04394c53a18c5beeded4655d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 15:07:25.226115 21272 tablet_bootstrap.cc:654] T 69c23627ab854000b360030c384dc966 P 3710b74ed2044ce5995533df32e67351: Neither blocks nor log segments found. Creating new log.
I20250112 15:07:25.226370 21281 tablet_bootstrap.cc:492] T 02c33d457b6b47138dd712850819b619 P 2537bbbd1a9947ae9facd87567e3ccb9: No bootstrap required, opened a new log
I20250112 15:07:25.226408 21281 ts_tablet_manager.cc:1397] T 02c33d457b6b47138dd712850819b619 P 2537bbbd1a9947ae9facd87567e3ccb9: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20250112 15:07:25.226568 21281 raft_consensus.cc:357] T 02c33d457b6b47138dd712850819b619 P 2537bbbd1a9947ae9facd87567e3ccb9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3710b74ed2044ce5995533df32e67351" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 38589 } } peers { permanent_uuid: "2537bbbd1a9947ae9facd87567e3ccb9" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 33173 } } peers { permanent_uuid: "c561ec3374b94b95be0c155d80ff0bff" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 33627 } }
I20250112 15:07:25.226624 21281 raft_consensus.cc:383] T 02c33d457b6b47138dd712850819b619 P 2537bbbd1a9947ae9facd87567e3ccb9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 15:07:25.226641 21281 raft_consensus.cc:738] T 02c33d457b6b47138dd712850819b619 P 2537bbbd1a9947ae9facd87567e3ccb9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2537bbbd1a9947ae9facd87567e3ccb9, State: Initialized, Role: FOLLOWER
I20250112 15:07:25.226691 21281 consensus_queue.cc:260] T 02c33d457b6b47138dd712850819b619 P 2537bbbd1a9947ae9facd87567e3ccb9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3710b74ed2044ce5995533df32e67351" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 38589 } } peers { permanent_uuid: "2537bbbd1a9947ae9facd87567e3ccb9" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 33173 } } peers { permanent_uuid: "c561ec3374b94b95be0c155d80ff0bff" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 33627 } }
I20250112 15:07:25.226861 21281 ts_tablet_manager.cc:1428] T 02c33d457b6b47138dd712850819b619 P 2537bbbd1a9947ae9facd87567e3ccb9: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20250112 15:07:25.227017 21061 tablet_service.cc:1467] Processing CreateTablet for tablet 57e47b07fdcf434e96ca5dd0354e93b2 (DEFAULT_TABLE table=test-workload [id=cdaf6d3a65a54ea08e8e2c0ecdfc9de7]), partition=RANGE (key) PARTITION 1968526670 <= VALUES
I20250112 15:07:25.227078 21061 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 57e47b07fdcf434e96ca5dd0354e93b2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 15:07:25.227087 21281 tablet_bootstrap.cc:492] T 8c7dbaf285a04da9b4eec92a29ab7d6c P 2537bbbd1a9947ae9facd87567e3ccb9: Bootstrap starting.
I20250112 15:07:25.227414 21063 tablet_service.cc:1467] Processing CreateTablet for tablet ef90b704208140a3983ffe6105939319 (DEFAULT_TABLE table=test-workload [id=cdaf6d3a65a54ea08e8e2c0ecdfc9de7]), partition=RANGE (key) PARTITION 536870910 <= VALUES < 715827880
I20250112 15:07:25.227499 21281 tablet_bootstrap.cc:654] T 8c7dbaf285a04da9b4eec92a29ab7d6c P 2537bbbd1a9947ae9facd87567e3ccb9: Neither blocks nor log segments found. Creating new log.
I20250112 15:07:25.227578 21062 tablet_service.cc:1467] Processing CreateTablet for tablet 5bcec60d3b974a9bb24aedfc738e1594 (DEFAULT_TABLE table=test-workload [id=cdaf6d3a65a54ea08e8e2c0ecdfc9de7]), partition=RANGE (key) PARTITION 715827880 <= VALUES < 894784850
I20250112 15:07:25.227730 21060 tablet_service.cc:1467] Processing CreateTablet for tablet 9601bcda24ed42e7b93ab9740f1e271e (DEFAULT_TABLE table=test-workload [id=cdaf6d3a65a54ea08e8e2c0ecdfc9de7]), partition=RANGE (key) PARTITION 1073741820 <= VALUES < 1252698790
I20250112 15:07:25.227507 21063 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ef90b704208140a3983ffe6105939319. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 15:07:25.227860 21059 tablet_service.cc:1467] Processing CreateTablet for tablet bfaeae584ec04772b2d6fcd088022117 (DEFAULT_TABLE table=test-workload [id=cdaf6d3a65a54ea08e8e2c0ecdfc9de7]), partition=RANGE (key) PARTITION 1252698790 <= VALUES < 1431655760
I20250112 15:07:25.227943 21062 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5bcec60d3b974a9bb24aedfc738e1594. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 15:07:25.227995 21058 tablet_service.cc:1467] Processing CreateTablet for tablet 69c23627ab854000b360030c384dc966 (DEFAULT_TABLE table=test-workload [id=cdaf6d3a65a54ea08e8e2c0ecdfc9de7]), partition=RANGE (key) PARTITION 1431655760 <= VALUES < 1610612730
I20250112 15:07:25.228068 21059 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bfaeae584ec04772b2d6fcd088022117. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 15:07:25.228199 21057 tablet_service.cc:1467] Processing CreateTablet for tablet 9d4bbba799324d7fbbac082520c65686 (DEFAULT_TABLE table=test-workload [id=cdaf6d3a65a54ea08e8e2c0ecdfc9de7]), partition=RANGE (key) PARTITION 1610612730 <= VALUES < 1789569700
I20250112 15:07:25.228266 21057 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9d4bbba799324d7fbbac082520c65686. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 15:07:25.229456 21281 tablet_bootstrap.cc:492] T 8c7dbaf285a04da9b4eec92a29ab7d6c P 2537bbbd1a9947ae9facd87567e3ccb9: No bootstrap required, opened a new log
I20250112 15:07:25.229485 21056 tablet_service.cc:1467] Processing CreateTablet for tablet 16f16fcd5cf347e5853189dc4da55c7f (DEFAULT_TABLE table=test-workload [id=cdaf6d3a65a54ea08e8e2c0ecdfc9de7]), partition=RANGE (key) PARTITION 1789569700 <= VALUES < 1968526670
I20250112 15:07:25.229508 21281 ts_tablet_manager.cc:1397] T 8c7dbaf285a04da9b4eec92a29ab7d6c P 2537bbbd1a9947ae9facd87567e3ccb9: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20250112 15:07:25.229566 21056 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 16f16fcd5cf347e5853189dc4da55c7f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 15:07:25.229636 21281 raft_consensus.cc:357] T 8c7dbaf285a04da9b4eec92a29ab7d6c P 2537bbbd1a9947ae9facd87567e3ccb9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2537bbbd1a9947ae9facd87567e3ccb9" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 33173 } } peers { permanent_uuid: "c561ec3374b94b95be0c155d80ff0bff" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 33627 } } peers { permanent_uuid: "3710b74ed2044ce5995533df32e67351" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 38589 } }
I20250112 15:07:25.229712 21281 raft_consensus.cc:383] T 8c7dbaf285a04da9b4eec92a29ab7d6c P 2537bbbd1a9947ae9facd87567e3ccb9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 15:07:25.229732 21281 raft_consensus.cc:738] T 8c7dbaf285a04da9b4eec92a29ab7d6c P 2537bbbd1a9947ae9facd87567e3ccb9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2537bbbd1a9947ae9facd87567e3ccb9, State: Initialized, Role: FOLLOWER
I20250112 15:07:25.229779 21281 consensus_queue.cc:260] T 8c7dbaf285a04da9b4eec92a29ab7d6c P 2537bbbd1a9947ae9facd87567e3ccb9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2537bbbd1a9947ae9facd87567e3ccb9" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 33173 } } peers { permanent_uuid: "c561ec3374b94b95be0c155d80ff0bff" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 33627 } } peers { permanent_uuid: "3710b74ed2044ce5995533df32e67351" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 38589 } }
I20250112 15:07:25.229861 21281 ts_tablet_manager.cc:1428] T 8c7dbaf285a04da9b4eec92a29ab7d6c P 2537bbbd1a9947ae9facd87567e3ccb9: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20250112 15:07:25.229916 21281 tablet_bootstrap.cc:492] T 80cc505a04394c53a18c5beeded4655d P 2537bbbd1a9947ae9facd87567e3ccb9: Bootstrap starting.
I20250112 15:07:25.230329 21281 tablet_bootstrap.cc:654] T 80cc505a04394c53a18c5beeded4655d P 2537bbbd1a9947ae9facd87567e3ccb9: Neither blocks nor log segments found. Creating new log.
I20250112 15:07:25.230762 21060 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9601bcda24ed42e7b93ab9740f1e271e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 15:07:25.230916 21281 tablet_bootstrap.cc:492] T 80cc505a04394c53a18c5beeded4655d P 2537bbbd1a9947ae9facd87567e3ccb9: No bootstrap required, opened a new log
I20250112 15:07:25.230964 21281 ts_tablet_manager.cc:1397] T 80cc505a04394c53a18c5beeded4655d P 2537bbbd1a9947ae9facd87567e3ccb9: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20250112 15:07:25.231092 21281 raft_consensus.cc:357] T 80cc505a04394c53a18c5beeded4655d P 2537bbbd1a9947ae9facd87567e3ccb9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2537bbbd1a9947ae9facd87567e3ccb9" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 33173 } } peers { permanent_uuid: "c561ec3374b94b95be0c155d80ff0bff" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 33627 } } peers { permanent_uuid: "3710b74ed2044ce5995533df32e67351" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 38589 } }
I20250112 15:07:25.231144 21281 raft_consensus.cc:383] T 80cc505a04394c53a18c5beeded4655d P 2537bbbd1a9947ae9facd87567e3ccb9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 15:07:25.231169 21281 raft_consensus.cc:738] T 80cc505a04394c53a18c5beeded4655d P 2537bbbd1a9947ae9facd87567e3ccb9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2537bbbd1a9947ae9facd87567e3ccb9, State: Initialized, Role: FOLLOWER
I20250112 15:07:25.231235 21281 consensus_queue.cc:260] T 80cc505a04394c53a18c5beeded4655d P 2537bbbd1a9947ae9facd87567e3ccb9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2537bbbd1a9947ae9facd87567e3ccb9" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 33173 } } peers { permanent_uuid: "c561ec3374b94b95be0c155d80ff0bff" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 33627 } } peers { permanent_uuid: "3710b74ed2044ce5995533df32e67351" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 38589 } }
I20250112 15:07:25.231339 21281 ts_tablet_manager.cc:1428] T 80cc505a04394c53a18c5beeded4655d P 2537bbbd1a9947ae9facd87567e3ccb9: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20250112 15:07:25.231393 21281 tablet_bootstrap.cc:492] T 57e47b07fdcf434e96ca5dd0354e93b2 P 2537bbbd1a9947ae9facd87567e3ccb9: Bootstrap starting.
I20250112 15:07:25.231788 21281 tablet_bootstrap.cc:654] T 57e47b07fdcf434e96ca5dd0354e93b2 P 2537bbbd1a9947ae9facd87567e3ccb9: Neither blocks nor log segments found. Creating new log.
I20250112 15:07:25.219651 21212 tablet_service.cc:1467] Processing CreateTablet for tablet 5bcec60d3b974a9bb24aedfc738e1594 (DEFAULT_TABLE table=test-workload [id=cdaf6d3a65a54ea08e8e2c0ecdfc9de7]), partition=RANGE (key) PARTITION 715827880 <= VALUES < 894784850
I20250112 15:07:25.232075 21212 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5bcec60d3b974a9bb24aedfc738e1594. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 15:07:25.232406 21281 tablet_bootstrap.cc:492] T 57e47b07fdcf434e96ca5dd0354e93b2 P 2537bbbd1a9947ae9facd87567e3ccb9: No bootstrap required, opened a new log
I20250112 15:07:25.232453 21281 ts_tablet_manager.cc:1397] T 57e47b07fdcf434e96ca5dd0354e93b2 P 2537bbbd1a9947ae9facd87567e3ccb9: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20250112 15:07:25.232635 21281 raft_consensus.cc:357] T 57e47b07fdcf434e96ca5dd0354e93b2 P 2537bbbd1a9947ae9facd87567e3ccb9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2537bbbd1a9947ae9facd87567e3ccb9" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 33173 } } peers { permanent_uuid: "c561ec3374b94b95be0c155d80ff0bff" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 33627 } } peers { permanent_uuid: "3710b74ed2044ce5995533df32e67351" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 38589 } }
I20250112 15:07:25.232697 21281 raft_consensus.cc:383] T 57e47b07fdcf434e96ca5dd0354e93b2 P 2537bbbd1a9947ae9facd87567e3ccb9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 15:07:25.232718 21281 raft_consensus.cc:738] T 57e47b07fdcf434e96ca5dd0354e93b2 P 2537bbbd1a9947ae9facd87567e3ccb9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2537bbbd1a9947ae9facd87567e3ccb9, State: Initialized, Role: FOLLOWER
I20250112 15:07:25.232765 21281 consensus_queue.cc:260] T 57e47b07fdcf434e96ca5dd0354e93b2 P 2537bbbd1a9947ae9facd87567e3ccb9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2537bbbd1a9947ae9facd87567e3ccb9" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 33173 } } peers { permanent_uuid: "c561ec3374b94b95be0c155d80ff0bff" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 33627 } } peers { permanent_uuid: "3710b74ed2044ce5995533df32e67351" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 38589 } }
I20250112 15:07:25.232873 21281 ts_tablet_manager.cc:1428] T 57e47b07fdcf434e96ca5dd0354e93b2 P 2537bbbd1a9947ae9facd87567e3ccb9: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20250112 15:07:25.232928 21281 tablet_bootstrap.cc:492] T ef90b704208140a3983ffe6105939319 P 2537bbbd1a9947ae9facd87567e3ccb9: Bootstrap starting.
I20250112 15:07:25.233253 21281 tablet_bootstrap.cc:654] T ef90b704208140a3983ffe6105939319 P 2537bbbd1a9947ae9facd87567e3ccb9: Neither blocks nor log segments found. Creating new log.
I20250112 15:07:25.233477 21058 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 69c23627ab854000b360030c384dc966. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 15:07:25.233915 21281 tablet_bootstrap.cc:492] T ef90b704208140a3983ffe6105939319 P 2537bbbd1a9947ae9facd87567e3ccb9: No bootstrap required, opened a new log
I20250112 15:07:25.233963 21281 ts_tablet_manager.cc:1397] T ef90b704208140a3983ffe6105939319 P 2537bbbd1a9947ae9facd87567e3ccb9: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20250112 15:07:25.234118 21281 raft_consensus.cc:357] T ef90b704208140a3983ffe6105939319 P 2537bbbd1a9947ae9facd87567e3ccb9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2537bbbd1a9947ae9facd87567e3ccb9" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 33173 } } peers { permanent_uuid: "c561ec3374b94b95be0c155d80ff0bff" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 33627 } } peers { permanent_uuid: "3710b74ed2044ce5995533df32e67351" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 38589 } }
I20250112 15:07:25.234189 21281 raft_consensus.cc:383] T ef90b704208140a3983ffe6105939319 P 2537bbbd1a9947ae9facd87567e3ccb9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 15:07:25.234216 21281 raft_consensus.cc:738] T ef90b704208140a3983ffe6105939319 P 2537bbbd1a9947ae9facd87567e3ccb9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2537bbbd1a9947ae9facd87567e3ccb9, State: Initialized, Role: FOLLOWER
I20250112 15:07:25.234251 21281 consensus_queue.cc:260] T ef90b704208140a3983ffe6105939319 P 2537bbbd1a9947ae9facd87567e3ccb9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2537bbbd1a9947ae9facd87567e3ccb9" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 33173 } } peers { permanent_uuid: "c561ec3374b94b95be0c155d80ff0bff" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 33627 } } peers { permanent_uuid: "3710b74ed2044ce5995533df32e67351" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 38589 } }
I20250112 15:07:25.234328 21281 ts_tablet_manager.cc:1428] T ef90b704208140a3983ffe6105939319 P 2537bbbd1a9947ae9facd87567e3ccb9: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20250112 15:07:25.234382 21281 tablet_bootstrap.cc:492] T bfaeae584ec04772b2d6fcd088022117 P 2537bbbd1a9947ae9facd87567e3ccb9: Bootstrap starting.
I20250112 15:07:25.218149 21135 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9601bcda24ed42e7b93ab9740f1e271e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 15:07:25.234716 21132 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9d4bbba799324d7fbbac082520c65686. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 15:07:25.234760 21281 tablet_bootstrap.cc:654] T bfaeae584ec04772b2d6fcd088022117 P 2537bbbd1a9947ae9facd87567e3ccb9: Neither blocks nor log segments found. Creating new log.
I20250112 15:07:25.235165 21139 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 80cc505a04394c53a18c5beeded4655d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 15:07:25.235255 21273 tablet_bootstrap.cc:654] T 5bcec60d3b974a9bb24aedfc738e1594 P c561ec3374b94b95be0c155d80ff0bff: Neither blocks nor log segments found. Creating new log.
I20250112 15:07:25.235864 21273 tablet_bootstrap.cc:492] T 5bcec60d3b974a9bb24aedfc738e1594 P c561ec3374b94b95be0c155d80ff0bff: No bootstrap required, opened a new log
I20250112 15:07:25.235919 21273 ts_tablet_manager.cc:1397] T 5bcec60d3b974a9bb24aedfc738e1594 P c561ec3374b94b95be0c155d80ff0bff: Time spent bootstrapping tablet: real 0.017s user 0.003s sys 0.000s
I20250112 15:07:25.236040 21133 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 69c23627ab854000b360030c384dc966. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 15:07:25.236064 21273 raft_consensus.cc:357] T 5bcec60d3b974a9bb24aedfc738e1594 P c561ec3374b94b95be0c155d80ff0bff [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2537bbbd1a9947ae9facd87567e3ccb9" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 33173 } } peers { permanent_uuid: "c561ec3374b94b95be0c155d80ff0bff" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 33627 } } peers { permanent_uuid: "3710b74ed2044ce5995533df32e67351" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 38589 } }
I20250112 15:07:25.236131 21273 raft_consensus.cc:383] T 5bcec60d3b974a9bb24aedfc738e1594 P c561ec3374b94b95be0c155d80ff0bff [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 15:07:25.236160 21273 raft_consensus.cc:738] T 5bcec60d3b974a9bb24aedfc738e1594 P c561ec3374b94b95be0c155d80ff0bff [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c561ec3374b94b95be0c155d80ff0bff, State: Initialized, Role: FOLLOWER
I20250112 15:07:25.236209 21273 consensus_queue.cc:260] T 5bcec60d3b974a9bb24aedfc738e1594 P c561ec3374b94b95be0c155d80ff0bff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2537bbbd1a9947ae9facd87567e3ccb9" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 33173 } } peers { permanent_uuid: "c561ec3374b94b95be0c155d80ff0bff" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 33627 } } peers { permanent_uuid: "3710b74ed2044ce5995533df32e67351" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 38589 } }
I20250112 15:07:25.236305 21273 ts_tablet_manager.cc:1428] T 5bcec60d3b974a9bb24aedfc738e1594 P c561ec3374b94b95be0c155d80ff0bff: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20250112 15:07:25.236361 21273 tablet_bootstrap.cc:492] T 1eb2b7e6edd34a7b83f9b918d3cdf2ed P c561ec3374b94b95be0c155d80ff0bff: Bootstrap starting.
I20250112 15:07:25.236779 21273 tablet_bootstrap.cc:654] T 1eb2b7e6edd34a7b83f9b918d3cdf2ed P c561ec3374b94b95be0c155d80ff0bff: Neither blocks nor log segments found. Creating new log.
I20250112 15:07:25.237355 21273 tablet_bootstrap.cc:492] T 1eb2b7e6edd34a7b83f9b918d3cdf2ed P c561ec3374b94b95be0c155d80ff0bff: No bootstrap required, opened a new log
I20250112 15:07:25.237404 21273 ts_tablet_manager.cc:1397] T 1eb2b7e6edd34a7b83f9b918d3cdf2ed P c561ec3374b94b95be0c155d80ff0bff: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20250112 15:07:25.237538 21273 raft_consensus.cc:357] T 1eb2b7e6edd34a7b83f9b918d3cdf2ed P c561ec3374b94b95be0c155d80ff0bff [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2537bbbd1a9947ae9facd87567e3ccb9" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 33173 } } peers { permanent_uuid: "c561ec3374b94b95be0c155d80ff0bff" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 33627 } } peers { permanent_uuid: "3710b74ed2044ce5995533df32e67351" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 38589 } }
I20250112 15:07:25.237602 21273 raft_consensus.cc:383] T 1eb2b7e6edd34a7b83f9b918d3cdf2ed P c561ec3374b94b95be0c155d80ff0bff [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 15:07:25.237632 21273 raft_consensus.cc:738] T 1eb2b7e6edd34a7b83f9b918d3cdf2ed P c561ec3374b94b95be0c155d80ff0bff [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c561ec3374b94b95be0c155d80ff0bff, State: Initialized, Role: FOLLOWER
I20250112 15:07:25.237686 21273 consensus_queue.cc:260] T 1eb2b7e6edd34a7b83f9b918d3cdf2ed P c561ec3374b94b95be0c155d80ff0bff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2537bbbd1a9947ae9facd87567e3ccb9" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 33173 } } peers { permanent_uuid: "c561ec3374b94b95be0c155d80ff0bff" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 33627 } } peers { permanent_uuid: "3710b74ed2044ce5995533df32e67351" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 38589 } }
I20250112 15:07:25.237812 21273 ts_tablet_manager.cc:1428] T 1eb2b7e6edd34a7b83f9b918d3cdf2ed P c561ec3374b94b95be0c155d80ff0bff: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20250112 15:07:25.237872 21273 tablet_bootstrap.cc:492] T bfaeae584ec04772b2d6fcd088022117 P c561ec3374b94b95be0c155d80ff0bff: Bootstrap starting.
I20250112 15:07:25.238296 21273 tablet_bootstrap.cc:654] T bfaeae584ec04772b2d6fcd088022117 P c561ec3374b94b95be0c155d80ff0bff: Neither blocks nor log segments found. Creating new log.
I20250112 15:07:25.215863 21137 tablet_service.cc:1467] Processing CreateTablet for tablet 57e47b07fdcf434e96ca5dd0354e93b2 (DEFAULT_TABLE table=test-workload [id=cdaf6d3a65a54ea08e8e2c0ecdfc9de7]), partition=RANGE (key) PARTITION 1968526670 <= VALUES
I20250112 15:07:25.238411 21137 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 57e47b07fdcf434e96ca5dd0354e93b2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 15:07:25.239234 21273 tablet_bootstrap.cc:492] T bfaeae584ec04772b2d6fcd088022117 P c561ec3374b94b95be0c155d80ff0bff: No bootstrap required, opened a new log
I20250112 15:07:25.239285 21273 ts_tablet_manager.cc:1397] T bfaeae584ec04772b2d6fcd088022117 P c561ec3374b94b95be0c155d80ff0bff: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20250112 15:07:25.239419 21273 raft_consensus.cc:357] T bfaeae584ec04772b2d6fcd088022117 P c561ec3374b94b95be0c155d80ff0bff [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2537bbbd1a9947ae9facd87567e3ccb9" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 33173 } } peers { permanent_uuid: "c561ec3374b94b95be0c155d80ff0bff" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 33627 } } peers { permanent_uuid: "3710b74ed2044ce5995533df32e67351" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 38589 } }
I20250112 15:07:25.239467 21273 raft_consensus.cc:383] T bfaeae584ec04772b2d6fcd088022117 P c561ec3374b94b95be0c155d80ff0bff [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 15:07:25.239490 21273 raft_consensus.cc:738] T bfaeae584ec04772b2d6fcd088022117 P c561ec3374b94b95be0c155d80ff0bff [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c561ec3374b94b95be0c155d80ff0bff, State: Initialized, Role: FOLLOWER
I20250112 15:07:25.239527 21273 consensus_queue.cc:260] T bfaeae584ec04772b2d6fcd088022117 P c561ec3374b94b95be0c155d80ff0bff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2537bbbd1a9947ae9facd87567e3ccb9" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 33173 } } peers { permanent_uuid: "c561ec3374b94b95be0c155d80ff0bff" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 33627 } } peers { permanent_uuid: "3710b74ed2044ce5995533df32e67351" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 38589 } }
I20250112 15:07:25.239631 21065 tablet_service.cc:1467] Processing CreateTablet for tablet 1eb2b7e6edd34a7b83f9b918d3cdf2ed (DEFAULT_TABLE table=test-workload [id=cdaf6d3a65a54ea08e8e2c0ecdfc9de7]), partition=RANGE (key) PARTITION 178956970 <= VALUES < 357913940
I20250112 15:07:25.239672 21273 ts_tablet_manager.cc:1428] T bfaeae584ec04772b2d6fcd088022117 P c561ec3374b94b95be0c155d80ff0bff: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20250112 15:07:25.239723 21273 tablet_bootstrap.cc:492] T 8c7dbaf285a04da9b4eec92a29ab7d6c P c561ec3374b94b95be0c155d80ff0bff: Bootstrap starting.
I20250112 15:07:25.239742 21065 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1eb2b7e6edd34a7b83f9b918d3cdf2ed. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 15:07:25.240216 21273 tablet_bootstrap.cc:654] T 8c7dbaf285a04da9b4eec92a29ab7d6c P c561ec3374b94b95be0c155d80ff0bff: Neither blocks nor log segments found. Creating new log.
I20250112 15:07:25.214550 21141 tablet_service.cc:1467] Processing CreateTablet for tablet 16f16fcd5cf347e5853189dc4da55c7f (DEFAULT_TABLE table=test-workload [id=cdaf6d3a65a54ea08e8e2c0ecdfc9de7]), partition=RANGE (key) PARTITION 1789569700 <= VALUES < 1968526670
I20250112 15:07:25.240728 21141 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 16f16fcd5cf347e5853189dc4da55c7f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 15:07:25.240833 21273 tablet_bootstrap.cc:492] T 8c7dbaf285a04da9b4eec92a29ab7d6c P c561ec3374b94b95be0c155d80ff0bff: No bootstrap required, opened a new log
I20250112 15:07:25.240878 21273 ts_tablet_manager.cc:1397] T 8c7dbaf285a04da9b4eec92a29ab7d6c P c561ec3374b94b95be0c155d80ff0bff: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20250112 15:07:25.241014 21273 raft_consensus.cc:357] T 8c7dbaf285a04da9b4eec92a29ab7d6c P c561ec3374b94b95be0c155d80ff0bff [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2537bbbd1a9947ae9facd87567e3ccb9" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 33173 } } peers { permanent_uuid: "c561ec3374b94b95be0c155d80ff0bff" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 33627 } } peers { permanent_uuid: "3710b74ed2044ce5995533df32e67351" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 38589 } }
I20250112 15:07:25.241056 21273 raft_consensus.cc:383] T 8c7dbaf285a04da9b4eec92a29ab7d6c P c561ec3374b94b95be0c155d80ff0bff [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 15:07:25.241070 21273 raft_consensus.cc:738] T 8c7dbaf285a04da9b4eec92a29ab7d6c P c561ec3374b94b95be0c155d80ff0bff [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c561ec3374b94b95be0c155d80ff0bff, State: Initialized, Role: FOLLOWER
I20250112 15:07:25.241129 21273 consensus_queue.cc:260] T 8c7dbaf285a04da9b4eec92a29ab7d6c P c561ec3374b94b95be0c155d80ff0bff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2537bbbd1a9947ae9facd87567e3ccb9" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 33173 } } peers { permanent_uuid: "c561ec3374b94b95be0c155d80ff0bff" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 33627 } } peers { permanent_uuid: "3710b74ed2044ce5995533df32e67351" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 38589 } }
I20250112 15:07:25.241214 21273 ts_tablet_manager.cc:1428] T 8c7dbaf285a04da9b4eec92a29ab7d6c P c561ec3374b94b95be0c155d80ff0bff: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20250112 15:07:25.241266 21273 tablet_bootstrap.cc:492] T ef90b704208140a3983ffe6105939319 P c561ec3374b94b95be0c155d80ff0bff: Bootstrap starting.
I20250112 15:07:25.241636 21273 tablet_bootstrap.cc:654] T ef90b704208140a3983ffe6105939319 P c561ec3374b94b95be0c155d80ff0bff: Neither blocks nor log segments found. Creating new log.
I20250112 15:07:25.242086 21281 tablet_bootstrap.cc:492] T bfaeae584ec04772b2d6fcd088022117 P 2537bbbd1a9947ae9facd87567e3ccb9: No bootstrap required, opened a new log
I20250112 15:07:25.242141 21281 ts_tablet_manager.cc:1397] T bfaeae584ec04772b2d6fcd088022117 P 2537bbbd1a9947ae9facd87567e3ccb9: Time spent bootstrapping tablet: real 0.008s user 0.001s sys 0.000s
I20250112 15:07:25.242218 21272 tablet_bootstrap.cc:492] T 69c23627ab854000b360030c384dc966 P 3710b74ed2044ce5995533df32e67351: No bootstrap required, opened a new log
I20250112 15:07:25.242260 21272 ts_tablet_manager.cc:1397] T 69c23627ab854000b360030c384dc966 P 3710b74ed2044ce5995533df32e67351: Time spent bootstrapping tablet: real 0.017s user 0.000s sys 0.001s
I20250112 15:07:25.242293 21281 raft_consensus.cc:357] T bfaeae584ec04772b2d6fcd088022117 P 2537bbbd1a9947ae9facd87567e3ccb9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2537bbbd1a9947ae9facd87567e3ccb9" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 33173 } } peers { permanent_uuid: "c561ec3374b94b95be0c155d80ff0bff" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 33627 } } peers { permanent_uuid: "3710b74ed2044ce5995533df32e67351" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 38589 } }
I20250112 15:07:25.242350 21281 raft_consensus.cc:383] T bfaeae584ec04772b2d6fcd088022117 P 2537bbbd1a9947ae9facd87567e3ccb9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 15:07:25.242372 21281 raft_consensus.cc:738] T bfaeae584ec04772b2d6fcd088022117 P 2537bbbd1a9947ae9facd87567e3ccb9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2537bbbd1a9947ae9facd87567e3ccb9, State: Initialized, Role: FOLLOWER
I20250112 15:07:25.242386 21272 raft_consensus.cc:357] T 69c23627ab854000b360030c384dc966 P 3710b74ed2044ce5995533df32e67351 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2537bbbd1a9947ae9facd87567e3ccb9" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 33173 } } peers { permanent_uuid: "c561ec3374b94b95be0c155d80ff0bff" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 33627 } } peers { permanent_uuid: "3710b74ed2044ce5995533df32e67351" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 38589 } }
I20250112 15:07:25.242422 21281 consensus_queue.cc:260] T bfaeae584ec04772b2d6fcd088022117 P 2537bbbd1a9947ae9facd87567e3ccb9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2537bbbd1a9947ae9facd87567e3ccb9" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 33173 } } peers { permanent_uuid: "c561ec3374b94b95be0c155d80ff0bff" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 33627 } } peers { permanent_uuid: "3710b74ed2044ce5995533df32e67351" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 38589 } }
I20250112 15:07:25.242455 21272 raft_consensus.cc:383] T 69c23627ab854000b360030c384dc966 P 3710b74ed2044ce5995533df32e67351 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 15:07:25.242471 21272 raft_consensus.cc:738] T 69c23627ab854000b360030c384dc966 P 3710b74ed2044ce5995533df32e67351 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3710b74ed2044ce5995533df32e67351, State: Initialized, Role: FOLLOWER
I20250112 15:07:25.242506 21281 ts_tablet_manager.cc:1428] T bfaeae584ec04772b2d6fcd088022117 P 2537bbbd1a9947ae9facd87567e3ccb9: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20250112 15:07:25.242522 21272 consensus_queue.cc:260] T 69c23627ab854000b360030c384dc966 P 3710b74ed2044ce5995533df32e67351 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2537bbbd1a9947ae9facd87567e3ccb9" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 33173 } } peers { permanent_uuid: "c561ec3374b94b95be0c155d80ff0bff" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 33627 } } peers { permanent_uuid: "3710b74ed2044ce5995533df32e67351" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 38589 } }
I20250112 15:07:25.242588 21281 tablet_bootstrap.cc:492] T 5bcec60d3b974a9bb24aedfc738e1594 P 2537bbbd1a9947ae9facd87567e3ccb9: Bootstrap starting.
I20250112 15:07:25.242617 21272 ts_tablet_manager.cc:1428] T 69c23627ab854000b360030c384dc966 P 3710b74ed2044ce5995533df32e67351: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20250112 15:07:25.242676 21272 tablet_bootstrap.cc:492] T 1eb2b7e6edd34a7b83f9b918d3cdf2ed P 3710b74ed2044ce5995533df32e67351: Bootstrap starting.
I20250112 15:07:25.243078 21281 tablet_bootstrap.cc:654] T 5bcec60d3b974a9bb24aedfc738e1594 P 2537bbbd1a9947ae9facd87567e3ccb9: Neither blocks nor log segments found. Creating new log.
I20250112 15:07:25.243134 21272 tablet_bootstrap.cc:654] T 1eb2b7e6edd34a7b83f9b918d3cdf2ed P 3710b74ed2044ce5995533df32e67351: Neither blocks nor log segments found. Creating new log.
I20250112 15:07:25.243690 21272 tablet_bootstrap.cc:492] T 1eb2b7e6edd34a7b83f9b918d3cdf2ed P 3710b74ed2044ce5995533df32e67351: No bootstrap required, opened a new log
I20250112 15:07:25.243769 21272 ts_tablet_manager.cc:1397] T 1eb2b7e6edd34a7b83f9b918d3cdf2ed P 3710b74ed2044ce5995533df32e67351: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20250112 15:07:25.243952 21272 raft_consensus.cc:357] T 1eb2b7e6edd34a7b83f9b918d3cdf2ed P 3710b74ed2044ce5995533df32e67351 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2537bbbd1a9947ae9facd87567e3ccb9" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 33173 } } peers { permanent_uuid: "c561ec3374b94b95be0c155d80ff0bff" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 33627 } } peers { permanent_uuid: "3710b74ed2044ce5995533df32e67351" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 38589 } }
I20250112 15:07:25.244019 21272 raft_consensus.cc:383] T 1eb2b7e6edd34a7b83f9b918d3cdf2ed P 3710b74ed2044ce5995533df32e67351 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 15:07:25.244040 21272 raft_consensus.cc:738] T 1eb2b7e6edd34a7b83f9b918d3cdf2ed P 3710b74ed2044ce5995533df32e67351 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3710b74ed2044ce5995533df32e67351, State: Initialized, Role: FOLLOWER
I20250112 15:07:25.244091 21272 consensus_queue.cc:260] T 1eb2b7e6edd34a7b83f9b918d3cdf2ed P 3710b74ed2044ce5995533df32e67351 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2537bbbd1a9947ae9facd87567e3ccb9" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 33173 } } peers { permanent_uuid: "c561ec3374b94b95be0c155d80ff0bff" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 33627 } } peers { permanent_uuid: "3710b74ed2044ce5995533df32e67351" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 38589 } }
I20250112 15:07:25.244180 21284 raft_consensus.cc:491] T ef90b704208140a3983ffe6105939319 P 2537bbbd1a9947ae9facd87567e3ccb9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 15:07:25.244202 21272 ts_tablet_manager.cc:1428] T 1eb2b7e6edd34a7b83f9b918d3cdf2ed P 3710b74ed2044ce5995533df32e67351: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20250112 15:07:25.244259 21284 raft_consensus.cc:513] T ef90b704208140a3983ffe6105939319 P 2537bbbd1a9947ae9facd87567e3ccb9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2537bbbd1a9947ae9facd87567e3ccb9" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 33173 } } peers { permanent_uuid: "c561ec3374b94b95be0c155d80ff0bff" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 33627 } } peers { permanent_uuid: "3710b74ed2044ce5995533df32e67351" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 38589 } }
I20250112 15:07:25.244292 21272 tablet_bootstrap.cc:492] T 80cc505a04394c53a18c5beeded4655d P 3710b74ed2044ce5995533df32e67351: Bootstrap starting.
I20250112 15:07:25.244448 21284 leader_election.cc:290] T ef90b704208140a3983ffe6105939319 P 2537bbbd1a9947ae9facd87567e3ccb9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c561ec3374b94b95be0c155d80ff0bff (127.17.60.194:33627), 3710b74ed2044ce5995533df32e67351 (127.17.60.195:38589)
I20250112 15:07:25.243678 21281 tablet_bootstrap.cc:492] T 5bcec60d3b974a9bb24aedfc738e1594 P 2537bbbd1a9947ae9facd87567e3ccb9: No bootstrap required, opened a new log
I20250112 15:07:25.244621 21281 ts_tablet_manager.cc:1397] T 5bcec60d3b974a9bb24aedfc738e1594 P 2537bbbd1a9947ae9facd87567e3ccb9: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20250112 15:07:25.244766 21272 tablet_bootstrap.cc:654] T 80cc505a04394c53a18c5beeded4655d P 3710b74ed2044ce5995533df32e67351: Neither blocks nor log segments found. Creating new log.
I20250112 15:07:25.244765 21281 raft_consensus.cc:357] T 5bcec60d3b974a9bb24aedfc738e1594 P 2537bbbd1a9947ae9facd87567e3ccb9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2537bbbd1a9947ae9facd87567e3ccb9" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 33173 } } peers { permanent_uuid: "c561ec3374b94b95be0c155d80ff0bff" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 33627 } } peers { permanent_uuid: "3710b74ed2044ce5995533df32e67351" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 38589 } }
I20250112 15:07:25.244920 21281 raft_consensus.cc:383] T 5bcec60d3b974a9bb24aedfc738e1594 P 2537bbbd1a9947ae9facd87567e3ccb9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 15:07:25.244962 21281 raft_consensus.cc:738] T 5bcec60d3b974a9bb24aedfc738e1594 P 2537bbbd1a9947ae9facd87567e3ccb9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2537bbbd1a9947ae9facd87567e3ccb9, State: Initialized, Role: FOLLOWER
I20250112 15:07:25.245002 21281 consensus_queue.cc:260] T 5bcec60d3b974a9bb24aedfc738e1594 P 2537bbbd1a9947ae9facd87567e3ccb9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2537bbbd1a9947ae9facd87567e3ccb9" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 33173 } } peers { permanent_uuid: "c561ec3374b94b95be0c155d80ff0bff" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 33627 } } peers { permanent_uuid: "3710b74ed2044ce5995533df32e67351" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 38589 } }
I20250112 15:07:25.245112 21281 ts_tablet_manager.cc:1428] T 5bcec60d3b974a9bb24aedfc738e1594 P 2537bbbd1a9947ae9facd87567e3ccb9: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20250112 15:07:25.244258 21273 tablet_bootstrap.cc:492] T ef90b704208140a3983ffe6105939319 P c561ec3374b94b95be0c155d80ff0bff: No bootstrap required, opened a new log
I20250112 15:07:25.245199 21281 tablet_bootstrap.cc:492] T 9d4bbba799324d7fbbac082520c65686 P 2537bbbd1a9947ae9facd87567e3ccb9: Bootstrap starting.
I20250112 15:07:25.245613 21281 tablet_bootstrap.cc:654] T 9d4bbba799324d7fbbac082520c65686 P 2537bbbd1a9947ae9facd87567e3ccb9: Neither blocks nor log segments found. Creating new log.
I20250112 15:07:25.245189 21273 ts_tablet_manager.cc:1397] T ef90b704208140a3983ffe6105939319 P c561ec3374b94b95be0c155d80ff0bff: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.001s
I20250112 15:07:25.246070 21273 raft_consensus.cc:357] T ef90b704208140a3983ffe6105939319 P c561ec3374b94b95be0c155d80ff0bff [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2537bbbd1a9947ae9facd87567e3ccb9" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 33173 } } peers { permanent_uuid: "c561ec3374b94b95be0c155d80ff0bff" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 33627 } } peers { permanent_uuid: "3710b74ed2044ce5995533df32e67351" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 38589 } }
I20250112 15:07:25.246141 21273 raft_consensus.cc:383] T ef90b704208140a3983ffe6105939319 P c561ec3374b94b95be0c155d80ff0bff [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 15:07:25.246181 21273 raft_consensus.cc:738] T ef90b704208140a3983ffe6105939319 P c561ec3374b94b95be0c155d80ff0bff [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c561ec3374b94b95be0c155d80ff0bff, State: Initialized, Role: FOLLOWER
I20250112 15:07:25.246235 21273 consensus_queue.cc:260] T ef90b704208140a3983ffe6105939319 P c561ec3374b94b95be0c155d80ff0bff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2537bbbd1a9947ae9facd87567e3ccb9" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 33173 } } peers { permanent_uuid: "c561ec3374b94b95be0c155d80ff0bff" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 33627 } } peers { permanent_uuid: "3710b74ed2044ce5995533df32e67351" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 38589 } }
I20250112 15:07:25.246259 21281 tablet_bootstrap.cc:492] T 9d4bbba799324d7fbbac082520c65686 P 2537bbbd1a9947ae9facd87567e3ccb9: No bootstrap required, opened a new log
I20250112 15:07:25.246295 21281 ts_tablet_manager.cc:1397] T 9d4bbba799324d7fbbac082520c65686 P 2537bbbd1a9947ae9facd87567e3ccb9: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20250112 15:07:25.246326 21273 ts_tablet_manager.cc:1428] T ef90b704208140a3983ffe6105939319 P c561ec3374b94b95be0c155d80ff0bff: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20250112 15:07:25.246373 21273 tablet_bootstrap.cc:492] T 9d4bbba799324d7fbbac082520c65686 P c561ec3374b94b95be0c155d80ff0bff: Bootstrap starting.
I20250112 15:07:25.246429 21281 raft_consensus.cc:357] T 9d4bbba799324d7fbbac082520c65686 P 2537bbbd1a9947ae9facd87567e3ccb9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2537bbbd1a9947ae9facd87567e3ccb9" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 33173 } } peers { permanent_uuid: "c561ec3374b94b95be0c155d80ff0bff" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 33627 } } peers { permanent_uuid: "3710b74ed2044ce5995533df32e67351" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 38589 } }
I20250112 15:07:25.246491 21281 raft_consensus.cc:383] T 9d4bbba799324d7fbbac082520c65686 P 2537bbbd1a9947ae9facd87567e3ccb9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 15:07:25.246515 21281 raft_consensus.cc:738] T 9d4bbba799324d7fbbac082520c65686 P 2537bbbd1a9947ae9facd87567e3ccb9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2537bbbd1a9947ae9facd87567e3ccb9, State: Initialized, Role: FOLLOWER
I20250112 15:07:25.246558 21281 consensus_queue.cc:260] T 9d4bbba799324d7fbbac082520c65686 P 2537bbbd1a9947ae9facd87567e3ccb9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2537bbbd1a9947ae9facd87567e3ccb9" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 33173 } } peers { permanent_uuid: "c561ec3374b94b95be0c155d80ff0bff" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 33627 } } peers { permanent_uuid: "3710b74ed2044ce5995533df32e67351" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 38589 } }
I20250112 15:07:25.246637 21281 ts_tablet_manager.cc:1428] T 9d4bbba799324d7fbbac082520c65686 P 2537bbbd1a9947ae9facd87567e3ccb9: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20250112 15:07:25.246683 21281 tablet_bootstrap.cc:492] T 16f16fcd5cf347e5853189dc4da55c7f P 2537bbbd1a9947ae9facd87567e3ccb9: Bootstrap starting.
I20250112 15:07:25.246786 21273 tablet_bootstrap.cc:654] T 9d4bbba799324d7fbbac082520c65686 P c561ec3374b94b95be0c155d80ff0bff: Neither blocks nor log segments found. Creating new log.
I20250112 15:07:25.247117 21281 tablet_bootstrap.cc:654] T 16f16fcd5cf347e5853189dc4da55c7f P 2537bbbd1a9947ae9facd87567e3ccb9: Neither blocks nor log segments found. Creating new log.
I20250112 15:07:25.247601 21272 tablet_bootstrap.cc:492] T 80cc505a04394c53a18c5beeded4655d P 3710b74ed2044ce5995533df32e67351: No bootstrap required, opened a new log
I20250112 15:07:25.247645 21272 ts_tablet_manager.cc:1397] T 80cc505a04394c53a18c5beeded4655d P 3710b74ed2044ce5995533df32e67351: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20250112 15:07:25.247859 21272 raft_consensus.cc:357] T 80cc505a04394c53a18c5beeded4655d P 3710b74ed2044ce5995533df32e67351 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2537bbbd1a9947ae9facd87567e3ccb9" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 33173 } } peers { permanent_uuid: "c561ec3374b94b95be0c155d80ff0bff" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 33627 } } peers { permanent_uuid: "3710b74ed2044ce5995533df32e67351" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 38589 } }
I20250112 15:07:25.247918 21272 raft_consensus.cc:383] T 80cc505a04394c53a18c5beeded4655d P 3710b74ed2044ce5995533df32e67351 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 15:07:25.247936 21272 raft_consensus.cc:738] T 80cc505a04394c53a18c5beeded4655d P 3710b74ed2044ce5995533df32e67351 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3710b74ed2044ce5995533df32e67351, State: Initialized, Role: FOLLOWER
I20250112 15:07:25.247989 21272 consensus_queue.cc:260] T 80cc505a04394c53a18c5beeded4655d P 3710b74ed2044ce5995533df32e67351 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2537bbbd1a9947ae9facd87567e3ccb9" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 33173 } } peers { permanent_uuid: "c561ec3374b94b95be0c155d80ff0bff" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 33627 } } peers { permanent_uuid: "3710b74ed2044ce5995533df32e67351" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 38589 } }
I20250112 15:07:25.248078 21272 ts_tablet_manager.cc:1428] T 80cc505a04394c53a18c5beeded4655d P 3710b74ed2044ce5995533df32e67351: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20250112 15:07:25.248139 21272 tablet_bootstrap.cc:492] T 8c7dbaf285a04da9b4eec92a29ab7d6c P 3710b74ed2044ce5995533df32e67351: Bootstrap starting.
I20250112 15:07:25.248478 21151 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef90b704208140a3983ffe6105939319" candidate_uuid: "2537bbbd1a9947ae9facd87567e3ccb9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c561ec3374b94b95be0c155d80ff0bff" is_pre_election: true
I20250112 15:07:25.248554 21151 raft_consensus.cc:2463] T ef90b704208140a3983ffe6105939319 P c561ec3374b94b95be0c155d80ff0bff [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2537bbbd1a9947ae9facd87567e3ccb9 in term 0.
I20250112 15:07:25.248664 21281 tablet_bootstrap.cc:492] T 16f16fcd5cf347e5853189dc4da55c7f P 2537bbbd1a9947ae9facd87567e3ccb9: No bootstrap required, opened a new log
I20250112 15:07:25.248703 21281 ts_tablet_manager.cc:1397] T 16f16fcd5cf347e5853189dc4da55c7f P 2537bbbd1a9947ae9facd87567e3ccb9: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20250112 15:07:25.248764 21040 leader_election.cc:304] T ef90b704208140a3983ffe6105939319 P 2537bbbd1a9947ae9facd87567e3ccb9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2537bbbd1a9947ae9facd87567e3ccb9, c561ec3374b94b95be0c155d80ff0bff; no voters:
I20250112 15:07:25.248732 21272 tablet_bootstrap.cc:654] T 8c7dbaf285a04da9b4eec92a29ab7d6c P 3710b74ed2044ce5995533df32e67351: Neither blocks nor log segments found. Creating new log.
I20250112 15:07:25.248870 21284 raft_consensus.cc:2798] T ef90b704208140a3983ffe6105939319 P 2537bbbd1a9947ae9facd87567e3ccb9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250112 15:07:25.248901 21284 raft_consensus.cc:491] T ef90b704208140a3983ffe6105939319 P 2537bbbd1a9947ae9facd87567e3ccb9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250112 15:07:25.248925 21284 raft_consensus.cc:3054] T ef90b704208140a3983ffe6105939319 P 2537bbbd1a9947ae9facd87567e3ccb9 [term 0 FOLLOWER]: Advancing to term 1
I20250112 15:07:25.249339 21272 tablet_bootstrap.cc:492] T 8c7dbaf285a04da9b4eec92a29ab7d6c P 3710b74ed2044ce5995533df32e67351: No bootstrap required, opened a new log
I20250112 15:07:25.249377 21272 ts_tablet_manager.cc:1397] T 8c7dbaf285a04da9b4eec92a29ab7d6c P 3710b74ed2044ce5995533df32e67351: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20250112 15:07:25.249521 21272 raft_consensus.cc:357] T 8c7dbaf285a04da9b4eec92a29ab7d6c P 3710b74ed2044ce5995533df32e67351 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2537bbbd1a9947ae9facd87567e3ccb9" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 33173 } } peers { permanent_uuid: "c561ec3374b94b95be0c155d80ff0bff" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 33627 } } peers { permanent_uuid: "3710b74ed2044ce5995533df32e67351" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 38589 } }
I20250112 15:07:25.249555 21273 tablet_bootstrap.cc:492] T 9d4bbba799324d7fbbac082520c65686 P c561ec3374b94b95be0c155d80ff0bff: No bootstrap required, opened a new log
I20250112 15:07:25.249579 21272 raft_consensus.cc:383] T 8c7dbaf285a04da9b4eec92a29ab7d6c P 3710b74ed2044ce5995533df32e67351 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 15:07:25.249605 21272 raft_consensus.cc:738] T 8c7dbaf285a04da9b4eec92a29ab7d6c P 3710b74ed2044ce5995533df32e67351 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3710b74ed2044ce5995533df32e67351, State: Initialized, Role: FOLLOWER
I20250112 15:07:25.249646 21284 raft_consensus.cc:513] T ef90b704208140a3983ffe6105939319 P 2537bbbd1a9947ae9facd87567e3ccb9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2537bbbd1a9947ae9facd87567e3ccb9" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 33173 } } peers { permanent_uuid: "c561ec3374b94b95be0c155d80ff0bff" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 33627 } } peers { permanent_uuid: "3710b74ed2044ce5995533df32e67351" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 38589 } }
I20250112 15:07:25.249676 21272 consensus_queue.cc:260] T 8c7dbaf285a04da9b4eec92a29ab7d6c P 3710b74ed2044ce5995533df32e67351 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2537bbbd1a9947ae9facd87567e3ccb9" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 33173 } } peers { permanent_uuid: "c561ec3374b94b95be0c155d80ff0bff" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 33627 } } peers { permanent_uuid: "3710b74ed2044ce5995533df32e67351" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 38589 } }
I20250112 15:07:25.249753 21284 leader_election.cc:290] T ef90b704208140a3983ffe6105939319 P 2537bbbd1a9947ae9facd87567e3ccb9 [CANDIDATE]: Term 1 election: Requested vote from peers c561ec3374b94b95be0c155d80ff0bff (127.17.60.194:33627), 3710b74ed2044ce5995533df32e67351 (127.17.60.195:38589)
I20250112 15:07:25.249596 21273 ts_tablet_manager.cc:1397] T 9d4bbba799324d7fbbac082520c65686 P c561ec3374b94b95be0c155d80ff0bff: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.001s
I20250112 15:07:25.249855 21151 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef90b704208140a3983ffe6105939319" candidate_uuid: "2537bbbd1a9947ae9facd87567e3ccb9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c561ec3374b94b95be0c155d80ff0bff"
I20250112 15:07:25.249924 21151 raft_consensus.cc:3054] T ef90b704208140a3983ffe6105939319 P c561ec3374b94b95be0c155d80ff0bff [term 0 FOLLOWER]: Advancing to term 1
I20250112 15:07:25.249954 21226 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef90b704208140a3983ffe6105939319" candidate_uuid: "2537bbbd1a9947ae9facd87567e3ccb9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3710b74ed2044ce5995533df32e67351" is_pre_election: true
I20250112 15:07:25.250027 21273 raft_consensus.cc:357] T 9d4bbba799324d7fbbac082520c65686 P c561ec3374b94b95be0c155d80ff0bff [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2537bbbd1a9947ae9facd87567e3ccb9" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 33173 } } peers { permanent_uuid: "c561ec3374b94b95be0c155d80ff0bff" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 33627 } } peers { permanent_uuid: "3710b74ed2044ce5995533df32e67351" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 38589 } }
I20250112 15:07:25.250094 21273 raft_consensus.cc:383] T 9d4bbba799324d7fbbac082520c65686 P c561ec3374b94b95be0c155d80ff0bff [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 15:07:25.250123 21273 raft_consensus.cc:738] T 9d4bbba799324d7fbbac082520c65686 P c561ec3374b94b95be0c155d80ff0bff [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c561ec3374b94b95be0c155d80ff0bff, State: Initialized, Role: FOLLOWER
I20250112 15:07:25.250080 21225 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef90b704208140a3983ffe6105939319" candidate_uuid: "2537bbbd1a9947ae9facd87567e3ccb9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3710b74ed2044ce5995533df32e67351"
I20250112 15:07:25.250175 21273 consensus_queue.cc:260] T 9d4bbba799324d7fbbac082520c65686 P c561ec3374b94b95be0c155d80ff0bff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2537bbbd1a9947ae9facd87567e3ccb9" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 33173 } } peers { permanent_uuid: "c561ec3374b94b95be0c155d80ff0bff" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 33627 } } peers { permanent_uuid: "3710b74ed2044ce5995533df32e67351" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 38589 } }
I20250112 15:07:25.250036 21226 raft_consensus.cc:2463] T ef90b704208140a3983ffe6105939319 P 3710b74ed2044ce5995533df32e67351 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2537bbbd1a9947ae9facd87567e3ccb9 in term 0.
I20250112 15:07:25.250265 21273 ts_tablet_manager.cc:1428] T 9d4bbba799324d7fbbac082520c65686 P c561ec3374b94b95be0c155d80ff0bff: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20250112 15:07:25.250310 21273 tablet_bootstrap.cc:492] T 80cc505a04394c53a18c5beeded4655d P c561ec3374b94b95be0c155d80ff0bff: Bootstrap starting.
I20250112 15:07:25.250398 21281 raft_consensus.cc:357] T 16f16fcd5cf347e5853189dc4da55c7f P 2537bbbd1a9947ae9facd87567e3ccb9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2537bbbd1a9947ae9facd87567e3ccb9" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 33173 } } peers { permanent_uuid: "c561ec3374b94b95be0c155d80ff0bff" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 33627 } } peers { permanent_uuid: "3710b74ed2044ce5995533df32e67351" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 38589 } }
I20250112 15:07:25.250509 21281 raft_consensus.cc:383] T 16f16fcd5cf347e5853189dc4da55c7f P 2537bbbd1a9947ae9facd87567e3ccb9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 15:07:25.250581 21281 raft_consensus.cc:738] T 16f16fcd5cf347e5853189dc4da55c7f P 2537bbbd1a9947ae9facd87567e3ccb9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2537bbbd1a9947ae9facd87567e3ccb9, State: Initialized, Role: FOLLOWER
I20250112 15:07:25.250612 21151 raft_consensus.cc:2463] T ef90b704208140a3983ffe6105939319 P c561ec3374b94b95be0c155d80ff0bff [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2537bbbd1a9947ae9facd87567e3ccb9 in term 1.
I20250112 15:07:25.249866 21272 ts_tablet_manager.cc:1428] T 8c7dbaf285a04da9b4eec92a29ab7d6c P 3710b74ed2044ce5995533df32e67351: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20250112 15:07:25.250702 21272 tablet_bootstrap.cc:492] T 9601bcda24ed42e7b93ab9740f1e271e P 3710b74ed2044ce5995533df32e67351: Bootstrap starting.
I20250112 15:07:25.250707 21273 tablet_bootstrap.cc:654] T 80cc505a04394c53a18c5beeded4655d P c561ec3374b94b95be0c155d80ff0bff: Neither blocks nor log segments found. Creating new log.
I20250112 15:07:25.250766 21040 leader_election.cc:304] T ef90b704208140a3983ffe6105939319 P 2537bbbd1a9947ae9facd87567e3ccb9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2537bbbd1a9947ae9facd87567e3ccb9, c561ec3374b94b95be0c155d80ff0bff; no voters:
I20250112 15:07:25.250909 21284 raft_consensus.cc:2798] T ef90b704208140a3983ffe6105939319 P 2537bbbd1a9947ae9facd87567e3ccb9 [term 1 FOLLOWER]: Leader election won for term 1
I20250112 15:07:25.251019 21284 raft_consensus.cc:695] T ef90b704208140a3983ffe6105939319 P 2537bbbd1a9947ae9facd87567e3ccb9 [term 1 LEADER]: Becoming Leader. State: Replica: 2537bbbd1a9947ae9facd87567e3ccb9, State: Running, Role: LEADER
I20250112 15:07:25.251065 21284 consensus_queue.cc:237] T ef90b704208140a3983ffe6105939319 P 2537bbbd1a9947ae9facd87567e3ccb9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2537bbbd1a9947ae9facd87567e3ccb9" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 33173 } } peers { permanent_uuid: "c561ec3374b94b95be0c155d80ff0bff" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 33627 } } peers { permanent_uuid: "3710b74ed2044ce5995533df32e67351" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 38589 } }
I20250112 15:07:25.251132 21272 tablet_bootstrap.cc:654] T 9601bcda24ed42e7b93ab9740f1e271e P 3710b74ed2044ce5995533df32e67351: Neither blocks nor log segments found. Creating new log.
I20250112 15:07:25.251441 21281 consensus_queue.cc:260] T 16f16fcd5cf347e5853189dc4da55c7f P 2537bbbd1a9947ae9facd87567e3ccb9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2537bbbd1a9947ae9facd87567e3ccb9" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 33173 } } peers { permanent_uuid: "c561ec3374b94b95be0c155d80ff0bff" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 33627 } } peers { permanent_uuid: "3710b74ed2044ce5995533df32e67351" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 38589 } }
I20250112 15:07:25.251551 21281 ts_tablet_manager.cc:1428] T 16f16fcd5cf347e5853189dc4da55c7f P 2537bbbd1a9947ae9facd87567e3ccb9: Time spent starting tablet: real 0.003s user 0.000s sys 0.001s
I20250112 15:07:25.251533 20972 catalog_manager.cc:5526] T ef90b704208140a3983ffe6105939319 P 2537bbbd1a9947ae9facd87567e3ccb9 reported cstate change: term changed from 0 to 1, leader changed from <none> to 2537bbbd1a9947ae9facd87567e3ccb9 (127.17.60.193). New cstate: current_term: 1 leader_uuid: "2537bbbd1a9947ae9facd87567e3ccb9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2537bbbd1a9947ae9facd87567e3ccb9" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 33173 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c561ec3374b94b95be0c155d80ff0bff" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 33627 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3710b74ed2044ce5995533df32e67351" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 38589 } health_report { overall_health: UNKNOWN } } }
I20250112 15:07:25.251627 21273 tablet_bootstrap.cc:492] T 80cc505a04394c53a18c5beeded4655d P c561ec3374b94b95be0c155d80ff0bff: No bootstrap required, opened a new log
I20250112 15:07:25.251673 21273 ts_tablet_manager.cc:1397] T 80cc505a04394c53a18c5beeded4655d P c561ec3374b94b95be0c155d80ff0bff: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20250112 15:07:25.251611 21281 tablet_bootstrap.cc:492] T 9601bcda24ed42e7b93ab9740f1e271e P 2537bbbd1a9947ae9facd87567e3ccb9: Bootstrap starting.
I20250112 15:07:25.251827 21273 raft_consensus.cc:357] T 80cc505a04394c53a18c5beeded4655d P c561ec3374b94b95be0c155d80ff0bff [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2537bbbd1a9947ae9facd87567e3ccb9" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 33173 } } peers { permanent_uuid: "c561ec3374b94b95be0c155d80ff0bff" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 33627 } } peers { permanent_uuid: "3710b74ed2044ce5995533df32e67351" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 38589 } }
I20250112 15:07:25.251896 21273 raft_consensus.cc:383] T 80cc505a04394c53a18c5beeded4655d P c561ec3374b94b95be0c155d80ff0bff [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 15:07:25.251942 21273 raft_consensus.cc:738] T 80cc505a04394c53a18c5beeded4655d P c561ec3374b94b95be0c155d80ff0bff [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c561ec3374b94b95be0c155d80ff0bff, State: Initialized, Role: FOLLOWER
I20250112 15:07:25.251991 21273 consensus_queue.cc:260] T 80cc505a04394c53a18c5beeded4655d P c561ec3374b94b95be0c155d80ff0bff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2537bbbd1a9947ae9facd87567e3ccb9" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 33173 } } peers { permanent_uuid: "c561ec3374b94b95be0c155d80ff0bff" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 33627 } } peers { permanent_uuid: "3710b74ed2044ce5995533df32e67351" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 38589 } }
I20250112 15:07:25.252077 21273 ts_tablet_manager.cc:1428] T 80cc505a04394c53a18c5beeded4655d P c561ec3374b94b95be0c155d80ff0bff: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20250112 15:07:25.252140 21281 tablet_bootstrap.cc:654] T 9601bcda24ed42e7b93ab9740f1e271e P 2537bbbd1a9947ae9facd87567e3ccb9: Neither blocks nor log segments found. Creating new log.
I20250112 15:07:25.252147 21273 tablet_bootstrap.cc:492] T 69c23627ab854000b360030c384dc966 P c561ec3374b94b95be0c155d80ff0bff: Bootstrap starting.
I20250112 15:07:25.252554 21273 tablet_bootstrap.cc:654] T 69c23627ab854000b360030c384dc966 P c561ec3374b94b95be0c155d80ff0bff: Neither blocks nor log segments found. Creating new log.
I20250112 15:07:25.252938 21272 tablet_bootstrap.cc:492] T 9601bcda24ed42e7b93ab9740f1e271e P 3710b74ed2044ce5995533df32e67351: No bootstrap required, opened a new log
I20250112 15:07:25.253029 21272 ts_tablet_manager.cc:1397] T 9601bcda24ed42e7b93ab9740f1e271e P 3710b74ed2044ce5995533df32e67351: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20250112 15:07:25.253167 21281 tablet_bootstrap.cc:492] T 9601bcda24ed42e7b93ab9740f1e271e P 2537bbbd1a9947ae9facd87567e3ccb9: No bootstrap required, opened a new log
I20250112 15:07:25.253218 21281 ts_tablet_manager.cc:1397] T 9601bcda24ed42e7b93ab9740f1e271e P 2537bbbd1a9947ae9facd87567e3ccb9: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20250112 15:07:25.253204 21272 raft_consensus.cc:357] T 9601bcda24ed42e7b93ab9740f1e271e P 3710b74ed2044ce5995533df32e67351 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2537bbbd1a9947ae9facd87567e3ccb9" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 33173 } } peers { permanent_uuid: "c561ec3374b94b95be0c155d80ff0bff" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 33627 } } peers { permanent_uuid: "3710b74ed2044ce5995533df32e67351" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 38589 } }
I20250112 15:07:25.253311 21272 raft_consensus.cc:383] T 9601bcda24ed42e7b93ab9740f1e271e P 3710b74ed2044ce5995533df32e67351 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 15:07:25.253340 21272 raft_consensus.cc:738] T 9601bcda24ed42e7b93ab9740f1e271e P 3710b74ed2044ce5995533df32e67351 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3710b74ed2044ce5995533df32e67351, State: Initialized, Role: FOLLOWER
I20250112 15:07:25.253394 21272 consensus_queue.cc:260] T 9601bcda24ed42e7b93ab9740f1e271e P 3710b74ed2044ce5995533df32e67351 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2537bbbd1a9947ae9facd87567e3ccb9" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 33173 } } peers { permanent_uuid: "c561ec3374b94b95be0c155d80ff0bff" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 33627 } } peers { permanent_uuid: "3710b74ed2044ce5995533df32e67351" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 38589 } }
I20250112 15:07:25.253386 21281 raft_consensus.cc:357] T 9601bcda24ed42e7b93ab9740f1e271e P 2537bbbd1a9947ae9facd87567e3ccb9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2537bbbd1a9947ae9facd87567e3ccb9" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 33173 } } peers { permanent_uuid: "c561ec3374b94b95be0c155d80ff0bff" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 33627 } } peers { permanent_uuid: "3710b74ed2044ce5995533df32e67351" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 38589 } }
I20250112 15:07:25.253479 21281 raft_consensus.cc:383] T 9601bcda24ed42e7b93ab9740f1e271e P 2537bbbd1a9947ae9facd87567e3ccb9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 15:07:25.253508 21272 ts_tablet_manager.cc:1428] T 9601bcda24ed42e7b93ab9740f1e271e P 3710b74ed2044ce5995533df32e67351: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20250112 15:07:25.253516 21281 raft_consensus.cc:738] T 9601bcda24ed42e7b93ab9740f1e271e P 2537bbbd1a9947ae9facd87567e3ccb9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2537bbbd1a9947ae9facd87567e3ccb9, State: Initialized, Role: FOLLOWER
I20250112 15:07:25.253576 21272 tablet_bootstrap.cc:492] T bfaeae584ec04772b2d6fcd088022117 P 3710b74ed2044ce5995533df32e67351: Bootstrap starting.
I20250112 15:07:25.253664 21281 consensus_queue.cc:260] T 9601bcda24ed42e7b93ab9740f1e271e P 2537bbbd1a9947ae9facd87567e3ccb9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2537bbbd1a9947ae9facd87567e3ccb9" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 33173 } } peers { permanent_uuid: "c561ec3374b94b95be0c155d80ff0bff" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 33627 } } peers { permanent_uuid: "3710b74ed2044ce5995533df32e67351" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 38589 } }
I20250112 15:07:25.253803 21273 tablet_bootstrap.cc:492] T 69c23627ab854000b360030c384dc966 P c561ec3374b94b95be0c155d80ff0bff: No bootstrap required, opened a new log
I20250112 15:07:25.253837 21273 ts_tablet_manager.cc:1397] T 69c23627ab854000b360030c384dc966 P c561ec3374b94b95be0c155d80ff0bff: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20250112 15:07:25.254020 21272 tablet_bootstrap.cc:654] T bfaeae584ec04772b2d6fcd088022117 P 3710b74ed2044ce5995533df32e67351: Neither blocks nor log segments found. Creating new log.
I20250112 15:07:25.253995 21273 raft_consensus.cc:357] T 69c23627ab854000b360030c384dc966 P c561ec3374b94b95be0c155d80ff0bff [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2537bbbd1a9947ae9facd87567e3ccb9" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 33173 } } peers { permanent_uuid: "c561ec3374b94b95be0c155d80ff0bff" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 33627 } } peers { permanent_uuid: "3710b74ed2044ce5995533df32e67351" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 38589 } }
I20250112 15:07:25.254082 21273 raft_consensus.cc:383] T 69c23627ab854000b360030c384dc966 P c561ec3374b94b95be0c155d80ff0bff [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 15:07:25.254109 21273 raft_consensus.cc:738] T 69c23627ab854000b360030c384dc966 P c561ec3374b94b95be0c155d80ff0bff [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c561ec3374b94b95be0c155d80ff0bff, State: Initialized, Role: FOLLOWER
I20250112 15:07:25.254155 21273 consensus_queue.cc:260] T 69c23627ab854000b360030c384dc966 P c561ec3374b94b95be0c155d80ff0bff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2537bbbd1a9947ae9facd87567e3ccb9" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 33173 } } peers { permanent_uuid: "c561ec3374b94b95be0c155d80ff0bff" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 33627 } } peers { permanent_uuid: "3710b74ed2044ce5995533df32e67351" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 38589 } }
I20250112 15:07:25.254266 21273 ts_tablet_manager.cc:1428] T 69c23627ab854000b360030c384dc966 P c561ec3374b94b95be0c155d80ff0bff: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20250112 15:07:25.254316 21273 tablet_bootstrap.cc:492] T 9601bcda24ed42e7b93ab9740f1e271e P c561ec3374b94b95be0c155d80ff0bff: Bootstrap starting.
I20250112 15:07:25.254719 21273 tablet_bootstrap.cc:654] T 9601bcda24ed42e7b93ab9740f1e271e P c561ec3374b94b95be0c155d80ff0bff: Neither blocks nor log segments found. Creating new log.
I20250112 15:07:25.254784 21272 tablet_bootstrap.cc:492] T bfaeae584ec04772b2d6fcd088022117 P 3710b74ed2044ce5995533df32e67351: No bootstrap required, opened a new log
I20250112 15:07:25.254822 21272 ts_tablet_manager.cc:1397] T bfaeae584ec04772b2d6fcd088022117 P 3710b74ed2044ce5995533df32e67351: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20250112 15:07:25.254961 21272 raft_consensus.cc:357] T bfaeae584ec04772b2d6fcd088022117 P 3710b74ed2044ce5995533df32e67351 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2537bbbd1a9947ae9facd87567e3ccb9" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 33173 } } peers { permanent_uuid: "c561ec3374b94b95be0c155d80ff0bff" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 33627 } } peers { permanent_uuid: "3710b74ed2044ce5995533df32e67351" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 38589 } }
I20250112 15:07:25.254149 21281 ts_tablet_manager.cc:1428] T 9601bcda24ed42e7b93ab9740f1e271e P 2537bbbd1a9947ae9facd87567e3ccb9: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20250112 15:07:25.255029 21272 raft_consensus.cc:383] T bfaeae584ec04772b2d6fcd088022117 P 3710b74ed2044ce5995533df32e67351 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 15:07:25.255088 21272 raft_consensus.cc:738] T bfaeae584ec04772b2d6fcd088022117 P 3710b74ed2044ce5995533df32e67351 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3710b74ed2044ce5995533df32e67351, State: Initialized, Role: FOLLOWER
I20250112 15:07:25.255154 21272 consensus_queue.cc:260] T bfaeae584ec04772b2d6fcd088022117 P 3710b74ed2044ce5995533df32e67351 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2537bbbd1a9947ae9facd87567e3ccb9" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 33173 } } peers { permanent_uuid: "c561ec3374b94b95be0c155d80ff0bff" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 33627 } } peers { permanent_uuid: "3710b74ed2044ce5995533df32e67351" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 38589 } }
I20250112 15:07:25.255239 21281 tablet_bootstrap.cc:492] T 69c23627ab854000b360030c384dc966 P 2537bbbd1a9947ae9facd87567e3ccb9: Bootstrap starting.
I20250112 15:07:25.255280 21272 ts_tablet_manager.cc:1428] T bfaeae584ec04772b2d6fcd088022117 P 3710b74ed2044ce5995533df32e67351: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20250112 15:07:25.255291 21273 tablet_bootstrap.cc:492] T 9601bcda24ed42e7b93ab9740f1e271e P c561ec3374b94b95be0c155d80ff0bff: No bootstrap required, opened a new log
I20250112 15:07:25.255331 21273 ts_tablet_manager.cc:1397] T 9601bcda24ed42e7b93ab9740f1e271e P c561ec3374b94b95be0c155d80ff0bff: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20250112 15:07:25.255510 21273 raft_consensus.cc:357] T 9601bcda24ed42e7b93ab9740f1e271e P c561ec3374b94b95be0c155d80ff0bff [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2537bbbd1a9947ae9facd87567e3ccb9" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 33173 } } peers { permanent_uuid: "c561ec3374b94b95be0c155d80ff0bff" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 33627 } } peers { permanent_uuid: "3710b74ed2044ce5995533df32e67351" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 38589 } }
I20250112 15:07:25.255565 21273 raft_consensus.cc:383] T 9601bcda24ed42e7b93ab9740f1e271e P c561ec3374b94b95be0c155d80ff0bff [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 15:07:25.255587 21273 raft_consensus.cc:738] T 9601bcda24ed42e7b93ab9740f1e271e P c561ec3374b94b95be0c155d80ff0bff [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c561ec3374b94b95be0c155d80ff0bff, State: Initialized, Role: FOLLOWER
I20250112 15:07:25.255630 21281 tablet_bootstrap.cc:654] T 69c23627ab854000b360030c384dc966 P 2537bbbd1a9947ae9facd87567e3ccb9: Neither blocks nor log segments found. Creating new log.
I20250112 15:07:25.255643 21273 consensus_queue.cc:260] T 9601bcda24ed42e7b93ab9740f1e271e P c561ec3374b94b95be0c155d80ff0bff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2537bbbd1a9947ae9facd87567e3ccb9" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 33173 } } peers { permanent_uuid: "c561ec3374b94b95be0c155d80ff0bff" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 33627 } } peers { permanent_uuid: "3710b74ed2044ce5995533df32e67351" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 38589 } }
I20250112 15:07:25.255324 21272 tablet_bootstrap.cc:492] T 9d4bbba799324d7fbbac082520c65686 P 3710b74ed2044ce5995533df32e67351: Bootstrap starting.
I20250112 15:07:25.255802 21277 raft_consensus.cc:491] T 16f16fcd5cf347e5853189dc4da55c7f P 3710b74ed2044ce5995533df32e67351 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 15:07:25.255857 21277 raft_consensus.cc:513] T 16f16fcd5cf347e5853189dc4da55c7f P 3710b74ed2044ce5995533df32e67351 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2537bbbd1a9947ae9facd87567e3ccb9" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 33173 } } peers { permanent_uuid: "c561ec3374b94b95be0c155d80ff0bff" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 33627 } } peers { permanent_uuid: "3710b74ed2044ce5995533df32e67351" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 38589 } }
I20250112 15:07:25.256013 21277 leader_election.cc:290] T 16f16fcd5cf347e5853189dc4da55c7f P 3710b74ed2044ce5995533df32e67351 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2537bbbd1a9947ae9facd87567e3ccb9 (127.17.60.193:33173), c561ec3374b94b95be0c155d80ff0bff (127.17.60.194:33627)
I20250112 15:07:25.255802 21273 ts_tablet_manager.cc:1428] T 9601bcda24ed42e7b93ab9740f1e271e P c561ec3374b94b95be0c155d80ff0bff: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20250112 15:07:25.256217 21273 tablet_bootstrap.cc:492] T 57e47b07fdcf434e96ca5dd0354e93b2 P c561ec3374b94b95be0c155d80ff0bff: Bootstrap starting.
I20250112 15:07:25.256261 21272 tablet_bootstrap.cc:654] T 9d4bbba799324d7fbbac082520c65686 P 3710b74ed2044ce5995533df32e67351: Neither blocks nor log segments found. Creating new log.
I20250112 15:07:25.256678 21273 tablet_bootstrap.cc:654] T 57e47b07fdcf434e96ca5dd0354e93b2 P c561ec3374b94b95be0c155d80ff0bff: Neither blocks nor log segments found. Creating new log.
I20250112 15:07:25.257022 21281 tablet_bootstrap.cc:492] T 69c23627ab854000b360030c384dc966 P 2537bbbd1a9947ae9facd87567e3ccb9: No bootstrap required, opened a new log
I20250112 15:07:25.257062 21281 ts_tablet_manager.cc:1397] T 69c23627ab854000b360030c384dc966 P 2537bbbd1a9947ae9facd87567e3ccb9: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20250112 15:07:25.257222 21281 raft_consensus.cc:357] T 69c23627ab854000b360030c384dc966 P 2537bbbd1a9947ae9facd87567e3ccb9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2537bbbd1a9947ae9facd87567e3ccb9" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 33173 } } peers { permanent_uuid: "c561ec3374b94b95be0c155d80ff0bff" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 33627 } } peers { permanent_uuid: "3710b74ed2044ce5995533df32e67351" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 38589 } }
I20250112 15:07:25.257275 21281 raft_consensus.cc:383] T 69c23627ab854000b360030c384dc966 P 2537bbbd1a9947ae9facd87567e3ccb9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 15:07:25.257294 21281 raft_consensus.cc:738] T 69c23627ab854000b360030c384dc966 P 2537bbbd1a9947ae9facd87567e3ccb9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2537bbbd1a9947ae9facd87567e3ccb9, State: Initialized, Role: FOLLOWER
I20250112 15:07:25.257328 21281 consensus_queue.cc:260] T 69c23627ab854000b360030c384dc966 P 2537bbbd1a9947ae9facd87567e3ccb9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2537bbbd1a9947ae9facd87567e3ccb9" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 33173 } } peers { permanent_uuid: "c561ec3374b94b95be0c155d80ff0bff" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 33627 } } peers { permanent_uuid: "3710b74ed2044ce5995533df32e67351" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 38589 } }
I20250112 15:07:25.257400 21281 ts_tablet_manager.cc:1428] T 69c23627ab854000b360030c384dc966 P 2537bbbd1a9947ae9facd87567e3ccb9: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20250112 15:07:25.257447 21281 tablet_bootstrap.cc:492] T 1eb2b7e6edd34a7b83f9b918d3cdf2ed P 2537bbbd1a9947ae9facd87567e3ccb9: Bootstrap starting.
I20250112 15:07:25.257473 21273 tablet_bootstrap.cc:492] T 57e47b07fdcf434e96ca5dd0354e93b2 P c561ec3374b94b95be0c155d80ff0bff: No bootstrap required, opened a new log
I20250112 15:07:25.257510 21273 ts_tablet_manager.cc:1397] T 57e47b07fdcf434e96ca5dd0354e93b2 P c561ec3374b94b95be0c155d80ff0bff: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20250112 15:07:25.257639 21273 raft_consensus.cc:357] T 57e47b07fdcf434e96ca5dd0354e93b2 P c561ec3374b94b95be0c155d80ff0bff [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2537bbbd1a9947ae9facd87567e3ccb9" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 33173 } } peers { permanent_uuid: "c561ec3374b94b95be0c155d80ff0bff" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 33627 } } peers { permanent_uuid: "3710b74ed2044ce5995533df32e67351" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 38589 } }
I20250112 15:07:25.257686 21273 raft_consensus.cc:383] T 57e47b07fdcf434e96ca5dd0354e93b2 P c561ec3374b94b95be0c155d80ff0bff [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 15:07:25.257704 21273 raft_consensus.cc:738] T 57e47b07fdcf434e96ca5dd0354e93b2 P c561ec3374b94b95be0c155d80ff0bff [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c561ec3374b94b95be0c155d80ff0bff, State: Initialized, Role: FOLLOWER
I20250112 15:07:25.257737 21273 consensus_queue.cc:260] T 57e47b07fdcf434e96ca5dd0354e93b2 P c561ec3374b94b95be0c155d80ff0bff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2537bbbd1a9947ae9facd87567e3ccb9" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 33173 } } peers { permanent_uuid: "c561ec3374b94b95be0c155d80ff0bff" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 33627 } } peers { permanent_uuid: "3710b74ed2044ce5995533df32e67351" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 38589 } }
I20250112 15:07:25.257850 21281 tablet_bootstrap.cc:654] T 1eb2b7e6edd34a7b83f9b918d3cdf2ed P 2537bbbd1a9947ae9facd87567e3ccb9: Neither blocks nor log segments found. Creating new log.
I20250112 15:07:25.258002 21273 ts_tablet_manager.cc:1428] T 57e47b07fdcf434e96ca5dd0354e93b2 P c561ec3374b94b95be0c155d80ff0bff: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20250112 15:07:25.258069 21273 tablet_bootstrap.cc:492] T 16f16fcd5cf347e5853189dc4da55c7f P c561ec3374b94b95be0c155d80ff0bff: Bootstrap starting.
I20250112 15:07:25.258455 21273 tablet_bootstrap.cc:654] T 16f16fcd5cf347e5853189dc4da55c7f P c561ec3374b94b95be0c155d80ff0bff: Neither blocks nor log segments found. Creating new log.
I20250112 15:07:25.259078 21273 tablet_bootstrap.cc:492] T 16f16fcd5cf347e5853189dc4da55c7f P c561ec3374b94b95be0c155d80ff0bff: No bootstrap required, opened a new log
I20250112 15:07:25.259136 21273 ts_tablet_manager.cc:1397] T 16f16fcd5cf347e5853189dc4da55c7f P c561ec3374b94b95be0c155d80ff0bff: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20250112 15:07:25.259261 21273 raft_consensus.cc:357] T 16f16fcd5cf347e5853189dc4da55c7f P c561ec3374b94b95be0c155d80ff0bff [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2537bbbd1a9947ae9facd87567e3ccb9" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 33173 } } peers { permanent_uuid: "c561ec3374b94b95be0c155d80ff0bff" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 33627 } } peers { permanent_uuid: "3710b74ed2044ce5995533df32e67351" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 38589 } }
I20250112 15:07:25.259305 21273 raft_consensus.cc:383] T 16f16fcd5cf347e5853189dc4da55c7f P c561ec3374b94b95be0c155d80ff0bff [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 15:07:25.259320 21273 raft_consensus.cc:738] T 16f16fcd5cf347e5853189dc4da55c7f P c561ec3374b94b95be0c155d80ff0bff [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c561ec3374b94b95be0c155d80ff0bff, State: Initialized, Role: FOLLOWER
I20250112 15:07:25.259378 21273 consensus_queue.cc:260] T 16f16fcd5cf347e5853189dc4da55c7f P c561ec3374b94b95be0c155d80ff0bff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2537bbbd1a9947ae9facd87567e3ccb9" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 33173 } } peers { permanent_uuid: "c561ec3374b94b95be0c155d80ff0bff" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 33627 } } peers { permanent_uuid: "3710b74ed2044ce5995533df32e67351" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 38589 } }
I20250112 15:07:25.259527 21281 tablet_bootstrap.cc:492] T 1eb2b7e6edd34a7b83f9b918d3cdf2ed P 2537bbbd1a9947ae9facd87567e3ccb9: No bootstrap required, opened a new log
I20250112 15:07:25.259568 21281 ts_tablet_manager.cc:1397] T 1eb2b7e6edd34a7b83f9b918d3cdf2ed P 2537bbbd1a9947ae9facd87567e3ccb9: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20250112 15:07:25.259629 21273 ts_tablet_manager.cc:1428] T 16f16fcd5cf347e5853189dc4da55c7f P c561ec3374b94b95be0c155d80ff0bff: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20250112 15:07:25.259702 21281 raft_consensus.cc:357] T 1eb2b7e6edd34a7b83f9b918d3cdf2ed P 2537bbbd1a9947ae9facd87567e3ccb9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2537bbbd1a9947ae9facd87567e3ccb9" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 33173 } } peers { permanent_uuid: "c561ec3374b94b95be0c155d80ff0bff" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 33627 } } peers { permanent_uuid: "3710b74ed2044ce5995533df32e67351" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 38589 } }
I20250112 15:07:25.259768 21281 raft_consensus.cc:383] T 1eb2b7e6edd34a7b83f9b918d3cdf2ed P 2537bbbd1a9947ae9facd87567e3ccb9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 15:07:25.259794 21281 raft_consensus.cc:738] T 1eb2b7e6edd34a7b83f9b918d3cdf2ed P 2537bbbd1a9947ae9facd87567e3ccb9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2537bbbd1a9947ae9facd87567e3ccb9, State: Initialized, Role: FOLLOWER
I20250112 15:07:25.259840 21281 consensus_queue.cc:260] T 1eb2b7e6edd34a7b83f9b918d3cdf2ed P 2537bbbd1a9947ae9facd87567e3ccb9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2537bbbd1a9947ae9facd87567e3ccb9" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 33173 } } peers { permanent_uuid: "c561ec3374b94b95be0c155d80ff0bff" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 33627 } } peers { permanent_uuid: "3710b74ed2044ce5995533df32e67351" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 38589 } }
I20250112 15:07:25.259944 21281 ts_tablet_manager.cc:1428] T 1eb2b7e6edd34a7b83f9b918d3cdf2ed P 2537bbbd1a9947ae9facd87567e3ccb9: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20250112 15:07:25.260098 21151 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "16f16fcd5cf347e5853189dc4da55c7f" candidate_uuid: "3710b74ed2044ce5995533df32e67351" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c561ec3374b94b95be0c155d80ff0bff" is_pre_election: true
I20250112 15:07:25.260162 21151 raft_consensus.cc:2463] T 16f16fcd5cf347e5853189dc4da55c7f P c561ec3374b94b95be0c155d80ff0bff [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3710b74ed2044ce5995533df32e67351 in term 0.
I20250112 15:07:25.260300 21191 leader_election.cc:304] T 16f16fcd5cf347e5853189dc4da55c7f P 3710b74ed2044ce5995533df32e67351 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3710b74ed2044ce5995533df32e67351, c561ec3374b94b95be0c155d80ff0bff; no voters:
I20250112 15:07:25.260382 21277 raft_consensus.cc:2798] T 16f16fcd5cf347e5853189dc4da55c7f P 3710b74ed2044ce5995533df32e67351 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250112 15:07:25.260411 21277 raft_consensus.cc:491] T 16f16fcd5cf347e5853189dc4da55c7f P 3710b74ed2044ce5995533df32e67351 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250112 15:07:25.260432 21277 raft_consensus.cc:3054] T 16f16fcd5cf347e5853189dc4da55c7f P 3710b74ed2044ce5995533df32e67351 [term 0 FOLLOWER]: Advancing to term 1
I20250112 15:07:25.260742 21284 raft_consensus.cc:491] T 9601bcda24ed42e7b93ab9740f1e271e P 2537bbbd1a9947ae9facd87567e3ccb9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 15:07:25.260797 21284 raft_consensus.cc:513] T 9601bcda24ed42e7b93ab9740f1e271e P 2537bbbd1a9947ae9facd87567e3ccb9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2537bbbd1a9947ae9facd87567e3ccb9" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 33173 } } peers { permanent_uuid: "c561ec3374b94b95be0c155d80ff0bff" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 33627 } } peers { permanent_uuid: "3710b74ed2044ce5995533df32e67351" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 38589 } }
I20250112 15:07:25.260905 21284 leader_election.cc:290] T 9601bcda24ed42e7b93ab9740f1e271e P 2537bbbd1a9947ae9facd87567e3ccb9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c561ec3374b94b95be0c155d80ff0bff (127.17.60.194:33627), 3710b74ed2044ce5995533df32e67351 (127.17.60.195:38589)
I20250112 15:07:25.261206 21075 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "16f16fcd5cf347e5853189dc4da55c7f" candidate_uuid: "3710b74ed2044ce5995533df32e67351" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2537bbbd1a9947ae9facd87567e3ccb9" is_pre_election: true
I20250112 15:07:25.261266 21226 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9601bcda24ed42e7b93ab9740f1e271e" candidate_uuid: "2537bbbd1a9947ae9facd87567e3ccb9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3710b74ed2044ce5995533df32e67351" is_pre_election: true
I20250112 15:07:25.261302 21075 raft_consensus.cc:2463] T 16f16fcd5cf347e5853189dc4da55c7f P 2537bbbd1a9947ae9facd87567e3ccb9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3710b74ed2044ce5995533df32e67351 in term 0.
I20250112 15:07:25.261330 21226 raft_consensus.cc:2463] T 9601bcda24ed42e7b93ab9740f1e271e P 3710b74ed2044ce5995533df32e67351 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2537bbbd1a9947ae9facd87567e3ccb9 in term 0.
I20250112 15:07:25.261371 21272 tablet_bootstrap.cc:492] T 9d4bbba799324d7fbbac082520c65686 P 3710b74ed2044ce5995533df32e67351: No bootstrap required, opened a new log
I20250112 15:07:25.261406 21272 ts_tablet_manager.cc:1397] T 9d4bbba799324d7fbbac082520c65686 P 3710b74ed2044ce5995533df32e67351: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.000s
I20250112 15:07:25.261472 21151 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9601bcda24ed42e7b93ab9740f1e271e" candidate_uuid: "2537bbbd1a9947ae9facd87567e3ccb9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c561ec3374b94b95be0c155d80ff0bff" is_pre_election: true
I20250112 15:07:25.261472 21039 leader_election.cc:304] T 9601bcda24ed42e7b93ab9740f1e271e P 2537bbbd1a9947ae9facd87567e3ccb9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2537bbbd1a9947ae9facd87567e3ccb9, 3710b74ed2044ce5995533df32e67351; no voters:
I20250112 15:07:25.261556 21151 raft_consensus.cc:2463] T 9601bcda24ed42e7b93ab9740f1e271e P c561ec3374b94b95be0c155d80ff0bff [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2537bbbd1a9947ae9facd87567e3ccb9 in term 0.
I20250112 15:07:25.261593 21284 raft_consensus.cc:2798] T 9601bcda24ed42e7b93ab9740f1e271e P 2537bbbd1a9947ae9facd87567e3ccb9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250112 15:07:25.261623 21284 raft_consensus.cc:491] T 9601bcda24ed42e7b93ab9740f1e271e P 2537bbbd1a9947ae9facd87567e3ccb9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250112 15:07:25.261644 21284 raft_consensus.cc:3054] T 9601bcda24ed42e7b93ab9740f1e271e P 2537bbbd1a9947ae9facd87567e3ccb9 [term 0 FOLLOWER]: Advancing to term 1
I20250112 15:07:25.261565 21272 raft_consensus.cc:357] T 9d4bbba799324d7fbbac082520c65686 P 3710b74ed2044ce5995533df32e67351 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2537bbbd1a9947ae9facd87567e3ccb9" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 33173 } } peers { permanent_uuid: "c561ec3374b94b95be0c155d80ff0bff" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 33627 } } peers { permanent_uuid: "3710b74ed2044ce5995533df32e67351" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 38589 } }
I20250112 15:07:25.261775 21272 raft_consensus.cc:383] T 9d4bbba799324d7fbbac082520c65686 P 3710b74ed2044ce5995533df32e67351 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 15:07:25.261801 21272 raft_consensus.cc:738] T 9d4bbba799324d7fbbac082520c65686 P 3710b74ed2044ce5995533df32e67351 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3710b74ed2044ce5995533df32e67351, State: Initialized, Role: FOLLOWER
I20250112 15:07:25.261840 21272 consensus_queue.cc:260] T 9d4bbba799324d7fbbac082520c65686 P 3710b74ed2044ce5995533df32e67351 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2537bbbd1a9947ae9facd87567e3ccb9" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 33173 } } peers { permanent_uuid: "c561ec3374b94b95be0c155d80ff0bff" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 33627 } } peers { permanent_uuid: "3710b74ed2044ce5995533df32e67351" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 38589 } }
I20250112 15:07:25.262022 21272 ts_tablet_manager.cc:1428] T 9d4bbba799324d7fbbac082520c65686 P 3710b74ed2044ce5995533df32e67351: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20250112 15:07:25.262089 21272 tablet_bootstrap.cc:492] T 5bcec60d3b974a9bb24aedfc738e1594 P 3710b74ed2044ce5995533df32e67351: Bootstrap starting.
I20250112 15:07:25.262060 21277 raft_consensus.cc:513] T 16f16fcd5cf347e5853189dc4da55c7f P 3710b74ed2044ce5995533df32e67351 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2537bbbd1a9947ae9facd87567e3ccb9" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 33173 } } peers { permanent_uuid: "c561ec3374b94b95be0c155d80ff0bff" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 33627 } } peers { permanent_uuid: "3710b74ed2044ce5995533df32e67351" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 38589 } }
I20250112 15:07:25.262189 21277 leader_election.cc:290] T 16f16fcd5cf347e5853189dc4da55c7f P 3710b74ed2044ce5995533df32e67351 [CANDIDATE]: Term 1 election: Requested vote from peers 2537bbbd1a9947ae9facd87567e3ccb9 (127.17.60.193:33173), c561ec3374b94b95be0c155d80ff0bff (127.17.60.194:33627)
I20250112 15:07:25.262307 21284 raft_consensus.cc:513] T 9601bcda24ed42e7b93ab9740f1e271e P 2537bbbd1a9947ae9facd87567e3ccb9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2537bbbd1a9947ae9facd87567e3ccb9" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 33173 } } peers { permanent_uuid: "c561ec3374b94b95be0c155d80ff0bff" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 33627 } } peers { permanent_uuid: "3710b74ed2044ce5995533df32e67351" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 38589 } }
I20250112 15:07:25.262373 21075 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "16f16fcd5cf347e5853189dc4da55c7f" candidate_uuid: "3710b74ed2044ce5995533df32e67351" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2537bbbd1a9947ae9facd87567e3ccb9"
I20250112 15:07:25.262430 21284 leader_election.cc:290] T 9601bcda24ed42e7b93ab9740f1e271e P 2537bbbd1a9947ae9facd87567e3ccb9 [CANDIDATE]: Term 1 election: Requested vote from peers c561ec3374b94b95be0c155d80ff0bff (127.17.60.194:33627), 3710b74ed2044ce5995533df32e67351 (127.17.60.195:38589)
I20250112 15:07:25.262449 21075 raft_consensus.cc:3054] T 16f16fcd5cf347e5853189dc4da55c7f P 2537bbbd1a9947ae9facd87567e3ccb9 [term 0 FOLLOWER]: Advancing to term 1
I20250112 15:07:25.262532 21272 tablet_bootstrap.cc:654] T 5bcec60d3b974a9bb24aedfc738e1594 P 3710b74ed2044ce5995533df32e67351: Neither blocks nor log segments found. Creating new log.
I20250112 15:07:25.262620 21151 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9601bcda24ed42e7b93ab9740f1e271e" candidate_uuid: "2537bbbd1a9947ae9facd87567e3ccb9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c561ec3374b94b95be0c155d80ff0bff"
I20250112 15:07:25.262676 21151 raft_consensus.cc:3054] T 9601bcda24ed42e7b93ab9740f1e271e P c561ec3374b94b95be0c155d80ff0bff [term 0 FOLLOWER]: Advancing to term 1
I20250112 15:07:25.262890 21150 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "16f16fcd5cf347e5853189dc4da55c7f" candidate_uuid: "3710b74ed2044ce5995533df32e67351" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c561ec3374b94b95be0c155d80ff0bff"
I20250112 15:07:25.262964 21150 raft_consensus.cc:3054] T 16f16fcd5cf347e5853189dc4da55c7f P c561ec3374b94b95be0c155d80ff0bff [term 0 FOLLOWER]: Advancing to term 1
I20250112 15:07:25.263147 21075 raft_consensus.cc:2463] T 16f16fcd5cf347e5853189dc4da55c7f P 2537bbbd1a9947ae9facd87567e3ccb9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3710b74ed2044ce5995533df32e67351 in term 1.
I20250112 15:07:25.263334 21189 leader_election.cc:304] T 16f16fcd5cf347e5853189dc4da55c7f P 3710b74ed2044ce5995533df32e67351 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2537bbbd1a9947ae9facd87567e3ccb9, 3710b74ed2044ce5995533df32e67351; no voters:
I20250112 15:07:25.263342 21151 raft_consensus.cc:2463] T 9601bcda24ed42e7b93ab9740f1e271e P c561ec3374b94b95be0c155d80ff0bff [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2537bbbd1a9947ae9facd87567e3ccb9 in term 1.
I20250112 15:07:25.263445 21277 raft_consensus.cc:2798] T 16f16fcd5cf347e5853189dc4da55c7f P 3710b74ed2044ce5995533df32e67351 [term 1 FOLLOWER]: Leader election won for term 1
I20250112 15:07:25.263489 21277 raft_consensus.cc:695] T 16f16fcd5cf347e5853189dc4da55c7f P 3710b74ed2044ce5995533df32e67351 [term 1 LEADER]: Becoming Leader. State: Replica: 3710b74ed2044ce5995533df32e67351, State: Running, Role: LEADER
I20250112 15:07:25.263537 21040 leader_election.cc:304] T 9601bcda24ed42e7b93ab9740f1e271e P 2537bbbd1a9947ae9facd87567e3ccb9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2537bbbd1a9947ae9facd87567e3ccb9, c561ec3374b94b95be0c155d80ff0bff; no voters:
I20250112 15:07:25.263535 21277 consensus_queue.cc:237] T 16f16fcd5cf347e5853189dc4da55c7f P 3710b74ed2044ce5995533df32e67351 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2537bbbd1a9947ae9facd87567e3ccb9" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 33173 } } peers { permanent_uuid: "c561ec3374b94b95be0c155d80ff0bff" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 33627 } } peers { permanent_uuid: "3710b74ed2044ce5995533df32e67351" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 38589 } }
I20250112 15:07:25.263608 21150 raft_consensus.cc:2463] T 16f16fcd5cf347e5853189dc4da55c7f P c561ec3374b94b95be0c155d80ff0bff [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3710b74ed2044ce5995533df32e67351 in term 1.
I20250112 15:07:25.263926 21226 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9601bcda24ed42e7b93ab9740f1e271e" candidate_uuid: "2537bbbd1a9947ae9facd87567e3ccb9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3710b74ed2044ce5995533df32e67351"
I20250112 15:07:25.263996 21226 raft_consensus.cc:3054] T 9601bcda24ed42e7b93ab9740f1e271e P 3710b74ed2044ce5995533df32e67351 [term 0 FOLLOWER]: Advancing to term 1
I20250112 15:07:25.264456 21226 raft_consensus.cc:2463] T 9601bcda24ed42e7b93ab9740f1e271e P 3710b74ed2044ce5995533df32e67351 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2537bbbd1a9947ae9facd87567e3ccb9 in term 1.
I20250112 15:07:25.264657 21284 raft_consensus.cc:2798] T 9601bcda24ed42e7b93ab9740f1e271e P 2537bbbd1a9947ae9facd87567e3ccb9 [term 1 FOLLOWER]: Leader election won for term 1
I20250112 15:07:25.264703 21284 raft_consensus.cc:695] T 9601bcda24ed42e7b93ab9740f1e271e P 2537bbbd1a9947ae9facd87567e3ccb9 [term 1 LEADER]: Becoming Leader. State: Replica: 2537bbbd1a9947ae9facd87567e3ccb9, State: Running, Role: LEADER
I20250112 15:07:25.264726 21284 consensus_queue.cc:237] T 9601bcda24ed42e7b93ab9740f1e271e P 2537bbbd1a9947ae9facd87567e3ccb9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2537bbbd1a9947ae9facd87567e3ccb9" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 33173 } } peers { permanent_uuid: "c561ec3374b94b95be0c155d80ff0bff" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 33627 } } peers { permanent_uuid: "3710b74ed2044ce5995533df32e67351" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 38589 } }
I20250112 15:07:25.265007 21272 tablet_bootstrap.cc:492] T 5bcec60d3b974a9bb24aedfc738e1594 P 3710b74ed2044ce5995533df32e67351: No bootstrap required, opened a new log
I20250112 15:07:25.265061 21272 ts_tablet_manager.cc:1397] T 5bcec60d3b974a9bb24aedfc738e1594 P 3710b74ed2044ce5995533df32e67351: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20250112 15:07:25.264719 20972 catalog_manager.cc:5526] T 16f16fcd5cf347e5853189dc4da55c7f P 3710b74ed2044ce5995533df32e67351 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3710b74ed2044ce5995533df32e67351 (127.17.60.195). New cstate: current_term: 1 leader_uuid: "3710b74ed2044ce5995533df32e67351" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2537bbbd1a9947ae9facd87567e3ccb9" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 33173 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c561ec3374b94b95be0c155d80ff0bff" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 33627 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3710b74ed2044ce5995533df32e67351" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 38589 } health_report { overall_health: HEALTHY } } }
I20250112 15:07:25.265218 21272 raft_consensus.cc:357] T 5bcec60d3b974a9bb24aedfc738e1594 P 3710b74ed2044ce5995533df32e67351 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2537bbbd1a9947ae9facd87567e3ccb9" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 33173 } } peers { permanent_uuid: "c561ec3374b94b95be0c155d80ff0bff" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 33627 } } peers { permanent_uuid: "3710b74ed2044ce5995533df32e67351" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 38589 } }
I20250112 15:07:25.265283 21272 raft_consensus.cc:383] T 5bcec60d3b974a9bb24aedfc738e1594 P 3710b74ed2044ce5995533df32e67351 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 15:07:25.265314 21272 raft_consensus.cc:738] T 5bcec60d3b974a9bb24aedfc738e1594 P 3710b74ed2044ce5995533df32e67351 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3710b74ed2044ce5995533df32e67351, State: Initialized, Role: FOLLOWER
I20250112 15:07:25.265380 21272 consensus_queue.cc:260] T 5bcec60d3b974a9bb24aedfc738e1594 P 3710b74ed2044ce5995533df32e67351 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2537bbbd1a9947ae9facd87567e3ccb9" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 33173 } } peers { permanent_uuid: "c561ec3374b94b95be0c155d80ff0bff" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 33627 } } peers { permanent_uuid: "3710b74ed2044ce5995533df32e67351" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 38589 } }
I20250112 15:07:25.265478 21272 ts_tablet_manager.cc:1428] T 5bcec60d3b974a9bb24aedfc738e1594 P 3710b74ed2044ce5995533df32e67351: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20250112 15:07:25.265717 20971 catalog_manager.cc:5526] T 9601bcda24ed42e7b93ab9740f1e271e P 2537bbbd1a9947ae9facd87567e3ccb9 reported cstate change: term changed from 0 to 1, leader changed from <none> to 2537bbbd1a9947ae9facd87567e3ccb9 (127.17.60.193). New cstate: current_term: 1 leader_uuid: "2537bbbd1a9947ae9facd87567e3ccb9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2537bbbd1a9947ae9facd87567e3ccb9" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 33173 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c561ec3374b94b95be0c155d80ff0bff" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 33627 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3710b74ed2044ce5995533df32e67351" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 38589 } health_report { overall_health: UNKNOWN } } }
I20250112 15:07:25.314379 21276 raft_consensus.cc:491] T 9d4bbba799324d7fbbac082520c65686 P c561ec3374b94b95be0c155d80ff0bff [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 15:07:25.314493 21276 raft_consensus.cc:513] T 9d4bbba799324d7fbbac082520c65686 P c561ec3374b94b95be0c155d80ff0bff [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2537bbbd1a9947ae9facd87567e3ccb9" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 33173 } } peers { permanent_uuid: "c561ec3374b94b95be0c155d80ff0bff" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 33627 } } peers { permanent_uuid: "3710b74ed2044ce5995533df32e67351" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 38589 } }
I20250112 15:07:25.314723 21276 leader_election.cc:290] T 9d4bbba799324d7fbbac082520c65686 P c561ec3374b94b95be0c155d80ff0bff [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2537bbbd1a9947ae9facd87567e3ccb9 (127.17.60.193:33173), 3710b74ed2044ce5995533df32e67351 (127.17.60.195:38589)
I20250112 15:07:25.317294 21226 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9d4bbba799324d7fbbac082520c65686" candidate_uuid: "c561ec3374b94b95be0c155d80ff0bff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3710b74ed2044ce5995533df32e67351" is_pre_election: true
I20250112 15:07:25.317329 21075 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9d4bbba799324d7fbbac082520c65686" candidate_uuid: "c561ec3374b94b95be0c155d80ff0bff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2537bbbd1a9947ae9facd87567e3ccb9" is_pre_election: true
I20250112 15:07:25.317392 21226 raft_consensus.cc:2463] T 9d4bbba799324d7fbbac082520c65686 P 3710b74ed2044ce5995533df32e67351 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c561ec3374b94b95be0c155d80ff0bff in term 0.
I20250112 15:07:25.317397 21075 raft_consensus.cc:2463] T 9d4bbba799324d7fbbac082520c65686 P 2537bbbd1a9947ae9facd87567e3ccb9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c561ec3374b94b95be0c155d80ff0bff in term 0.
I20250112 15:07:25.317826 21115 leader_election.cc:304] T 9d4bbba799324d7fbbac082520c65686 P c561ec3374b94b95be0c155d80ff0bff [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3710b74ed2044ce5995533df32e67351, c561ec3374b94b95be0c155d80ff0bff; no voters:
I20250112 15:07:25.317914 21276 raft_consensus.cc:2798] T 9d4bbba799324d7fbbac082520c65686 P c561ec3374b94b95be0c155d80ff0bff [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250112 15:07:25.317955 21276 raft_consensus.cc:491] T 9d4bbba799324d7fbbac082520c65686 P c561ec3374b94b95be0c155d80ff0bff [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250112 15:07:25.317981 21276 raft_consensus.cc:3054] T 9d4bbba799324d7fbbac082520c65686 P c561ec3374b94b95be0c155d80ff0bff [term 0 FOLLOWER]: Advancing to term 1
I20250112 15:07:25.318604 21276 raft_consensus.cc:513] T 9d4bbba799324d7fbbac082520c65686 P c561ec3374b94b95be0c155d80ff0bff [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2537bbbd1a9947ae9facd87567e3ccb9" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 33173 } } peers { permanent_uuid: "c561ec3374b94b95be0c155d80ff0bff" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 33627 } } peers { permanent_uuid: "3710b74ed2044ce5995533df32e67351" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 38589 } }
I20250112 15:07:25.318717 21276 leader_election.cc:290] T 9d4bbba799324d7fbbac082520c65686 P c561ec3374b94b95be0c155d80ff0bff [CANDIDATE]: Term 1 election: Requested vote from peers 2537bbbd1a9947ae9facd87567e3ccb9 (127.17.60.193:33173), 3710b74ed2044ce5995533df32e67351 (127.17.60.195:38589)
I20250112 15:07:25.318979 21226 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9d4bbba799324d7fbbac082520c65686" candidate_uuid: "c561ec3374b94b95be0c155d80ff0bff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3710b74ed2044ce5995533df32e67351"
I20250112 15:07:25.318992 21075 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9d4bbba799324d7fbbac082520c65686" candidate_uuid: "c561ec3374b94b95be0c155d80ff0bff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2537bbbd1a9947ae9facd87567e3ccb9"
I20250112 15:07:25.319070 21226 raft_consensus.cc:3054] T 9d4bbba799324d7fbbac082520c65686 P 3710b74ed2044ce5995533df32e67351 [term 0 FOLLOWER]: Advancing to term 1
I20250112 15:07:25.319082 21075 raft_consensus.cc:3054] T 9d4bbba799324d7fbbac082520c65686 P 2537bbbd1a9947ae9facd87567e3ccb9 [term 0 FOLLOWER]: Advancing to term 1
I20250112 15:07:25.319696 21226 raft_consensus.cc:2463] T 9d4bbba799324d7fbbac082520c65686 P 3710b74ed2044ce5995533df32e67351 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c561ec3374b94b95be0c155d80ff0bff in term 1.
I20250112 15:07:25.319696 21075 raft_consensus.cc:2463] T 9d4bbba799324d7fbbac082520c65686 P 2537bbbd1a9947ae9facd87567e3ccb9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c561ec3374b94b95be0c155d80ff0bff in term 1.
I20250112 15:07:25.319928 21114 leader_election.cc:304] T 9d4bbba799324d7fbbac082520c65686 P c561ec3374b94b95be0c155d80ff0bff [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2537bbbd1a9947ae9facd87567e3ccb9, c561ec3374b94b95be0c155d80ff0bff; no voters:
I20250112 15:07:25.320019 21276 raft_consensus.cc:2798] T 9d4bbba799324d7fbbac082520c65686 P c561ec3374b94b95be0c155d80ff0bff [term 1 FOLLOWER]: Leader election won for term 1
I20250112 15:07:25.320195 21276 raft_consensus.cc:695] T 9d4bbba799324d7fbbac082520c65686 P c561ec3374b94b95be0c155d80ff0bff [term 1 LEADER]: Becoming Leader. State: Replica: c561ec3374b94b95be0c155d80ff0bff, State: Running, Role: LEADER
I20250112 15:07:25.320281 21276 consensus_queue.cc:237] T 9d4bbba799324d7fbbac082520c65686 P c561ec3374b94b95be0c155d80ff0bff [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2537bbbd1a9947ae9facd87567e3ccb9" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 33173 } } peers { permanent_uuid: "c561ec3374b94b95be0c155d80ff0bff" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 33627 } } peers { permanent_uuid: "3710b74ed2044ce5995533df32e67351" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 38589 } }
I20250112 15:07:25.320853 20971 catalog_manager.cc:5526] T 9d4bbba799324d7fbbac082520c65686 P c561ec3374b94b95be0c155d80ff0bff reported cstate change: term changed from 0 to 1, leader changed from <none> to c561ec3374b94b95be0c155d80ff0bff (127.17.60.194). New cstate: current_term: 1 leader_uuid: "c561ec3374b94b95be0c155d80ff0bff" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2537bbbd1a9947ae9facd87567e3ccb9" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 33173 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c561ec3374b94b95be0c155d80ff0bff" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 33627 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3710b74ed2044ce5995533df32e67351" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 38589 } health_report { overall_health: UNKNOWN } } }
I20250112 15:07:25.378268 21277 raft_consensus.cc:491] T 8c7dbaf285a04da9b4eec92a29ab7d6c P 3710b74ed2044ce5995533df32e67351 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 15:07:25.378381 21277 raft_consensus.cc:513] T 8c7dbaf285a04da9b4eec92a29ab7d6c P 3710b74ed2044ce5995533df32e67351 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2537bbbd1a9947ae9facd87567e3ccb9" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 33173 } } peers { permanent_uuid: "c561ec3374b94b95be0c155d80ff0bff" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 33627 } } peers { permanent_uuid: "3710b74ed2044ce5995533df32e67351" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 38589 } }
I20250112 15:07:25.378557 21277 leader_election.cc:290] T 8c7dbaf285a04da9b4eec92a29ab7d6c P 3710b74ed2044ce5995533df32e67351 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2537bbbd1a9947ae9facd87567e3ccb9 (127.17.60.193:33173), c561ec3374b94b95be0c155d80ff0bff (127.17.60.194:33627)
I20250112 15:07:25.378790 21075 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c7dbaf285a04da9b4eec92a29ab7d6c" candidate_uuid: "3710b74ed2044ce5995533df32e67351" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2537bbbd1a9947ae9facd87567e3ccb9" is_pre_election: true
I20250112 15:07:25.378798 21150 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c7dbaf285a04da9b4eec92a29ab7d6c" candidate_uuid: "3710b74ed2044ce5995533df32e67351" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c561ec3374b94b95be0c155d80ff0bff" is_pre_election: true
I20250112 15:07:25.378885 21075 raft_consensus.cc:2463] T 8c7dbaf285a04da9b4eec92a29ab7d6c P 2537bbbd1a9947ae9facd87567e3ccb9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3710b74ed2044ce5995533df32e67351 in term 0.
I20250112 15:07:25.378893 21150 raft_consensus.cc:2463] T 8c7dbaf285a04da9b4eec92a29ab7d6c P c561ec3374b94b95be0c155d80ff0bff [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3710b74ed2044ce5995533df32e67351 in term 0.
I20250112 15:07:25.379058 21189 leader_election.cc:304] T 8c7dbaf285a04da9b4eec92a29ab7d6c P 3710b74ed2044ce5995533df32e67351 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2537bbbd1a9947ae9facd87567e3ccb9, 3710b74ed2044ce5995533df32e67351; no voters:
I20250112 15:07:25.379144 21277 raft_consensus.cc:2798] T 8c7dbaf285a04da9b4eec92a29ab7d6c P 3710b74ed2044ce5995533df32e67351 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250112 15:07:25.379215 21277 raft_consensus.cc:491] T 8c7dbaf285a04da9b4eec92a29ab7d6c P 3710b74ed2044ce5995533df32e67351 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250112 15:07:25.379240 21277 raft_consensus.cc:3054] T 8c7dbaf285a04da9b4eec92a29ab7d6c P 3710b74ed2044ce5995533df32e67351 [term 0 FOLLOWER]: Advancing to term 1
I20250112 15:07:25.379884 21277 raft_consensus.cc:513] T 8c7dbaf285a04da9b4eec92a29ab7d6c P 3710b74ed2044ce5995533df32e67351 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2537bbbd1a9947ae9facd87567e3ccb9" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 33173 } } peers { permanent_uuid: "c561ec3374b94b95be0c155d80ff0bff" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 33627 } } peers { permanent_uuid: "3710b74ed2044ce5995533df32e67351" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 38589 } }
I20250112 15:07:25.380018 21277 leader_election.cc:290] T 8c7dbaf285a04da9b4eec92a29ab7d6c P 3710b74ed2044ce5995533df32e67351 [CANDIDATE]: Term 1 election: Requested vote from peers 2537bbbd1a9947ae9facd87567e3ccb9 (127.17.60.193:33173), c561ec3374b94b95be0c155d80ff0bff (127.17.60.194:33627)
I20250112 15:07:25.380180 21075 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c7dbaf285a04da9b4eec92a29ab7d6c" candidate_uuid: "3710b74ed2044ce5995533df32e67351" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2537bbbd1a9947ae9facd87567e3ccb9"
I20250112 15:07:25.380191 21150 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c7dbaf285a04da9b4eec92a29ab7d6c" candidate_uuid: "3710b74ed2044ce5995533df32e67351" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c561ec3374b94b95be0c155d80ff0bff"
I20250112 15:07:25.380249 21075 raft_consensus.cc:3054] T 8c7dbaf285a04da9b4eec92a29ab7d6c P 2537bbbd1a9947ae9facd87567e3ccb9 [term 0 FOLLOWER]: Advancing to term 1
I20250112 15:07:25.380268 21150 raft_consensus.cc:3054] T 8c7dbaf285a04da9b4eec92a29ab7d6c P c561ec3374b94b95be0c155d80ff0bff [term 0 FOLLOWER]: Advancing to term 1
I20250112 15:07:25.380803 21075 raft_consensus.cc:2463] T 8c7dbaf285a04da9b4eec92a29ab7d6c P 2537bbbd1a9947ae9facd87567e3ccb9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3710b74ed2044ce5995533df32e67351 in term 1.
I20250112 15:07:25.380846 21150 raft_consensus.cc:2463] T 8c7dbaf285a04da9b4eec92a29ab7d6c P c561ec3374b94b95be0c155d80ff0bff [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3710b74ed2044ce5995533df32e67351 in term 1.
I20250112 15:07:25.380992 21189 leader_election.cc:304] T 8c7dbaf285a04da9b4eec92a29ab7d6c P 3710b74ed2044ce5995533df32e67351 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2537bbbd1a9947ae9facd87567e3ccb9, 3710b74ed2044ce5995533df32e67351; no voters:
I20250112 15:07:25.381079 21277 raft_consensus.cc:2798] T 8c7dbaf285a04da9b4eec92a29ab7d6c P 3710b74ed2044ce5995533df32e67351 [term 1 FOLLOWER]: Leader election won for term 1
I20250112 15:07:25.381139 21277 raft_consensus.cc:695] T 8c7dbaf285a04da9b4eec92a29ab7d6c P 3710b74ed2044ce5995533df32e67351 [term 1 LEADER]: Becoming Leader. State: Replica: 3710b74ed2044ce5995533df32e67351, State: Running, Role: LEADER
I20250112 15:07:25.381184 21277 consensus_queue.cc:237] T 8c7dbaf285a04da9b4eec92a29ab7d6c P 3710b74ed2044ce5995533df32e67351 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2537bbbd1a9947ae9facd87567e3ccb9" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 33173 } } peers { permanent_uuid: "c561ec3374b94b95be0c155d80ff0bff" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 33627 } } peers { permanent_uuid: "3710b74ed2044ce5995533df32e67351" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 38589 } }
I20250112 15:07:25.381750 20971 catalog_manager.cc:5526] T 8c7dbaf285a04da9b4eec92a29ab7d6c P 3710b74ed2044ce5995533df32e67351 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3710b74ed2044ce5995533df32e67351 (127.17.60.195). New cstate: current_term: 1 leader_uuid: "3710b74ed2044ce5995533df32e67351" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2537bbbd1a9947ae9facd87567e3ccb9" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 33173 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c561ec3374b94b95be0c155d80ff0bff" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 33627 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3710b74ed2044ce5995533df32e67351" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 38589 } health_report { overall_health: HEALTHY } } }
I20250112 15:07:25.392184 21276 raft_consensus.cc:491] T 57e47b07fdcf434e96ca5dd0354e93b2 P c561ec3374b94b95be0c155d80ff0bff [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 15:07:25.392278 21276 raft_consensus.cc:513] T 57e47b07fdcf434e96ca5dd0354e93b2 P c561ec3374b94b95be0c155d80ff0bff [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2537bbbd1a9947ae9facd87567e3ccb9" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 33173 } } peers { permanent_uuid: "c561ec3374b94b95be0c155d80ff0bff" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 33627 } } peers { permanent_uuid: "3710b74ed2044ce5995533df32e67351" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 38589 } }
I20250112 15:07:25.392438 21276 leader_election.cc:290] T 57e47b07fdcf434e96ca5dd0354e93b2 P c561ec3374b94b95be0c155d80ff0bff [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2537bbbd1a9947ae9facd87567e3ccb9 (127.17.60.193:33173), 3710b74ed2044ce5995533df32e67351 (127.17.60.195:38589)
I20250112 15:07:25.392654 21075 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "57e47b07fdcf434e96ca5dd0354e93b2" candidate_uuid: "c561ec3374b94b95be0c155d80ff0bff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2537bbbd1a9947ae9facd87567e3ccb9" is_pre_election: true
I20250112 15:07:25.392618 21226 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "57e47b07fdcf434e96ca5dd0354e93b2" candidate_uuid: "c561ec3374b94b95be0c155d80ff0bff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3710b74ed2044ce5995533df32e67351" is_pre_election: true
I20250112 15:07:25.392755 21075 raft_consensus.cc:2463] T 57e47b07fdcf434e96ca5dd0354e93b2 P 2537bbbd1a9947ae9facd87567e3ccb9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c561ec3374b94b95be0c155d80ff0bff in term 0.
I20250112 15:07:25.392760 21226 raft_consensus.cc:2463] T 57e47b07fdcf434e96ca5dd0354e93b2 P 3710b74ed2044ce5995533df32e67351 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c561ec3374b94b95be0c155d80ff0bff in term 0.
I20250112 15:07:25.392899 21115 leader_election.cc:304] T 57e47b07fdcf434e96ca5dd0354e93b2 P c561ec3374b94b95be0c155d80ff0bff [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3710b74ed2044ce5995533df32e67351, c561ec3374b94b95be0c155d80ff0bff; no voters:
I20250112 15:07:25.392978 21276 raft_consensus.cc:2798] T 57e47b07fdcf434e96ca5dd0354e93b2 P c561ec3374b94b95be0c155d80ff0bff [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250112 15:07:25.393029 21276 raft_consensus.cc:491] T 57e47b07fdcf434e96ca5dd0354e93b2 P c561ec3374b94b95be0c155d80ff0bff [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250112 15:07:25.393052 21276 raft_consensus.cc:3054] T 57e47b07fdcf434e96ca5dd0354e93b2 P c561ec3374b94b95be0c155d80ff0bff [term 0 FOLLOWER]: Advancing to term 1
I20250112 15:07:25.393656 21276 raft_consensus.cc:513] T 57e47b07fdcf434e96ca5dd0354e93b2 P c561ec3374b94b95be0c155d80ff0bff [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2537bbbd1a9947ae9facd87567e3ccb9" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 33173 } } peers { permanent_uuid: "c561ec3374b94b95be0c155d80ff0bff" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 33627 } } peers { permanent_uuid: "3710b74ed2044ce5995533df32e67351" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 38589 } }
I20250112 15:07:25.393766 21276 leader_election.cc:290] T 57e47b07fdcf434e96ca5dd0354e93b2 P c561ec3374b94b95be0c155d80ff0bff [CANDIDATE]: Term 1 election: Requested vote from peers 2537bbbd1a9947ae9facd87567e3ccb9 (127.17.60.193:33173), 3710b74ed2044ce5995533df32e67351 (127.17.60.195:38589)
I20250112 15:07:25.393908 21075 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "57e47b07fdcf434e96ca5dd0354e93b2" candidate_uuid: "c561ec3374b94b95be0c155d80ff0bff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2537bbbd1a9947ae9facd87567e3ccb9"
I20250112 15:07:25.393952 21226 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "57e47b07fdcf434e96ca5dd0354e93b2" candidate_uuid: "c561ec3374b94b95be0c155d80ff0bff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3710b74ed2044ce5995533df32e67351"
I20250112 15:07:25.394023 21226 raft_consensus.cc:3054] T 57e47b07fdcf434e96ca5dd0354e93b2 P 3710b74ed2044ce5995533df32e67351 [term 0 FOLLOWER]: Advancing to term 1
I20250112 15:07:25.393971 21075 raft_consensus.cc:3054] T 57e47b07fdcf434e96ca5dd0354e93b2 P 2537bbbd1a9947ae9facd87567e3ccb9 [term 0 FOLLOWER]: Advancing to term 1
I20250112 15:07:25.394557 21226 raft_consensus.cc:2463] T 57e47b07fdcf434e96ca5dd0354e93b2 P 3710b74ed2044ce5995533df32e67351 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c561ec3374b94b95be0c155d80ff0bff in term 1.
I20250112 15:07:25.394562 21075 raft_consensus.cc:2463] T 57e47b07fdcf434e96ca5dd0354e93b2 P 2537bbbd1a9947ae9facd87567e3ccb9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c561ec3374b94b95be0c155d80ff0bff in term 1.
I20250112 15:07:25.394723 21115 leader_election.cc:304] T 57e47b07fdcf434e96ca5dd0354e93b2 P c561ec3374b94b95be0c155d80ff0bff [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3710b74ed2044ce5995533df32e67351, c561ec3374b94b95be0c155d80ff0bff; no voters:
I20250112 15:07:25.394837 21276 raft_consensus.cc:2798] T 57e47b07fdcf434e96ca5dd0354e93b2 P c561ec3374b94b95be0c155d80ff0bff [term 1 FOLLOWER]: Leader election won for term 1
I20250112 15:07:25.394905 21276 raft_consensus.cc:695] T 57e47b07fdcf434e96ca5dd0354e93b2 P c561ec3374b94b95be0c155d80ff0bff [term 1 LEADER]: Becoming Leader. State: Replica: c561ec3374b94b95be0c155d80ff0bff, State: Running, Role: LEADER
I20250112 15:07:25.394954 21276 consensus_queue.cc:237] T 57e47b07fdcf434e96ca5dd0354e93b2 P c561ec3374b94b95be0c155d80ff0bff [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2537bbbd1a9947ae9facd87567e3ccb9" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 33173 } } peers { permanent_uuid: "c561ec3374b94b95be0c155d80ff0bff" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 33627 } } peers { permanent_uuid: "3710b74ed2044ce5995533df32e67351" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 38589 } }
I20250112 15:07:25.395539 20971 catalog_manager.cc:5526] T 57e47b07fdcf434e96ca5dd0354e93b2 P c561ec3374b94b95be0c155d80ff0bff reported cstate change: term changed from 0 to 1, leader changed from <none> to c561ec3374b94b95be0c155d80ff0bff (127.17.60.194). New cstate: current_term: 1 leader_uuid: "c561ec3374b94b95be0c155d80ff0bff" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2537bbbd1a9947ae9facd87567e3ccb9" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 33173 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c561ec3374b94b95be0c155d80ff0bff" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 33627 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3710b74ed2044ce5995533df32e67351" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 38589 } health_report { overall_health: UNKNOWN } } }
I20250112 15:07:25.395941 21276 raft_consensus.cc:491] T 80cc505a04394c53a18c5beeded4655d P c561ec3374b94b95be0c155d80ff0bff [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 15:07:25.395987 21276 raft_consensus.cc:513] T 80cc505a04394c53a18c5beeded4655d P c561ec3374b94b95be0c155d80ff0bff [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2537bbbd1a9947ae9facd87567e3ccb9" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 33173 } } peers { permanent_uuid: "c561ec3374b94b95be0c155d80ff0bff" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 33627 } } peers { permanent_uuid: "3710b74ed2044ce5995533df32e67351" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 38589 } }
I20250112 15:07:25.396087 21276 leader_election.cc:290] T 80cc505a04394c53a18c5beeded4655d P c561ec3374b94b95be0c155d80ff0bff [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2537bbbd1a9947ae9facd87567e3ccb9 (127.17.60.193:33173), 3710b74ed2044ce5995533df32e67351 (127.17.60.195:38589)
I20250112 15:07:25.396224 21226 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "80cc505a04394c53a18c5beeded4655d" candidate_uuid: "c561ec3374b94b95be0c155d80ff0bff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3710b74ed2044ce5995533df32e67351" is_pre_election: true
I20250112 15:07:25.396353 21226 raft_consensus.cc:2463] T 80cc505a04394c53a18c5beeded4655d P 3710b74ed2044ce5995533df32e67351 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c561ec3374b94b95be0c155d80ff0bff in term 0.
I20250112 15:07:25.396412 21075 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "80cc505a04394c53a18c5beeded4655d" candidate_uuid: "c561ec3374b94b95be0c155d80ff0bff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2537bbbd1a9947ae9facd87567e3ccb9" is_pre_election: true
I20250112 15:07:25.396450 21115 leader_election.cc:304] T 80cc505a04394c53a18c5beeded4655d P c561ec3374b94b95be0c155d80ff0bff [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3710b74ed2044ce5995533df32e67351, c561ec3374b94b95be0c155d80ff0bff; no voters:
I20250112 15:07:25.396453 21075 raft_consensus.cc:2463] T 80cc505a04394c53a18c5beeded4655d P 2537bbbd1a9947ae9facd87567e3ccb9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c561ec3374b94b95be0c155d80ff0bff in term 0.
I20250112 15:07:25.396553 21276 raft_consensus.cc:2798] T 80cc505a04394c53a18c5beeded4655d P c561ec3374b94b95be0c155d80ff0bff [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250112 15:07:25.396608 21276 raft_consensus.cc:491] T 80cc505a04394c53a18c5beeded4655d P c561ec3374b94b95be0c155d80ff0bff [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250112 15:07:25.396628 21276 raft_consensus.cc:3054] T 80cc505a04394c53a18c5beeded4655d P c561ec3374b94b95be0c155d80ff0bff [term 0 FOLLOWER]: Advancing to term 1
I20250112 15:07:25.397071 21276 raft_consensus.cc:513] T 80cc505a04394c53a18c5beeded4655d P c561ec3374b94b95be0c155d80ff0bff [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2537bbbd1a9947ae9facd87567e3ccb9" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 33173 } } peers { permanent_uuid: "c561ec3374b94b95be0c155d80ff0bff" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 33627 } } peers { permanent_uuid: "3710b74ed2044ce5995533df32e67351" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 38589 } }
I20250112 15:07:25.397169 21276 leader_election.cc:290] T 80cc505a04394c53a18c5beeded4655d P c561ec3374b94b95be0c155d80ff0bff [CANDIDATE]: Term 1 election: Requested vote from peers 2537bbbd1a9947ae9facd87567e3ccb9 (127.17.60.193:33173), 3710b74ed2044ce5995533df32e67351 (127.17.60.195:38589)
I20250112 15:07:25.397310 21226 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "80cc505a04394c53a18c5beeded4655d" candidate_uuid: "c561ec3374b94b95be0c155d80ff0bff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3710b74ed2044ce5995533df32e67351"
I20250112 15:07:25.397362 21226 raft_consensus.cc:3054] T 80cc505a04394c53a18c5beeded4655d P 3710b74ed2044ce5995533df32e67351 [term 0 FOLLOWER]: Advancing to term 1
I20250112 15:07:25.397747 21226 raft_consensus.cc:2463] T 80cc505a04394c53a18c5beeded4655d P 3710b74ed2044ce5995533df32e67351 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c561ec3374b94b95be0c155d80ff0bff in term 1.
I20250112 15:07:25.397822 21075 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "80cc505a04394c53a18c5beeded4655d" candidate_uuid: "c561ec3374b94b95be0c155d80ff0bff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2537bbbd1a9947ae9facd87567e3ccb9"
I20250112 15:07:25.397902 21075 raft_consensus.cc:3054] T 80cc505a04394c53a18c5beeded4655d P 2537bbbd1a9947ae9facd87567e3ccb9 [term 0 FOLLOWER]: Advancing to term 1
I20250112 15:07:25.397926 21115 leader_election.cc:304] T 80cc505a04394c53a18c5beeded4655d P c561ec3374b94b95be0c155d80ff0bff [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3710b74ed2044ce5995533df32e67351, c561ec3374b94b95be0c155d80ff0bff; no voters:
I20250112 15:07:25.398026 21276 raft_consensus.cc:2798] T 80cc505a04394c53a18c5beeded4655d P c561ec3374b94b95be0c155d80ff0bff [term 1 FOLLOWER]: Leader election won for term 1
I20250112 15:07:25.398072 21276 raft_consensus.cc:695] T 80cc505a04394c53a18c5beeded4655d P c561ec3374b94b95be0c155d80ff0bff [term 1 LEADER]: Becoming Leader. State: Replica: c561ec3374b94b95be0c155d80ff0bff, State: Running, Role: LEADER
I20250112 15:07:25.398180 21276 consensus_queue.cc:237] T 80cc505a04394c53a18c5beeded4655d P c561ec3374b94b95be0c155d80ff0bff [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2537bbbd1a9947ae9facd87567e3ccb9" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 33173 } } peers { permanent_uuid: "c561ec3374b94b95be0c155d80ff0bff" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 33627 } } peers { permanent_uuid: "3710b74ed2044ce5995533df32e67351" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 38589 } }
I20250112 15:07:25.398480 21075 raft_consensus.cc:2463] T 80cc505a04394c53a18c5beeded4655d P 2537bbbd1a9947ae9facd87567e3ccb9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c561ec3374b94b95be0c155d80ff0bff in term 1.
I20250112 15:07:25.398725 20971 catalog_manager.cc:5526] T 80cc505a04394c53a18c5beeded4655d P c561ec3374b94b95be0c155d80ff0bff reported cstate change: term changed from 0 to 1, leader changed from <none> to c561ec3374b94b95be0c155d80ff0bff (127.17.60.194). New cstate: current_term: 1 leader_uuid: "c561ec3374b94b95be0c155d80ff0bff" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2537bbbd1a9947ae9facd87567e3ccb9" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 33173 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c561ec3374b94b95be0c155d80ff0bff" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 33627 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3710b74ed2044ce5995533df32e67351" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 38589 } health_report { overall_health: UNKNOWN } } }
I20250112 15:07:25.437764 21276 raft_consensus.cc:491] T 02c33d457b6b47138dd712850819b619 P c561ec3374b94b95be0c155d80ff0bff [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 15:07:25.437870 21276 raft_consensus.cc:513] T 02c33d457b6b47138dd712850819b619 P c561ec3374b94b95be0c155d80ff0bff [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3710b74ed2044ce5995533df32e67351" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 38589 } } peers { permanent_uuid: "2537bbbd1a9947ae9facd87567e3ccb9" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 33173 } } peers { permanent_uuid: "c561ec3374b94b95be0c155d80ff0bff" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 33627 } }
I20250112 15:07:25.438005 21276 leader_election.cc:290] T 02c33d457b6b47138dd712850819b619 P c561ec3374b94b95be0c155d80ff0bff [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3710b74ed2044ce5995533df32e67351 (127.17.60.195:38589), 2537bbbd1a9947ae9facd87567e3ccb9 (127.17.60.193:33173)
I20250112 15:07:25.438226 21226 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "02c33d457b6b47138dd712850819b619" candidate_uuid: "c561ec3374b94b95be0c155d80ff0bff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3710b74ed2044ce5995533df32e67351" is_pre_election: true
I20250112 15:07:25.438297 21226 raft_consensus.cc:2463] T 02c33d457b6b47138dd712850819b619 P 3710b74ed2044ce5995533df32e67351 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c561ec3374b94b95be0c155d80ff0bff in term 0.
I20250112 15:07:25.438254 21075 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "02c33d457b6b47138dd712850819b619" candidate_uuid: "c561ec3374b94b95be0c155d80ff0bff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2537bbbd1a9947ae9facd87567e3ccb9" is_pre_election: true
I20250112 15:07:25.438359 21075 raft_consensus.cc:2463] T 02c33d457b6b47138dd712850819b619 P 2537bbbd1a9947ae9facd87567e3ccb9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c561ec3374b94b95be0c155d80ff0bff in term 0.
I20250112 15:07:25.438457 21115 leader_election.cc:304] T 02c33d457b6b47138dd712850819b619 P c561ec3374b94b95be0c155d80ff0bff [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3710b74ed2044ce5995533df32e67351, c561ec3374b94b95be0c155d80ff0bff; no voters:
I20250112 15:07:25.438545 21276 raft_consensus.cc:2798] T 02c33d457b6b47138dd712850819b619 P c561ec3374b94b95be0c155d80ff0bff [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250112 15:07:25.438580 21276 raft_consensus.cc:491] T 02c33d457b6b47138dd712850819b619 P c561ec3374b94b95be0c155d80ff0bff [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250112 15:07:25.438592 21276 raft_consensus.cc:3054] T 02c33d457b6b47138dd712850819b619 P c561ec3374b94b95be0c155d80ff0bff [term 0 FOLLOWER]: Advancing to term 1
I20250112 15:07:25.439195 21276 raft_consensus.cc:513] T 02c33d457b6b47138dd712850819b619 P c561ec3374b94b95be0c155d80ff0bff [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3710b74ed2044ce5995533df32e67351" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 38589 } } peers { permanent_uuid: "2537bbbd1a9947ae9facd87567e3ccb9" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 33173 } } peers { permanent_uuid: "c561ec3374b94b95be0c155d80ff0bff" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 33627 } }
I20250112 15:07:25.439311 21276 leader_election.cc:290] T 02c33d457b6b47138dd712850819b619 P c561ec3374b94b95be0c155d80ff0bff [CANDIDATE]: Term 1 election: Requested vote from peers 3710b74ed2044ce5995533df32e67351 (127.17.60.195:38589), 2537bbbd1a9947ae9facd87567e3ccb9 (127.17.60.193:33173)
I20250112 15:07:25.439455 21226 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "02c33d457b6b47138dd712850819b619" candidate_uuid: "c561ec3374b94b95be0c155d80ff0bff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3710b74ed2044ce5995533df32e67351"
I20250112 15:07:25.439463 21075 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "02c33d457b6b47138dd712850819b619" candidate_uuid: "c561ec3374b94b95be0c155d80ff0bff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2537bbbd1a9947ae9facd87567e3ccb9"
I20250112 15:07:25.439514 21226 raft_consensus.cc:3054] T 02c33d457b6b47138dd712850819b619 P 3710b74ed2044ce5995533df32e67351 [term 0 FOLLOWER]: Advancing to term 1
I20250112 15:07:25.439531 21075 raft_consensus.cc:3054] T 02c33d457b6b47138dd712850819b619 P 2537bbbd1a9947ae9facd87567e3ccb9 [term 0 FOLLOWER]: Advancing to term 1
I20250112 15:07:25.440004 21075 raft_consensus.cc:2463] T 02c33d457b6b47138dd712850819b619 P 2537bbbd1a9947ae9facd87567e3ccb9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c561ec3374b94b95be0c155d80ff0bff in term 1.
I20250112 15:07:25.440060 21226 raft_consensus.cc:2463] T 02c33d457b6b47138dd712850819b619 P 3710b74ed2044ce5995533df32e67351 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c561ec3374b94b95be0c155d80ff0bff in term 1.
I20250112 15:07:25.440184 21114 leader_election.cc:304] T 02c33d457b6b47138dd712850819b619 P c561ec3374b94b95be0c155d80ff0bff [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2537bbbd1a9947ae9facd87567e3ccb9, c561ec3374b94b95be0c155d80ff0bff; no voters:
I20250112 15:07:25.440280 21276 raft_consensus.cc:2798] T 02c33d457b6b47138dd712850819b619 P c561ec3374b94b95be0c155d80ff0bff [term 1 FOLLOWER]: Leader election won for term 1
I20250112 15:07:25.440326 21276 raft_consensus.cc:695] T 02c33d457b6b47138dd712850819b619 P c561ec3374b94b95be0c155d80ff0bff [term 1 LEADER]: Becoming Leader. State: Replica: c561ec3374b94b95be0c155d80ff0bff, State: Running, Role: LEADER
I20250112 15:07:25.440369 21276 consensus_queue.cc:237] T 02c33d457b6b47138dd712850819b619 P c561ec3374b94b95be0c155d80ff0bff [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3710b74ed2044ce5995533df32e67351" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 38589 } } peers { permanent_uuid: "2537bbbd1a9947ae9facd87567e3ccb9" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 33173 } } peers { permanent_uuid: "c561ec3374b94b95be0c155d80ff0bff" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 33627 } }
I20250112 15:07:25.440980 20971 catalog_manager.cc:5526] T 02c33d457b6b47138dd712850819b619 P c561ec3374b94b95be0c155d80ff0bff reported cstate change: term changed from 0 to 1, leader changed from <none> to c561ec3374b94b95be0c155d80ff0bff (127.17.60.194). New cstate: current_term: 1 leader_uuid: "c561ec3374b94b95be0c155d80ff0bff" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3710b74ed2044ce5995533df32e67351" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 38589 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2537bbbd1a9947ae9facd87567e3ccb9" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 33173 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c561ec3374b94b95be0c155d80ff0bff" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 33627 } health_report { overall_health: HEALTHY } } }
I20250112 15:07:25.443236 21277 raft_consensus.cc:491] T bfaeae584ec04772b2d6fcd088022117 P 3710b74ed2044ce5995533df32e67351 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 15:07:25.443286 21277 raft_consensus.cc:513] T bfaeae584ec04772b2d6fcd088022117 P 3710b74ed2044ce5995533df32e67351 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2537bbbd1a9947ae9facd87567e3ccb9" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 33173 } } peers { permanent_uuid: "c561ec3374b94b95be0c155d80ff0bff" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 33627 } } peers { permanent_uuid: "3710b74ed2044ce5995533df32e67351" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 38589 } }
I20250112 15:07:25.443383 21277 leader_election.cc:290] T bfaeae584ec04772b2d6fcd088022117 P 3710b74ed2044ce5995533df32e67351 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2537bbbd1a9947ae9facd87567e3ccb9 (127.17.60.193:33173), c561ec3374b94b95be0c155d80ff0bff (127.17.60.194:33627)
I20250112 15:07:25.443535 21075 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bfaeae584ec04772b2d6fcd088022117" candidate_uuid: "3710b74ed2044ce5995533df32e67351" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2537bbbd1a9947ae9facd87567e3ccb9" is_pre_election: true
I20250112 15:07:25.443527 21150 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bfaeae584ec04772b2d6fcd088022117" candidate_uuid: "3710b74ed2044ce5995533df32e67351" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c561ec3374b94b95be0c155d80ff0bff" is_pre_election: true
I20250112 15:07:25.443622 21075 raft_consensus.cc:2463] T bfaeae584ec04772b2d6fcd088022117 P 2537bbbd1a9947ae9facd87567e3ccb9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3710b74ed2044ce5995533df32e67351 in term 0.
I20250112 15:07:25.443627 21150 raft_consensus.cc:2463] T bfaeae584ec04772b2d6fcd088022117 P c561ec3374b94b95be0c155d80ff0bff [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3710b74ed2044ce5995533df32e67351 in term 0.
I20250112 15:07:25.443765 21191 leader_election.cc:304] T bfaeae584ec04772b2d6fcd088022117 P 3710b74ed2044ce5995533df32e67351 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3710b74ed2044ce5995533df32e67351, c561ec3374b94b95be0c155d80ff0bff; no voters:
I20250112 15:07:25.443859 21277 raft_consensus.cc:2798] T bfaeae584ec04772b2d6fcd088022117 P 3710b74ed2044ce5995533df32e67351 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250112 15:07:25.443894 21277 raft_consensus.cc:491] T bfaeae584ec04772b2d6fcd088022117 P 3710b74ed2044ce5995533df32e67351 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250112 15:07:25.443924 21277 raft_consensus.cc:3054] T bfaeae584ec04772b2d6fcd088022117 P 3710b74ed2044ce5995533df32e67351 [term 0 FOLLOWER]: Advancing to term 1
I20250112 15:07:25.444501 21277 raft_consensus.cc:513] T bfaeae584ec04772b2d6fcd088022117 P 3710b74ed2044ce5995533df32e67351 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2537bbbd1a9947ae9facd87567e3ccb9" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 33173 } } peers { permanent_uuid: "c561ec3374b94b95be0c155d80ff0bff" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 33627 } } peers { permanent_uuid: "3710b74ed2044ce5995533df32e67351" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 38589 } }
I20250112 15:07:25.444626 21277 leader_election.cc:290] T bfaeae584ec04772b2d6fcd088022117 P 3710b74ed2044ce5995533df32e67351 [CANDIDATE]: Term 1 election: Requested vote from peers 2537bbbd1a9947ae9facd87567e3ccb9 (127.17.60.193:33173), c561ec3374b94b95be0c155d80ff0bff (127.17.60.194:33627)
I20250112 15:07:25.444756 21150 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bfaeae584ec04772b2d6fcd088022117" candidate_uuid: "3710b74ed2044ce5995533df32e67351" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c561ec3374b94b95be0c155d80ff0bff"
I20250112 15:07:25.444808 21150 raft_consensus.cc:3054] T bfaeae584ec04772b2d6fcd088022117 P c561ec3374b94b95be0c155d80ff0bff [term 0 FOLLOWER]: Advancing to term 1
I20250112 15:07:25.444801 21075 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bfaeae584ec04772b2d6fcd088022117" candidate_uuid: "3710b74ed2044ce5995533df32e67351" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2537bbbd1a9947ae9facd87567e3ccb9"
I20250112 15:07:25.444871 21075 raft_consensus.cc:3054] T bfaeae584ec04772b2d6fcd088022117 P 2537bbbd1a9947ae9facd87567e3ccb9 [term 0 FOLLOWER]: Advancing to term 1
I20250112 15:07:25.445255 21150 raft_consensus.cc:2463] T bfaeae584ec04772b2d6fcd088022117 P c561ec3374b94b95be0c155d80ff0bff [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3710b74ed2044ce5995533df32e67351 in term 1.
I20250112 15:07:25.445350 21075 raft_consensus.cc:2463] T bfaeae584ec04772b2d6fcd088022117 P 2537bbbd1a9947ae9facd87567e3ccb9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3710b74ed2044ce5995533df32e67351 in term 1.
I20250112 15:07:25.445421 21191 leader_election.cc:304] T bfaeae584ec04772b2d6fcd088022117 P 3710b74ed2044ce5995533df32e67351 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3710b74ed2044ce5995533df32e67351, c561ec3374b94b95be0c155d80ff0bff; no voters:
I20250112 15:07:25.445505 21277 raft_consensus.cc:2798] T bfaeae584ec04772b2d6fcd088022117 P 3710b74ed2044ce5995533df32e67351 [term 1 FOLLOWER]: Leader election won for term 1
I20250112 15:07:25.445564 21277 raft_consensus.cc:695] T bfaeae584ec04772b2d6fcd088022117 P 3710b74ed2044ce5995533df32e67351 [term 1 LEADER]: Becoming Leader. State: Replica: 3710b74ed2044ce5995533df32e67351, State: Running, Role: LEADER
I20250112 15:07:25.445616 21277 consensus_queue.cc:237] T bfaeae584ec04772b2d6fcd088022117 P 3710b74ed2044ce5995533df32e67351 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2537bbbd1a9947ae9facd87567e3ccb9" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 33173 } } peers { permanent_uuid: "c561ec3374b94b95be0c155d80ff0bff" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 33627 } } peers { permanent_uuid: "3710b74ed2044ce5995533df32e67351" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 38589 } }
I20250112 15:07:25.446166 20971 catalog_manager.cc:5526] T bfaeae584ec04772b2d6fcd088022117 P 3710b74ed2044ce5995533df32e67351 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3710b74ed2044ce5995533df32e67351 (127.17.60.195). New cstate: current_term: 1 leader_uuid: "3710b74ed2044ce5995533df32e67351" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2537bbbd1a9947ae9facd87567e3ccb9" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 33173 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c561ec3374b94b95be0c155d80ff0bff" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 33627 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3710b74ed2044ce5995533df32e67351" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 38589 } health_report { overall_health: HEALTHY } } }
I20250112 15:07:25.473337 21276 raft_consensus.cc:491] T 69c23627ab854000b360030c384dc966 P c561ec3374b94b95be0c155d80ff0bff [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 15:07:25.473450 21276 raft_consensus.cc:513] T 69c23627ab854000b360030c384dc966 P c561ec3374b94b95be0c155d80ff0bff [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2537bbbd1a9947ae9facd87567e3ccb9" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 33173 } } peers { permanent_uuid: "c561ec3374b94b95be0c155d80ff0bff" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 33627 } } peers { permanent_uuid: "3710b74ed2044ce5995533df32e67351" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 38589 } }
I20250112 15:07:25.473629 21276 leader_election.cc:290] T 69c23627ab854000b360030c384dc966 P c561ec3374b94b95be0c155d80ff0bff [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2537bbbd1a9947ae9facd87567e3ccb9 (127.17.60.193:33173), 3710b74ed2044ce5995533df32e67351 (127.17.60.195:38589)
I20250112 15:07:25.473858 21226 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "69c23627ab854000b360030c384dc966" candidate_uuid: "c561ec3374b94b95be0c155d80ff0bff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3710b74ed2044ce5995533df32e67351" is_pre_election: true
I20250112 15:07:25.473877 21075 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "69c23627ab854000b360030c384dc966" candidate_uuid: "c561ec3374b94b95be0c155d80ff0bff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2537bbbd1a9947ae9facd87567e3ccb9" is_pre_election: true
I20250112 15:07:25.473950 21226 raft_consensus.cc:2463] T 69c23627ab854000b360030c384dc966 P 3710b74ed2044ce5995533df32e67351 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c561ec3374b94b95be0c155d80ff0bff in term 0.
I20250112 15:07:25.473956 21075 raft_consensus.cc:2463] T 69c23627ab854000b360030c384dc966 P 2537bbbd1a9947ae9facd87567e3ccb9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c561ec3374b94b95be0c155d80ff0bff in term 0.
I20250112 15:07:25.474144 21115 leader_election.cc:304] T 69c23627ab854000b360030c384dc966 P c561ec3374b94b95be0c155d80ff0bff [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3710b74ed2044ce5995533df32e67351, c561ec3374b94b95be0c155d80ff0bff; no voters:
I20250112 15:07:25.474216 21276 raft_consensus.cc:2798] T 69c23627ab854000b360030c384dc966 P c561ec3374b94b95be0c155d80ff0bff [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250112 15:07:25.474272 21276 raft_consensus.cc:491] T 69c23627ab854000b360030c384dc966 P c561ec3374b94b95be0c155d80ff0bff [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250112 15:07:25.474298 21276 raft_consensus.cc:3054] T 69c23627ab854000b360030c384dc966 P c561ec3374b94b95be0c155d80ff0bff [term 0 FOLLOWER]: Advancing to term 1
I20250112 15:07:25.474858 21276 raft_consensus.cc:513] T 69c23627ab854000b360030c384dc966 P c561ec3374b94b95be0c155d80ff0bff [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2537bbbd1a9947ae9facd87567e3ccb9" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 33173 } } peers { permanent_uuid: "c561ec3374b94b95be0c155d80ff0bff" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 33627 } } peers { permanent_uuid: "3710b74ed2044ce5995533df32e67351" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 38589 } }
I20250112 15:07:25.475004 21276 leader_election.cc:290] T 69c23627ab854000b360030c384dc966 P c561ec3374b94b95be0c155d80ff0bff [CANDIDATE]: Term 1 election: Requested vote from peers 2537bbbd1a9947ae9facd87567e3ccb9 (127.17.60.193:33173), 3710b74ed2044ce5995533df32e67351 (127.17.60.195:38589)
I20250112 15:07:25.475212 21226 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "69c23627ab854000b360030c384dc966" candidate_uuid: "c561ec3374b94b95be0c155d80ff0bff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3710b74ed2044ce5995533df32e67351"
I20250112 15:07:25.475242 21075 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "69c23627ab854000b360030c384dc966" candidate_uuid: "c561ec3374b94b95be0c155d80ff0bff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2537bbbd1a9947ae9facd87567e3ccb9"
I20250112 15:07:25.475294 21226 raft_consensus.cc:3054] T 69c23627ab854000b360030c384dc966 P 3710b74ed2044ce5995533df32e67351 [term 0 FOLLOWER]: Advancing to term 1
I20250112 15:07:25.475306 21075 raft_consensus.cc:3054] T 69c23627ab854000b360030c384dc966 P 2537bbbd1a9947ae9facd87567e3ccb9 [term 0 FOLLOWER]: Advancing to term 1
I20250112 15:07:25.475904 21075 raft_consensus.cc:2463] T 69c23627ab854000b360030c384dc966 P 2537bbbd1a9947ae9facd87567e3ccb9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c561ec3374b94b95be0c155d80ff0bff in term 1.
I20250112 15:07:25.475904 21226 raft_consensus.cc:2463] T 69c23627ab854000b360030c384dc966 P 3710b74ed2044ce5995533df32e67351 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c561ec3374b94b95be0c155d80ff0bff in term 1.
I20250112 15:07:25.476089 21115 leader_election.cc:304] T 69c23627ab854000b360030c384dc966 P c561ec3374b94b95be0c155d80ff0bff [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3710b74ed2044ce5995533df32e67351, c561ec3374b94b95be0c155d80ff0bff; no voters:
I20250112 15:07:25.476183 21276 raft_consensus.cc:2798] T 69c23627ab854000b360030c384dc966 P c561ec3374b94b95be0c155d80ff0bff [term 1 FOLLOWER]: Leader election won for term 1
I20250112 15:07:25.476243 21276 raft_consensus.cc:695] T 69c23627ab854000b360030c384dc966 P c561ec3374b94b95be0c155d80ff0bff [term 1 LEADER]: Becoming Leader. State: Replica: c561ec3374b94b95be0c155d80ff0bff, State: Running, Role: LEADER
I20250112 15:07:25.476301 21276 consensus_queue.cc:237] T 69c23627ab854000b360030c384dc966 P c561ec3374b94b95be0c155d80ff0bff [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2537bbbd1a9947ae9facd87567e3ccb9" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 33173 } } peers { permanent_uuid: "c561ec3374b94b95be0c155d80ff0bff" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 33627 } } peers { permanent_uuid: "3710b74ed2044ce5995533df32e67351" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 38589 } }
I20250112 15:07:25.476986 20971 catalog_manager.cc:5526] T 69c23627ab854000b360030c384dc966 P c561ec3374b94b95be0c155d80ff0bff reported cstate change: term changed from 0 to 1, leader changed from <none> to c561ec3374b94b95be0c155d80ff0bff (127.17.60.194). New cstate: current_term: 1 leader_uuid: "c561ec3374b94b95be0c155d80ff0bff" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2537bbbd1a9947ae9facd87567e3ccb9" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 33173 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c561ec3374b94b95be0c155d80ff0bff" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 33627 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3710b74ed2044ce5995533df32e67351" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 38589 } health_report { overall_health: UNKNOWN } } }
I20250112 15:07:25.519449 21276 raft_consensus.cc:491] T 1eb2b7e6edd34a7b83f9b918d3cdf2ed P c561ec3374b94b95be0c155d80ff0bff [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 15:07:25.519558 21276 raft_consensus.cc:513] T 1eb2b7e6edd34a7b83f9b918d3cdf2ed P c561ec3374b94b95be0c155d80ff0bff [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2537bbbd1a9947ae9facd87567e3ccb9" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 33173 } } peers { permanent_uuid: "c561ec3374b94b95be0c155d80ff0bff" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 33627 } } peers { permanent_uuid: "3710b74ed2044ce5995533df32e67351" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 38589 } }
I20250112 15:07:25.519732 21276 leader_election.cc:290] T 1eb2b7e6edd34a7b83f9b918d3cdf2ed P c561ec3374b94b95be0c155d80ff0bff [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2537bbbd1a9947ae9facd87567e3ccb9 (127.17.60.193:33173), 3710b74ed2044ce5995533df32e67351 (127.17.60.195:38589)
I20250112 15:07:25.519959 21226 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1eb2b7e6edd34a7b83f9b918d3cdf2ed" candidate_uuid: "c561ec3374b94b95be0c155d80ff0bff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3710b74ed2044ce5995533df32e67351" is_pre_election: true
I20250112 15:07:25.519960 21075 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1eb2b7e6edd34a7b83f9b918d3cdf2ed" candidate_uuid: "c561ec3374b94b95be0c155d80ff0bff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2537bbbd1a9947ae9facd87567e3ccb9" is_pre_election: true
I20250112 15:07:25.520061 21226 raft_consensus.cc:2463] T 1eb2b7e6edd34a7b83f9b918d3cdf2ed P 3710b74ed2044ce5995533df32e67351 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c561ec3374b94b95be0c155d80ff0bff in term 0.
I20250112 15:07:25.520067 21075 raft_consensus.cc:2463] T 1eb2b7e6edd34a7b83f9b918d3cdf2ed P 2537bbbd1a9947ae9facd87567e3ccb9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c561ec3374b94b95be0c155d80ff0bff in term 0.
I20250112 15:07:25.520272 21114 leader_election.cc:304] T 1eb2b7e6edd34a7b83f9b918d3cdf2ed P c561ec3374b94b95be0c155d80ff0bff [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2537bbbd1a9947ae9facd87567e3ccb9, c561ec3374b94b95be0c155d80ff0bff; no voters:
I20250112 15:07:25.520393 21276 raft_consensus.cc:2798] T 1eb2b7e6edd34a7b83f9b918d3cdf2ed P c561ec3374b94b95be0c155d80ff0bff [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250112 15:07:25.520439 21276 raft_consensus.cc:491] T 1eb2b7e6edd34a7b83f9b918d3cdf2ed P c561ec3374b94b95be0c155d80ff0bff [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250112 15:07:25.520454 21276 raft_consensus.cc:3054] T 1eb2b7e6edd34a7b83f9b918d3cdf2ed P c561ec3374b94b95be0c155d80ff0bff [term 0 FOLLOWER]: Advancing to term 1
I20250112 15:07:25.521140 21276 raft_consensus.cc:513] T 1eb2b7e6edd34a7b83f9b918d3cdf2ed P c561ec3374b94b95be0c155d80ff0bff [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2537bbbd1a9947ae9facd87567e3ccb9" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 33173 } } peers { permanent_uuid: "c561ec3374b94b95be0c155d80ff0bff" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 33627 } } peers { permanent_uuid: "3710b74ed2044ce5995533df32e67351" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 38589 } }
I20250112 15:07:25.521260 21276 leader_election.cc:290] T 1eb2b7e6edd34a7b83f9b918d3cdf2ed P c561ec3374b94b95be0c155d80ff0bff [CANDIDATE]: Term 1 election: Requested vote from peers 2537bbbd1a9947ae9facd87567e3ccb9 (127.17.60.193:33173), 3710b74ed2044ce5995533df32e67351 (127.17.60.195:38589)
I20250112 15:07:25.521378 21226 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1eb2b7e6edd34a7b83f9b918d3cdf2ed" candidate_uuid: "c561ec3374b94b95be0c155d80ff0bff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3710b74ed2044ce5995533df32e67351"
I20250112 15:07:25.521421 21226 raft_consensus.cc:3054] T 1eb2b7e6edd34a7b83f9b918d3cdf2ed P 3710b74ed2044ce5995533df32e67351 [term 0 FOLLOWER]: Advancing to term 1
I20250112 15:07:25.521394 21075 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1eb2b7e6edd34a7b83f9b918d3cdf2ed" candidate_uuid: "c561ec3374b94b95be0c155d80ff0bff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2537bbbd1a9947ae9facd87567e3ccb9"
I20250112 15:07:25.521461 21075 raft_consensus.cc:3054] T 1eb2b7e6edd34a7b83f9b918d3cdf2ed P 2537bbbd1a9947ae9facd87567e3ccb9 [term 0 FOLLOWER]: Advancing to term 1
I20250112 15:07:25.521896 21226 raft_consensus.cc:2463] T 1eb2b7e6edd34a7b83f9b918d3cdf2ed P 3710b74ed2044ce5995533df32e67351 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c561ec3374b94b95be0c155d80ff0bff in term 1.
I20250112 15:07:25.521971 21075 raft_consensus.cc:2463] T 1eb2b7e6edd34a7b83f9b918d3cdf2ed P 2537bbbd1a9947ae9facd87567e3ccb9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c561ec3374b94b95be0c155d80ff0bff in term 1.
I20250112 15:07:25.522046 21115 leader_election.cc:304] T 1eb2b7e6edd34a7b83f9b918d3cdf2ed P c561ec3374b94b95be0c155d80ff0bff [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3710b74ed2044ce5995533df32e67351, c561ec3374b94b95be0c155d80ff0bff; no voters:
I20250112 15:07:25.522133 21276 raft_consensus.cc:2798] T 1eb2b7e6edd34a7b83f9b918d3cdf2ed P c561ec3374b94b95be0c155d80ff0bff [term 1 FOLLOWER]: Leader election won for term 1
I20250112 15:07:25.522192 21276 raft_consensus.cc:695] T 1eb2b7e6edd34a7b83f9b918d3cdf2ed P c561ec3374b94b95be0c155d80ff0bff [term 1 LEADER]: Becoming Leader. State: Replica: c561ec3374b94b95be0c155d80ff0bff, State: Running, Role: LEADER
I20250112 15:07:25.522249 21276 consensus_queue.cc:237] T 1eb2b7e6edd34a7b83f9b918d3cdf2ed P c561ec3374b94b95be0c155d80ff0bff [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2537bbbd1a9947ae9facd87567e3ccb9" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 33173 } } peers { permanent_uuid: "c561ec3374b94b95be0c155d80ff0bff" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 33627 } } peers { permanent_uuid: "3710b74ed2044ce5995533df32e67351" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 38589 } }
I20250112 15:07:25.522778 20971 catalog_manager.cc:5526] T 1eb2b7e6edd34a7b83f9b918d3cdf2ed P c561ec3374b94b95be0c155d80ff0bff reported cstate change: term changed from 0 to 1, leader changed from <none> to c561ec3374b94b95be0c155d80ff0bff (127.17.60.194). New cstate: current_term: 1 leader_uuid: "c561ec3374b94b95be0c155d80ff0bff" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2537bbbd1a9947ae9facd87567e3ccb9" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 33173 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c561ec3374b94b95be0c155d80ff0bff" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 33627 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3710b74ed2044ce5995533df32e67351" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 38589 } health_report { overall_health: UNKNOWN } } }
I20250112 15:07:25.555373 21284 raft_consensus.cc:491] T 5bcec60d3b974a9bb24aedfc738e1594 P 2537bbbd1a9947ae9facd87567e3ccb9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 15:07:25.555486 21284 raft_consensus.cc:513] T 5bcec60d3b974a9bb24aedfc738e1594 P 2537bbbd1a9947ae9facd87567e3ccb9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2537bbbd1a9947ae9facd87567e3ccb9" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 33173 } } peers { permanent_uuid: "c561ec3374b94b95be0c155d80ff0bff" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 33627 } } peers { permanent_uuid: "3710b74ed2044ce5995533df32e67351" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 38589 } }
I20250112 15:07:25.555691 21284 leader_election.cc:290] T 5bcec60d3b974a9bb24aedfc738e1594 P 2537bbbd1a9947ae9facd87567e3ccb9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c561ec3374b94b95be0c155d80ff0bff (127.17.60.194:33627), 3710b74ed2044ce5995533df32e67351 (127.17.60.195:38589)
I20250112 15:07:25.555932 21226 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5bcec60d3b974a9bb24aedfc738e1594" candidate_uuid: "2537bbbd1a9947ae9facd87567e3ccb9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3710b74ed2044ce5995533df32e67351" is_pre_election: true
I20250112 15:07:25.555950 21150 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5bcec60d3b974a9bb24aedfc738e1594" candidate_uuid: "2537bbbd1a9947ae9facd87567e3ccb9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c561ec3374b94b95be0c155d80ff0bff" is_pre_election: true
I20250112 15:07:25.556011 21226 raft_consensus.cc:2463] T 5bcec60d3b974a9bb24aedfc738e1594 P 3710b74ed2044ce5995533df32e67351 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2537bbbd1a9947ae9facd87567e3ccb9 in term 0.
I20250112 15:07:25.556036 21150 raft_consensus.cc:2463] T 5bcec60d3b974a9bb24aedfc738e1594 P c561ec3374b94b95be0c155d80ff0bff [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2537bbbd1a9947ae9facd87567e3ccb9 in term 0.
I20250112 15:07:25.556139 21039 leader_election.cc:304] T 5bcec60d3b974a9bb24aedfc738e1594 P 2537bbbd1a9947ae9facd87567e3ccb9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2537bbbd1a9947ae9facd87567e3ccb9, 3710b74ed2044ce5995533df32e67351; no voters:
I20250112 15:07:25.556206 21284 raft_consensus.cc:2798] T 5bcec60d3b974a9bb24aedfc738e1594 P 2537bbbd1a9947ae9facd87567e3ccb9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250112 15:07:25.556259 21284 raft_consensus.cc:491] T 5bcec60d3b974a9bb24aedfc738e1594 P 2537bbbd1a9947ae9facd87567e3ccb9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250112 15:07:25.556288 21284 raft_consensus.cc:3054] T 5bcec60d3b974a9bb24aedfc738e1594 P 2537bbbd1a9947ae9facd87567e3ccb9 [term 0 FOLLOWER]: Advancing to term 1
I20250112 15:07:25.556919 21284 raft_consensus.cc:513] T 5bcec60d3b974a9bb24aedfc738e1594 P 2537bbbd1a9947ae9facd87567e3ccb9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2537bbbd1a9947ae9facd87567e3ccb9" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 33173 } } peers { permanent_uuid: "c561ec3374b94b95be0c155d80ff0bff" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 33627 } } peers { permanent_uuid: "3710b74ed2044ce5995533df32e67351" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 38589 } }
I20250112 15:07:25.557035 21284 leader_election.cc:290] T 5bcec60d3b974a9bb24aedfc738e1594 P 2537bbbd1a9947ae9facd87567e3ccb9 [CANDIDATE]: Term 1 election: Requested vote from peers c561ec3374b94b95be0c155d80ff0bff (127.17.60.194:33627), 3710b74ed2044ce5995533df32e67351 (127.17.60.195:38589)
I20250112 15:07:25.557152 21150 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5bcec60d3b974a9bb24aedfc738e1594" candidate_uuid: "2537bbbd1a9947ae9facd87567e3ccb9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c561ec3374b94b95be0c155d80ff0bff"
I20250112 15:07:25.557176 21226 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5bcec60d3b974a9bb24aedfc738e1594" candidate_uuid: "2537bbbd1a9947ae9facd87567e3ccb9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3710b74ed2044ce5995533df32e67351"
I20250112 15:07:25.557217 21150 raft_consensus.cc:3054] T 5bcec60d3b974a9bb24aedfc738e1594 P c561ec3374b94b95be0c155d80ff0bff [term 0 FOLLOWER]: Advancing to term 1
I20250112 15:07:25.557232 21226 raft_consensus.cc:3054] T 5bcec60d3b974a9bb24aedfc738e1594 P 3710b74ed2044ce5995533df32e67351 [term 0 FOLLOWER]: Advancing to term 1
I20250112 15:07:25.557699 21226 raft_consensus.cc:2463] T 5bcec60d3b974a9bb24aedfc738e1594 P 3710b74ed2044ce5995533df32e67351 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2537bbbd1a9947ae9facd87567e3ccb9 in term 1.
I20250112 15:07:25.557739 21150 raft_consensus.cc:2463] T 5bcec60d3b974a9bb24aedfc738e1594 P c561ec3374b94b95be0c155d80ff0bff [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2537bbbd1a9947ae9facd87567e3ccb9 in term 1.
I20250112 15:07:25.557868 21039 leader_election.cc:304] T 5bcec60d3b974a9bb24aedfc738e1594 P 2537bbbd1a9947ae9facd87567e3ccb9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2537bbbd1a9947ae9facd87567e3ccb9, 3710b74ed2044ce5995533df32e67351; no voters:
I20250112 15:07:25.557955 21284 raft_consensus.cc:2798] T 5bcec60d3b974a9bb24aedfc738e1594 P 2537bbbd1a9947ae9facd87567e3ccb9 [term 1 FOLLOWER]: Leader election won for term 1
I20250112 15:07:25.558004 21284 raft_consensus.cc:695] T 5bcec60d3b974a9bb24aedfc738e1594 P 2537bbbd1a9947ae9facd87567e3ccb9 [term 1 LEADER]: Becoming Leader. State: Replica: 2537bbbd1a9947ae9facd87567e3ccb9, State: Running, Role: LEADER
I20250112 15:07:25.558056 21284 consensus_queue.cc:237] T 5bcec60d3b974a9bb24aedfc738e1594 P 2537bbbd1a9947ae9facd87567e3ccb9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2537bbbd1a9947ae9facd87567e3ccb9" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 33173 } } peers { permanent_uuid: "c561ec3374b94b95be0c155d80ff0bff" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 33627 } } peers { permanent_uuid: "3710b74ed2044ce5995533df32e67351" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 38589 } }
I20250112 15:07:25.558655 20971 catalog_manager.cc:5526] T 5bcec60d3b974a9bb24aedfc738e1594 P 2537bbbd1a9947ae9facd87567e3ccb9 reported cstate change: term changed from 0 to 1, leader changed from <none> to 2537bbbd1a9947ae9facd87567e3ccb9 (127.17.60.193). New cstate: current_term: 1 leader_uuid: "2537bbbd1a9947ae9facd87567e3ccb9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2537bbbd1a9947ae9facd87567e3ccb9" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 33173 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c561ec3374b94b95be0c155d80ff0bff" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 33627 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3710b74ed2044ce5995533df32e67351" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 38589 } health_report { overall_health: UNKNOWN } } }
I20250112 15:07:25.579244 21075 raft_consensus.cc:1270] T 1eb2b7e6edd34a7b83f9b918d3cdf2ed P 2537bbbd1a9947ae9facd87567e3ccb9 [term 1 FOLLOWER]: Refusing update from remote peer c561ec3374b94b95be0c155d80ff0bff: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250112 15:07:25.579401 21226 raft_consensus.cc:1270] T 1eb2b7e6edd34a7b83f9b918d3cdf2ed P 3710b74ed2044ce5995533df32e67351 [term 1 FOLLOWER]: Refusing update from remote peer c561ec3374b94b95be0c155d80ff0bff: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250112 15:07:25.579720 21074 raft_consensus.cc:1270] T 02c33d457b6b47138dd712850819b619 P 2537bbbd1a9947ae9facd87567e3ccb9 [term 1 FOLLOWER]: Refusing update from remote peer c561ec3374b94b95be0c155d80ff0bff: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250112 15:07:25.579860 21073 raft_consensus.cc:1270] T 57e47b07fdcf434e96ca5dd0354e93b2 P 2537bbbd1a9947ae9facd87567e3ccb9 [term 1 FOLLOWER]: Refusing update from remote peer c561ec3374b94b95be0c155d80ff0bff: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250112 15:07:25.579928 21224 raft_consensus.cc:1270] T 57e47b07fdcf434e96ca5dd0354e93b2 P 3710b74ed2044ce5995533df32e67351 [term 1 FOLLOWER]: Refusing update from remote peer c561ec3374b94b95be0c155d80ff0bff: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250112 15:07:25.579878 21225 raft_consensus.cc:1270] T 02c33d457b6b47138dd712850819b619 P 3710b74ed2044ce5995533df32e67351 [term 1 FOLLOWER]: Refusing update from remote peer c561ec3374b94b95be0c155d80ff0bff: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250112 15:07:25.580885 21073 raft_consensus.cc:1270] T 16f16fcd5cf347e5853189dc4da55c7f P 2537bbbd1a9947ae9facd87567e3ccb9 [term 1 FOLLOWER]: Refusing update from remote peer 3710b74ed2044ce5995533df32e67351: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250112 15:07:25.581128 21277 consensus_queue.cc:1035] T 16f16fcd5cf347e5853189dc4da55c7f P 3710b74ed2044ce5995533df32e67351 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2537bbbd1a9947ae9facd87567e3ccb9" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 33173 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250112 15:07:25.581243 21225 raft_consensus.cc:1270] T 5bcec60d3b974a9bb24aedfc738e1594 P 3710b74ed2044ce5995533df32e67351 [term 1 FOLLOWER]: Refusing update from remote peer 2537bbbd1a9947ae9facd87567e3ccb9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250112 15:07:25.581288 21073 raft_consensus.cc:1270] T 8c7dbaf285a04da9b4eec92a29ab7d6c P 2537bbbd1a9947ae9facd87567e3ccb9 [term 1 FOLLOWER]: Refusing update from remote peer 3710b74ed2044ce5995533df32e67351: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250112 15:07:25.581288 21224 raft_consensus.cc:1270] T 9601bcda24ed42e7b93ab9740f1e271e P 3710b74ed2044ce5995533df32e67351 [term 1 FOLLOWER]: Refusing update from remote peer 2537bbbd1a9947ae9facd87567e3ccb9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250112 15:07:25.581410 21150 raft_consensus.cc:1270] T 5bcec60d3b974a9bb24aedfc738e1594 P c561ec3374b94b95be0c155d80ff0bff [term 1 FOLLOWER]: Refusing update from remote peer 2537bbbd1a9947ae9facd87567e3ccb9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250112 15:07:25.581449 21276 consensus_queue.cc:1035] T 1eb2b7e6edd34a7b83f9b918d3cdf2ed P c561ec3374b94b95be0c155d80ff0bff [LEADER]: Connected to new peer: Peer: permanent_uuid: "3710b74ed2044ce5995533df32e67351" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 38589 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250112 15:07:25.581528 21325 consensus_queue.cc:1035] T 57e47b07fdcf434e96ca5dd0354e93b2 P c561ec3374b94b95be0c155d80ff0bff [LEADER]: Connected to new peer: Peer: permanent_uuid: "3710b74ed2044ce5995533df32e67351" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 38589 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250112 15:07:25.581557 21276 consensus_queue.cc:1035] T 02c33d457b6b47138dd712850819b619 P c561ec3374b94b95be0c155d80ff0bff [LEADER]: Connected to new peer: Peer: permanent_uuid: "3710b74ed2044ce5995533df32e67351" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 38589 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250112 15:07:25.581581 21151 raft_consensus.cc:1270] T 8c7dbaf285a04da9b4eec92a29ab7d6c P c561ec3374b94b95be0c155d80ff0bff [term 1 FOLLOWER]: Refusing update from remote peer 3710b74ed2044ce5995533df32e67351: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250112 15:07:25.581619 21325 consensus_queue.cc:1035] T 1eb2b7e6edd34a7b83f9b918d3cdf2ed P c561ec3374b94b95be0c155d80ff0bff [LEADER]: Connected to new peer: Peer: permanent_uuid: "2537bbbd1a9947ae9facd87567e3ccb9" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 33173 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250112 15:07:25.581683 21149 raft_consensus.cc:1270] T bfaeae584ec04772b2d6fcd088022117 P c561ec3374b94b95be0c155d80ff0bff [term 1 FOLLOWER]: Refusing update from remote peer 3710b74ed2044ce5995533df32e67351: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250112 15:07:25.581713 21325 consensus_queue.cc:1035] T 57e47b07fdcf434e96ca5dd0354e93b2 P c561ec3374b94b95be0c155d80ff0bff [LEADER]: Connected to new peer: Peer: permanent_uuid: "2537bbbd1a9947ae9facd87567e3ccb9" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 33173 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250112 15:07:25.581820 21150 raft_consensus.cc:1270] T 16f16fcd5cf347e5853189dc4da55c7f P c561ec3374b94b95be0c155d80ff0bff [term 1 FOLLOWER]: Refusing update from remote peer 3710b74ed2044ce5995533df32e67351: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250112 15:07:25.581871 21226 raft_consensus.cc:1270] T 69c23627ab854000b360030c384dc966 P 3710b74ed2044ce5995533df32e67351 [term 1 FOLLOWER]: Refusing update from remote peer c561ec3374b94b95be0c155d80ff0bff: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250112 15:07:25.581902 21224 raft_consensus.cc:1270] T 80cc505a04394c53a18c5beeded4655d P 3710b74ed2044ce5995533df32e67351 [term 1 FOLLOWER]: Refusing update from remote peer c561ec3374b94b95be0c155d80ff0bff: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250112 15:07:25.581912 21319 consensus_queue.cc:1035] T 8c7dbaf285a04da9b4eec92a29ab7d6c P 3710b74ed2044ce5995533df32e67351 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c561ec3374b94b95be0c155d80ff0bff" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 33627 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250112 15:07:25.581960 21225 raft_consensus.cc:1270] T 9d4bbba799324d7fbbac082520c65686 P 3710b74ed2044ce5995533df32e67351 [term 1 FOLLOWER]: Refusing update from remote peer c561ec3374b94b95be0c155d80ff0bff: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250112 15:07:25.581952 21277 consensus_queue.cc:1035] T bfaeae584ec04772b2d6fcd088022117 P 3710b74ed2044ce5995533df32e67351 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c561ec3374b94b95be0c155d80ff0bff" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 33627 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250112 15:07:25.581629 21276 consensus_queue.cc:1035] T 02c33d457b6b47138dd712850819b619 P c561ec3374b94b95be0c155d80ff0bff [LEADER]: Connected to new peer: Peer: permanent_uuid: "2537bbbd1a9947ae9facd87567e3ccb9" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 33173 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250112 15:07:25.582266 21284 consensus_queue.cc:1035] T 5bcec60d3b974a9bb24aedfc738e1594 P 2537bbbd1a9947ae9facd87567e3ccb9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3710b74ed2044ce5995533df32e67351" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 38589 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250112 15:07:25.582329 21304 consensus_queue.cc:1035] T 9601bcda24ed42e7b93ab9740f1e271e P 2537bbbd1a9947ae9facd87567e3ccb9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3710b74ed2044ce5995533df32e67351" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 38589 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250112 15:07:25.583438 21070 raft_consensus.cc:1270] T bfaeae584ec04772b2d6fcd088022117 P 2537bbbd1a9947ae9facd87567e3ccb9 [term 1 FOLLOWER]: Refusing update from remote peer 3710b74ed2044ce5995533df32e67351: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250112 15:07:25.583886 21075 raft_consensus.cc:1270] T 69c23627ab854000b360030c384dc966 P 2537bbbd1a9947ae9facd87567e3ccb9 [term 1 FOLLOWER]: Refusing update from remote peer c561ec3374b94b95be0c155d80ff0bff: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250112 15:07:25.584025 21074 raft_consensus.cc:1270] T 9d4bbba799324d7fbbac082520c65686 P 2537bbbd1a9947ae9facd87567e3ccb9 [term 1 FOLLOWER]: Refusing update from remote peer c561ec3374b94b95be0c155d80ff0bff: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250112 15:07:25.584192 21277 consensus_queue.cc:1035] T 8c7dbaf285a04da9b4eec92a29ab7d6c P 3710b74ed2044ce5995533df32e67351 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2537bbbd1a9947ae9facd87567e3ccb9" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 33173 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250112 15:07:25.584210 21319 consensus_queue.cc:1035] T bfaeae584ec04772b2d6fcd088022117 P 3710b74ed2044ce5995533df32e67351 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2537bbbd1a9947ae9facd87567e3ccb9" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 33173 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250112 15:07:25.584391 21150 raft_consensus.cc:1270] T 9601bcda24ed42e7b93ab9740f1e271e P c561ec3374b94b95be0c155d80ff0bff [term 1 FOLLOWER]: Refusing update from remote peer 2537bbbd1a9947ae9facd87567e3ccb9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250112 15:07:25.584432 21149 raft_consensus.cc:1270] T ef90b704208140a3983ffe6105939319 P c561ec3374b94b95be0c155d80ff0bff [term 1 FOLLOWER]: Refusing update from remote peer 2537bbbd1a9947ae9facd87567e3ccb9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250112 15:07:25.584522 21304 consensus_queue.cc:1035] T 5bcec60d3b974a9bb24aedfc738e1594 P 2537bbbd1a9947ae9facd87567e3ccb9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c561ec3374b94b95be0c155d80ff0bff" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 33627 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250112 15:07:25.584676 21073 raft_consensus.cc:1270] T 80cc505a04394c53a18c5beeded4655d P 2537bbbd1a9947ae9facd87567e3ccb9 [term 1 FOLLOWER]: Refusing update from remote peer c561ec3374b94b95be0c155d80ff0bff: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250112 15:07:25.584877 21226 raft_consensus.cc:3054] T ef90b704208140a3983ffe6105939319 P 3710b74ed2044ce5995533df32e67351 [term 0 FOLLOWER]: Advancing to term 1
I20250112 15:07:25.585603 21226 raft_consensus.cc:1270] T ef90b704208140a3983ffe6105939319 P 3710b74ed2044ce5995533df32e67351 [term 1 FOLLOWER]: Refusing update from remote peer 2537bbbd1a9947ae9facd87567e3ccb9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250112 15:07:25.586863 21359 consensus_queue.cc:1035] T 69c23627ab854000b360030c384dc966 P c561ec3374b94b95be0c155d80ff0bff [LEADER]: Connected to new peer: Peer: permanent_uuid: "3710b74ed2044ce5995533df32e67351" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 38589 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250112 15:07:25.586911 21353 consensus_queue.cc:1035] T 80cc505a04394c53a18c5beeded4655d P c561ec3374b94b95be0c155d80ff0bff [LEADER]: Connected to new peer: Peer: permanent_uuid: "3710b74ed2044ce5995533df32e67351" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 38589 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250112 15:07:25.586944 21357 consensus_queue.cc:1035] T 9d4bbba799324d7fbbac082520c65686 P c561ec3374b94b95be0c155d80ff0bff [LEADER]: Connected to new peer: Peer: permanent_uuid: "3710b74ed2044ce5995533df32e67351" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 38589 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250112 15:07:25.588495 21277 consensus_queue.cc:1035] T 16f16fcd5cf347e5853189dc4da55c7f P 3710b74ed2044ce5995533df32e67351 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c561ec3374b94b95be0c155d80ff0bff" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 33627 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250112 15:07:25.588728 21387 consensus_queue.cc:1035] T 9601bcda24ed42e7b93ab9740f1e271e P 2537bbbd1a9947ae9facd87567e3ccb9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c561ec3374b94b95be0c155d80ff0bff" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 33627 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250112 15:07:25.588851 21389 consensus_queue.cc:1035] T ef90b704208140a3983ffe6105939319 P 2537bbbd1a9947ae9facd87567e3ccb9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3710b74ed2044ce5995533df32e67351" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 38589 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250112 15:07:25.589051 21284 consensus_queue.cc:1035] T ef90b704208140a3983ffe6105939319 P 2537bbbd1a9947ae9facd87567e3ccb9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c561ec3374b94b95be0c155d80ff0bff" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 33627 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250112 15:07:25.590397 21351 consensus_queue.cc:1035] T 69c23627ab854000b360030c384dc966 P c561ec3374b94b95be0c155d80ff0bff [LEADER]: Connected to new peer: Peer: permanent_uuid: "2537bbbd1a9947ae9facd87567e3ccb9" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 33173 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250112 15:07:25.590449 21356 mvcc.cc:204] Tried to move back new op lower bound from 7113500449105534976 to 7113500448281866240. Current Snapshot: MvccSnapshot[applied={T|T < 7113500449098321920}]
I20250112 15:07:25.590546 21351 consensus_queue.cc:1035] T 9d4bbba799324d7fbbac082520c65686 P c561ec3374b94b95be0c155d80ff0bff [LEADER]: Connected to new peer: Peer: permanent_uuid: "2537bbbd1a9947ae9facd87567e3ccb9" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 33173 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250112 15:07:25.590611 21351 consensus_queue.cc:1035] T 80cc505a04394c53a18c5beeded4655d P c561ec3374b94b95be0c155d80ff0bff [LEADER]: Connected to new peer: Peer: permanent_uuid: "2537bbbd1a9947ae9facd87567e3ccb9" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 33173 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250112 15:07:25.596031 21354 mvcc.cc:204] Tried to move back new op lower bound from 7113500449097117696 to 7113500448859512832. Current Snapshot: MvccSnapshot[applied={T|T < 7113500449097117696 or (T in {7113500449097117696})}]
I20250112 15:07:25.676810 17651 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250112 15:07:25.679369 21429 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250112 15:07:25.679628 21428 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250112 15:07:25.680701 21431 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250112 15:07:25.681553 17651 server_base.cc:1034] running on GCE node
I20250112 15:07:25.681654 17651 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250112 15:07:25.681694 17651 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250112 15:07:25.681720 17651 hybrid_clock.cc:648] HybridClock initialized: now 1736694445681721 us; error 0 us; skew 500 ppm
I20250112 15:07:25.682592 17651 webserver.cc:458] Webserver started at http://127.17.60.196:40217/ using document root <none> and password file <none>
I20250112 15:07:25.682688 17651 fs_manager.cc:362] Metadata directory not provided
I20250112 15:07:25.682739 17651 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250112 15:07:25.682792 17651 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250112 15:07:25.683068 17651 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task95dLSD/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1736694351070047-17651-0/minicluster-data/ts-3-root/instance:
uuid: "a2a5bba4ac1145d5bab94a69c9f4df40"
format_stamp: "Formatted at 2025-01-12 15:07:25 on dist-test-slave-b4nz"
I20250112 15:07:25.683983 17651 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20250112 15:07:25.684448 21436 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250112 15:07:25.684615 17651 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20250112 15:07:25.684659 17651 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task95dLSD/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1736694351070047-17651-0/minicluster-data/ts-3-root
uuid: "a2a5bba4ac1145d5bab94a69c9f4df40"
format_stamp: "Formatted at 2025-01-12 15:07:25 on dist-test-slave-b4nz"
I20250112 15:07:25.684703 17651 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task95dLSD/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1736694351070047-17651-0/minicluster-data/ts-3-root
metadata directory: /tmp/dist-test-task95dLSD/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1736694351070047-17651-0/minicluster-data/ts-3-root
1 data directories: /tmp/dist-test-task95dLSD/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1736694351070047-17651-0/minicluster-data/ts-3-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250112 15:07:25.694219 17651 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250112 15:07:25.694459 17651 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250112 15:07:25.694598 17651 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250112 15:07:25.694789 17651 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250112 15:07:25.694816 17651 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250112 15:07:25.694847 17651 ts_tablet_manager.cc:610] Registered 0 tablets
I20250112 15:07:25.694871 17651 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250112 15:07:25.698318 17651 rpc_server.cc:307] RPC server started. Bound to: 127.17.60.196:41807
I20250112 15:07:25.698444 21499 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.17.60.196:41807 every 8 connection(s)
I20250112 15:07:25.698704 17651 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250112 15:07:25.700187 21503 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250112 15:07:25.701020 21504 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250112 15:07:25.701831 21506 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250112 15:07:25.701828 21500 heartbeater.cc:346] Connected to a master server at 127.17.60.254:42121
I20250112 15:07:25.701926 21500 heartbeater.cc:463] Registering TS with master...
I20250112 15:07:25.702032 21500 heartbeater.cc:510] Master 127.17.60.254:42121 requested a full tablet report, sending...
I20250112 15:07:25.702258 20966 ts_manager.cc:194] Registered new tserver with Master: a2a5bba4ac1145d5bab94a69c9f4df40 (127.17.60.196:41807)
I20250112 15:07:25.702486 17651 server_base.cc:1034] running on GCE node
I20250112 15:07:25.702569 17651 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250112 15:07:25.702597 17651 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250112 15:07:25.702613 17651 hybrid_clock.cc:648] HybridClock initialized: now 1736694445702613 us; error 0 us; skew 500 ppm
I20250112 15:07:25.702728 20966 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:39644
I20250112 15:07:25.703436 17651 webserver.cc:458] Webserver started at http://127.17.60.197:35457/ using document root <none> and password file <none>
I20250112 15:07:25.703506 17651 fs_manager.cc:362] Metadata directory not provided
I20250112 15:07:25.703541 17651 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250112 15:07:25.703574 17651 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250112 15:07:25.703800 17651 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task95dLSD/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1736694351070047-17651-0/minicluster-data/ts-4-root/instance:
uuid: "b5e9178a7a83402e8c4a6943ab78589d"
format_stamp: "Formatted at 2025-01-12 15:07:25 on dist-test-slave-b4nz"
I20250112 15:07:25.704612 17651 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20250112 15:07:25.705036 21511 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250112 15:07:25.705153 17651 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20250112 15:07:25.705196 17651 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task95dLSD/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1736694351070047-17651-0/minicluster-data/ts-4-root
uuid: "b5e9178a7a83402e8c4a6943ab78589d"
format_stamp: "Formatted at 2025-01-12 15:07:25 on dist-test-slave-b4nz"
I20250112 15:07:25.705243 17651 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task95dLSD/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1736694351070047-17651-0/minicluster-data/ts-4-root
metadata directory: /tmp/dist-test-task95dLSD/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1736694351070047-17651-0/minicluster-data/ts-4-root
1 data directories: /tmp/dist-test-task95dLSD/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1736694351070047-17651-0/minicluster-data/ts-4-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250112 15:07:25.717731 17651 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250112 15:07:25.717944 17651 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250112 15:07:25.718058 17651 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250112 15:07:25.718214 17651 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250112 15:07:25.718240 17651 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250112 15:07:25.718276 17651 ts_tablet_manager.cc:610] Registered 0 tablets
I20250112 15:07:25.718298 17651 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250112 15:07:25.721763 17651 rpc_server.cc:307] RPC server started. Bound to: 127.17.60.197:36573
I20250112 15:07:25.721855 21574 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.17.60.197:36573 every 8 connection(s)
I20250112 15:07:25.722239 17651 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250112 15:07:25.723682 21578 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250112 15:07:25.724308 21581 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250112 15:07:25.725138 21575 heartbeater.cc:346] Connected to a master server at 127.17.60.254:42121
I20250112 15:07:25.725188 21575 heartbeater.cc:463] Registering TS with master...
W20250112 15:07:25.725271 21579 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250112 15:07:25.725292 21575 heartbeater.cc:510] Master 127.17.60.254:42121 requested a full tablet report, sending...
I20250112 15:07:25.725512 20966 ts_manager.cc:194] Registered new tserver with Master: b5e9178a7a83402e8c4a6943ab78589d (127.17.60.197:36573)
I20250112 15:07:25.725819 20966 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:39646
I20250112 15:07:25.726051 17651 server_base.cc:1034] running on GCE node
I20250112 15:07:25.726168 17651 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250112 15:07:25.726202 17651 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250112 15:07:25.726260 17651 hybrid_clock.cc:648] HybridClock initialized: now 1736694445726260 us; error 0 us; skew 500 ppm
I20250112 15:07:25.726987 17651 webserver.cc:458] Webserver started at http://127.17.60.198:36817/ using document root <none> and password file <none>
I20250112 15:07:25.727072 17651 fs_manager.cc:362] Metadata directory not provided
I20250112 15:07:25.727118 17651 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250112 15:07:25.727178 17651 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250112 15:07:25.727438 17651 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task95dLSD/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1736694351070047-17651-0/minicluster-data/ts-5-root/instance:
uuid: "25286c485fa04c86a5d07c8f88643a47"
format_stamp: "Formatted at 2025-01-12 15:07:25 on dist-test-slave-b4nz"
I20250112 15:07:25.728338 17651 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20250112 15:07:25.728801 21586 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250112 15:07:25.728926 17651 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20250112 15:07:25.728972 17651 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task95dLSD/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1736694351070047-17651-0/minicluster-data/ts-5-root
uuid: "25286c485fa04c86a5d07c8f88643a47"
format_stamp: "Formatted at 2025-01-12 15:07:25 on dist-test-slave-b4nz"
I20250112 15:07:25.729017 17651 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task95dLSD/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1736694351070047-17651-0/minicluster-data/ts-5-root
metadata directory: /tmp/dist-test-task95dLSD/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1736694351070047-17651-0/minicluster-data/ts-5-root
1 data directories: /tmp/dist-test-task95dLSD/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1736694351070047-17651-0/minicluster-data/ts-5-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250112 15:07:25.756232 17651 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250112 15:07:25.756464 17651 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250112 15:07:25.756630 17651 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250112 15:07:25.756815 17651 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250112 15:07:25.756851 17651 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250112 15:07:25.756888 17651 ts_tablet_manager.cc:610] Registered 0 tablets
I20250112 15:07:25.756913 17651 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250112 15:07:25.760322 17651 rpc_server.cc:307] RPC server started. Bound to: 127.17.60.198:37089
I20250112 15:07:25.760490 21649 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.17.60.198:37089 every 8 connection(s)
I20250112 15:07:25.763387 21650 heartbeater.cc:346] Connected to a master server at 127.17.60.254:42121
I20250112 15:07:25.763449 21650 heartbeater.cc:463] Registering TS with master...
I20250112 15:07:25.763535 21650 heartbeater.cc:510] Master 127.17.60.254:42121 requested a full tablet report, sending...
I20250112 15:07:25.763757 20966 ts_manager.cc:194] Registered new tserver with Master: 25286c485fa04c86a5d07c8f88643a47 (127.17.60.198:37089)
I20250112 15:07:25.764109 20966 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:39648
I20250112 15:07:26.066207 21026 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250112 15:07:26.069731 21151 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "02c33d457b6b47138dd712850819b619"
dest_uuid: "c561ec3374b94b95be0c155d80ff0bff"
mode: GRACEFUL
new_leader_uuid: "3710b74ed2044ce5995533df32e67351"
from {username='slave'} at 127.0.0.1:48104
I20250112 15:07:26.069825 21151 raft_consensus.cc:604] T 02c33d457b6b47138dd712850819b619 P c561ec3374b94b95be0c155d80ff0bff [term 1 LEADER]: Received request to transfer leadership to TS 3710b74ed2044ce5995533df32e67351
I20250112 15:07:26.070079 21151 consensus_queue.cc:237] T 1eb2b7e6edd34a7b83f9b918d3cdf2ed P c561ec3374b94b95be0c155d80ff0bff [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 25, Committed index: 25, Last appended: 1.25, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 26 OBSOLETE_local: false peers { permanent_uuid: "2537bbbd1a9947ae9facd87567e3ccb9" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 33173 } } peers { permanent_uuid: "c561ec3374b94b95be0c155d80ff0bff" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 33627 } } peers { permanent_uuid: "3710b74ed2044ce5995533df32e67351" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 38589 } attrs { replace: true } } peers { permanent_uuid: "25286c485fa04c86a5d07c8f88643a47" member_type: NON_VOTER last_known_addr { host: "127.17.60.198" port: 37089 } attrs { promote: true } }
I20250112 15:07:26.070534 21151 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "57e47b07fdcf434e96ca5dd0354e93b2"
dest_uuid: "c561ec3374b94b95be0c155d80ff0bff"
mode: GRACEFUL
new_leader_uuid: "2537bbbd1a9947ae9facd87567e3ccb9"
from {username='slave'} at 127.0.0.1:48104
I20250112 15:07:26.070550 21226 raft_consensus.cc:1270] T 1eb2b7e6edd34a7b83f9b918d3cdf2ed P 3710b74ed2044ce5995533df32e67351 [term 1 FOLLOWER]: Refusing update from remote peer c561ec3374b94b95be0c155d80ff0bff: Log matching property violated. Preceding OpId in replica: term: 1 index: 25. Preceding OpId from leader: term: 1 index: 26. (index mismatch)
I20250112 15:07:26.070611 21151 raft_consensus.cc:604] T 57e47b07fdcf434e96ca5dd0354e93b2 P c561ec3374b94b95be0c155d80ff0bff [term 1 LEADER]: Received request to transfer leadership to TS 2537bbbd1a9947ae9facd87567e3ccb9
I20250112 15:07:26.070644 21068 raft_consensus.cc:1270] T 1eb2b7e6edd34a7b83f9b918d3cdf2ed P 2537bbbd1a9947ae9facd87567e3ccb9 [term 1 FOLLOWER]: Refusing update from remote peer c561ec3374b94b95be0c155d80ff0bff: Log matching property violated. Preceding OpId in replica: term: 1 index: 25. Preceding OpId from leader: term: 1 index: 26. (index mismatch)
I20250112 15:07:26.070752 21026 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 2
I20250112 15:07:26.070819 21026 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20250112 15:07:26.070860 21351 consensus_queue.cc:1035] T 1eb2b7e6edd34a7b83f9b918d3cdf2ed P c561ec3374b94b95be0c155d80ff0bff [LEADER]: Connected to new peer: Peer: permanent_uuid: "3710b74ed2044ce5995533df32e67351" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 38589 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 26, Last known committed idx: 25, Time since last communication: 0.000s
I20250112 15:07:26.070973 21351 consensus_queue.cc:1035] T 1eb2b7e6edd34a7b83f9b918d3cdf2ed P c561ec3374b94b95be0c155d80ff0bff [LEADER]: Connected to new peer: Peer: permanent_uuid: "2537bbbd1a9947ae9facd87567e3ccb9" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 33173 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 26, Last known committed idx: 25, Time since last communication: 0.000s
I20250112 15:07:26.071430 21351 raft_consensus.cc:2949] T 1eb2b7e6edd34a7b83f9b918d3cdf2ed P c561ec3374b94b95be0c155d80ff0bff [term 1 LEADER]: Committing config change with OpId 1.26: config changed from index -1 to 26, NON_VOTER 25286c485fa04c86a5d07c8f88643a47 (127.17.60.198) added. New config: { opid_index: 26 OBSOLETE_local: false peers { permanent_uuid: "2537bbbd1a9947ae9facd87567e3ccb9" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 33173 } } peers { permanent_uuid: "c561ec3374b94b95be0c155d80ff0bff" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 33627 } } peers { permanent_uuid: "3710b74ed2044ce5995533df32e67351" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 38589 } attrs { replace: true } } peers { permanent_uuid: "25286c485fa04c86a5d07c8f88643a47" member_type: NON_VOTER last_known_addr { host: "127.17.60.198" port: 37089 } attrs { promote: true } } }
I20250112 15:07:26.071605 21226 raft_consensus.cc:2949] T 1eb2b7e6edd34a7b83f9b918d3cdf2ed P 3710b74ed2044ce5995533df32e67351 [term 1 FOLLOWER]: Committing config change with OpId 1.26: config changed from index -1 to 26, NON_VOTER 25286c485fa04c86a5d07c8f88643a47 (127.17.60.198) added. New config: { opid_index: 26 OBSOLETE_local: false peers { permanent_uuid: "2537bbbd1a9947ae9facd87567e3ccb9" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 33173 } } peers { permanent_uuid: "c561ec3374b94b95be0c155d80ff0bff" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 33627 } } peers { permanent_uuid: "3710b74ed2044ce5995533df32e67351" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 38589 } attrs { replace: true } } peers { permanent_uuid: "25286c485fa04c86a5d07c8f88643a47" member_type: NON_VOTER last_known_addr { host: "127.17.60.198" port: 37089 } attrs { promote: true } } }
I20250112 15:07:26.072554 21068 raft_consensus.cc:2949] T 1eb2b7e6edd34a7b83f9b918d3cdf2ed P 2537bbbd1a9947ae9facd87567e3ccb9 [term 1 FOLLOWER]: Committing config change with OpId 1.26: config changed from index -1 to 26, NON_VOTER 25286c485fa04c86a5d07c8f88643a47 (127.17.60.198) added. New config: { opid_index: 26 OBSOLETE_local: false peers { permanent_uuid: "2537bbbd1a9947ae9facd87567e3ccb9" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 33173 } } peers { permanent_uuid: "c561ec3374b94b95be0c155d80ff0bff" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 33627 } } peers { permanent_uuid: "3710b74ed2044ce5995533df32e67351" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 38589 } attrs { replace: true } } peers { permanent_uuid: "25286c485fa04c86a5d07c8f88643a47" member_type: NON_VOTER last_known_addr { host: "127.17.60.198" port: 37089 } attrs { promote: true } } }
I20250112 15:07:26.072796 20966 catalog_manager.cc:5526] T 1eb2b7e6edd34a7b83f9b918d3cdf2ed P 3710b74ed2044ce5995533df32e67351 reported cstate change: config changed from index -1 to 26, NON_VOTER 25286c485fa04c86a5d07c8f88643a47 (127.17.60.198) added. New cstate: current_term: 1 leader_uuid: "c561ec3374b94b95be0c155d80ff0bff" committed_config { opid_index: 26 OBSOLETE_local: false peers { permanent_uuid: "2537bbbd1a9947ae9facd87567e3ccb9" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 33173 } } peers { permanent_uuid: "c561ec3374b94b95be0c155d80ff0bff" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 33627 } } peers { permanent_uuid: "3710b74ed2044ce5995533df32e67351" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 38589 } attrs { replace: true } } peers { permanent_uuid: "25286c485fa04c86a5d07c8f88643a47" member_type: NON_VOTER last_known_addr { host: "127.17.60.198" port: 37089 } attrs { promote: true } } }
W20250112 15:07:26.073864 21021 auto_rebalancer.cc:253] failed to send replica move request: Remote error: Service unavailable: leader transfer in progress
W20250112 15:07:26.074687 21021 auto_rebalancer.cc:663] Could not move replica: Incomplete: tablet 57e47b07fdcf434e96ca5dd0354e93b2, TS 2537bbbd1a9947ae9facd87567e3ccb9 -> TS a2a5bba4ac1145d5bab94a69c9f4df40 move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2537bbbd1a9947ae9facd87567e3ccb9" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 33173 } } peers { permanent_uuid: "c561ec3374b94b95be0c155d80ff0bff" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 33627 } } peers { permanent_uuid: "3710b74ed2044ce5995533df32e67351" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 38589 } }
W20250112 15:07:26.074694 21114 consensus_peers.cc:487] T 1eb2b7e6edd34a7b83f9b918d3cdf2ed P c561ec3374b94b95be0c155d80ff0bff -> Peer 25286c485fa04c86a5d07c8f88643a47 (127.17.60.198:37089): Couldn't send request to peer 25286c485fa04c86a5d07c8f88643a47. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 1eb2b7e6edd34a7b83f9b918d3cdf2ed. This is attempt 1: this message will repeat every 5th retry.
W20250112 15:07:26.074736 21021 auto_rebalancer.cc:263] scheduled replica move failed to complete: Incomplete: tablet 57e47b07fdcf434e96ca5dd0354e93b2, TS 2537bbbd1a9947ae9facd87567e3ccb9 -> TS a2a5bba4ac1145d5bab94a69c9f4df40 move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2537bbbd1a9947ae9facd87567e3ccb9" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 33173 } } peers { permanent_uuid: "c561ec3374b94b95be0c155d80ff0bff" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 33627 } } peers { permanent_uuid: "3710b74ed2044ce5995533df32e67351" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 38589 } }
W20250112 15:07:26.075227 21021 auto_rebalancer.cc:663] Could not move replica: Incomplete: tablet 5bcec60d3b974a9bb24aedfc738e1594, TS c561ec3374b94b95be0c155d80ff0bff -> TS b5e9178a7a83402e8c4a6943ab78589d move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2537bbbd1a9947ae9facd87567e3ccb9" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 33173 } } peers { permanent_uuid: "c561ec3374b94b95be0c155d80ff0bff" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 33627 } } peers { permanent_uuid: "3710b74ed2044ce5995533df32e67351" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 38589 } }
W20250112 15:07:26.075261 21021 auto_rebalancer.cc:263] scheduled replica move failed to complete: Incomplete: tablet 5bcec60d3b974a9bb24aedfc738e1594, TS c561ec3374b94b95be0c155d80ff0bff -> TS b5e9178a7a83402e8c4a6943ab78589d move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2537bbbd1a9947ae9facd87567e3ccb9" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 33173 } } peers { permanent_uuid: "c561ec3374b94b95be0c155d80ff0bff" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 33627 } } peers { permanent_uuid: "3710b74ed2044ce5995533df32e67351" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 38589 } }
W20250112 15:07:26.075652 21021 auto_rebalancer.cc:663] Could not move replica: Incomplete: tablet bfaeae584ec04772b2d6fcd088022117, TS 2537bbbd1a9947ae9facd87567e3ccb9 -> TS a2a5bba4ac1145d5bab94a69c9f4df40 move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2537bbbd1a9947ae9facd87567e3ccb9" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 33173 } } peers { permanent_uuid: "c561ec3374b94b95be0c155d80ff0bff" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 33627 } } peers { permanent_uuid: "3710b74ed2044ce5995533df32e67351" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 38589 } }
W20250112 15:07:26.075683 21021 auto_rebalancer.cc:263] scheduled replica move failed to complete: Incomplete: tablet bfaeae584ec04772b2d6fcd088022117, TS 2537bbbd1a9947ae9facd87567e3ccb9 -> TS a2a5bba4ac1145d5bab94a69c9f4df40 move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2537bbbd1a9947ae9facd87567e3ccb9" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 33173 } } peers { permanent_uuid: "c561ec3374b94b95be0c155d80ff0bff" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 33627 } } peers { permanent_uuid: "3710b74ed2044ce5995533df32e67351" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 38589 } }
W20250112 15:07:26.076171 21021 auto_rebalancer.cc:663] Could not move replica: Incomplete: tablet ef90b704208140a3983ffe6105939319, TS 3710b74ed2044ce5995533df32e67351 -> TS 25286c485fa04c86a5d07c8f88643a47 move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2537bbbd1a9947ae9facd87567e3ccb9" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 33173 } } peers { permanent_uuid: "c561ec3374b94b95be0c155d80ff0bff" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 33627 } } peers { permanent_uuid: "3710b74ed2044ce5995533df32e67351" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 38589 } }
W20250112 15:07:26.076206 21021 auto_rebalancer.cc:263] scheduled replica move failed to complete: Incomplete: tablet ef90b704208140a3983ffe6105939319, TS 3710b74ed2044ce5995533df32e67351 -> TS 25286c485fa04c86a5d07c8f88643a47 move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2537bbbd1a9947ae9facd87567e3ccb9" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 33173 } } peers { permanent_uuid: "c561ec3374b94b95be0c155d80ff0bff" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 33627 } } peers { permanent_uuid: "3710b74ed2044ce5995533df32e67351" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 38589 } }
W20250112 15:07:26.076799 21021 auto_rebalancer.cc:663] Could not move replica: Incomplete: tablet 9601bcda24ed42e7b93ab9740f1e271e, TS c561ec3374b94b95be0c155d80ff0bff -> TS b5e9178a7a83402e8c4a6943ab78589d move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2537bbbd1a9947ae9facd87567e3ccb9" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 33173 } } peers { permanent_uuid: "c561ec3374b94b95be0c155d80ff0bff" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 33627 } } peers { permanent_uuid: "3710b74ed2044ce5995533df32e67351" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 38589 } }
W20250112 15:07:26.076836 21021 auto_rebalancer.cc:263] scheduled replica move failed to complete: Incomplete: tablet 9601bcda24ed42e7b93ab9740f1e271e, TS c561ec3374b94b95be0c155d80ff0bff -> TS b5e9178a7a83402e8c4a6943ab78589d move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2537bbbd1a9947ae9facd87567e3ccb9" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 33173 } } peers { permanent_uuid: "c561ec3374b94b95be0c155d80ff0bff" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 33627 } } peers { permanent_uuid: "3710b74ed2044ce5995533df32e67351" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 38589 } }
I20250112 15:07:26.204340 21426 raft_consensus.cc:988] T 02c33d457b6b47138dd712850819b619 P c561ec3374b94b95be0c155d80ff0bff: : Instructing follower 3710b74ed2044ce5995533df32e67351 to start an election
I20250112 15:07:26.204437 21426 raft_consensus.cc:1076] T 02c33d457b6b47138dd712850819b619 P c561ec3374b94b95be0c155d80ff0bff [term 1 LEADER]: Signalling peer 3710b74ed2044ce5995533df32e67351 to start an election
I20250112 15:07:26.204613 21226 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "02c33d457b6b47138dd712850819b619"
dest_uuid: "3710b74ed2044ce5995533df32e67351"
from {username='slave'} at 127.0.0.1:33202
I20250112 15:07:26.204700 21226 raft_consensus.cc:491] T 02c33d457b6b47138dd712850819b619 P 3710b74ed2044ce5995533df32e67351 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20250112 15:07:26.204733 21226 raft_consensus.cc:3054] T 02c33d457b6b47138dd712850819b619 P 3710b74ed2044ce5995533df32e67351 [term 1 FOLLOWER]: Advancing to term 2
I20250112 15:07:26.205562 21226 raft_consensus.cc:513] T 02c33d457b6b47138dd712850819b619 P 3710b74ed2044ce5995533df32e67351 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3710b74ed2044ce5995533df32e67351" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 38589 } } peers { permanent_uuid: "2537bbbd1a9947ae9facd87567e3ccb9" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 33173 } } peers { permanent_uuid: "c561ec3374b94b95be0c155d80ff0bff" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 33627 } }
I20250112 15:07:26.205752 21226 leader_election.cc:290] T 02c33d457b6b47138dd712850819b619 P 3710b74ed2044ce5995533df32e67351 [CANDIDATE]: Term 2 election: Requested vote from peers 2537bbbd1a9947ae9facd87567e3ccb9 (127.17.60.193:33173), c561ec3374b94b95be0c155d80ff0bff (127.17.60.194:33627)
I20250112 15:07:26.205916 21149 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "02c33d457b6b47138dd712850819b619" candidate_uuid: "3710b74ed2044ce5995533df32e67351" candidate_term: 2 candidate_status { last_received { term: 1 index: 25 } } ignore_live_leader: true dest_uuid: "c561ec3374b94b95be0c155d80ff0bff"
I20250112 15:07:26.205926 21068 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "02c33d457b6b47138dd712850819b619" candidate_uuid: "3710b74ed2044ce5995533df32e67351" candidate_term: 2 candidate_status { last_received { term: 1 index: 25 } } ignore_live_leader: true dest_uuid: "2537bbbd1a9947ae9facd87567e3ccb9"
I20250112 15:07:26.205988 21149 raft_consensus.cc:3049] T 02c33d457b6b47138dd712850819b619 P c561ec3374b94b95be0c155d80ff0bff [term 1 LEADER]: Stepping down as leader of term 1
I20250112 15:07:26.206029 21149 raft_consensus.cc:738] T 02c33d457b6b47138dd712850819b619 P c561ec3374b94b95be0c155d80ff0bff [term 1 LEADER]: Becoming Follower/Learner. State: Replica: c561ec3374b94b95be0c155d80ff0bff, State: Running, Role: LEADER
I20250112 15:07:26.206085 21149 consensus_queue.cc:260] T 02c33d457b6b47138dd712850819b619 P c561ec3374b94b95be0c155d80ff0bff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 25, Committed index: 25, Last appended: 1.25, Last appended by leader: 25, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3710b74ed2044ce5995533df32e67351" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 38589 } } peers { permanent_uuid: "2537bbbd1a9947ae9facd87567e3ccb9" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 33173 } } peers { permanent_uuid: "c561ec3374b94b95be0c155d80ff0bff" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 33627 } }
I20250112 15:07:26.206178 21149 raft_consensus.cc:3054] T 02c33d457b6b47138dd712850819b619 P c561ec3374b94b95be0c155d80ff0bff [term 1 FOLLOWER]: Advancing to term 2
I20250112 15:07:26.206913 21149 raft_consensus.cc:2463] T 02c33d457b6b47138dd712850819b619 P c561ec3374b94b95be0c155d80ff0bff [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3710b74ed2044ce5995533df32e67351 in term 2.
I20250112 15:07:26.205993 21068 raft_consensus.cc:3054] T 02c33d457b6b47138dd712850819b619 P 2537bbbd1a9947ae9facd87567e3ccb9 [term 1 FOLLOWER]: Advancing to term 2
I20250112 15:07:26.207598 21068 raft_consensus.cc:2463] T 02c33d457b6b47138dd712850819b619 P 2537bbbd1a9947ae9facd87567e3ccb9 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3710b74ed2044ce5995533df32e67351 in term 2.
I20250112 15:07:26.207698 21191 leader_election.cc:304] T 02c33d457b6b47138dd712850819b619 P 3710b74ed2044ce5995533df32e67351 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3710b74ed2044ce5995533df32e67351, c561ec3374b94b95be0c155d80ff0bff; no voters:
I20250112 15:07:26.207787 21382 raft_consensus.cc:2798] T 02c33d457b6b47138dd712850819b619 P 3710b74ed2044ce5995533df32e67351 [term 2 FOLLOWER]: Leader election won for term 2
I20250112 15:07:26.207943 21382 raft_consensus.cc:695] T 02c33d457b6b47138dd712850819b619 P 3710b74ed2044ce5995533df32e67351 [term 2 LEADER]: Becoming Leader. State: Replica: 3710b74ed2044ce5995533df32e67351, State: Running, Role: LEADER
I20250112 15:07:26.208007 21382 consensus_queue.cc:237] T 02c33d457b6b47138dd712850819b619 P 3710b74ed2044ce5995533df32e67351 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 25, Committed index: 25, Last appended: 1.25, Last appended by leader: 25, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3710b74ed2044ce5995533df32e67351" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 38589 } } peers { permanent_uuid: "2537bbbd1a9947ae9facd87567e3ccb9" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 33173 } } peers { permanent_uuid: "c561ec3374b94b95be0c155d80ff0bff" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 33627 } }
I20250112 15:07:26.208420 20964 catalog_manager.cc:5526] T 02c33d457b6b47138dd712850819b619 P 3710b74ed2044ce5995533df32e67351 reported cstate change: term changed from 1 to 2, leader changed from c561ec3374b94b95be0c155d80ff0bff (127.17.60.194) to 3710b74ed2044ce5995533df32e67351 (127.17.60.195). New cstate: current_term: 2 leader_uuid: "3710b74ed2044ce5995533df32e67351" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3710b74ed2044ce5995533df32e67351" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 38589 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2537bbbd1a9947ae9facd87567e3ccb9" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 33173 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c561ec3374b94b95be0c155d80ff0bff" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 33627 } health_report { overall_health: UNKNOWN } } }
I20250112 15:07:26.598258 21662 ts_tablet_manager.cc:927] T 1eb2b7e6edd34a7b83f9b918d3cdf2ed P 25286c485fa04c86a5d07c8f88643a47: Initiating tablet copy from peer c561ec3374b94b95be0c155d80ff0bff (127.17.60.194:33627)
I20250112 15:07:26.598426 21662 tablet_copy_client.cc:323] T 1eb2b7e6edd34a7b83f9b918d3cdf2ed P 25286c485fa04c86a5d07c8f88643a47: tablet copy: Beginning tablet copy session from remote peer at address 127.17.60.194:33627
I20250112 15:07:26.601519 21161 tablet_copy_service.cc:140] P c561ec3374b94b95be0c155d80ff0bff: Received BeginTabletCopySession request for tablet 1eb2b7e6edd34a7b83f9b918d3cdf2ed from peer 25286c485fa04c86a5d07c8f88643a47 ({username='slave'} at 127.0.0.1:48116)
I20250112 15:07:26.601594 21161 tablet_copy_service.cc:161] P c561ec3374b94b95be0c155d80ff0bff: Beginning new tablet copy session on tablet 1eb2b7e6edd34a7b83f9b918d3cdf2ed from peer 25286c485fa04c86a5d07c8f88643a47 at {username='slave'} at 127.0.0.1:48116: session id = 25286c485fa04c86a5d07c8f88643a47-1eb2b7e6edd34a7b83f9b918d3cdf2ed
I20250112 15:07:26.602249 21161 tablet_copy_source_session.cc:215] T 1eb2b7e6edd34a7b83f9b918d3cdf2ed P c561ec3374b94b95be0c155d80ff0bff: Tablet Copy: opened 0 blocks and 1 log segments
I20250112 15:07:26.602577 21662 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1eb2b7e6edd34a7b83f9b918d3cdf2ed. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 15:07:26.604012 21662 tablet_copy_client.cc:806] T 1eb2b7e6edd34a7b83f9b918d3cdf2ed P 25286c485fa04c86a5d07c8f88643a47: tablet copy: Starting download of 0 data blocks...
I20250112 15:07:26.604151 21662 tablet_copy_client.cc:670] T 1eb2b7e6edd34a7b83f9b918d3cdf2ed P 25286c485fa04c86a5d07c8f88643a47: tablet copy: Starting download of 1 WAL segments...
I20250112 15:07:26.604647 21663 tablet_copy_client.cc:1025] T 1eb2b7e6edd34a7b83f9b918d3cdf2ed P 25286c485fa04c86a5d07c8f88643a47: tablet copy: Injecting latency into file download: 2000
I20250112 15:07:26.669953 21424 raft_consensus.cc:988] T 57e47b07fdcf434e96ca5dd0354e93b2 P c561ec3374b94b95be0c155d80ff0bff: : Instructing follower 2537bbbd1a9947ae9facd87567e3ccb9 to start an election
I20250112 15:07:26.670053 21424 raft_consensus.cc:1076] T 57e47b07fdcf434e96ca5dd0354e93b2 P c561ec3374b94b95be0c155d80ff0bff [term 1 LEADER]: Signalling peer 2537bbbd1a9947ae9facd87567e3ccb9 to start an election
I20250112 15:07:26.670215 21068 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "57e47b07fdcf434e96ca5dd0354e93b2"
dest_uuid: "2537bbbd1a9947ae9facd87567e3ccb9"
from {username='slave'} at 127.0.0.1:37216
I20250112 15:07:26.670308 21068 raft_consensus.cc:491] T 57e47b07fdcf434e96ca5dd0354e93b2 P 2537bbbd1a9947ae9facd87567e3ccb9 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20250112 15:07:26.670346 21068 raft_consensus.cc:3054] T 57e47b07fdcf434e96ca5dd0354e93b2 P 2537bbbd1a9947ae9facd87567e3ccb9 [term 1 FOLLOWER]: Advancing to term 2
I20250112 15:07:26.671254 21068 raft_consensus.cc:513] T 57e47b07fdcf434e96ca5dd0354e93b2 P 2537bbbd1a9947ae9facd87567e3ccb9 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2537bbbd1a9947ae9facd87567e3ccb9" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 33173 } } peers { permanent_uuid: "c561ec3374b94b95be0c155d80ff0bff" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 33627 } } peers { permanent_uuid: "3710b74ed2044ce5995533df32e67351" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 38589 } }
I20250112 15:07:26.671401 21068 leader_election.cc:290] T 57e47b07fdcf434e96ca5dd0354e93b2 P 2537bbbd1a9947ae9facd87567e3ccb9 [CANDIDATE]: Term 2 election: Requested vote from peers c561ec3374b94b95be0c155d80ff0bff (127.17.60.194:33627), 3710b74ed2044ce5995533df32e67351 (127.17.60.195:38589)
I20250112 15:07:26.671483 21149 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "57e47b07fdcf434e96ca5dd0354e93b2" candidate_uuid: "2537bbbd1a9947ae9facd87567e3ccb9" candidate_term: 2 candidate_status { last_received { term: 1 index: 25 } } ignore_live_leader: true dest_uuid: "c561ec3374b94b95be0c155d80ff0bff"
I20250112 15:07:26.671602 21226 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "57e47b07fdcf434e96ca5dd0354e93b2" candidate_uuid: "2537bbbd1a9947ae9facd87567e3ccb9" candidate_term: 2 candidate_status { last_received { term: 1 index: 25 } } ignore_live_leader: true dest_uuid: "3710b74ed2044ce5995533df32e67351"
I20250112 15:07:26.671679 21226 raft_consensus.cc:3054] T 57e47b07fdcf434e96ca5dd0354e93b2 P 3710b74ed2044ce5995533df32e67351 [term 1 FOLLOWER]: Advancing to term 2
I20250112 15:07:26.672439 21226 raft_consensus.cc:2463] T 57e47b07fdcf434e96ca5dd0354e93b2 P 3710b74ed2044ce5995533df32e67351 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2537bbbd1a9947ae9facd87567e3ccb9 in term 2.
I20250112 15:07:26.672617 21039 leader_election.cc:304] T 57e47b07fdcf434e96ca5dd0354e93b2 P 2537bbbd1a9947ae9facd87567e3ccb9 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2537bbbd1a9947ae9facd87567e3ccb9, 3710b74ed2044ce5995533df32e67351; no voters:
I20250112 15:07:26.672750 21408 raft_consensus.cc:2798] T 57e47b07fdcf434e96ca5dd0354e93b2 P 2537bbbd1a9947ae9facd87567e3ccb9 [term 2 FOLLOWER]: Leader election won for term 2
I20250112 15:07:26.672960 21408 raft_consensus.cc:695] T 57e47b07fdcf434e96ca5dd0354e93b2 P 2537bbbd1a9947ae9facd87567e3ccb9 [term 2 LEADER]: Becoming Leader. State: Replica: 2537bbbd1a9947ae9facd87567e3ccb9, State: Running, Role: LEADER
I20250112 15:07:26.673053 21408 consensus_queue.cc:237] T 57e47b07fdcf434e96ca5dd0354e93b2 P 2537bbbd1a9947ae9facd87567e3ccb9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 25, Committed index: 25, Last appended: 1.25, Last appended by leader: 25, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2537bbbd1a9947ae9facd87567e3ccb9" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 33173 } } peers { permanent_uuid: "c561ec3374b94b95be0c155d80ff0bff" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 33627 } } peers { permanent_uuid: "3710b74ed2044ce5995533df32e67351" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 38589 } }
I20250112 15:07:26.671607 21149 raft_consensus.cc:3049] T 57e47b07fdcf434e96ca5dd0354e93b2 P c561ec3374b94b95be0c155d80ff0bff [term 1 LEADER]: Stepping down as leader of term 1
I20250112 15:07:26.673538 21149 raft_consensus.cc:738] T 57e47b07fdcf434e96ca5dd0354e93b2 P c561ec3374b94b95be0c155d80ff0bff [term 1 LEADER]: Becoming Follower/Learner. State: Replica: c561ec3374b94b95be0c155d80ff0bff, State: Running, Role: LEADER
I20250112 15:07:26.673592 21149 consensus_queue.cc:260] T 57e47b07fdcf434e96ca5dd0354e93b2 P c561ec3374b94b95be0c155d80ff0bff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 25, Committed index: 25, Last appended: 1.25, Last appended by leader: 25, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2537bbbd1a9947ae9facd87567e3ccb9" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 33173 } } peers { permanent_uuid: "c561ec3374b94b95be0c155d80ff0bff" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 33627 } } peers { permanent_uuid: "3710b74ed2044ce5995533df32e67351" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 38589 } }
I20250112 15:07:26.673559 20964 catalog_manager.cc:5526] T 57e47b07fdcf434e96ca5dd0354e93b2 P 2537bbbd1a9947ae9facd87567e3ccb9 reported cstate change: term changed from 1 to 2, leader changed from c561ec3374b94b95be0c155d80ff0bff (127.17.60.194) to 2537bbbd1a9947ae9facd87567e3ccb9 (127.17.60.193). New cstate: current_term: 2 leader_uuid: "2537bbbd1a9947ae9facd87567e3ccb9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2537bbbd1a9947ae9facd87567e3ccb9" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 33173 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c561ec3374b94b95be0c155d80ff0bff" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 33627 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3710b74ed2044ce5995533df32e67351" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 38589 } health_report { overall_health: UNKNOWN } } }
I20250112 15:07:26.673650 21149 raft_consensus.cc:3054] T 57e47b07fdcf434e96ca5dd0354e93b2 P c561ec3374b94b95be0c155d80ff0bff [term 1 FOLLOWER]: Advancing to term 2
I20250112 15:07:26.674671 21149 raft_consensus.cc:2463] T 57e47b07fdcf434e96ca5dd0354e93b2 P c561ec3374b94b95be0c155d80ff0bff [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2537bbbd1a9947ae9facd87567e3ccb9 in term 2.
I20250112 15:07:26.703305 21500 heartbeater.cc:502] Master 127.17.60.254:42121 was elected leader, sending a full tablet report...
I20250112 15:07:26.712013 21068 raft_consensus.cc:1270] T 02c33d457b6b47138dd712850819b619 P 2537bbbd1a9947ae9facd87567e3ccb9 [term 2 FOLLOWER]: Refusing update from remote peer 3710b74ed2044ce5995533df32e67351: Log matching property violated. Preceding OpId in replica: term: 1 index: 25. Preceding OpId from leader: term: 2 index: 26. (index mismatch)
I20250112 15:07:26.712229 21382 consensus_queue.cc:1035] T 02c33d457b6b47138dd712850819b619 P 3710b74ed2044ce5995533df32e67351 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2537bbbd1a9947ae9facd87567e3ccb9" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 33173 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 26, Last known committed idx: 25, Time since last communication: 0.000s
I20250112 15:07:26.713270 21151 raft_consensus.cc:1270] T 02c33d457b6b47138dd712850819b619 P c561ec3374b94b95be0c155d80ff0bff [term 2 FOLLOWER]: Refusing update from remote peer 3710b74ed2044ce5995533df32e67351: Log matching property violated. Preceding OpId in replica: term: 1 index: 25. Preceding OpId from leader: term: 2 index: 26. (index mismatch)
I20250112 15:07:26.713590 21382 consensus_queue.cc:1035] T 02c33d457b6b47138dd712850819b619 P 3710b74ed2044ce5995533df32e67351 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c561ec3374b94b95be0c155d80ff0bff" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 33627 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 26, Last known committed idx: 25, Time since last communication: 0.000s
I20250112 15:07:26.726455 21575 heartbeater.cc:502] Master 127.17.60.254:42121 was elected leader, sending a full tablet report...
I20250112 15:07:26.764569 21650 heartbeater.cc:502] Master 127.17.60.254:42121 was elected leader, sending a full tablet report...
I20250112 15:07:27.101542 21226 raft_consensus.cc:1270] T 57e47b07fdcf434e96ca5dd0354e93b2 P 3710b74ed2044ce5995533df32e67351 [term 2 FOLLOWER]: Refusing update from remote peer 2537bbbd1a9947ae9facd87567e3ccb9: Log matching property violated. Preceding OpId in replica: term: 1 index: 25. Preceding OpId from leader: term: 2 index: 26. (index mismatch)
I20250112 15:07:27.101789 21666 consensus_queue.cc:1035] T 57e47b07fdcf434e96ca5dd0354e93b2 P 2537bbbd1a9947ae9facd87567e3ccb9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3710b74ed2044ce5995533df32e67351" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 38589 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 26, Last known committed idx: 25, Time since last communication: 0.000s
I20250112 15:07:27.103159 21149 raft_consensus.cc:1270] T 57e47b07fdcf434e96ca5dd0354e93b2 P c561ec3374b94b95be0c155d80ff0bff [term 2 FOLLOWER]: Refusing update from remote peer 2537bbbd1a9947ae9facd87567e3ccb9: Log matching property violated. Preceding OpId in replica: term: 1 index: 25. Preceding OpId from leader: term: 2 index: 26. (index mismatch)
I20250112 15:07:27.103408 21408 consensus_queue.cc:1035] T 57e47b07fdcf434e96ca5dd0354e93b2 P 2537bbbd1a9947ae9facd87567e3ccb9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c561ec3374b94b95be0c155d80ff0bff" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 33627 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 26, Last known committed idx: 25, Time since last communication: 0.000s
I20250112 15:07:28.070967 21026 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250112 15:07:28.071213 21026 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20250112 15:07:28.071277 21026 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20250112 15:07:28.604907 21662 tablet_copy_client.cc:538] T 1eb2b7e6edd34a7b83f9b918d3cdf2ed P 25286c485fa04c86a5d07c8f88643a47: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250112 15:07:28.606024 21662 tablet_bootstrap.cc:492] T 1eb2b7e6edd34a7b83f9b918d3cdf2ed P 25286c485fa04c86a5d07c8f88643a47: Bootstrap starting.
I20250112 15:07:28.609196 21662 tablet_bootstrap.cc:492] T 1eb2b7e6edd34a7b83f9b918d3cdf2ed P 25286c485fa04c86a5d07c8f88643a47: Bootstrap replayed 1/1 log segments. Stats: ops{read=26 overwritten=0 applied=26 ignored=0} inserts{seen=114 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250112 15:07:28.609385 21662 tablet_bootstrap.cc:492] T 1eb2b7e6edd34a7b83f9b918d3cdf2ed P 25286c485fa04c86a5d07c8f88643a47: Bootstrap complete.
I20250112 15:07:28.609508 21662 ts_tablet_manager.cc:1397] T 1eb2b7e6edd34a7b83f9b918d3cdf2ed P 25286c485fa04c86a5d07c8f88643a47: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.001s
I20250112 15:07:28.609663 21662 raft_consensus.cc:357] T 1eb2b7e6edd34a7b83f9b918d3cdf2ed P 25286c485fa04c86a5d07c8f88643a47 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 26 OBSOLETE_local: false peers { permanent_uuid: "2537bbbd1a9947ae9facd87567e3ccb9" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 33173 } } peers { permanent_uuid: "c561ec3374b94b95be0c155d80ff0bff" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 33627 } } peers { permanent_uuid: "3710b74ed2044ce5995533df32e67351" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 38589 } attrs { replace: true } } peers { permanent_uuid: "25286c485fa04c86a5d07c8f88643a47" member_type: NON_VOTER last_known_addr { host: "127.17.60.198" port: 37089 } attrs { promote: true } }
I20250112 15:07:28.609760 21662 raft_consensus.cc:738] T 1eb2b7e6edd34a7b83f9b918d3cdf2ed P 25286c485fa04c86a5d07c8f88643a47 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 25286c485fa04c86a5d07c8f88643a47, State: Initialized, Role: LEARNER
I20250112 15:07:28.609802 21662 consensus_queue.cc:260] T 1eb2b7e6edd34a7b83f9b918d3cdf2ed P 25286c485fa04c86a5d07c8f88643a47 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 26, Last appended: 1.26, Last appended by leader: 26, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 26 OBSOLETE_local: false peers { permanent_uuid: "2537bbbd1a9947ae9facd87567e3ccb9" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 33173 } } peers { permanent_uuid: "c561ec3374b94b95be0c155d80ff0bff" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 33627 } } peers { permanent_uuid: "3710b74ed2044ce5995533df32e67351" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 38589 } attrs { replace: true } } peers { permanent_uuid: "25286c485fa04c86a5d07c8f88643a47" member_type: NON_VOTER last_known_addr { host: "127.17.60.198" port: 37089 } attrs { promote: true } }
I20250112 15:07:28.609927 21662 ts_tablet_manager.cc:1428] T 1eb2b7e6edd34a7b83f9b918d3cdf2ed P 25286c485fa04c86a5d07c8f88643a47: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20250112 15:07:28.610113 21161 tablet_copy_service.cc:342] P c561ec3374b94b95be0c155d80ff0bff: Request end of tablet copy session 25286c485fa04c86a5d07c8f88643a47-1eb2b7e6edd34a7b83f9b918d3cdf2ed received from {username='slave'} at 127.0.0.1:48116
I20250112 15:07:28.610159 21161 tablet_copy_service.cc:434] P c561ec3374b94b95be0c155d80ff0bff: ending tablet copy session 25286c485fa04c86a5d07c8f88643a47-1eb2b7e6edd34a7b83f9b918d3cdf2ed on tablet 1eb2b7e6edd34a7b83f9b918d3cdf2ed with peer 25286c485fa04c86a5d07c8f88643a47
I20250112 15:07:28.651865 21624 raft_consensus.cc:1212] T 1eb2b7e6edd34a7b83f9b918d3cdf2ed P 25286c485fa04c86a5d07c8f88643a47 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.25->[1.26-1.26] Dedup: 1.26->[]
I20250112 15:07:29.214854 21680 raft_consensus.cc:1059] T 1eb2b7e6edd34a7b83f9b918d3cdf2ed P c561ec3374b94b95be0c155d80ff0bff: attempting to promote NON_VOTER 25286c485fa04c86a5d07c8f88643a47 to VOTER
I20250112 15:07:29.215133 21680 consensus_queue.cc:237] T 1eb2b7e6edd34a7b83f9b918d3cdf2ed P c561ec3374b94b95be0c155d80ff0bff [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 26, Committed index: 26, Last appended: 1.26, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "2537bbbd1a9947ae9facd87567e3ccb9" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 33173 } } peers { permanent_uuid: "c561ec3374b94b95be0c155d80ff0bff" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 33627 } } peers { permanent_uuid: "3710b74ed2044ce5995533df32e67351" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 38589 } attrs { replace: true } } peers { permanent_uuid: "25286c485fa04c86a5d07c8f88643a47" member_type: VOTER last_known_addr { host: "127.17.60.198" port: 37089 } attrs { promote: false } }
I20250112 15:07:29.215612 21624 raft_consensus.cc:1270] T 1eb2b7e6edd34a7b83f9b918d3cdf2ed P 25286c485fa04c86a5d07c8f88643a47 [term 1 LEARNER]: Refusing update from remote peer c561ec3374b94b95be0c155d80ff0bff: Log matching property violated. Preceding OpId in replica: term: 1 index: 26. Preceding OpId from leader: term: 1 index: 27. (index mismatch)
I20250112 15:07:29.215657 21068 raft_consensus.cc:1270] T 1eb2b7e6edd34a7b83f9b918d3cdf2ed P 2537bbbd1a9947ae9facd87567e3ccb9 [term 1 FOLLOWER]: Refusing update from remote peer c561ec3374b94b95be0c155d80ff0bff: Log matching property violated. Preceding OpId in replica: term: 1 index: 26. Preceding OpId from leader: term: 1 index: 27. (index mismatch)
I20250112 15:07:29.215724 21226 raft_consensus.cc:1270] T 1eb2b7e6edd34a7b83f9b918d3cdf2ed P 3710b74ed2044ce5995533df32e67351 [term 1 FOLLOWER]: Refusing update from remote peer c561ec3374b94b95be0c155d80ff0bff: Log matching property violated. Preceding OpId in replica: term: 1 index: 26. Preceding OpId from leader: term: 1 index: 27. (index mismatch)
I20250112 15:07:29.215840 21683 consensus_queue.cc:1035] T 1eb2b7e6edd34a7b83f9b918d3cdf2ed P c561ec3374b94b95be0c155d80ff0bff [LEADER]: Connected to new peer: Peer: permanent_uuid: "2537bbbd1a9947ae9facd87567e3ccb9" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 33173 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 27, Last known committed idx: 26, Time since last communication: 0.000s
I20250112 15:07:29.215932 21424 consensus_queue.cc:1035] T 1eb2b7e6edd34a7b83f9b918d3cdf2ed P c561ec3374b94b95be0c155d80ff0bff [LEADER]: Connected to new peer: Peer: permanent_uuid: "25286c485fa04c86a5d07c8f88643a47" member_type: VOTER last_known_addr { host: "127.17.60.198" port: 37089 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 27, Last known committed idx: 26, Time since last communication: 0.000s
I20250112 15:07:29.216010 21680 consensus_queue.cc:1035] T 1eb2b7e6edd34a7b83f9b918d3cdf2ed P c561ec3374b94b95be0c155d80ff0bff [LEADER]: Connected to new peer: Peer: permanent_uuid: "3710b74ed2044ce5995533df32e67351" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 38589 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 27, Last known committed idx: 26, Time since last communication: 0.000s
I20250112 15:07:29.218202 21683 raft_consensus.cc:2949] T 1eb2b7e6edd34a7b83f9b918d3cdf2ed P c561ec3374b94b95be0c155d80ff0bff [term 1 LEADER]: Committing config change with OpId 1.27: config changed from index 26 to 27, 25286c485fa04c86a5d07c8f88643a47 (127.17.60.198) changed from NON_VOTER to VOTER. New config: { opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "2537bbbd1a9947ae9facd87567e3ccb9" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 33173 } } peers { permanent_uuid: "c561ec3374b94b95be0c155d80ff0bff" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 33627 } } peers { permanent_uuid: "3710b74ed2044ce5995533df32e67351" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 38589 } attrs { replace: true } } peers { permanent_uuid: "25286c485fa04c86a5d07c8f88643a47" member_type: VOTER last_known_addr { host: "127.17.60.198" port: 37089 } attrs { promote: false } } }
I20250112 15:07:29.218343 21068 raft_consensus.cc:2949] T 1eb2b7e6edd34a7b83f9b918d3cdf2ed P 2537bbbd1a9947ae9facd87567e3ccb9 [term 1 FOLLOWER]: Committing config change with OpId 1.27: config changed from index 26 to 27, 25286c485fa04c86a5d07c8f88643a47 (127.17.60.198) changed from NON_VOTER to VOTER. New config: { opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "2537bbbd1a9947ae9facd87567e3ccb9" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 33173 } } peers { permanent_uuid: "c561ec3374b94b95be0c155d80ff0bff" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 33627 } } peers { permanent_uuid: "3710b74ed2044ce5995533df32e67351" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 38589 } attrs { replace: true } } peers { permanent_uuid: "25286c485fa04c86a5d07c8f88643a47" member_type: VOTER last_known_addr { host: "127.17.60.198" port: 37089 } attrs { promote: false } } }
I20250112 15:07:29.219374 21226 raft_consensus.cc:2949] T 1eb2b7e6edd34a7b83f9b918d3cdf2ed P 3710b74ed2044ce5995533df32e67351 [term 1 FOLLOWER]: Committing config change with OpId 1.27: config changed from index 26 to 27, 25286c485fa04c86a5d07c8f88643a47 (127.17.60.198) changed from NON_VOTER to VOTER. New config: { opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "2537bbbd1a9947ae9facd87567e3ccb9" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 33173 } } peers { permanent_uuid: "c561ec3374b94b95be0c155d80ff0bff" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 33627 } } peers { permanent_uuid: "3710b74ed2044ce5995533df32e67351" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 38589 } attrs { replace: true } } peers { permanent_uuid: "25286c485fa04c86a5d07c8f88643a47" member_type: VOTER last_known_addr { host: "127.17.60.198" port: 37089 } attrs { promote: false } } }
I20250112 15:07:29.219540 20965 catalog_manager.cc:5526] T 1eb2b7e6edd34a7b83f9b918d3cdf2ed P 2537bbbd1a9947ae9facd87567e3ccb9 reported cstate change: config changed from index 26 to 27, 25286c485fa04c86a5d07c8f88643a47 (127.17.60.198) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "c561ec3374b94b95be0c155d80ff0bff" committed_config { opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "2537bbbd1a9947ae9facd87567e3ccb9" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 33173 } } peers { permanent_uuid: "c561ec3374b94b95be0c155d80ff0bff" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 33627 } } peers { permanent_uuid: "3710b74ed2044ce5995533df32e67351" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 38589 } attrs { replace: true } } peers { permanent_uuid: "25286c485fa04c86a5d07c8f88643a47" member_type: VOTER last_known_addr { host: "127.17.60.198" port: 37089 } attrs { promote: false } } }
I20250112 15:07:29.219342 21624 raft_consensus.cc:2949] T 1eb2b7e6edd34a7b83f9b918d3cdf2ed P 25286c485fa04c86a5d07c8f88643a47 [term 1 FOLLOWER]: Committing config change with OpId 1.27: config changed from index 26 to 27, 25286c485fa04c86a5d07c8f88643a47 (127.17.60.198) changed from NON_VOTER to VOTER. New config: { opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "2537bbbd1a9947ae9facd87567e3ccb9" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 33173 } } peers { permanent_uuid: "c561ec3374b94b95be0c155d80ff0bff" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 33627 } } peers { permanent_uuid: "3710b74ed2044ce5995533df32e67351" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 38589 } attrs { replace: true } } peers { permanent_uuid: "25286c485fa04c86a5d07c8f88643a47" member_type: VOTER last_known_addr { host: "127.17.60.198" port: 37089 } attrs { promote: false } } }
I20250112 15:07:29.221709 21149 consensus_queue.cc:237] T 1eb2b7e6edd34a7b83f9b918d3cdf2ed P c561ec3374b94b95be0c155d80ff0bff [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 27, Committed index: 27, Last appended: 1.27, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "2537bbbd1a9947ae9facd87567e3ccb9" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 33173 } } peers { permanent_uuid: "c561ec3374b94b95be0c155d80ff0bff" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 33627 } } peers { permanent_uuid: "25286c485fa04c86a5d07c8f88643a47" member_type: VOTER last_known_addr { host: "127.17.60.198" port: 37089 } attrs { promote: false } }
I20250112 15:07:29.222008 21624 raft_consensus.cc:1270] T 1eb2b7e6edd34a7b83f9b918d3cdf2ed P 25286c485fa04c86a5d07c8f88643a47 [term 1 FOLLOWER]: Refusing update from remote peer c561ec3374b94b95be0c155d80ff0bff: Log matching property violated. Preceding OpId in replica: term: 1 index: 27. Preceding OpId from leader: term: 1 index: 28. (index mismatch)
I20250112 15:07:29.222075 21068 raft_consensus.cc:1270] T 1eb2b7e6edd34a7b83f9b918d3cdf2ed P 2537bbbd1a9947ae9facd87567e3ccb9 [term 1 FOLLOWER]: Refusing update from remote peer c561ec3374b94b95be0c155d80ff0bff: Log matching property violated. Preceding OpId in replica: term: 1 index: 27. Preceding OpId from leader: term: 1 index: 28. (index mismatch)
I20250112 15:07:29.222146 21682 consensus_queue.cc:1035] T 1eb2b7e6edd34a7b83f9b918d3cdf2ed P c561ec3374b94b95be0c155d80ff0bff [LEADER]: Connected to new peer: Peer: permanent_uuid: "25286c485fa04c86a5d07c8f88643a47" member_type: VOTER last_known_addr { host: "127.17.60.198" port: 37089 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 28, Last known committed idx: 27, Time since last communication: 0.000s
I20250112 15:07:29.222299 21682 consensus_queue.cc:1035] T 1eb2b7e6edd34a7b83f9b918d3cdf2ed P c561ec3374b94b95be0c155d80ff0bff [LEADER]: Connected to new peer: Peer: permanent_uuid: "2537bbbd1a9947ae9facd87567e3ccb9" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 33173 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 28, Last known committed idx: 27, Time since last communication: 0.000s
I20250112 15:07:29.222714 21683 raft_consensus.cc:2949] T 1eb2b7e6edd34a7b83f9b918d3cdf2ed P c561ec3374b94b95be0c155d80ff0bff [term 1 LEADER]: Committing config change with OpId 1.28: config changed from index 27 to 28, VOTER 3710b74ed2044ce5995533df32e67351 (127.17.60.195) evicted. New config: { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "2537bbbd1a9947ae9facd87567e3ccb9" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 33173 } } peers { permanent_uuid: "c561ec3374b94b95be0c155d80ff0bff" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 33627 } } peers { permanent_uuid: "25286c485fa04c86a5d07c8f88643a47" member_type: VOTER last_known_addr { host: "127.17.60.198" port: 37089 } attrs { promote: false } } }
I20250112 15:07:29.222815 21068 raft_consensus.cc:2949] T 1eb2b7e6edd34a7b83f9b918d3cdf2ed P 2537bbbd1a9947ae9facd87567e3ccb9 [term 1 FOLLOWER]: Committing config change with OpId 1.28: config changed from index 27 to 28, VOTER 3710b74ed2044ce5995533df32e67351 (127.17.60.195) evicted. New config: { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "2537bbbd1a9947ae9facd87567e3ccb9" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 33173 } } peers { permanent_uuid: "c561ec3374b94b95be0c155d80ff0bff" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 33627 } } peers { permanent_uuid: "25286c485fa04c86a5d07c8f88643a47" member_type: VOTER last_known_addr { host: "127.17.60.198" port: 37089 } attrs { promote: false } } }
I20250112 15:07:29.222820 21624 raft_consensus.cc:2949] T 1eb2b7e6edd34a7b83f9b918d3cdf2ed P 25286c485fa04c86a5d07c8f88643a47 [term 1 FOLLOWER]: Committing config change with OpId 1.28: config changed from index 27 to 28, VOTER 3710b74ed2044ce5995533df32e67351 (127.17.60.195) evicted. New config: { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "2537bbbd1a9947ae9facd87567e3ccb9" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 33173 } } peers { permanent_uuid: "c561ec3374b94b95be0c155d80ff0bff" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 33627 } } peers { permanent_uuid: "25286c485fa04c86a5d07c8f88643a47" member_type: VOTER last_known_addr { host: "127.17.60.198" port: 37089 } attrs { promote: false } } }
I20250112 15:07:29.223466 20958 catalog_manager.cc:5039] ChangeConfig:REMOVE_PEER RPC for tablet 1eb2b7e6edd34a7b83f9b918d3cdf2ed with cas_config_opid_index 27: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250112 15:07:29.223670 20970 catalog_manager.cc:5526] T 1eb2b7e6edd34a7b83f9b918d3cdf2ed P c561ec3374b94b95be0c155d80ff0bff reported cstate change: config changed from index 27 to 28, VOTER 3710b74ed2044ce5995533df32e67351 (127.17.60.195) evicted. New cstate: current_term: 1 leader_uuid: "c561ec3374b94b95be0c155d80ff0bff" committed_config { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "2537bbbd1a9947ae9facd87567e3ccb9" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 33173 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c561ec3374b94b95be0c155d80ff0bff" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 33627 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "25286c485fa04c86a5d07c8f88643a47" member_type: VOTER last_known_addr { host: "127.17.60.198" port: 37089 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20250112 15:07:29.224463 21212 tablet_service.cc:1514] Processing DeleteTablet for tablet 1eb2b7e6edd34a7b83f9b918d3cdf2ed with delete_type TABLET_DATA_TOMBSTONED (TS 3710b74ed2044ce5995533df32e67351 not found in new config with opid_index 28) from {username='slave'} at 127.0.0.1:33192
I20250112 15:07:29.224720 21692 tablet_replica.cc:331] T 1eb2b7e6edd34a7b83f9b918d3cdf2ed P 3710b74ed2044ce5995533df32e67351: stopping tablet replica
I20250112 15:07:29.224776 21692 raft_consensus.cc:2238] T 1eb2b7e6edd34a7b83f9b918d3cdf2ed P 3710b74ed2044ce5995533df32e67351 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250112 15:07:29.224845 21692 raft_consensus.cc:2267] T 1eb2b7e6edd34a7b83f9b918d3cdf2ed P 3710b74ed2044ce5995533df32e67351 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 15:07:29.225075 21692 ts_tablet_manager.cc:1905] T 1eb2b7e6edd34a7b83f9b918d3cdf2ed P 3710b74ed2044ce5995533df32e67351: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250112 15:07:29.226284 21692 ts_tablet_manager.cc:1918] T 1eb2b7e6edd34a7b83f9b918d3cdf2ed P 3710b74ed2044ce5995533df32e67351: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.27
I20250112 15:07:29.226329 21692 log.cc:1198] T 1eb2b7e6edd34a7b83f9b918d3cdf2ed P 3710b74ed2044ce5995533df32e67351: Deleting WAL directory at /tmp/dist-test-task95dLSD/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1736694351070047-17651-0/minicluster-data/ts-2-root/wals/1eb2b7e6edd34a7b83f9b918d3cdf2ed
I20250112 15:07:29.226637 20957 catalog_manager.cc:4872] TS 3710b74ed2044ce5995533df32e67351 (127.17.60.195:38589): tablet 1eb2b7e6edd34a7b83f9b918d3cdf2ed (table test-workload [id=cdaf6d3a65a54ea08e8e2c0ecdfc9de7]) successfully deleted
I20250112 15:07:30.071538 21026 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250112 15:07:30.072052 21147 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "1eb2b7e6edd34a7b83f9b918d3cdf2ed"
dest_uuid: "c561ec3374b94b95be0c155d80ff0bff"
mode: GRACEFUL
new_leader_uuid: "25286c485fa04c86a5d07c8f88643a47"
from {username='slave'} at 127.0.0.1:48104
I20250112 15:07:30.072130 21147 raft_consensus.cc:604] T 1eb2b7e6edd34a7b83f9b918d3cdf2ed P c561ec3374b94b95be0c155d80ff0bff [term 1 LEADER]: Received request to transfer leadership to TS 25286c485fa04c86a5d07c8f88643a47
I20250112 15:07:30.080899 21069 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "ef90b704208140a3983ffe6105939319"
dest_uuid: "2537bbbd1a9947ae9facd87567e3ccb9"
mode: GRACEFUL
new_leader_uuid: "c561ec3374b94b95be0c155d80ff0bff"
from {username='slave'} at 127.0.0.1:54414
I20250112 15:07:30.081018 21069 raft_consensus.cc:604] T ef90b704208140a3983ffe6105939319 P 2537bbbd1a9947ae9facd87567e3ccb9 [term 1 LEADER]: Received request to transfer leadership to TS c561ec3374b94b95be0c155d80ff0bff
I20250112 15:07:30.081266 21026 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 2
I20250112 15:07:30.081339 21026 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20250112 15:07:30.227449 21069 consensus_queue.cc:237] T 5bcec60d3b974a9bb24aedfc738e1594 P 2537bbbd1a9947ae9facd87567e3ccb9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 25, Committed index: 25, Last appended: 1.25, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 26 OBSOLETE_local: false peers { permanent_uuid: "2537bbbd1a9947ae9facd87567e3ccb9" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 33173 } } peers { permanent_uuid: "c561ec3374b94b95be0c155d80ff0bff" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 33627 } attrs { replace: true } } peers { permanent_uuid: "3710b74ed2044ce5995533df32e67351" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 38589 } } peers { permanent_uuid: "a2a5bba4ac1145d5bab94a69c9f4df40" member_type: NON_VOTER last_known_addr { host: "127.17.60.196" port: 41807 } attrs { promote: true } }
I20250112 15:07:30.227980 21147 raft_consensus.cc:1270] T 5bcec60d3b974a9bb24aedfc738e1594 P c561ec3374b94b95be0c155d80ff0bff [term 1 FOLLOWER]: Refusing update from remote peer 2537bbbd1a9947ae9facd87567e3ccb9: Log matching property violated. Preceding OpId in replica: term: 1 index: 25. Preceding OpId from leader: term: 1 index: 26. (index mismatch)
I20250112 15:07:30.228021 21226 raft_consensus.cc:1270] T 5bcec60d3b974a9bb24aedfc738e1594 P 3710b74ed2044ce5995533df32e67351 [term 1 FOLLOWER]: Refusing update from remote peer 2537bbbd1a9947ae9facd87567e3ccb9: Log matching property violated. Preceding OpId in replica: term: 1 index: 25. Preceding OpId from leader: term: 1 index: 26. (index mismatch)
I20250112 15:07:30.228199 21699 consensus_queue.cc:1035] T 5bcec60d3b974a9bb24aedfc738e1594 P 2537bbbd1a9947ae9facd87567e3ccb9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c561ec3374b94b95be0c155d80ff0bff" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 33627 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 26, Last known committed idx: 25, Time since last communication: 0.000s
I20250112 15:07:30.228323 21698 consensus_queue.cc:1035] T 5bcec60d3b974a9bb24aedfc738e1594 P 2537bbbd1a9947ae9facd87567e3ccb9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3710b74ed2044ce5995533df32e67351" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 38589 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 26, Last known committed idx: 25, Time since last communication: 0.000s
I20250112 15:07:30.229161 21693 raft_consensus.cc:2949] T 5bcec60d3b974a9bb24aedfc738e1594 P 2537bbbd1a9947ae9facd87567e3ccb9 [term 1 LEADER]: Committing config change with OpId 1.26: config changed from index -1 to 26, NON_VOTER a2a5bba4ac1145d5bab94a69c9f4df40 (127.17.60.196) added. New config: { opid_index: 26 OBSOLETE_local: false peers { permanent_uuid: "2537bbbd1a9947ae9facd87567e3ccb9" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 33173 } } peers { permanent_uuid: "c561ec3374b94b95be0c155d80ff0bff" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 33627 } attrs { replace: true } } peers { permanent_uuid: "3710b74ed2044ce5995533df32e67351" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 38589 } } peers { permanent_uuid: "a2a5bba4ac1145d5bab94a69c9f4df40" member_type: NON_VOTER last_known_addr { host: "127.17.60.196" port: 41807 } attrs { promote: true } } }
I20250112 15:07:30.229266 21226 raft_consensus.cc:2949] T 5bcec60d3b974a9bb24aedfc738e1594 P 3710b74ed2044ce5995533df32e67351 [term 1 FOLLOWER]: Committing config change with OpId 1.26: config changed from index -1 to 26, NON_VOTER a2a5bba4ac1145d5bab94a69c9f4df40 (127.17.60.196) added. New config: { opid_index: 26 OBSOLETE_local: false peers { permanent_uuid: "2537bbbd1a9947ae9facd87567e3ccb9" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 33173 } } peers { permanent_uuid: "c561ec3374b94b95be0c155d80ff0bff" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 33627 } attrs { replace: true } } peers { permanent_uuid: "3710b74ed2044ce5995533df32e67351" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 38589 } } peers { permanent_uuid: "a2a5bba4ac1145d5bab94a69c9f4df40" member_type: NON_VOTER last_known_addr { host: "127.17.60.196" port: 41807 } attrs { promote: true } } }
I20250112 15:07:30.229911 21147 raft_consensus.cc:2949] T 5bcec60d3b974a9bb24aedfc738e1594 P c561ec3374b94b95be0c155d80ff0bff [term 1 FOLLOWER]: Committing config change with OpId 1.26: config changed from index -1 to 26, NON_VOTER a2a5bba4ac1145d5bab94a69c9f4df40 (127.17.60.196) added. New config: { opid_index: 26 OBSOLETE_local: false peers { permanent_uuid: "2537bbbd1a9947ae9facd87567e3ccb9" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 33173 } } peers { permanent_uuid: "c561ec3374b94b95be0c155d80ff0bff" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 33627 } attrs { replace: true } } peers { permanent_uuid: "3710b74ed2044ce5995533df32e67351" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 38589 } } peers { permanent_uuid: "a2a5bba4ac1145d5bab94a69c9f4df40" member_type: NON_VOTER last_known_addr { host: "127.17.60.196" port: 41807 } attrs { promote: true } } }
I20250112 15:07:30.230368 20964 catalog_manager.cc:5526] T 5bcec60d3b974a9bb24aedfc738e1594 P 3710b74ed2044ce5995533df32e67351 reported cstate change: config changed from index -1 to 26, NON_VOTER a2a5bba4ac1145d5bab94a69c9f4df40 (127.17.60.196) added. New cstate: current_term: 1 leader_uuid: "2537bbbd1a9947ae9facd87567e3ccb9" committed_config { opid_index: 26 OBSOLETE_local: false peers { permanent_uuid: "2537bbbd1a9947ae9facd87567e3ccb9" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 33173 } } peers { permanent_uuid: "c561ec3374b94b95be0c155d80ff0bff" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 33627 } attrs { replace: true } } peers { permanent_uuid: "3710b74ed2044ce5995533df32e67351" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 38589 } } peers { permanent_uuid: "a2a5bba4ac1145d5bab94a69c9f4df40" member_type: NON_VOTER last_known_addr { host: "127.17.60.196" port: 41807 } attrs { promote: true } } }
W20250112 15:07:30.231527 21040 consensus_peers.cc:487] T 5bcec60d3b974a9bb24aedfc738e1594 P 2537bbbd1a9947ae9facd87567e3ccb9 -> Peer a2a5bba4ac1145d5bab94a69c9f4df40 (127.17.60.196:41807): Couldn't send request to peer a2a5bba4ac1145d5bab94a69c9f4df40. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 5bcec60d3b974a9bb24aedfc738e1594. This is attempt 1: this message will repeat every 5th retry.
I20250112 15:07:30.231540 21147 consensus_queue.cc:237] T 69c23627ab854000b360030c384dc966 P c561ec3374b94b95be0c155d80ff0bff [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 24, Committed index: 24, Last appended: 1.24, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 25 OBSOLETE_local: false peers { permanent_uuid: "2537bbbd1a9947ae9facd87567e3ccb9" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 33173 } attrs { replace: true } } peers { permanent_uuid: "c561ec3374b94b95be0c155d80ff0bff" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 33627 } } peers { permanent_uuid: "3710b74ed2044ce5995533df32e67351" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 38589 } } peers { permanent_uuid: "b5e9178a7a83402e8c4a6943ab78589d" member_type: NON_VOTER last_known_addr { host: "127.17.60.197" port: 36573 } attrs { promote: true } }
I20250112 15:07:30.232230 21226 raft_consensus.cc:1270] T 69c23627ab854000b360030c384dc966 P 3710b74ed2044ce5995533df32e67351 [term 1 FOLLOWER]: Refusing update from remote peer c561ec3374b94b95be0c155d80ff0bff: Log matching property violated. Preceding OpId in replica: term: 1 index: 24. Preceding OpId from leader: term: 1 index: 25. (index mismatch)
I20250112 15:07:30.232280 21069 raft_consensus.cc:1270] T 69c23627ab854000b360030c384dc966 P 2537bbbd1a9947ae9facd87567e3ccb9 [term 1 FOLLOWER]: Refusing update from remote peer c561ec3374b94b95be0c155d80ff0bff: Log matching property violated. Preceding OpId in replica: term: 1 index: 24. Preceding OpId from leader: term: 1 index: 25. (index mismatch)
I20250112 15:07:30.232457 21710 consensus_queue.cc:1035] T 69c23627ab854000b360030c384dc966 P c561ec3374b94b95be0c155d80ff0bff [LEADER]: Connected to new peer: Peer: permanent_uuid: "3710b74ed2044ce5995533df32e67351" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 38589 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 25, Last known committed idx: 24, Time since last communication: 0.000s
I20250112 15:07:30.232609 21709 consensus_queue.cc:1035] T 69c23627ab854000b360030c384dc966 P c561ec3374b94b95be0c155d80ff0bff [LEADER]: Connected to new peer: Peer: permanent_uuid: "2537bbbd1a9947ae9facd87567e3ccb9" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 33173 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 25, Last known committed idx: 24, Time since last communication: 0.000s
I20250112 15:07:30.233196 21424 raft_consensus.cc:2949] T 69c23627ab854000b360030c384dc966 P c561ec3374b94b95be0c155d80ff0bff [term 1 LEADER]: Committing config change with OpId 1.25: config changed from index -1 to 25, NON_VOTER b5e9178a7a83402e8c4a6943ab78589d (127.17.60.197) added. New config: { opid_index: 25 OBSOLETE_local: false peers { permanent_uuid: "2537bbbd1a9947ae9facd87567e3ccb9" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 33173 } attrs { replace: true } } peers { permanent_uuid: "c561ec3374b94b95be0c155d80ff0bff" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 33627 } } peers { permanent_uuid: "3710b74ed2044ce5995533df32e67351" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 38589 } } peers { permanent_uuid: "b5e9178a7a83402e8c4a6943ab78589d" member_type: NON_VOTER last_known_addr { host: "127.17.60.197" port: 36573 } attrs { promote: true } } }
I20250112 15:07:30.234297 20970 catalog_manager.cc:5526] T 69c23627ab854000b360030c384dc966 P c561ec3374b94b95be0c155d80ff0bff reported cstate change: config changed from index -1 to 25, NON_VOTER b5e9178a7a83402e8c4a6943ab78589d (127.17.60.197) added. New cstate: current_term: 1 leader_uuid: "c561ec3374b94b95be0c155d80ff0bff" committed_config { opid_index: 25 OBSOLETE_local: false peers { permanent_uuid: "2537bbbd1a9947ae9facd87567e3ccb9" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 33173 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c561ec3374b94b95be0c155d80ff0bff" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 33627 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3710b74ed2044ce5995533df32e67351" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 38589 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b5e9178a7a83402e8c4a6943ab78589d" member_type: NON_VOTER last_known_addr { host: "127.17.60.197" port: 36573 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20250112 15:07:30.234445 21068 raft_consensus.cc:2949] T 69c23627ab854000b360030c384dc966 P 2537bbbd1a9947ae9facd87567e3ccb9 [term 1 FOLLOWER]: Committing config change with OpId 1.25: config changed from index -1 to 25, NON_VOTER b5e9178a7a83402e8c4a6943ab78589d (127.17.60.197) added. New config: { opid_index: 25 OBSOLETE_local: false peers { permanent_uuid: "2537bbbd1a9947ae9facd87567e3ccb9" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 33173 } attrs { replace: true } } peers { permanent_uuid: "c561ec3374b94b95be0c155d80ff0bff" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 33627 } } peers { permanent_uuid: "3710b74ed2044ce5995533df32e67351" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 38589 } } peers { permanent_uuid: "b5e9178a7a83402e8c4a6943ab78589d" member_type: NON_VOTER last_known_addr { host: "127.17.60.197" port: 36573 } attrs { promote: true } } }
I20250112 15:07:30.234493 21226 raft_consensus.cc:2949] T 69c23627ab854000b360030c384dc966 P 3710b74ed2044ce5995533df32e67351 [term 1 FOLLOWER]: Committing config change with OpId 1.25: config changed from index -1 to 25, NON_VOTER b5e9178a7a83402e8c4a6943ab78589d (127.17.60.197) added. New config: { opid_index: 25 OBSOLETE_local: false peers { permanent_uuid: "2537bbbd1a9947ae9facd87567e3ccb9" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 33173 } attrs { replace: true } } peers { permanent_uuid: "c561ec3374b94b95be0c155d80ff0bff" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 33627 } } peers { permanent_uuid: "3710b74ed2044ce5995533df32e67351" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 38589 } } peers { permanent_uuid: "b5e9178a7a83402e8c4a6943ab78589d" member_type: NON_VOTER last_known_addr { host: "127.17.60.197" port: 36573 } attrs { promote: true } } }
I20250112 15:07:30.234463 21069 consensus_queue.cc:237] T 57e47b07fdcf434e96ca5dd0354e93b2 P 2537bbbd1a9947ae9facd87567e3ccb9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 26, Committed index: 26, Last appended: 2.26, Last appended by leader: 25, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "2537bbbd1a9947ae9facd87567e3ccb9" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 33173 } } peers { permanent_uuid: "c561ec3374b94b95be0c155d80ff0bff" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 33627 } } peers { permanent_uuid: "3710b74ed2044ce5995533df32e67351" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 38589 } attrs { replace: true } } peers { permanent_uuid: "b5e9178a7a83402e8c4a6943ab78589d" member_type: NON_VOTER last_known_addr { host: "127.17.60.197" port: 36573 } attrs { promote: true } }
I20250112 15:07:30.235899 21698 raft_consensus.cc:988] T ef90b704208140a3983ffe6105939319 P 2537bbbd1a9947ae9facd87567e3ccb9: : Instructing follower c561ec3374b94b95be0c155d80ff0bff to start an election
I20250112 15:07:30.235963 21698 raft_consensus.cc:1076] T ef90b704208140a3983ffe6105939319 P 2537bbbd1a9947ae9facd87567e3ccb9 [term 1 LEADER]: Signalling peer c561ec3374b94b95be0c155d80ff0bff to start an election
I20250112 15:07:30.236315 21149 raft_consensus.cc:1270] T 57e47b07fdcf434e96ca5dd0354e93b2 P c561ec3374b94b95be0c155d80ff0bff [term 2 FOLLOWER]: Refusing update from remote peer 2537bbbd1a9947ae9facd87567e3ccb9: Log matching property violated. Preceding OpId in replica: term: 2 index: 26. Preceding OpId from leader: term: 2 index: 27. (index mismatch)
W20250112 15:07:30.236374 21114 consensus_peers.cc:487] T 69c23627ab854000b360030c384dc966 P c561ec3374b94b95be0c155d80ff0bff -> Peer b5e9178a7a83402e8c4a6943ab78589d (127.17.60.197:36573): Couldn't send request to peer b5e9178a7a83402e8c4a6943ab78589d. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 69c23627ab854000b360030c384dc966. This is attempt 1: this message will repeat every 5th retry.
I20250112 15:07:30.236477 21147 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "ef90b704208140a3983ffe6105939319"
dest_uuid: "c561ec3374b94b95be0c155d80ff0bff"
from {username='slave'} at 127.0.0.1:48082
I20250112 15:07:30.236496 21226 raft_consensus.cc:1270] T 57e47b07fdcf434e96ca5dd0354e93b2 P 3710b74ed2044ce5995533df32e67351 [term 2 FOLLOWER]: Refusing update from remote peer 2537bbbd1a9947ae9facd87567e3ccb9: Log matching property violated. Preceding OpId in replica: term: 2 index: 26. Preceding OpId from leader: term: 2 index: 27. (index mismatch)
I20250112 15:07:30.236554 21147 raft_consensus.cc:491] T ef90b704208140a3983ffe6105939319 P c561ec3374b94b95be0c155d80ff0bff [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20250112 15:07:30.236622 21147 raft_consensus.cc:3054] T ef90b704208140a3983ffe6105939319 P c561ec3374b94b95be0c155d80ff0bff [term 1 FOLLOWER]: Advancing to term 2
I20250112 15:07:30.236735 21716 consensus_queue.cc:1035] T 57e47b07fdcf434e96ca5dd0354e93b2 P 2537bbbd1a9947ae9facd87567e3ccb9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3710b74ed2044ce5995533df32e67351" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 38589 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 27, Last known committed idx: 26, Time since last communication: 0.000s
I20250112 15:07:30.237231 21147 raft_consensus.cc:513] T ef90b704208140a3983ffe6105939319 P c561ec3374b94b95be0c155d80ff0bff [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2537bbbd1a9947ae9facd87567e3ccb9" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 33173 } } peers { permanent_uuid: "c561ec3374b94b95be0c155d80ff0bff" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 33627 } } peers { permanent_uuid: "3710b74ed2044ce5995533df32e67351" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 38589 } }
I20250112 15:07:30.237335 21147 leader_election.cc:290] T ef90b704208140a3983ffe6105939319 P c561ec3374b94b95be0c155d80ff0bff [CANDIDATE]: Term 2 election: Requested vote from peers 2537bbbd1a9947ae9facd87567e3ccb9 (127.17.60.193:33173), 3710b74ed2044ce5995533df32e67351 (127.17.60.195:38589)
I20250112 15:07:30.237316 21693 raft_consensus.cc:2949] T 57e47b07fdcf434e96ca5dd0354e93b2 P 2537bbbd1a9947ae9facd87567e3ccb9 [term 2 LEADER]: Committing config change with OpId 2.27: config changed from index -1 to 27, NON_VOTER b5e9178a7a83402e8c4a6943ab78589d (127.17.60.197) added. New config: { opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "2537bbbd1a9947ae9facd87567e3ccb9" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 33173 } } peers { permanent_uuid: "c561ec3374b94b95be0c155d80ff0bff" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 33627 } } peers { permanent_uuid: "3710b74ed2044ce5995533df32e67351" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 38589 } attrs { replace: true } } peers { permanent_uuid: "b5e9178a7a83402e8c4a6943ab78589d" member_type: NON_VOTER last_known_addr { host: "127.17.60.197" port: 36573 } attrs { promote: true } } }
I20250112 15:07:30.237413 21226 raft_consensus.cc:2949] T 57e47b07fdcf434e96ca5dd0354e93b2 P 3710b74ed2044ce5995533df32e67351 [term 2 FOLLOWER]: Committing config change with OpId 2.27: config changed from index -1 to 27, NON_VOTER b5e9178a7a83402e8c4a6943ab78589d (127.17.60.197) added. New config: { opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "2537bbbd1a9947ae9facd87567e3ccb9" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 33173 } } peers { permanent_uuid: "c561ec3374b94b95be0c155d80ff0bff" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 33627 } } peers { permanent_uuid: "3710b74ed2044ce5995533df32e67351" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 38589 } attrs { replace: true } } peers { permanent_uuid: "b5e9178a7a83402e8c4a6943ab78589d" member_type: NON_VOTER last_known_addr { host: "127.17.60.197" port: 36573 } attrs { promote: true } } }
I20250112 15:07:30.237779 21069 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef90b704208140a3983ffe6105939319" candidate_uuid: "c561ec3374b94b95be0c155d80ff0bff" candidate_term: 2 candidate_status { last_received { term: 1 index: 25 } } ignore_live_leader: true dest_uuid: "2537bbbd1a9947ae9facd87567e3ccb9"
I20250112 15:07:30.237823 21222 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef90b704208140a3983ffe6105939319" candidate_uuid: "c561ec3374b94b95be0c155d80ff0bff" candidate_term: 2 candidate_status { last_received { term: 1 index: 25 } } ignore_live_leader: true dest_uuid: "3710b74ed2044ce5995533df32e67351"
I20250112 15:07:30.237856 21069 raft_consensus.cc:3049] T ef90b704208140a3983ffe6105939319 P 2537bbbd1a9947ae9facd87567e3ccb9 [term 1 LEADER]: Stepping down as leader of term 1
I20250112 15:07:30.237866 21222 raft_consensus.cc:3054] T ef90b704208140a3983ffe6105939319 P 3710b74ed2044ce5995533df32e67351 [term 1 FOLLOWER]: Advancing to term 2
I20250112 15:07:30.237879 21069 raft_consensus.cc:738] T ef90b704208140a3983ffe6105939319 P 2537bbbd1a9947ae9facd87567e3ccb9 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 2537bbbd1a9947ae9facd87567e3ccb9, State: Running, Role: LEADER
I20250112 15:07:30.237920 21069 consensus_queue.cc:260] T ef90b704208140a3983ffe6105939319 P 2537bbbd1a9947ae9facd87567e3ccb9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 25, Committed index: 25, Last appended: 1.25, Last appended by leader: 25, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2537bbbd1a9947ae9facd87567e3ccb9" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 33173 } } peers { permanent_uuid: "c561ec3374b94b95be0c155d80ff0bff" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 33627 } } peers { permanent_uuid: "3710b74ed2044ce5995533df32e67351" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 38589 } }
I20250112 15:07:30.237982 21069 raft_consensus.cc:3054] T ef90b704208140a3983ffe6105939319 P 2537bbbd1a9947ae9facd87567e3ccb9 [term 1 FOLLOWER]: Advancing to term 2
I20250112 15:07:30.238062 21693 consensus_queue.cc:1035] T 57e47b07fdcf434e96ca5dd0354e93b2 P 2537bbbd1a9947ae9facd87567e3ccb9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c561ec3374b94b95be0c155d80ff0bff" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 33627 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 27, Last known committed idx: 26, Time since last communication: 0.000s
I20250112 15:07:30.238461 21068 consensus_queue.cc:237] T 9601bcda24ed42e7b93ab9740f1e271e P 2537bbbd1a9947ae9facd87567e3ccb9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 24, Committed index: 24, Last appended: 1.24, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 25 OBSOLETE_local: false peers { permanent_uuid: "2537bbbd1a9947ae9facd87567e3ccb9" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 33173 } } peers { permanent_uuid: "c561ec3374b94b95be0c155d80ff0bff" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 33627 } attrs { replace: true } } peers { permanent_uuid: "3710b74ed2044ce5995533df32e67351" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 38589 } } peers { permanent_uuid: "a2a5bba4ac1145d5bab94a69c9f4df40" member_type: NON_VOTER last_known_addr { host: "127.17.60.196" port: 41807 } attrs { promote: true } }
I20250112 15:07:30.238550 21069 raft_consensus.cc:2463] T ef90b704208140a3983ffe6105939319 P 2537bbbd1a9947ae9facd87567e3ccb9 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c561ec3374b94b95be0c155d80ff0bff in term 2.
I20250112 15:07:30.238567 21222 raft_consensus.cc:2463] T ef90b704208140a3983ffe6105939319 P 3710b74ed2044ce5995533df32e67351 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c561ec3374b94b95be0c155d80ff0bff in term 2.
I20250112 15:07:30.238777 21114 leader_election.cc:304] T ef90b704208140a3983ffe6105939319 P c561ec3374b94b95be0c155d80ff0bff [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2537bbbd1a9947ae9facd87567e3ccb9, c561ec3374b94b95be0c155d80ff0bff; no voters:
I20250112 15:07:30.238859 21710 raft_consensus.cc:2798] T ef90b704208140a3983ffe6105939319 P c561ec3374b94b95be0c155d80ff0bff [term 2 FOLLOWER]: Leader election won for term 2
I20250112 15:07:30.238807 20966 catalog_manager.cc:5526] T 57e47b07fdcf434e96ca5dd0354e93b2 P 2537bbbd1a9947ae9facd87567e3ccb9 reported cstate change: config changed from index -1 to 27, NON_VOTER b5e9178a7a83402e8c4a6943ab78589d (127.17.60.197) added. New cstate: current_term: 2 leader_uuid: "2537bbbd1a9947ae9facd87567e3ccb9" committed_config { opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "2537bbbd1a9947ae9facd87567e3ccb9" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 33173 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c561ec3374b94b95be0c155d80ff0bff" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 33627 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3710b74ed2044ce5995533df32e67351" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 38589 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b5e9178a7a83402e8c4a6943ab78589d" member_type: NON_VOTER last_known_addr { host: "127.17.60.197" port: 36573 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20250112 15:07:30.238922 21710 raft_consensus.cc:695] T ef90b704208140a3983ffe6105939319 P c561ec3374b94b95be0c155d80ff0bff [term 2 LEADER]: Becoming Leader. State: Replica: c561ec3374b94b95be0c155d80ff0bff, State: Running, Role: LEADER
I20250112 15:07:30.238988 21710 consensus_queue.cc:237] T ef90b704208140a3983ffe6105939319 P c561ec3374b94b95be0c155d80ff0bff [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 25, Committed index: 25, Last appended: 1.25, Last appended by leader: 25, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2537bbbd1a9947ae9facd87567e3ccb9" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 33173 } } peers { permanent_uuid: "c561ec3374b94b95be0c155d80ff0bff" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 33627 } } peers { permanent_uuid: "3710b74ed2044ce5995533df32e67351" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 38589 } }
I20250112 15:07:30.239035 21222 raft_consensus.cc:1270] T 9601bcda24ed42e7b93ab9740f1e271e P 3710b74ed2044ce5995533df32e67351 [term 1 FOLLOWER]: Refusing update from remote peer 2537bbbd1a9947ae9facd87567e3ccb9: Log matching property violated. Preceding OpId in replica: term: 1 index: 24. Preceding OpId from leader: term: 1 index: 25. (index mismatch)
W20250112 15:07:30.239516 21040 consensus_peers.cc:487] T 9601bcda24ed42e7b93ab9740f1e271e P 2537bbbd1a9947ae9facd87567e3ccb9 -> Peer a2a5bba4ac1145d5bab94a69c9f4df40 (127.17.60.196:41807): Couldn't send request to peer a2a5bba4ac1145d5bab94a69c9f4df40. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 9601bcda24ed42e7b93ab9740f1e271e. This is attempt 1: this message will repeat every 5th retry.
I20250112 15:07:30.239539 21149 raft_consensus.cc:1270] T 9601bcda24ed42e7b93ab9740f1e271e P c561ec3374b94b95be0c155d80ff0bff [term 1 FOLLOWER]: Refusing update from remote peer 2537bbbd1a9947ae9facd87567e3ccb9: Log matching property violated. Preceding OpId in replica: term: 1 index: 24. Preceding OpId from leader: term: 1 index: 25. (index mismatch)
I20250112 15:07:30.239645 21693 consensus_queue.cc:1035] T 9601bcda24ed42e7b93ab9740f1e271e P 2537bbbd1a9947ae9facd87567e3ccb9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3710b74ed2044ce5995533df32e67351" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 38589 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 25, Last known committed idx: 24, Time since last communication: 0.000s
I20250112 15:07:30.239534 20966 catalog_manager.cc:5526] T ef90b704208140a3983ffe6105939319 P c561ec3374b94b95be0c155d80ff0bff reported cstate change: term changed from 1 to 2, leader changed from 2537bbbd1a9947ae9facd87567e3ccb9 (127.17.60.193) to c561ec3374b94b95be0c155d80ff0bff (127.17.60.194). New cstate: current_term: 2 leader_uuid: "c561ec3374b94b95be0c155d80ff0bff" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2537bbbd1a9947ae9facd87567e3ccb9" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 33173 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c561ec3374b94b95be0c155d80ff0bff" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 33627 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3710b74ed2044ce5995533df32e67351" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 38589 } health_report { overall_health: UNKNOWN } } }
I20250112 15:07:30.239713 21147 raft_consensus.cc:2949] T 57e47b07fdcf434e96ca5dd0354e93b2 P c561ec3374b94b95be0c155d80ff0bff [term 2 FOLLOWER]: Committing config change with OpId 2.27: config changed from index -1 to 27, NON_VOTER b5e9178a7a83402e8c4a6943ab78589d (127.17.60.197) added. New config: { opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "2537bbbd1a9947ae9facd87567e3ccb9" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 33173 } } peers { permanent_uuid: "c561ec3374b94b95be0c155d80ff0bff" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 33627 } } peers { permanent_uuid: "3710b74ed2044ce5995533df32e67351" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 38589 } attrs { replace: true } } peers { permanent_uuid: "b5e9178a7a83402e8c4a6943ab78589d" member_type: NON_VOTER last_known_addr { host: "127.17.60.197" port: 36573 } attrs { promote: true } } }
I20250112 15:07:30.240784 21693 consensus_queue.cc:1035] T 9601bcda24ed42e7b93ab9740f1e271e P 2537bbbd1a9947ae9facd87567e3ccb9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c561ec3374b94b95be0c155d80ff0bff" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 33627 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 25, Last known committed idx: 24, Time since last communication: 0.000s
W20250112 15:07:30.240912 21038 consensus_peers.cc:487] T 57e47b07fdcf434e96ca5dd0354e93b2 P 2537bbbd1a9947ae9facd87567e3ccb9 -> Peer b5e9178a7a83402e8c4a6943ab78589d (127.17.60.197:36573): Couldn't send request to peer b5e9178a7a83402e8c4a6943ab78589d. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 57e47b07fdcf434e96ca5dd0354e93b2. This is attempt 1: this message will repeat every 5th retry.
I20250112 15:07:30.240908 21698 raft_consensus.cc:2949] T 9601bcda24ed42e7b93ab9740f1e271e P 2537bbbd1a9947ae9facd87567e3ccb9 [term 1 LEADER]: Committing config change with OpId 1.25: config changed from index -1 to 25, NON_VOTER a2a5bba4ac1145d5bab94a69c9f4df40 (127.17.60.196) added. New config: { opid_index: 25 OBSOLETE_local: false peers { permanent_uuid: "2537bbbd1a9947ae9facd87567e3ccb9" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 33173 } } peers { permanent_uuid: "c561ec3374b94b95be0c155d80ff0bff" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 33627 } attrs { replace: true } } peers { permanent_uuid: "3710b74ed2044ce5995533df32e67351" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 38589 } } peers { permanent_uuid: "a2a5bba4ac1145d5bab94a69c9f4df40" member_type: NON_VOTER last_known_addr { host: "127.17.60.196" port: 41807 } attrs { promote: true } } }
I20250112 15:07:30.241031 21222 raft_consensus.cc:2949] T 9601bcda24ed42e7b93ab9740f1e271e P 3710b74ed2044ce5995533df32e67351 [term 1 FOLLOWER]: Committing config change with OpId 1.25: config changed from index -1 to 25, NON_VOTER a2a5bba4ac1145d5bab94a69c9f4df40 (127.17.60.196) added. New config: { opid_index: 25 OBSOLETE_local: false peers { permanent_uuid: "2537bbbd1a9947ae9facd87567e3ccb9" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 33173 } } peers { permanent_uuid: "c561ec3374b94b95be0c155d80ff0bff" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 33627 } attrs { replace: true } } peers { permanent_uuid: "3710b74ed2044ce5995533df32e67351" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 38589 } } peers { permanent_uuid: "a2a5bba4ac1145d5bab94a69c9f4df40" member_type: NON_VOTER last_known_addr { host: "127.17.60.196" port: 41807 } attrs { promote: true } } }
I20250112 15:07:30.241125 21149 raft_consensus.cc:2949] T 9601bcda24ed42e7b93ab9740f1e271e P c561ec3374b94b95be0c155d80ff0bff [term 1 FOLLOWER]: Committing config change with OpId 1.25: config changed from index -1 to 25, NON_VOTER a2a5bba4ac1145d5bab94a69c9f4df40 (127.17.60.196) added. New config: { opid_index: 25 OBSOLETE_local: false peers { permanent_uuid: "2537bbbd1a9947ae9facd87567e3ccb9" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 33173 } } peers { permanent_uuid: "c561ec3374b94b95be0c155d80ff0bff" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 33627 } attrs { replace: true } } peers { permanent_uuid: "3710b74ed2044ce5995533df32e67351" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 38589 } } peers { permanent_uuid: "a2a5bba4ac1145d5bab94a69c9f4df40" member_type: NON_VOTER last_known_addr { host: "127.17.60.196" port: 41807 } attrs { promote: true } } }
I20250112 15:07:30.242106 20966 catalog_manager.cc:5526] T 9601bcda24ed42e7b93ab9740f1e271e P 2537bbbd1a9947ae9facd87567e3ccb9 reported cstate change: config changed from index -1 to 25, NON_VOTER a2a5bba4ac1145d5bab94a69c9f4df40 (127.17.60.196) added. New cstate: current_term: 1 leader_uuid: "2537bbbd1a9947ae9facd87567e3ccb9" committed_config { opid_index: 25 OBSOLETE_local: false peers { permanent_uuid: "2537bbbd1a9947ae9facd87567e3ccb9" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 33173 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c561ec3374b94b95be0c155d80ff0bff" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 33627 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3710b74ed2044ce5995533df32e67351" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 38589 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a2a5bba4ac1145d5bab94a69c9f4df40" member_type: NON_VOTER last_known_addr { host: "127.17.60.196" port: 41807 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20250112 15:07:30.242486 21147 consensus_queue.cc:237] T 80cc505a04394c53a18c5beeded4655d P c561ec3374b94b95be0c155d80ff0bff [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 25, Committed index: 25, Last appended: 1.25, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 26 OBSOLETE_local: false peers { permanent_uuid: "2537bbbd1a9947ae9facd87567e3ccb9" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 33173 } attrs { replace: true } } peers { permanent_uuid: "c561ec3374b94b95be0c155d80ff0bff" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 33627 } } peers { permanent_uuid: "3710b74ed2044ce5995533df32e67351" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 38589 } } peers { permanent_uuid: "25286c485fa04c86a5d07c8f88643a47" member_type: NON_VOTER last_known_addr { host: "127.17.60.198" port: 37089 } attrs { promote: true } }
I20250112 15:07:30.242926 21068 raft_consensus.cc:1270] T 80cc505a04394c53a18c5beeded4655d P 2537bbbd1a9947ae9facd87567e3ccb9 [term 1 FOLLOWER]: Refusing update from remote peer c561ec3374b94b95be0c155d80ff0bff: Log matching property violated. Preceding OpId in replica: term: 1 index: 25. Preceding OpId from leader: term: 1 index: 26. (index mismatch)
W20250112 15:07:30.242974 21114 consensus_peers.cc:487] T 80cc505a04394c53a18c5beeded4655d P c561ec3374b94b95be0c155d80ff0bff -> Peer 25286c485fa04c86a5d07c8f88643a47 (127.17.60.198:37089): Couldn't send request to peer 25286c485fa04c86a5d07c8f88643a47. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 80cc505a04394c53a18c5beeded4655d. This is attempt 1: this message will repeat every 5th retry.
I20250112 15:07:30.243062 21222 raft_consensus.cc:1270] T 80cc505a04394c53a18c5beeded4655d P 3710b74ed2044ce5995533df32e67351 [term 1 FOLLOWER]: Refusing update from remote peer c561ec3374b94b95be0c155d80ff0bff: Log matching property violated. Preceding OpId in replica: term: 1 index: 25. Preceding OpId from leader: term: 1 index: 26. (index mismatch)
I20250112 15:07:30.243135 21424 consensus_queue.cc:1035] T 80cc505a04394c53a18c5beeded4655d P c561ec3374b94b95be0c155d80ff0bff [LEADER]: Connected to new peer: Peer: permanent_uuid: "2537bbbd1a9947ae9facd87567e3ccb9" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 33173 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 26, Last known committed idx: 25, Time since last communication: 0.000s
I20250112 15:07:30.243225 21710 consensus_queue.cc:1035] T 80cc505a04394c53a18c5beeded4655d P c561ec3374b94b95be0c155d80ff0bff [LEADER]: Connected to new peer: Peer: permanent_uuid: "3710b74ed2044ce5995533df32e67351" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 38589 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 26, Last known committed idx: 25, Time since last communication: 0.000s
I20250112 15:07:30.243850 21709 raft_consensus.cc:2949] T 80cc505a04394c53a18c5beeded4655d P c561ec3374b94b95be0c155d80ff0bff [term 1 LEADER]: Committing config change with OpId 1.26: config changed from index -1 to 26, NON_VOTER 25286c485fa04c86a5d07c8f88643a47 (127.17.60.198) added. New config: { opid_index: 26 OBSOLETE_local: false peers { permanent_uuid: "2537bbbd1a9947ae9facd87567e3ccb9" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 33173 } attrs { replace: true } } peers { permanent_uuid: "c561ec3374b94b95be0c155d80ff0bff" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 33627 } } peers { permanent_uuid: "3710b74ed2044ce5995533df32e67351" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 38589 } } peers { permanent_uuid: "25286c485fa04c86a5d07c8f88643a47" member_type: NON_VOTER last_known_addr { host: "127.17.60.198" port: 37089 } attrs { promote: true } } }
I20250112 15:07:30.243966 21068 raft_consensus.cc:2949] T 80cc505a04394c53a18c5beeded4655d P 2537bbbd1a9947ae9facd87567e3ccb9 [term 1 FOLLOWER]: Committing config change with OpId 1.26: config changed from index -1 to 26, NON_VOTER 25286c485fa04c86a5d07c8f88643a47 (127.17.60.198) added. New config: { opid_index: 26 OBSOLETE_local: false peers { permanent_uuid: "2537bbbd1a9947ae9facd87567e3ccb9" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 33173 } attrs { replace: true } } peers { permanent_uuid: "c561ec3374b94b95be0c155d80ff0bff" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 33627 } } peers { permanent_uuid: "3710b74ed2044ce5995533df32e67351" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 38589 } } peers { permanent_uuid: "25286c485fa04c86a5d07c8f88643a47" member_type: NON_VOTER last_known_addr { host: "127.17.60.198" port: 37089 } attrs { promote: true } } }
I20250112 15:07:30.243973 21222 raft_consensus.cc:2949] T 80cc505a04394c53a18c5beeded4655d P 3710b74ed2044ce5995533df32e67351 [term 1 FOLLOWER]: Committing config change with OpId 1.26: config changed from index -1 to 26, NON_VOTER 25286c485fa04c86a5d07c8f88643a47 (127.17.60.198) added. New config: { opid_index: 26 OBSOLETE_local: false peers { permanent_uuid: "2537bbbd1a9947ae9facd87567e3ccb9" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 33173 } attrs { replace: true } } peers { permanent_uuid: "c561ec3374b94b95be0c155d80ff0bff" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 33627 } } peers { permanent_uuid: "3710b74ed2044ce5995533df32e67351" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 38589 } } peers { permanent_uuid: "25286c485fa04c86a5d07c8f88643a47" member_type: NON_VOTER last_known_addr { host: "127.17.60.198" port: 37089 } attrs { promote: true } } }
I20250112 15:07:30.245087 20964 catalog_manager.cc:5526] T 80cc505a04394c53a18c5beeded4655d P 2537bbbd1a9947ae9facd87567e3ccb9 reported cstate change: config changed from index -1 to 26, NON_VOTER 25286c485fa04c86a5d07c8f88643a47 (127.17.60.198) added. New cstate: current_term: 1 leader_uuid: "c561ec3374b94b95be0c155d80ff0bff" committed_config { opid_index: 26 OBSOLETE_local: false peers { permanent_uuid: "2537bbbd1a9947ae9facd87567e3ccb9" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 33173 } attrs { replace: true } } peers { permanent_uuid: "c561ec3374b94b95be0c155d80ff0bff" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 33627 } } peers { permanent_uuid: "3710b74ed2044ce5995533df32e67351" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 38589 } } peers { permanent_uuid: "25286c485fa04c86a5d07c8f88643a47" member_type: NON_VOTER last_known_addr { host: "127.17.60.198" port: 37089 } attrs { promote: true } } }
W20250112 15:07:30.245347 21021 auto_rebalancer.cc:253] failed to send replica move request: Illegal state: Leader has not yet committed an operation in its own term
W20250112 15:07:30.246874 21021 auto_rebalancer.cc:663] Could not move replica: Incomplete: tablet ef90b704208140a3983ffe6105939319, TS c561ec3374b94b95be0c155d80ff0bff -> TS b5e9178a7a83402e8c4a6943ab78589d move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2537bbbd1a9947ae9facd87567e3ccb9" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 33173 } } peers { permanent_uuid: "c561ec3374b94b95be0c155d80ff0bff" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 33627 } } peers { permanent_uuid: "3710b74ed2044ce5995533df32e67351" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 38589 } }
W20250112 15:07:30.246913 21021 auto_rebalancer.cc:263] scheduled replica move failed to complete: Incomplete: tablet ef90b704208140a3983ffe6105939319, TS c561ec3374b94b95be0c155d80ff0bff -> TS b5e9178a7a83402e8c4a6943ab78589d move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2537bbbd1a9947ae9facd87567e3ccb9" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 33173 } } peers { permanent_uuid: "c561ec3374b94b95be0c155d80ff0bff" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 33627 } } peers { permanent_uuid: "3710b74ed2044ce5995533df32e67351" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 38589 } }
I20250112 15:07:30.317844 21424 raft_consensus.cc:988] T 1eb2b7e6edd34a7b83f9b918d3cdf2ed P c561ec3374b94b95be0c155d80ff0bff: : Instructing follower 25286c485fa04c86a5d07c8f88643a47 to start an election
I20250112 15:07:30.317956 21424 raft_consensus.cc:1076] T 1eb2b7e6edd34a7b83f9b918d3cdf2ed P c561ec3374b94b95be0c155d80ff0bff [term 1 LEADER]: Signalling peer 25286c485fa04c86a5d07c8f88643a47 to start an election
I20250112 15:07:30.318115 21624 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "1eb2b7e6edd34a7b83f9b918d3cdf2ed"
dest_uuid: "25286c485fa04c86a5d07c8f88643a47"
from {username='slave'} at 127.0.0.1:35738
I20250112 15:07:30.318205 21624 raft_consensus.cc:491] T 1eb2b7e6edd34a7b83f9b918d3cdf2ed P 25286c485fa04c86a5d07c8f88643a47 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20250112 15:07:30.318235 21624 raft_consensus.cc:3054] T 1eb2b7e6edd34a7b83f9b918d3cdf2ed P 25286c485fa04c86a5d07c8f88643a47 [term 1 FOLLOWER]: Advancing to term 2
I20250112 15:07:30.319186 21624 raft_consensus.cc:513] T 1eb2b7e6edd34a7b83f9b918d3cdf2ed P 25286c485fa04c86a5d07c8f88643a47 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "2537bbbd1a9947ae9facd87567e3ccb9" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 33173 } } peers { permanent_uuid: "c561ec3374b94b95be0c155d80ff0bff" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 33627 } } peers { permanent_uuid: "25286c485fa04c86a5d07c8f88643a47" member_type: VOTER last_known_addr { host: "127.17.60.198" port: 37089 } attrs { promote: false } }
I20250112 15:07:30.319365 21624 leader_election.cc:290] T 1eb2b7e6edd34a7b83f9b918d3cdf2ed P 25286c485fa04c86a5d07c8f88643a47 [CANDIDATE]: Term 2 election: Requested vote from peers 2537bbbd1a9947ae9facd87567e3ccb9 (127.17.60.193:33173), c561ec3374b94b95be0c155d80ff0bff (127.17.60.194:33627)
I20250112 15:07:30.319459 21147 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1eb2b7e6edd34a7b83f9b918d3cdf2ed" candidate_uuid: "25286c485fa04c86a5d07c8f88643a47" candidate_term: 2 candidate_status { last_received { term: 1 index: 28 } } ignore_live_leader: true dest_uuid: "c561ec3374b94b95be0c155d80ff0bff"
I20250112 15:07:30.319548 21147 raft_consensus.cc:3049] T 1eb2b7e6edd34a7b83f9b918d3cdf2ed P c561ec3374b94b95be0c155d80ff0bff [term 1 LEADER]: Stepping down as leader of term 1
I20250112 15:07:30.319579 21147 raft_consensus.cc:738] T 1eb2b7e6edd34a7b83f9b918d3cdf2ed P c561ec3374b94b95be0c155d80ff0bff [term 1 LEADER]: Becoming Follower/Learner. State: Replica: c561ec3374b94b95be0c155d80ff0bff, State: Running, Role: LEADER
I20250112 15:07:30.319628 21147 consensus_queue.cc:260] T 1eb2b7e6edd34a7b83f9b918d3cdf2ed P c561ec3374b94b95be0c155d80ff0bff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 28, Committed index: 28, Last appended: 1.28, Last appended by leader: 28, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "2537bbbd1a9947ae9facd87567e3ccb9" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 33173 } } peers { permanent_uuid: "c561ec3374b94b95be0c155d80ff0bff" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 33627 } } peers { permanent_uuid: "25286c485fa04c86a5d07c8f88643a47" member_type: VOTER last_known_addr { host: "127.17.60.198" port: 37089 } attrs { promote: false } }
I20250112 15:07:30.319713 21147 raft_consensus.cc:3054] T 1eb2b7e6edd34a7b83f9b918d3cdf2ed P c561ec3374b94b95be0c155d80ff0bff [term 1 FOLLOWER]: Advancing to term 2
I20250112 15:07:30.320426 21147 raft_consensus.cc:2463] T 1eb2b7e6edd34a7b83f9b918d3cdf2ed P c561ec3374b94b95be0c155d80ff0bff [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 25286c485fa04c86a5d07c8f88643a47 in term 2.
I20250112 15:07:30.321128 21589 leader_election.cc:304] T 1eb2b7e6edd34a7b83f9b918d3cdf2ed P 25286c485fa04c86a5d07c8f88643a47 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 25286c485fa04c86a5d07c8f88643a47, c561ec3374b94b95be0c155d80ff0bff; no voters:
I20250112 15:07:30.321331 21727 raft_consensus.cc:2798] T 1eb2b7e6edd34a7b83f9b918d3cdf2ed P 25286c485fa04c86a5d07c8f88643a47 [term 2 FOLLOWER]: Leader election won for term 2
I20250112 15:07:30.321463 21727 raft_consensus.cc:695] T 1eb2b7e6edd34a7b83f9b918d3cdf2ed P 25286c485fa04c86a5d07c8f88643a47 [term 2 LEADER]: Becoming Leader. State: Replica: 25286c485fa04c86a5d07c8f88643a47, State: Running, Role: LEADER
I20250112 15:07:30.321535 21727 consensus_queue.cc:237] T 1eb2b7e6edd34a7b83f9b918d3cdf2ed P 25286c485fa04c86a5d07c8f88643a47 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 28, Committed index: 28, Last appended: 1.28, Last appended by leader: 28, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "2537bbbd1a9947ae9facd87567e3ccb9" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 33173 } } peers { permanent_uuid: "c561ec3374b94b95be0c155d80ff0bff" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 33627 } } peers { permanent_uuid: "25286c485fa04c86a5d07c8f88643a47" member_type: VOTER last_known_addr { host: "127.17.60.198" port: 37089 } attrs { promote: false } }
I20250112 15:07:30.321997 20970 catalog_manager.cc:5526] T 1eb2b7e6edd34a7b83f9b918d3cdf2ed P 25286c485fa04c86a5d07c8f88643a47 reported cstate change: term changed from 1 to 2, leader changed from c561ec3374b94b95be0c155d80ff0bff (127.17.60.194) to 25286c485fa04c86a5d07c8f88643a47 (127.17.60.198). New cstate: current_term: 2 leader_uuid: "25286c485fa04c86a5d07c8f88643a47" committed_config { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "2537bbbd1a9947ae9facd87567e3ccb9" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 33173 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c561ec3374b94b95be0c155d80ff0bff" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 33627 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "25286c485fa04c86a5d07c8f88643a47" member_type: VOTER last_known_addr { host: "127.17.60.198" port: 37089 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20250112 15:07:30.322620 21069 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1eb2b7e6edd34a7b83f9b918d3cdf2ed" candidate_uuid: "25286c485fa04c86a5d07c8f88643a47" candidate_term: 2 candidate_status { last_received { term: 1 index: 28 } } ignore_live_leader: true dest_uuid: "2537bbbd1a9947ae9facd87567e3ccb9"
I20250112 15:07:30.322690 21069 raft_consensus.cc:3054] T 1eb2b7e6edd34a7b83f9b918d3cdf2ed P 2537bbbd1a9947ae9facd87567e3ccb9 [term 1 FOLLOWER]: Advancing to term 2
I20250112 15:07:30.323304 21069 raft_consensus.cc:2463] T 1eb2b7e6edd34a7b83f9b918d3cdf2ed P 2537bbbd1a9947ae9facd87567e3ccb9 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 25286c485fa04c86a5d07c8f88643a47 in term 2.
I20250112 15:07:30.646237 21730 ts_tablet_manager.cc:927] T 80cc505a04394c53a18c5beeded4655d P 25286c485fa04c86a5d07c8f88643a47: Initiating tablet copy from peer c561ec3374b94b95be0c155d80ff0bff (127.17.60.194:33627)
I20250112 15:07:30.646466 21730 tablet_copy_client.cc:323] T 80cc505a04394c53a18c5beeded4655d P 25286c485fa04c86a5d07c8f88643a47: tablet copy: Beginning tablet copy session from remote peer at address 127.17.60.194:33627
I20250112 15:07:30.646718 21161 tablet_copy_service.cc:140] P c561ec3374b94b95be0c155d80ff0bff: Received BeginTabletCopySession request for tablet 80cc505a04394c53a18c5beeded4655d from peer 25286c485fa04c86a5d07c8f88643a47 ({username='slave'} at 127.0.0.1:48116)
I20250112 15:07:30.646783 21161 tablet_copy_service.cc:161] P c561ec3374b94b95be0c155d80ff0bff: Beginning new tablet copy session on tablet 80cc505a04394c53a18c5beeded4655d from peer 25286c485fa04c86a5d07c8f88643a47 at {username='slave'} at 127.0.0.1:48116: session id = 25286c485fa04c86a5d07c8f88643a47-80cc505a04394c53a18c5beeded4655d
I20250112 15:07:30.647231 21161 tablet_copy_source_session.cc:215] T 80cc505a04394c53a18c5beeded4655d P c561ec3374b94b95be0c155d80ff0bff: Tablet Copy: opened 0 blocks and 1 log segments
I20250112 15:07:30.647657 21730 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 80cc505a04394c53a18c5beeded4655d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 15:07:30.648815 21730 tablet_copy_client.cc:806] T 80cc505a04394c53a18c5beeded4655d P 25286c485fa04c86a5d07c8f88643a47: tablet copy: Starting download of 0 data blocks...
I20250112 15:07:30.648941 21730 tablet_copy_client.cc:670] T 80cc505a04394c53a18c5beeded4655d P 25286c485fa04c86a5d07c8f88643a47: tablet copy: Starting download of 1 WAL segments...
I20250112 15:07:30.649425 21731 tablet_copy_client.cc:1025] T 80cc505a04394c53a18c5beeded4655d P 25286c485fa04c86a5d07c8f88643a47: tablet copy: Injecting latency into file download: 2000
I20250112 15:07:30.767957 21732 ts_tablet_manager.cc:927] T 69c23627ab854000b360030c384dc966 P b5e9178a7a83402e8c4a6943ab78589d: Initiating tablet copy from peer c561ec3374b94b95be0c155d80ff0bff (127.17.60.194:33627)
I20250112 15:07:30.768158 21732 tablet_copy_client.cc:323] T 69c23627ab854000b360030c384dc966 P b5e9178a7a83402e8c4a6943ab78589d: tablet copy: Beginning tablet copy session from remote peer at address 127.17.60.194:33627
I20250112 15:07:30.771425 21161 tablet_copy_service.cc:140] P c561ec3374b94b95be0c155d80ff0bff: Received BeginTabletCopySession request for tablet 69c23627ab854000b360030c384dc966 from peer b5e9178a7a83402e8c4a6943ab78589d ({username='slave'} at 127.0.0.1:58510)
I20250112 15:07:30.771535 21161 tablet_copy_service.cc:161] P c561ec3374b94b95be0c155d80ff0bff: Beginning new tablet copy session on tablet 69c23627ab854000b360030c384dc966 from peer b5e9178a7a83402e8c4a6943ab78589d at {username='slave'} at 127.0.0.1:58510: session id = b5e9178a7a83402e8c4a6943ab78589d-69c23627ab854000b360030c384dc966
I20250112 15:07:30.772223 21161 tablet_copy_source_session.cc:215] T 69c23627ab854000b360030c384dc966 P c561ec3374b94b95be0c155d80ff0bff: Tablet Copy: opened 0 blocks and 1 log segments
I20250112 15:07:30.772600 21732 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 69c23627ab854000b360030c384dc966. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 15:07:30.774070 21732 tablet_copy_client.cc:806] T 69c23627ab854000b360030c384dc966 P b5e9178a7a83402e8c4a6943ab78589d: tablet copy: Starting download of 0 data blocks...
I20250112 15:07:30.774259 21732 tablet_copy_client.cc:670] T 69c23627ab854000b360030c384dc966 P b5e9178a7a83402e8c4a6943ab78589d: tablet copy: Starting download of 1 WAL segments...
I20250112 15:07:30.774792 21733 tablet_copy_client.cc:1025] T 69c23627ab854000b360030c384dc966 P b5e9178a7a83402e8c4a6943ab78589d: tablet copy: Injecting latency into file download: 2000
I20250112 15:07:30.795411 21068 raft_consensus.cc:1270] T 1eb2b7e6edd34a7b83f9b918d3cdf2ed P 2537bbbd1a9947ae9facd87567e3ccb9 [term 2 FOLLOWER]: Refusing update from remote peer 25286c485fa04c86a5d07c8f88643a47: Log matching property violated. Preceding OpId in replica: term: 1 index: 28. Preceding OpId from leader: term: 2 index: 29. (index mismatch)
I20250112 15:07:30.795627 21727 consensus_queue.cc:1035] T 1eb2b7e6edd34a7b83f9b918d3cdf2ed P 25286c485fa04c86a5d07c8f88643a47 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2537bbbd1a9947ae9facd87567e3ccb9" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 33173 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.000s
I20250112 15:07:30.796890 21147 raft_consensus.cc:1270] T 1eb2b7e6edd34a7b83f9b918d3cdf2ed P c561ec3374b94b95be0c155d80ff0bff [term 2 FOLLOWER]: Refusing update from remote peer 25286c485fa04c86a5d07c8f88643a47: Log matching property violated. Preceding OpId in replica: term: 1 index: 28. Preceding OpId from leader: term: 2 index: 29. (index mismatch)
I20250112 15:07:30.797178 21727 consensus_queue.cc:1035] T 1eb2b7e6edd34a7b83f9b918d3cdf2ed P 25286c485fa04c86a5d07c8f88643a47 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c561ec3374b94b95be0c155d80ff0bff" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 33627 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.000s
I20250112 15:07:30.798110 21741 ts_tablet_manager.cc:927] T 9601bcda24ed42e7b93ab9740f1e271e P a2a5bba4ac1145d5bab94a69c9f4df40: Initiating tablet copy from peer 2537bbbd1a9947ae9facd87567e3ccb9 (127.17.60.193:33173)
I20250112 15:07:30.798282 21741 tablet_copy_client.cc:323] T 9601bcda24ed42e7b93ab9740f1e271e P a2a5bba4ac1145d5bab94a69c9f4df40: tablet copy: Beginning tablet copy session from remote peer at address 127.17.60.193:33173
I20250112 15:07:30.801625 21085 tablet_copy_service.cc:140] P 2537bbbd1a9947ae9facd87567e3ccb9: Received BeginTabletCopySession request for tablet 9601bcda24ed42e7b93ab9740f1e271e from peer a2a5bba4ac1145d5bab94a69c9f4df40 ({username='slave'} at 127.0.0.1:54442)
I20250112 15:07:30.801689 21085 tablet_copy_service.cc:161] P 2537bbbd1a9947ae9facd87567e3ccb9: Beginning new tablet copy session on tablet 9601bcda24ed42e7b93ab9740f1e271e from peer a2a5bba4ac1145d5bab94a69c9f4df40 at {username='slave'} at 127.0.0.1:54442: session id = a2a5bba4ac1145d5bab94a69c9f4df40-9601bcda24ed42e7b93ab9740f1e271e
I20250112 15:07:30.802188 21085 tablet_copy_source_session.cc:215] T 9601bcda24ed42e7b93ab9740f1e271e P 2537bbbd1a9947ae9facd87567e3ccb9: Tablet Copy: opened 0 blocks and 1 log segments
I20250112 15:07:30.802482 21741 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9601bcda24ed42e7b93ab9740f1e271e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 15:07:30.803686 21741 tablet_copy_client.cc:806] T 9601bcda24ed42e7b93ab9740f1e271e P a2a5bba4ac1145d5bab94a69c9f4df40: tablet copy: Starting download of 0 data blocks...
I20250112 15:07:30.803848 21741 tablet_copy_client.cc:670] T 9601bcda24ed42e7b93ab9740f1e271e P a2a5bba4ac1145d5bab94a69c9f4df40: tablet copy: Starting download of 1 WAL segments...
I20250112 15:07:30.804451 21742 tablet_copy_client.cc:1025] T 9601bcda24ed42e7b93ab9740f1e271e P a2a5bba4ac1145d5bab94a69c9f4df40: tablet copy: Injecting latency into file download: 2000
I20250112 15:07:30.839686 21222 raft_consensus.cc:1270] T ef90b704208140a3983ffe6105939319 P 3710b74ed2044ce5995533df32e67351 [term 2 FOLLOWER]: Refusing update from remote peer c561ec3374b94b95be0c155d80ff0bff: Log matching property violated. Preceding OpId in replica: term: 1 index: 25. Preceding OpId from leader: term: 2 index: 26. (index mismatch)
I20250112 15:07:30.839965 21424 consensus_queue.cc:1035] T ef90b704208140a3983ffe6105939319 P c561ec3374b94b95be0c155d80ff0bff [LEADER]: Connected to new peer: Peer: permanent_uuid: "3710b74ed2044ce5995533df32e67351" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 38589 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 26, Last known committed idx: 25, Time since last communication: 0.000s
I20250112 15:07:30.841326 21068 raft_consensus.cc:1270] T ef90b704208140a3983ffe6105939319 P 2537bbbd1a9947ae9facd87567e3ccb9 [term 2 FOLLOWER]: Refusing update from remote peer c561ec3374b94b95be0c155d80ff0bff: Log matching property violated. Preceding OpId in replica: term: 1 index: 25. Preceding OpId from leader: term: 2 index: 26. (index mismatch)
I20250112 15:07:30.841529 21424 consensus_queue.cc:1035] T ef90b704208140a3983ffe6105939319 P c561ec3374b94b95be0c155d80ff0bff [LEADER]: Connected to new peer: Peer: permanent_uuid: "2537bbbd1a9947ae9facd87567e3ccb9" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 33173 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 26, Last known committed idx: 25, Time since last communication: 0.000s
I20250112 15:07:30.844756 21748 ts_tablet_manager.cc:927] T 5bcec60d3b974a9bb24aedfc738e1594 P a2a5bba4ac1145d5bab94a69c9f4df40: Initiating tablet copy from peer 2537bbbd1a9947ae9facd87567e3ccb9 (127.17.60.193:33173)
I20250112 15:07:30.844942 21748 tablet_copy_client.cc:323] T 5bcec60d3b974a9bb24aedfc738e1594 P a2a5bba4ac1145d5bab94a69c9f4df40: tablet copy: Beginning tablet copy session from remote peer at address 127.17.60.193:33173
I20250112 15:07:30.845122 21085 tablet_copy_service.cc:140] P 2537bbbd1a9947ae9facd87567e3ccb9: Received BeginTabletCopySession request for tablet 5bcec60d3b974a9bb24aedfc738e1594 from peer a2a5bba4ac1145d5bab94a69c9f4df40 ({username='slave'} at 127.0.0.1:54442)
I20250112 15:07:30.845209 21085 tablet_copy_service.cc:161] P 2537bbbd1a9947ae9facd87567e3ccb9: Beginning new tablet copy session on tablet 5bcec60d3b974a9bb24aedfc738e1594 from peer a2a5bba4ac1145d5bab94a69c9f4df40 at {username='slave'} at 127.0.0.1:54442: session id = a2a5bba4ac1145d5bab94a69c9f4df40-5bcec60d3b974a9bb24aedfc738e1594
I20250112 15:07:30.845849 21085 tablet_copy_source_session.cc:215] T 5bcec60d3b974a9bb24aedfc738e1594 P 2537bbbd1a9947ae9facd87567e3ccb9: Tablet Copy: opened 0 blocks and 1 log segments
I20250112 15:07:30.846158 21748 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5bcec60d3b974a9bb24aedfc738e1594. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 15:07:30.847507 21748 tablet_copy_client.cc:806] T 5bcec60d3b974a9bb24aedfc738e1594 P a2a5bba4ac1145d5bab94a69c9f4df40: tablet copy: Starting download of 0 data blocks...
I20250112 15:07:30.847664 21748 tablet_copy_client.cc:670] T 5bcec60d3b974a9bb24aedfc738e1594 P a2a5bba4ac1145d5bab94a69c9f4df40: tablet copy: Starting download of 1 WAL segments...
I20250112 15:07:30.848244 21749 tablet_copy_client.cc:1025] T 5bcec60d3b974a9bb24aedfc738e1594 P a2a5bba4ac1145d5bab94a69c9f4df40: tablet copy: Injecting latency into file download: 2000
I20250112 15:07:30.849855 21750 ts_tablet_manager.cc:927] T 57e47b07fdcf434e96ca5dd0354e93b2 P b5e9178a7a83402e8c4a6943ab78589d: Initiating tablet copy from peer 2537bbbd1a9947ae9facd87567e3ccb9 (127.17.60.193:33173)
I20250112 15:07:30.850020 21750 tablet_copy_client.cc:323] T 57e47b07fdcf434e96ca5dd0354e93b2 P b5e9178a7a83402e8c4a6943ab78589d: tablet copy: Beginning tablet copy session from remote peer at address 127.17.60.193:33173
I20250112 15:07:30.852375 21085 tablet_copy_service.cc:140] P 2537bbbd1a9947ae9facd87567e3ccb9: Received BeginTabletCopySession request for tablet 57e47b07fdcf434e96ca5dd0354e93b2 from peer b5e9178a7a83402e8c4a6943ab78589d ({username='slave'} at 127.0.0.1:54448)
I20250112 15:07:30.852429 21085 tablet_copy_service.cc:161] P 2537bbbd1a9947ae9facd87567e3ccb9: Beginning new tablet copy session on tablet 57e47b07fdcf434e96ca5dd0354e93b2 from peer b5e9178a7a83402e8c4a6943ab78589d at {username='slave'} at 127.0.0.1:54448: session id = b5e9178a7a83402e8c4a6943ab78589d-57e47b07fdcf434e96ca5dd0354e93b2
I20250112 15:07:30.852984 21085 tablet_copy_source_session.cc:215] T 57e47b07fdcf434e96ca5dd0354e93b2 P 2537bbbd1a9947ae9facd87567e3ccb9: Tablet Copy: opened 0 blocks and 1 log segments
I20250112 15:07:30.853299 21750 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 57e47b07fdcf434e96ca5dd0354e93b2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 15:07:30.854700 21750 tablet_copy_client.cc:806] T 57e47b07fdcf434e96ca5dd0354e93b2 P b5e9178a7a83402e8c4a6943ab78589d: tablet copy: Starting download of 0 data blocks...
I20250112 15:07:30.854878 21750 tablet_copy_client.cc:670] T 57e47b07fdcf434e96ca5dd0354e93b2 P b5e9178a7a83402e8c4a6943ab78589d: tablet copy: Starting download of 1 WAL segments...
I20250112 15:07:30.855369 21751 tablet_copy_client.cc:1025] T 57e47b07fdcf434e96ca5dd0354e93b2 P b5e9178a7a83402e8c4a6943ab78589d: tablet copy: Injecting latency into file download: 2000
I20250112 15:07:32.081460 21026 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250112 15:07:32.081851 21147 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "9d4bbba799324d7fbbac082520c65686"
dest_uuid: "c561ec3374b94b95be0c155d80ff0bff"
mode: GRACEFUL
new_leader_uuid: "2537bbbd1a9947ae9facd87567e3ccb9"
from {username='slave'} at 127.0.0.1:48104
I20250112 15:07:32.081949 21147 raft_consensus.cc:604] T 9d4bbba799324d7fbbac082520c65686 P c561ec3374b94b95be0c155d80ff0bff [term 1 LEADER]: Received request to transfer leadership to TS 2537bbbd1a9947ae9facd87567e3ccb9
I20250112 15:07:32.082110 21026 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 1
I20250112 15:07:32.082185 21026 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20250112 15:07:32.172391 21745 raft_consensus.cc:988] T 9d4bbba799324d7fbbac082520c65686 P c561ec3374b94b95be0c155d80ff0bff: : Instructing follower 2537bbbd1a9947ae9facd87567e3ccb9 to start an election
I20250112 15:07:32.172501 21745 raft_consensus.cc:1076] T 9d4bbba799324d7fbbac082520c65686 P c561ec3374b94b95be0c155d80ff0bff [term 1 LEADER]: Signalling peer 2537bbbd1a9947ae9facd87567e3ccb9 to start an election
I20250112 15:07:32.172677 21068 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "9d4bbba799324d7fbbac082520c65686"
dest_uuid: "2537bbbd1a9947ae9facd87567e3ccb9"
from {username='slave'} at 127.0.0.1:37216
I20250112 15:07:32.172780 21068 raft_consensus.cc:491] T 9d4bbba799324d7fbbac082520c65686 P 2537bbbd1a9947ae9facd87567e3ccb9 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20250112 15:07:32.172816 21068 raft_consensus.cc:3054] T 9d4bbba799324d7fbbac082520c65686 P 2537bbbd1a9947ae9facd87567e3ccb9 [term 1 FOLLOWER]: Advancing to term 2
I20250112 15:07:32.173712 21068 raft_consensus.cc:513] T 9d4bbba799324d7fbbac082520c65686 P 2537bbbd1a9947ae9facd87567e3ccb9 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2537bbbd1a9947ae9facd87567e3ccb9" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 33173 } } peers { permanent_uuid: "c561ec3374b94b95be0c155d80ff0bff" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 33627 } } peers { permanent_uuid: "3710b74ed2044ce5995533df32e67351" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 38589 } }
I20250112 15:07:32.173862 21068 leader_election.cc:290] T 9d4bbba799324d7fbbac082520c65686 P 2537bbbd1a9947ae9facd87567e3ccb9 [CANDIDATE]: Term 2 election: Requested vote from peers c561ec3374b94b95be0c155d80ff0bff (127.17.60.194:33627), 3710b74ed2044ce5995533df32e67351 (127.17.60.195:38589)
I20250112 15:07:32.173942 21149 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9d4bbba799324d7fbbac082520c65686" candidate_uuid: "2537bbbd1a9947ae9facd87567e3ccb9" candidate_term: 2 candidate_status { last_received { term: 1 index: 24 } } ignore_live_leader: true dest_uuid: "c561ec3374b94b95be0c155d80ff0bff"
I20250112 15:07:32.174033 21149 raft_consensus.cc:3049] T 9d4bbba799324d7fbbac082520c65686 P c561ec3374b94b95be0c155d80ff0bff [term 1 LEADER]: Stepping down as leader of term 1
I20250112 15:07:32.174077 21149 raft_consensus.cc:738] T 9d4bbba799324d7fbbac082520c65686 P c561ec3374b94b95be0c155d80ff0bff [term 1 LEADER]: Becoming Follower/Learner. State: Replica: c561ec3374b94b95be0c155d80ff0bff, State: Running, Role: LEADER
I20250112 15:07:32.174036 21222 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9d4bbba799324d7fbbac082520c65686" candidate_uuid: "2537bbbd1a9947ae9facd87567e3ccb9" candidate_term: 2 candidate_status { last_received { term: 1 index: 24 } } ignore_live_leader: true dest_uuid: "3710b74ed2044ce5995533df32e67351"
I20250112 15:07:32.174108 21222 raft_consensus.cc:3054] T 9d4bbba799324d7fbbac082520c65686 P 3710b74ed2044ce5995533df32e67351 [term 1 FOLLOWER]: Advancing to term 2
I20250112 15:07:32.174125 21149 consensus_queue.cc:260] T 9d4bbba799324d7fbbac082520c65686 P c561ec3374b94b95be0c155d80ff0bff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 24, Committed index: 24, Last appended: 1.24, Last appended by leader: 24, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2537bbbd1a9947ae9facd87567e3ccb9" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 33173 } } peers { permanent_uuid: "c561ec3374b94b95be0c155d80ff0bff" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 33627 } } peers { permanent_uuid: "3710b74ed2044ce5995533df32e67351" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 38589 } }
I20250112 15:07:32.174198 21149 raft_consensus.cc:3054] T 9d4bbba799324d7fbbac082520c65686 P c561ec3374b94b95be0c155d80ff0bff [term 1 FOLLOWER]: Advancing to term 2
I20250112 15:07:32.174832 21222 raft_consensus.cc:2463] T 9d4bbba799324d7fbbac082520c65686 P 3710b74ed2044ce5995533df32e67351 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2537bbbd1a9947ae9facd87567e3ccb9 in term 2.
I20250112 15:07:32.174832 21149 raft_consensus.cc:2463] T 9d4bbba799324d7fbbac082520c65686 P c561ec3374b94b95be0c155d80ff0bff [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2537bbbd1a9947ae9facd87567e3ccb9 in term 2.
I20250112 15:07:32.175004 21039 leader_election.cc:304] T 9d4bbba799324d7fbbac082520c65686 P 2537bbbd1a9947ae9facd87567e3ccb9 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2537bbbd1a9947ae9facd87567e3ccb9, 3710b74ed2044ce5995533df32e67351; no voters:
I20250112 15:07:32.175093 21754 raft_consensus.cc:2798] T 9d4bbba799324d7fbbac082520c65686 P 2537bbbd1a9947ae9facd87567e3ccb9 [term 2 FOLLOWER]: Leader election won for term 2
I20250112 15:07:32.175158 21754 raft_consensus.cc:695] T 9d4bbba799324d7fbbac082520c65686 P 2537bbbd1a9947ae9facd87567e3ccb9 [term 2 LEADER]: Becoming Leader. State: Replica: 2537bbbd1a9947ae9facd87567e3ccb9, State: Running, Role: LEADER
I20250112 15:07:32.175208 21754 consensus_queue.cc:237] T 9d4bbba799324d7fbbac082520c65686 P 2537bbbd1a9947ae9facd87567e3ccb9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 24, Committed index: 24, Last appended: 1.24, Last appended by leader: 24, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2537bbbd1a9947ae9facd87567e3ccb9" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 33173 } } peers { permanent_uuid: "c561ec3374b94b95be0c155d80ff0bff" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 33627 } } peers { permanent_uuid: "3710b74ed2044ce5995533df32e67351" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 38589 } }
I20250112 15:07:32.175784 20966 catalog_manager.cc:5526] T 9d4bbba799324d7fbbac082520c65686 P 2537bbbd1a9947ae9facd87567e3ccb9 reported cstate change: term changed from 1 to 2, leader changed from c561ec3374b94b95be0c155d80ff0bff (127.17.60.194) to 2537bbbd1a9947ae9facd87567e3ccb9 (127.17.60.193). New cstate: current_term: 2 leader_uuid: "2537bbbd1a9947ae9facd87567e3ccb9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2537bbbd1a9947ae9facd87567e3ccb9" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 33173 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c561ec3374b94b95be0c155d80ff0bff" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 33627 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3710b74ed2044ce5995533df32e67351" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 38589 } health_report { overall_health: UNKNOWN } } }
I20250112 15:07:32.649652 21730 tablet_copy_client.cc:538] T 80cc505a04394c53a18c5beeded4655d P 25286c485fa04c86a5d07c8f88643a47: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250112 15:07:32.650792 21730 tablet_bootstrap.cc:492] T 80cc505a04394c53a18c5beeded4655d P 25286c485fa04c86a5d07c8f88643a47: Bootstrap starting.
I20250112 15:07:32.655125 21730 tablet_bootstrap.cc:492] T 80cc505a04394c53a18c5beeded4655d P 25286c485fa04c86a5d07c8f88643a47: Bootstrap replayed 1/1 log segments. Stats: ops{read=26 overwritten=0 applied=26 ignored=0} inserts{seen=125 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250112 15:07:32.655329 21730 tablet_bootstrap.cc:492] T 80cc505a04394c53a18c5beeded4655d P 25286c485fa04c86a5d07c8f88643a47: Bootstrap complete.
I20250112 15:07:32.655431 21730 ts_tablet_manager.cc:1397] T 80cc505a04394c53a18c5beeded4655d P 25286c485fa04c86a5d07c8f88643a47: Time spent bootstrapping tablet: real 0.005s user 0.005s sys 0.000s
I20250112 15:07:32.655584 21730 raft_consensus.cc:357] T 80cc505a04394c53a18c5beeded4655d P 25286c485fa04c86a5d07c8f88643a47 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 26 OBSOLETE_local: false peers { permanent_uuid: "2537bbbd1a9947ae9facd87567e3ccb9" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 33173 } attrs { replace: true } } peers { permanent_uuid: "c561ec3374b94b95be0c155d80ff0bff" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 33627 } } peers { permanent_uuid: "3710b74ed2044ce5995533df32e67351" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 38589 } } peers { permanent_uuid: "25286c485fa04c86a5d07c8f88643a47" member_type: NON_VOTER last_known_addr { host: "127.17.60.198" port: 37089 } attrs { promote: true } }
I20250112 15:07:32.655663 21730 raft_consensus.cc:738] T 80cc505a04394c53a18c5beeded4655d P 25286c485fa04c86a5d07c8f88643a47 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 25286c485fa04c86a5d07c8f88643a47, State: Initialized, Role: LEARNER
I20250112 15:07:32.655690 21730 consensus_queue.cc:260] T 80cc505a04394c53a18c5beeded4655d P 25286c485fa04c86a5d07c8f88643a47 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 26, Last appended: 1.26, Last appended by leader: 26, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 26 OBSOLETE_local: false peers { permanent_uuid: "2537bbbd1a9947ae9facd87567e3ccb9" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 33173 } attrs { replace: true } } peers { permanent_uuid: "c561ec3374b94b95be0c155d80ff0bff" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 33627 } } peers { permanent_uuid: "3710b74ed2044ce5995533df32e67351" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 38589 } } peers { permanent_uuid: "25286c485fa04c86a5d07c8f88643a47" member_type: NON_VOTER last_known_addr { host: "127.17.60.198" port: 37089 } attrs { promote: true } }
I20250112 15:07:32.655848 21730 ts_tablet_manager.cc:1428] T 80cc505a04394c53a18c5beeded4655d P 25286c485fa04c86a5d07c8f88643a47: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20250112 15:07:32.656006 21161 tablet_copy_service.cc:342] P c561ec3374b94b95be0c155d80ff0bff: Request end of tablet copy session 25286c485fa04c86a5d07c8f88643a47-80cc505a04394c53a18c5beeded4655d received from {username='slave'} at 127.0.0.1:48116
I20250112 15:07:32.656056 21161 tablet_copy_service.cc:434] P c561ec3374b94b95be0c155d80ff0bff: ending tablet copy session 25286c485fa04c86a5d07c8f88643a47-80cc505a04394c53a18c5beeded4655d on tablet 80cc505a04394c53a18c5beeded4655d with peer 25286c485fa04c86a5d07c8f88643a47
I20250112 15:07:32.666532 21222 raft_consensus.cc:1270] T 9d4bbba799324d7fbbac082520c65686 P 3710b74ed2044ce5995533df32e67351 [term 2 FOLLOWER]: Refusing update from remote peer 2537bbbd1a9947ae9facd87567e3ccb9: Log matching property violated. Preceding OpId in replica: term: 1 index: 24. Preceding OpId from leader: term: 2 index: 25. (index mismatch)
I20250112 15:07:32.666705 21754 consensus_queue.cc:1035] T 9d4bbba799324d7fbbac082520c65686 P 2537bbbd1a9947ae9facd87567e3ccb9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3710b74ed2044ce5995533df32e67351" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 38589 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 25, Last known committed idx: 24, Time since last communication: 0.000s
I20250112 15:07:32.667670 21149 raft_consensus.cc:1270] T 9d4bbba799324d7fbbac082520c65686 P c561ec3374b94b95be0c155d80ff0bff [term 2 FOLLOWER]: Refusing update from remote peer 2537bbbd1a9947ae9facd87567e3ccb9: Log matching property violated. Preceding OpId in replica: term: 1 index: 24. Preceding OpId from leader: term: 2 index: 25. (index mismatch)
I20250112 15:07:32.667857 21716 consensus_queue.cc:1035] T 9d4bbba799324d7fbbac082520c65686 P 2537bbbd1a9947ae9facd87567e3ccb9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c561ec3374b94b95be0c155d80ff0bff" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 33627 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 25, Last known committed idx: 24, Time since last communication: 0.000s
I20250112 15:07:32.774992 21732 tablet_copy_client.cc:538] T 69c23627ab854000b360030c384dc966 P b5e9178a7a83402e8c4a6943ab78589d: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250112 15:07:32.775947 21732 tablet_bootstrap.cc:492] T 69c23627ab854000b360030c384dc966 P b5e9178a7a83402e8c4a6943ab78589d: Bootstrap starting.
I20250112 15:07:32.779723 21732 tablet_bootstrap.cc:492] T 69c23627ab854000b360030c384dc966 P b5e9178a7a83402e8c4a6943ab78589d: Bootstrap replayed 1/1 log segments. Stats: ops{read=25 overwritten=0 applied=25 ignored=0} inserts{seen=103 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250112 15:07:32.779899 21732 tablet_bootstrap.cc:492] T 69c23627ab854000b360030c384dc966 P b5e9178a7a83402e8c4a6943ab78589d: Bootstrap complete.
I20250112 15:07:32.780001 21732 ts_tablet_manager.cc:1397] T 69c23627ab854000b360030c384dc966 P b5e9178a7a83402e8c4a6943ab78589d: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.002s
I20250112 15:07:32.780169 21732 raft_consensus.cc:357] T 69c23627ab854000b360030c384dc966 P b5e9178a7a83402e8c4a6943ab78589d [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 25 OBSOLETE_local: false peers { permanent_uuid: "2537bbbd1a9947ae9facd87567e3ccb9" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 33173 } attrs { replace: true } } peers { permanent_uuid: "c561ec3374b94b95be0c155d80ff0bff" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 33627 } } peers { permanent_uuid: "3710b74ed2044ce5995533df32e67351" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 38589 } } peers { permanent_uuid: "b5e9178a7a83402e8c4a6943ab78589d" member_type: NON_VOTER last_known_addr { host: "127.17.60.197" port: 36573 } attrs { promote: true } }
I20250112 15:07:32.780273 21732 raft_consensus.cc:738] T 69c23627ab854000b360030c384dc966 P b5e9178a7a83402e8c4a6943ab78589d [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: b5e9178a7a83402e8c4a6943ab78589d, State: Initialized, Role: LEARNER
I20250112 15:07:32.780305 21732 consensus_queue.cc:260] T 69c23627ab854000b360030c384dc966 P b5e9178a7a83402e8c4a6943ab78589d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 25, Last appended: 1.25, Last appended by leader: 25, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 25 OBSOLETE_local: false peers { permanent_uuid: "2537bbbd1a9947ae9facd87567e3ccb9" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 33173 } attrs { replace: true } } peers { permanent_uuid: "c561ec3374b94b95be0c155d80ff0bff" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 33627 } } peers { permanent_uuid: "3710b74ed2044ce5995533df32e67351" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 38589 } } peers { permanent_uuid: "b5e9178a7a83402e8c4a6943ab78589d" member_type: NON_VOTER last_known_addr { host: "127.17.60.197" port: 36573 } attrs { promote: true } }
I20250112 15:07:32.780442 21732 ts_tablet_manager.cc:1428] T 69c23627ab854000b360030c384dc966 P b5e9178a7a83402e8c4a6943ab78589d: Time spent starting tablet: real 0.000s user 0.001s sys 0.001s
I20250112 15:07:32.780634 21161 tablet_copy_service.cc:342] P c561ec3374b94b95be0c155d80ff0bff: Request end of tablet copy session b5e9178a7a83402e8c4a6943ab78589d-69c23627ab854000b360030c384dc966 received from {username='slave'} at 127.0.0.1:58510
I20250112 15:07:32.780697 21161 tablet_copy_service.cc:434] P c561ec3374b94b95be0c155d80ff0bff: ending tablet copy session b5e9178a7a83402e8c4a6943ab78589d-69c23627ab854000b360030c384dc966 on tablet 69c23627ab854000b360030c384dc966 with peer b5e9178a7a83402e8c4a6943ab78589d
I20250112 15:07:32.804678 21741 tablet_copy_client.cc:538] T 9601bcda24ed42e7b93ab9740f1e271e P a2a5bba4ac1145d5bab94a69c9f4df40: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250112 15:07:32.805678 21741 tablet_bootstrap.cc:492] T 9601bcda24ed42e7b93ab9740f1e271e P a2a5bba4ac1145d5bab94a69c9f4df40: Bootstrap starting.
I20250112 15:07:32.809664 21741 tablet_bootstrap.cc:492] T 9601bcda24ed42e7b93ab9740f1e271e P a2a5bba4ac1145d5bab94a69c9f4df40: Bootstrap replayed 1/1 log segments. Stats: ops{read=25 overwritten=0 applied=25 ignored=0} inserts{seen=96 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250112 15:07:32.809868 21741 tablet_bootstrap.cc:492] T 9601bcda24ed42e7b93ab9740f1e271e P a2a5bba4ac1145d5bab94a69c9f4df40: Bootstrap complete.
I20250112 15:07:32.809983 21741 ts_tablet_manager.cc:1397] T 9601bcda24ed42e7b93ab9740f1e271e P a2a5bba4ac1145d5bab94a69c9f4df40: Time spent bootstrapping tablet: real 0.004s user 0.004s sys 0.000s
I20250112 15:07:32.810148 21741 raft_consensus.cc:357] T 9601bcda24ed42e7b93ab9740f1e271e P a2a5bba4ac1145d5bab94a69c9f4df40 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 25 OBSOLETE_local: false peers { permanent_uuid: "2537bbbd1a9947ae9facd87567e3ccb9" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 33173 } } peers { permanent_uuid: "c561ec3374b94b95be0c155d80ff0bff" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 33627 } attrs { replace: true } } peers { permanent_uuid: "3710b74ed2044ce5995533df32e67351" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 38589 } } peers { permanent_uuid: "a2a5bba4ac1145d5bab94a69c9f4df40" member_type: NON_VOTER last_known_addr { host: "127.17.60.196" port: 41807 } attrs { promote: true } }
I20250112 15:07:32.810247 21741 raft_consensus.cc:738] T 9601bcda24ed42e7b93ab9740f1e271e P a2a5bba4ac1145d5bab94a69c9f4df40 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: a2a5bba4ac1145d5bab94a69c9f4df40, State: Initialized, Role: LEARNER
I20250112 15:07:32.810286 21741 consensus_queue.cc:260] T 9601bcda24ed42e7b93ab9740f1e271e P a2a5bba4ac1145d5bab94a69c9f4df40 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 25, Last appended: 1.25, Last appended by leader: 25, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 25 OBSOLETE_local: false peers { permanent_uuid: "2537bbbd1a9947ae9facd87567e3ccb9" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 33173 } } peers { permanent_uuid: "c561ec3374b94b95be0c155d80ff0bff" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 33627 } attrs { replace: true } } peers { permanent_uuid: "3710b74ed2044ce5995533df32e67351" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 38589 } } peers { permanent_uuid: "a2a5bba4ac1145d5bab94a69c9f4df40" member_type: NON_VOTER last_known_addr { host: "127.17.60.196" port: 41807 } attrs { promote: true } }
I20250112 15:07:32.810451 21741 ts_tablet_manager.cc:1428] T 9601bcda24ed42e7b93ab9740f1e271e P a2a5bba4ac1145d5bab94a69c9f4df40: Time spent starting tablet: real 0.000s user 0.002s sys 0.000s
I20250112 15:07:32.810650 21085 tablet_copy_service.cc:342] P 2537bbbd1a9947ae9facd87567e3ccb9: Request end of tablet copy session a2a5bba4ac1145d5bab94a69c9f4df40-9601bcda24ed42e7b93ab9740f1e271e received from {username='slave'} at 127.0.0.1:54442
I20250112 15:07:32.810710 21085 tablet_copy_service.cc:434] P 2537bbbd1a9947ae9facd87567e3ccb9: ending tablet copy session a2a5bba4ac1145d5bab94a69c9f4df40-9601bcda24ed42e7b93ab9740f1e271e on tablet 9601bcda24ed42e7b93ab9740f1e271e with peer a2a5bba4ac1145d5bab94a69c9f4df40
I20250112 15:07:32.818756 21549 raft_consensus.cc:1212] T 69c23627ab854000b360030c384dc966 P b5e9178a7a83402e8c4a6943ab78589d [term 1 LEARNER]: Deduplicated request from leader. Original: 1.24->[1.25-1.25] Dedup: 1.25->[]
I20250112 15:07:32.839630 21624 raft_consensus.cc:1212] T 80cc505a04394c53a18c5beeded4655d P 25286c485fa04c86a5d07c8f88643a47 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.25->[1.26-1.26] Dedup: 1.26->[]
I20250112 15:07:32.848481 21748 tablet_copy_client.cc:538] T 5bcec60d3b974a9bb24aedfc738e1594 P a2a5bba4ac1145d5bab94a69c9f4df40: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250112 15:07:32.849550 21748 tablet_bootstrap.cc:492] T 5bcec60d3b974a9bb24aedfc738e1594 P a2a5bba4ac1145d5bab94a69c9f4df40: Bootstrap starting.
I20250112 15:07:32.853782 21748 tablet_bootstrap.cc:492] T 5bcec60d3b974a9bb24aedfc738e1594 P a2a5bba4ac1145d5bab94a69c9f4df40: Bootstrap replayed 1/1 log segments. Stats: ops{read=26 overwritten=0 applied=26 ignored=0} inserts{seen=88 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250112 15:07:32.854005 21748 tablet_bootstrap.cc:492] T 5bcec60d3b974a9bb24aedfc738e1594 P a2a5bba4ac1145d5bab94a69c9f4df40: Bootstrap complete.
I20250112 15:07:32.854121 21748 ts_tablet_manager.cc:1397] T 5bcec60d3b974a9bb24aedfc738e1594 P a2a5bba4ac1145d5bab94a69c9f4df40: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.002s
I20250112 15:07:32.854298 21748 raft_consensus.cc:357] T 5bcec60d3b974a9bb24aedfc738e1594 P a2a5bba4ac1145d5bab94a69c9f4df40 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 26 OBSOLETE_local: false peers { permanent_uuid: "2537bbbd1a9947ae9facd87567e3ccb9" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 33173 } } peers { permanent_uuid: "c561ec3374b94b95be0c155d80ff0bff" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 33627 } attrs { replace: true } } peers { permanent_uuid: "3710b74ed2044ce5995533df32e67351" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 38589 } } peers { permanent_uuid: "a2a5bba4ac1145d5bab94a69c9f4df40" member_type: NON_VOTER last_known_addr { host: "127.17.60.196" port: 41807 } attrs { promote: true } }
I20250112 15:07:32.854395 21748 raft_consensus.cc:738] T 5bcec60d3b974a9bb24aedfc738e1594 P a2a5bba4ac1145d5bab94a69c9f4df40 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: a2a5bba4ac1145d5bab94a69c9f4df40, State: Initialized, Role: LEARNER
I20250112 15:07:32.854434 21748 consensus_queue.cc:260] T 5bcec60d3b974a9bb24aedfc738e1594 P a2a5bba4ac1145d5bab94a69c9f4df40 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 26, Last appended: 1.26, Last appended by leader: 26, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 26 OBSOLETE_local: false peers { permanent_uuid: "2537bbbd1a9947ae9facd87567e3ccb9" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 33173 } } peers { permanent_uuid: "c561ec3374b94b95be0c155d80ff0bff" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 33627 } attrs { replace: true } } peers { permanent_uuid: "3710b74ed2044ce5995533df32e67351" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 38589 } } peers { permanent_uuid: "a2a5bba4ac1145d5bab94a69c9f4df40" member_type: NON_VOTER last_known_addr { host: "127.17.60.196" port: 41807 } attrs { promote: true } }
I20250112 15:07:32.854528 21748 ts_tablet_manager.cc:1428] T 5bcec60d3b974a9bb24aedfc738e1594 P a2a5bba4ac1145d5bab94a69c9f4df40: Time spent starting tablet: real 0.000s user 0.000s sys 0.002s
I20250112 15:07:32.854717 21085 tablet_copy_service.cc:342] P 2537bbbd1a9947ae9facd87567e3ccb9: Request end of tablet copy session a2a5bba4ac1145d5bab94a69c9f4df40-5bcec60d3b974a9bb24aedfc738e1594 received from {username='slave'} at 127.0.0.1:54442
I20250112 15:07:32.854766 21085 tablet_copy_service.cc:434] P 2537bbbd1a9947ae9facd87567e3ccb9: ending tablet copy session a2a5bba4ac1145d5bab94a69c9f4df40-5bcec60d3b974a9bb24aedfc738e1594 on tablet 5bcec60d3b974a9bb24aedfc738e1594 with peer a2a5bba4ac1145d5bab94a69c9f4df40
I20250112 15:07:32.855558 21750 tablet_copy_client.cc:538] T 57e47b07fdcf434e96ca5dd0354e93b2 P b5e9178a7a83402e8c4a6943ab78589d: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250112 15:07:32.856360 21750 tablet_bootstrap.cc:492] T 57e47b07fdcf434e96ca5dd0354e93b2 P b5e9178a7a83402e8c4a6943ab78589d: Bootstrap starting.
I20250112 15:07:32.859007 21750 tablet_bootstrap.cc:492] T 57e47b07fdcf434e96ca5dd0354e93b2 P b5e9178a7a83402e8c4a6943ab78589d: Bootstrap replayed 1/1 log segments. Stats: ops{read=27 overwritten=0 applied=27 ignored=0} inserts{seen=83 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250112 15:07:32.859187 21750 tablet_bootstrap.cc:492] T 57e47b07fdcf434e96ca5dd0354e93b2 P b5e9178a7a83402e8c4a6943ab78589d: Bootstrap complete.
I20250112 15:07:32.859284 21750 ts_tablet_manager.cc:1397] T 57e47b07fdcf434e96ca5dd0354e93b2 P b5e9178a7a83402e8c4a6943ab78589d: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20250112 15:07:32.859423 21750 raft_consensus.cc:357] T 57e47b07fdcf434e96ca5dd0354e93b2 P b5e9178a7a83402e8c4a6943ab78589d [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "2537bbbd1a9947ae9facd87567e3ccb9" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 33173 } } peers { permanent_uuid: "c561ec3374b94b95be0c155d80ff0bff" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 33627 } } peers { permanent_uuid: "3710b74ed2044ce5995533df32e67351" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 38589 } attrs { replace: true } } peers { permanent_uuid: "b5e9178a7a83402e8c4a6943ab78589d" member_type: NON_VOTER last_known_addr { host: "127.17.60.197" port: 36573 } attrs { promote: true } }
I20250112 15:07:32.859483 21750 raft_consensus.cc:738] T 57e47b07fdcf434e96ca5dd0354e93b2 P b5e9178a7a83402e8c4a6943ab78589d [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: b5e9178a7a83402e8c4a6943ab78589d, State: Initialized, Role: LEARNER
I20250112 15:07:32.859515 21750 consensus_queue.cc:260] T 57e47b07fdcf434e96ca5dd0354e93b2 P b5e9178a7a83402e8c4a6943ab78589d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 27, Last appended: 2.27, Last appended by leader: 27, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "2537bbbd1a9947ae9facd87567e3ccb9" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 33173 } } peers { permanent_uuid: "c561ec3374b94b95be0c155d80ff0bff" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 33627 } } peers { permanent_uuid: "3710b74ed2044ce5995533df32e67351" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 38589 } attrs { replace: true } } peers { permanent_uuid: "b5e9178a7a83402e8c4a6943ab78589d" member_type: NON_VOTER last_known_addr { host: "127.17.60.197" port: 36573 } attrs { promote: true } }
I20250112 15:07:32.859679 21750 ts_tablet_manager.cc:1428] T 57e47b07fdcf434e96ca5dd0354e93b2 P b5e9178a7a83402e8c4a6943ab78589d: Time spent starting tablet: real 0.000s user 0.000s sys 0.002s
I20250112 15:07:32.859841 21085 tablet_copy_service.cc:342] P 2537bbbd1a9947ae9facd87567e3ccb9: Request end of tablet copy session b5e9178a7a83402e8c4a6943ab78589d-57e47b07fdcf434e96ca5dd0354e93b2 received from {username='slave'} at 127.0.0.1:54448
I20250112 15:07:32.859891 21085 tablet_copy_service.cc:434] P 2537bbbd1a9947ae9facd87567e3ccb9: ending tablet copy session b5e9178a7a83402e8c4a6943ab78589d-57e47b07fdcf434e96ca5dd0354e93b2 on tablet 57e47b07fdcf434e96ca5dd0354e93b2 with peer b5e9178a7a83402e8c4a6943ab78589d
I20250112 15:07:32.992524 21474 raft_consensus.cc:1212] T 9601bcda24ed42e7b93ab9740f1e271e P a2a5bba4ac1145d5bab94a69c9f4df40 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.24->[1.25-1.25] Dedup: 1.25->[]
I20250112 15:07:33.127925 21549 raft_consensus.cc:1212] T 57e47b07fdcf434e96ca5dd0354e93b2 P b5e9178a7a83402e8c4a6943ab78589d [term 2 LEARNER]: Deduplicated request from leader. Original: 2.26->[2.27-2.27] Dedup: 2.27->[]
I20250112 15:07:33.219456 21745 raft_consensus.cc:1059] T 80cc505a04394c53a18c5beeded4655d P c561ec3374b94b95be0c155d80ff0bff: attempting to promote NON_VOTER 25286c485fa04c86a5d07c8f88643a47 to VOTER
I20250112 15:07:33.219682 21745 consensus_queue.cc:237] T 80cc505a04394c53a18c5beeded4655d P c561ec3374b94b95be0c155d80ff0bff [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 26, Committed index: 26, Last appended: 1.26, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "2537bbbd1a9947ae9facd87567e3ccb9" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 33173 } attrs { replace: true } } peers { permanent_uuid: "c561ec3374b94b95be0c155d80ff0bff" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 33627 } } peers { permanent_uuid: "3710b74ed2044ce5995533df32e67351" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 38589 } } peers { permanent_uuid: "25286c485fa04c86a5d07c8f88643a47" member_type: VOTER last_known_addr { host: "127.17.60.198" port: 37089 } attrs { promote: false } }
I20250112 15:07:33.220129 21624 raft_consensus.cc:1270] T 80cc505a04394c53a18c5beeded4655d P 25286c485fa04c86a5d07c8f88643a47 [term 1 LEARNER]: Refusing update from remote peer c561ec3374b94b95be0c155d80ff0bff: Log matching property violated. Preceding OpId in replica: term: 1 index: 26. Preceding OpId from leader: term: 1 index: 27. (index mismatch)
I20250112 15:07:33.220280 21069 raft_consensus.cc:1270] T 80cc505a04394c53a18c5beeded4655d P 2537bbbd1a9947ae9facd87567e3ccb9 [term 1 FOLLOWER]: Refusing update from remote peer c561ec3374b94b95be0c155d80ff0bff: Log matching property violated. Preceding OpId in replica: term: 1 index: 26. Preceding OpId from leader: term: 1 index: 27. (index mismatch)
I20250112 15:07:33.220371 21222 raft_consensus.cc:1270] T 80cc505a04394c53a18c5beeded4655d P 3710b74ed2044ce5995533df32e67351 [term 1 FOLLOWER]: Refusing update from remote peer c561ec3374b94b95be0c155d80ff0bff: Log matching property violated. Preceding OpId in replica: term: 1 index: 26. Preceding OpId from leader: term: 1 index: 27. (index mismatch)
I20250112 15:07:33.220475 21776 consensus_queue.cc:1035] T 80cc505a04394c53a18c5beeded4655d P c561ec3374b94b95be0c155d80ff0bff [LEADER]: Connected to new peer: Peer: permanent_uuid: "25286c485fa04c86a5d07c8f88643a47" member_type: VOTER last_known_addr { host: "127.17.60.198" port: 37089 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 27, Last known committed idx: 26, Time since last communication: 0.000s
I20250112 15:07:33.220618 21775 consensus_queue.cc:1035] T 80cc505a04394c53a18c5beeded4655d P c561ec3374b94b95be0c155d80ff0bff [LEADER]: Connected to new peer: Peer: permanent_uuid: "3710b74ed2044ce5995533df32e67351" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 38589 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 27, Last known committed idx: 26, Time since last communication: 0.000s
I20250112 15:07:33.220693 21745 consensus_queue.cc:1035] T 80cc505a04394c53a18c5beeded4655d P c561ec3374b94b95be0c155d80ff0bff [LEADER]: Connected to new peer: Peer: permanent_uuid: "2537bbbd1a9947ae9facd87567e3ccb9" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 33173 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 27, Last known committed idx: 26, Time since last communication: 0.000s
I20250112 15:07:33.221454 21745 raft_consensus.cc:2949] T 80cc505a04394c53a18c5beeded4655d P c561ec3374b94b95be0c155d80ff0bff [term 1 LEADER]: Committing config change with OpId 1.27: config changed from index 26 to 27, 25286c485fa04c86a5d07c8f88643a47 (127.17.60.198) changed from NON_VOTER to VOTER. New config: { opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "2537bbbd1a9947ae9facd87567e3ccb9" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 33173 } attrs { replace: true } } peers { permanent_uuid: "c561ec3374b94b95be0c155d80ff0bff" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 33627 } } peers { permanent_uuid: "3710b74ed2044ce5995533df32e67351" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 38589 } } peers { permanent_uuid: "25286c485fa04c86a5d07c8f88643a47" member_type: VOTER last_known_addr { host: "127.17.60.198" port: 37089 } attrs { promote: false } } }
I20250112 15:07:33.221536 21624 raft_consensus.cc:2949] T 80cc505a04394c53a18c5beeded4655d P 25286c485fa04c86a5d07c8f88643a47 [term 1 FOLLOWER]: Committing config change with OpId 1.27: config changed from index 26 to 27, 25286c485fa04c86a5d07c8f88643a47 (127.17.60.198) changed from NON_VOTER to VOTER. New config: { opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "2537bbbd1a9947ae9facd87567e3ccb9" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 33173 } attrs { replace: true } } peers { permanent_uuid: "c561ec3374b94b95be0c155d80ff0bff" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 33627 } } peers { permanent_uuid: "3710b74ed2044ce5995533df32e67351" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 38589 } } peers { permanent_uuid: "25286c485fa04c86a5d07c8f88643a47" member_type: VOTER last_known_addr { host: "127.17.60.198" port: 37089 } attrs { promote: false } } }
I20250112 15:07:33.222340 21068 raft_consensus.cc:2949] T 80cc505a04394c53a18c5beeded4655d P 2537bbbd1a9947ae9facd87567e3ccb9 [term 1 FOLLOWER]: Committing config change with OpId 1.27: config changed from index 26 to 27, 25286c485fa04c86a5d07c8f88643a47 (127.17.60.198) changed from NON_VOTER to VOTER. New config: { opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "2537bbbd1a9947ae9facd87567e3ccb9" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 33173 } attrs { replace: true } } peers { permanent_uuid: "c561ec3374b94b95be0c155d80ff0bff" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 33627 } } peers { permanent_uuid: "3710b74ed2044ce5995533df32e67351" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 38589 } } peers { permanent_uuid: "25286c485fa04c86a5d07c8f88643a47" member_type: VOTER last_known_addr { host: "127.17.60.198" port: 37089 } attrs { promote: false } } }
I20250112 15:07:33.222620 20966 catalog_manager.cc:5526] T 80cc505a04394c53a18c5beeded4655d P c561ec3374b94b95be0c155d80ff0bff reported cstate change: config changed from index 26 to 27, 25286c485fa04c86a5d07c8f88643a47 (127.17.60.198) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "c561ec3374b94b95be0c155d80ff0bff" committed_config { opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "2537bbbd1a9947ae9facd87567e3ccb9" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 33173 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c561ec3374b94b95be0c155d80ff0bff" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 33627 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3710b74ed2044ce5995533df32e67351" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 38589 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "25286c485fa04c86a5d07c8f88643a47" member_type: VOTER last_known_addr { host: "127.17.60.198" port: 37089 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20250112 15:07:33.223484 21222 raft_consensus.cc:2949] T 80cc505a04394c53a18c5beeded4655d P 3710b74ed2044ce5995533df32e67351 [term 1 FOLLOWER]: Committing config change with OpId 1.27: config changed from index 26 to 27, 25286c485fa04c86a5d07c8f88643a47 (127.17.60.198) changed from NON_VOTER to VOTER. New config: { opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "2537bbbd1a9947ae9facd87567e3ccb9" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 33173 } attrs { replace: true } } peers { permanent_uuid: "c561ec3374b94b95be0c155d80ff0bff" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 33627 } } peers { permanent_uuid: "3710b74ed2044ce5995533df32e67351" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 38589 } } peers { permanent_uuid: "25286c485fa04c86a5d07c8f88643a47" member_type: VOTER last_known_addr { host: "127.17.60.198" port: 37089 } attrs { promote: false } } }
I20250112 15:07:33.224094 21149 consensus_queue.cc:237] T 80cc505a04394c53a18c5beeded4655d P c561ec3374b94b95be0c155d80ff0bff [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 27, Committed index: 27, Last appended: 1.27, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "c561ec3374b94b95be0c155d80ff0bff" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 33627 } } peers { permanent_uuid: "3710b74ed2044ce5995533df32e67351" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 38589 } } peers { permanent_uuid: "25286c485fa04c86a5d07c8f88643a47" member_type: VOTER last_known_addr { host: "127.17.60.198" port: 37089 } attrs { promote: false } }
I20250112 15:07:33.224474 21624 raft_consensus.cc:1270] T 80cc505a04394c53a18c5beeded4655d P 25286c485fa04c86a5d07c8f88643a47 [term 1 FOLLOWER]: Refusing update from remote peer c561ec3374b94b95be0c155d80ff0bff: Log matching property violated. Preceding OpId in replica: term: 1 index: 27. Preceding OpId from leader: term: 1 index: 28. (index mismatch)
I20250112 15:07:33.224602 21222 raft_consensus.cc:1270] T 80cc505a04394c53a18c5beeded4655d P 3710b74ed2044ce5995533df32e67351 [term 1 FOLLOWER]: Refusing update from remote peer c561ec3374b94b95be0c155d80ff0bff: Log matching property violated. Preceding OpId in replica: term: 1 index: 27. Preceding OpId from leader: term: 1 index: 28. (index mismatch)
I20250112 15:07:33.224747 21773 consensus_queue.cc:1035] T 80cc505a04394c53a18c5beeded4655d P c561ec3374b94b95be0c155d80ff0bff [LEADER]: Connected to new peer: Peer: permanent_uuid: "25286c485fa04c86a5d07c8f88643a47" member_type: VOTER last_known_addr { host: "127.17.60.198" port: 37089 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 28, Last known committed idx: 27, Time since last communication: 0.000s
I20250112 15:07:33.224833 21776 consensus_queue.cc:1035] T 80cc505a04394c53a18c5beeded4655d P c561ec3374b94b95be0c155d80ff0bff [LEADER]: Connected to new peer: Peer: permanent_uuid: "3710b74ed2044ce5995533df32e67351" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 38589 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 28, Last known committed idx: 27, Time since last communication: 0.000s
I20250112 15:07:33.225431 21776 raft_consensus.cc:2949] T 80cc505a04394c53a18c5beeded4655d P c561ec3374b94b95be0c155d80ff0bff [term 1 LEADER]: Committing config change with OpId 1.28: config changed from index 27 to 28, VOTER 2537bbbd1a9947ae9facd87567e3ccb9 (127.17.60.193) evicted. New config: { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "c561ec3374b94b95be0c155d80ff0bff" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 33627 } } peers { permanent_uuid: "3710b74ed2044ce5995533df32e67351" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 38589 } } peers { permanent_uuid: "25286c485fa04c86a5d07c8f88643a47" member_type: VOTER last_known_addr { host: "127.17.60.198" port: 37089 } attrs { promote: false } } }
I20250112 15:07:33.226325 20958 catalog_manager.cc:5039] ChangeConfig:REMOVE_PEER RPC for tablet 80cc505a04394c53a18c5beeded4655d with cas_config_opid_index 27: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250112 15:07:33.226284 21222 raft_consensus.cc:2949] T 80cc505a04394c53a18c5beeded4655d P 3710b74ed2044ce5995533df32e67351 [term 1 FOLLOWER]: Committing config change with OpId 1.28: config changed from index 27 to 28, VOTER 2537bbbd1a9947ae9facd87567e3ccb9 (127.17.60.193) evicted. New config: { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "c561ec3374b94b95be0c155d80ff0bff" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 33627 } } peers { permanent_uuid: "3710b74ed2044ce5995533df32e67351" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 38589 } } peers { permanent_uuid: "25286c485fa04c86a5d07c8f88643a47" member_type: VOTER last_known_addr { host: "127.17.60.198" port: 37089 } attrs { promote: false } } }
I20250112 15:07:33.226488 20970 catalog_manager.cc:5526] T 80cc505a04394c53a18c5beeded4655d P c561ec3374b94b95be0c155d80ff0bff reported cstate change: config changed from index 27 to 28, VOTER 2537bbbd1a9947ae9facd87567e3ccb9 (127.17.60.193) evicted. New cstate: current_term: 1 leader_uuid: "c561ec3374b94b95be0c155d80ff0bff" committed_config { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "c561ec3374b94b95be0c155d80ff0bff" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 33627 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3710b74ed2044ce5995533df32e67351" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 38589 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "25286c485fa04c86a5d07c8f88643a47" member_type: VOTER last_known_addr { host: "127.17.60.198" port: 37089 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20250112 15:07:33.226750 21624 raft_consensus.cc:2949] T 80cc505a04394c53a18c5beeded4655d P 25286c485fa04c86a5d07c8f88643a47 [term 1 FOLLOWER]: Committing config change with OpId 1.28: config changed from index 27 to 28, VOTER 2537bbbd1a9947ae9facd87567e3ccb9 (127.17.60.193) evicted. New config: { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "c561ec3374b94b95be0c155d80ff0bff" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 33627 } } peers { permanent_uuid: "3710b74ed2044ce5995533df32e67351" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 38589 } } peers { permanent_uuid: "25286c485fa04c86a5d07c8f88643a47" member_type: VOTER last_known_addr { host: "127.17.60.198" port: 37089 } attrs { promote: false } } }
I20250112 15:07:33.227763 21065 tablet_service.cc:1514] Processing DeleteTablet for tablet 80cc505a04394c53a18c5beeded4655d with delete_type TABLET_DATA_TOMBSTONED (TS 2537bbbd1a9947ae9facd87567e3ccb9 not found in new config with opid_index 28) from {username='slave'} at 127.0.0.1:37188
I20250112 15:07:33.228019 21784 tablet_replica.cc:331] T 80cc505a04394c53a18c5beeded4655d P 2537bbbd1a9947ae9facd87567e3ccb9: stopping tablet replica
I20250112 15:07:33.228096 21784 raft_consensus.cc:2238] T 80cc505a04394c53a18c5beeded4655d P 2537bbbd1a9947ae9facd87567e3ccb9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250112 15:07:33.228178 21784 raft_consensus.cc:2267] T 80cc505a04394c53a18c5beeded4655d P 2537bbbd1a9947ae9facd87567e3ccb9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 15:07:33.228415 21784 ts_tablet_manager.cc:1905] T 80cc505a04394c53a18c5beeded4655d P 2537bbbd1a9947ae9facd87567e3ccb9: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250112 15:07:33.229607 21784 ts_tablet_manager.cc:1918] T 80cc505a04394c53a18c5beeded4655d P 2537bbbd1a9947ae9facd87567e3ccb9: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.27
I20250112 15:07:33.229671 21784 log.cc:1198] T 80cc505a04394c53a18c5beeded4655d P 2537bbbd1a9947ae9facd87567e3ccb9: Deleting WAL directory at /tmp/dist-test-task95dLSD/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1736694351070047-17651-0/minicluster-data/ts-0-root/wals/80cc505a04394c53a18c5beeded4655d
I20250112 15:07:33.230010 20956 catalog_manager.cc:4872] TS 2537bbbd1a9947ae9facd87567e3ccb9 (127.17.60.193:33173): tablet 80cc505a04394c53a18c5beeded4655d (table test-workload [id=cdaf6d3a65a54ea08e8e2c0ecdfc9de7]) successfully deleted
I20250112 15:07:33.325680 21776 raft_consensus.cc:1059] T 69c23627ab854000b360030c384dc966 P c561ec3374b94b95be0c155d80ff0bff: attempting to promote NON_VOTER b5e9178a7a83402e8c4a6943ab78589d to VOTER
I20250112 15:07:33.325892 21776 consensus_queue.cc:237] T 69c23627ab854000b360030c384dc966 P c561ec3374b94b95be0c155d80ff0bff [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 25, Committed index: 25, Last appended: 1.25, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 26 OBSOLETE_local: false peers { permanent_uuid: "2537bbbd1a9947ae9facd87567e3ccb9" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 33173 } attrs { replace: true } } peers { permanent_uuid: "c561ec3374b94b95be0c155d80ff0bff" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 33627 } } peers { permanent_uuid: "3710b74ed2044ce5995533df32e67351" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 38589 } } peers { permanent_uuid: "b5e9178a7a83402e8c4a6943ab78589d" member_type: VOTER last_known_addr { host: "127.17.60.197" port: 36573 } attrs { promote: false } }
I20250112 15:07:33.326331 21222 raft_consensus.cc:1270] T 69c23627ab854000b360030c384dc966 P 3710b74ed2044ce5995533df32e67351 [term 1 FOLLOWER]: Refusing update from remote peer c561ec3374b94b95be0c155d80ff0bff: Log matching property violated. Preceding OpId in replica: term: 1 index: 25. Preceding OpId from leader: term: 1 index: 26. (index mismatch)
I20250112 15:07:33.326454 21069 raft_consensus.cc:1270] T 69c23627ab854000b360030c384dc966 P 2537bbbd1a9947ae9facd87567e3ccb9 [term 1 FOLLOWER]: Refusing update from remote peer c561ec3374b94b95be0c155d80ff0bff: Log matching property violated. Preceding OpId in replica: term: 1 index: 25. Preceding OpId from leader: term: 1 index: 26. (index mismatch)
I20250112 15:07:33.326511 21549 raft_consensus.cc:1270] T 69c23627ab854000b360030c384dc966 P b5e9178a7a83402e8c4a6943ab78589d [term 1 LEARNER]: Refusing update from remote peer c561ec3374b94b95be0c155d80ff0bff: Log matching property violated. Preceding OpId in replica: term: 1 index: 25. Preceding OpId from leader: term: 1 index: 26. (index mismatch)
I20250112 15:07:33.326627 21773 consensus_queue.cc:1035] T 69c23627ab854000b360030c384dc966 P c561ec3374b94b95be0c155d80ff0bff [LEADER]: Connected to new peer: Peer: permanent_uuid: "3710b74ed2044ce5995533df32e67351" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 38589 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 26, Last known committed idx: 25, Time since last communication: 0.000s
I20250112 15:07:33.326714 21775 consensus_queue.cc:1035] T 69c23627ab854000b360030c384dc966 P c561ec3374b94b95be0c155d80ff0bff [LEADER]: Connected to new peer: Peer: permanent_uuid: "b5e9178a7a83402e8c4a6943ab78589d" member_type: VOTER last_known_addr { host: "127.17.60.197" port: 36573 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 26, Last known committed idx: 25, Time since last communication: 0.000s
I20250112 15:07:33.326805 21745 consensus_queue.cc:1035] T 69c23627ab854000b360030c384dc966 P c561ec3374b94b95be0c155d80ff0bff [LEADER]: Connected to new peer: Peer: permanent_uuid: "2537bbbd1a9947ae9facd87567e3ccb9" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 33173 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 26, Last known committed idx: 25, Time since last communication: 0.000s
I20250112 15:07:33.327421 21776 raft_consensus.cc:2949] T 69c23627ab854000b360030c384dc966 P c561ec3374b94b95be0c155d80ff0bff [term 1 LEADER]: Committing config change with OpId 1.26: config changed from index 25 to 26, b5e9178a7a83402e8c4a6943ab78589d (127.17.60.197) changed from NON_VOTER to VOTER. New config: { opid_index: 26 OBSOLETE_local: false peers { permanent_uuid: "2537bbbd1a9947ae9facd87567e3ccb9" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 33173 } attrs { replace: true } } peers { permanent_uuid: "c561ec3374b94b95be0c155d80ff0bff" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 33627 } } peers { permanent_uuid: "3710b74ed2044ce5995533df32e67351" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 38589 } } peers { permanent_uuid: "b5e9178a7a83402e8c4a6943ab78589d" member_type: VOTER last_known_addr { host: "127.17.60.197" port: 36573 } attrs { promote: false } } }
I20250112 15:07:33.327549 21549 raft_consensus.cc:2949] T 69c23627ab854000b360030c384dc966 P b5e9178a7a83402e8c4a6943ab78589d [term 1 FOLLOWER]: Committing config change with OpId 1.26: config changed from index 25 to 26, b5e9178a7a83402e8c4a6943ab78589d (127.17.60.197) changed from NON_VOTER to VOTER. New config: { opid_index: 26 OBSOLETE_local: false peers { permanent_uuid: "2537bbbd1a9947ae9facd87567e3ccb9" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 33173 } attrs { replace: true } } peers { permanent_uuid: "c561ec3374b94b95be0c155d80ff0bff" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 33627 } } peers { permanent_uuid: "3710b74ed2044ce5995533df32e67351" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 38589 } } peers { permanent_uuid: "b5e9178a7a83402e8c4a6943ab78589d" member_type: VOTER last_known_addr { host: "127.17.60.197" port: 36573 } attrs { promote: false } } }
I20250112 15:07:33.328419 21222 raft_consensus.cc:2949] T 69c23627ab854000b360030c384dc966 P 3710b74ed2044ce5995533df32e67351 [term 1 FOLLOWER]: Committing config change with OpId 1.26: config changed from index 25 to 26, b5e9178a7a83402e8c4a6943ab78589d (127.17.60.197) changed from NON_VOTER to VOTER. New config: { opid_index: 26 OBSOLETE_local: false peers { permanent_uuid: "2537bbbd1a9947ae9facd87567e3ccb9" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 33173 } attrs { replace: true } } peers { permanent_uuid: "c561ec3374b94b95be0c155d80ff0bff" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 33627 } } peers { permanent_uuid: "3710b74ed2044ce5995533df32e67351" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 38589 } } peers { permanent_uuid: "b5e9178a7a83402e8c4a6943ab78589d" member_type: VOTER last_known_addr { host: "127.17.60.197" port: 36573 } attrs { promote: false } } }
I20250112 15:07:33.328485 20970 catalog_manager.cc:5526] T 69c23627ab854000b360030c384dc966 P c561ec3374b94b95be0c155d80ff0bff reported cstate change: config changed from index 25 to 26, b5e9178a7a83402e8c4a6943ab78589d (127.17.60.197) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "c561ec3374b94b95be0c155d80ff0bff" committed_config { opid_index: 26 OBSOLETE_local: false peers { permanent_uuid: "2537bbbd1a9947ae9facd87567e3ccb9" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 33173 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c561ec3374b94b95be0c155d80ff0bff" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 33627 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3710b74ed2044ce5995533df32e67351" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 38589 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b5e9178a7a83402e8c4a6943ab78589d" member_type: VOTER last_known_addr { host: "127.17.60.197" port: 36573 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20250112 15:07:33.328747 21069 raft_consensus.cc:2949] T 69c23627ab854000b360030c384dc966 P 2537bbbd1a9947ae9facd87567e3ccb9 [term 1 FOLLOWER]: Committing config change with OpId 1.26: config changed from index 25 to 26, b5e9178a7a83402e8c4a6943ab78589d (127.17.60.197) changed from NON_VOTER to VOTER. New config: { opid_index: 26 OBSOLETE_local: false peers { permanent_uuid: "2537bbbd1a9947ae9facd87567e3ccb9" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 33173 } attrs { replace: true } } peers { permanent_uuid: "c561ec3374b94b95be0c155d80ff0bff" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 33627 } } peers { permanent_uuid: "3710b74ed2044ce5995533df32e67351" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 38589 } } peers { permanent_uuid: "b5e9178a7a83402e8c4a6943ab78589d" member_type: VOTER last_known_addr { host: "127.17.60.197" port: 36573 } attrs { promote: false } } }
I20250112 15:07:33.329747 21149 consensus_queue.cc:237] T 69c23627ab854000b360030c384dc966 P c561ec3374b94b95be0c155d80ff0bff [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 26, Committed index: 26, Last appended: 1.26, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "c561ec3374b94b95be0c155d80ff0bff" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 33627 } } peers { permanent_uuid: "3710b74ed2044ce5995533df32e67351" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 38589 } } peers { permanent_uuid: "b5e9178a7a83402e8c4a6943ab78589d" member_type: VOTER last_known_addr { host: "127.17.60.197" port: 36573 } attrs { promote: false } }
I20250112 15:07:33.330057 21549 raft_consensus.cc:1270] T 69c23627ab854000b360030c384dc966 P b5e9178a7a83402e8c4a6943ab78589d [term 1 FOLLOWER]: Refusing update from remote peer c561ec3374b94b95be0c155d80ff0bff: Log matching property violated. Preceding OpId in replica: term: 1 index: 26. Preceding OpId from leader: term: 1 index: 27. (index mismatch)
I20250112 15:07:33.330147 21222 raft_consensus.cc:1270] T 69c23627ab854000b360030c384dc966 P 3710b74ed2044ce5995533df32e67351 [term 1 FOLLOWER]: Refusing update from remote peer c561ec3374b94b95be0c155d80ff0bff: Log matching property violated. Preceding OpId in replica: term: 1 index: 26. Preceding OpId from leader: term: 1 index: 27. (index mismatch)
I20250112 15:07:33.330216 21775 consensus_queue.cc:1035] T 69c23627ab854000b360030c384dc966 P c561ec3374b94b95be0c155d80ff0bff [LEADER]: Connected to new peer: Peer: permanent_uuid: "b5e9178a7a83402e8c4a6943ab78589d" member_type: VOTER last_known_addr { host: "127.17.60.197" port: 36573 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 27, Last known committed idx: 26, Time since last communication: 0.000s
I20250112 15:07:33.330305 21773 consensus_queue.cc:1035] T 69c23627ab854000b360030c384dc966 P c561ec3374b94b95be0c155d80ff0bff [LEADER]: Connected to new peer: Peer: permanent_uuid: "3710b74ed2044ce5995533df32e67351" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 38589 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 27, Last known committed idx: 26, Time since last communication: 0.000s
I20250112 15:07:33.330713 21745 raft_consensus.cc:2949] T 69c23627ab854000b360030c384dc966 P c561ec3374b94b95be0c155d80ff0bff [term 1 LEADER]: Committing config change with OpId 1.27: config changed from index 26 to 27, VOTER 2537bbbd1a9947ae9facd87567e3ccb9 (127.17.60.193) evicted. New config: { opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "c561ec3374b94b95be0c155d80ff0bff" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 33627 } } peers { permanent_uuid: "3710b74ed2044ce5995533df32e67351" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 38589 } } peers { permanent_uuid: "b5e9178a7a83402e8c4a6943ab78589d" member_type: VOTER last_known_addr { host: "127.17.60.197" port: 36573 } attrs { promote: false } } }
I20250112 15:07:33.330837 21549 raft_consensus.cc:2949] T 69c23627ab854000b360030c384dc966 P b5e9178a7a83402e8c4a6943ab78589d [term 1 FOLLOWER]: Committing config change with OpId 1.27: config changed from index 26 to 27, VOTER 2537bbbd1a9947ae9facd87567e3ccb9 (127.17.60.193) evicted. New config: { opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "c561ec3374b94b95be0c155d80ff0bff" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 33627 } } peers { permanent_uuid: "3710b74ed2044ce5995533df32e67351" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 38589 } } peers { permanent_uuid: "b5e9178a7a83402e8c4a6943ab78589d" member_type: VOTER last_known_addr { host: "127.17.60.197" port: 36573 } attrs { promote: false } } }
I20250112 15:07:33.331532 20958 catalog_manager.cc:5039] ChangeConfig:REMOVE_PEER RPC for tablet 69c23627ab854000b360030c384dc966 with cas_config_opid_index 26: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250112 15:07:33.331583 20964 catalog_manager.cc:5526] T 69c23627ab854000b360030c384dc966 P c561ec3374b94b95be0c155d80ff0bff reported cstate change: config changed from index 26 to 27, VOTER 2537bbbd1a9947ae9facd87567e3ccb9 (127.17.60.193) evicted. New cstate: current_term: 1 leader_uuid: "c561ec3374b94b95be0c155d80ff0bff" committed_config { opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "c561ec3374b94b95be0c155d80ff0bff" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 33627 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3710b74ed2044ce5995533df32e67351" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 38589 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b5e9178a7a83402e8c4a6943ab78589d" member_type: VOTER last_known_addr { host: "127.17.60.197" port: 36573 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20250112 15:07:33.331532 21222 raft_consensus.cc:2949] T 69c23627ab854000b360030c384dc966 P 3710b74ed2044ce5995533df32e67351 [term 1 FOLLOWER]: Committing config change with OpId 1.27: config changed from index 26 to 27, VOTER 2537bbbd1a9947ae9facd87567e3ccb9 (127.17.60.193) evicted. New config: { opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "c561ec3374b94b95be0c155d80ff0bff" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 33627 } } peers { permanent_uuid: "3710b74ed2044ce5995533df32e67351" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 38589 } } peers { permanent_uuid: "b5e9178a7a83402e8c4a6943ab78589d" member_type: VOTER last_known_addr { host: "127.17.60.197" port: 36573 } attrs { promote: false } } }
I20250112 15:07:33.332602 21065 tablet_service.cc:1514] Processing DeleteTablet for tablet 69c23627ab854000b360030c384dc966 with delete_type TABLET_DATA_TOMBSTONED (TS 2537bbbd1a9947ae9facd87567e3ccb9 not found in new config with opid_index 27) from {username='slave'} at 127.0.0.1:37188
I20250112 15:07:33.332686 21784 tablet_replica.cc:331] T 69c23627ab854000b360030c384dc966 P 2537bbbd1a9947ae9facd87567e3ccb9: stopping tablet replica
I20250112 15:07:33.332742 21784 raft_consensus.cc:2238] T 69c23627ab854000b360030c384dc966 P 2537bbbd1a9947ae9facd87567e3ccb9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250112 15:07:33.332803 21784 raft_consensus.cc:2267] T 69c23627ab854000b360030c384dc966 P 2537bbbd1a9947ae9facd87567e3ccb9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 15:07:33.333036 21784 ts_tablet_manager.cc:1905] T 69c23627ab854000b360030c384dc966 P 2537bbbd1a9947ae9facd87567e3ccb9: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250112 15:07:33.334323 21784 ts_tablet_manager.cc:1918] T 69c23627ab854000b360030c384dc966 P 2537bbbd1a9947ae9facd87567e3ccb9: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.26
I20250112 15:07:33.334383 21784 log.cc:1198] T 69c23627ab854000b360030c384dc966 P 2537bbbd1a9947ae9facd87567e3ccb9: Deleting WAL directory at /tmp/dist-test-task95dLSD/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1736694351070047-17651-0/minicluster-data/ts-0-root/wals/69c23627ab854000b360030c384dc966
I20250112 15:07:33.334688 20956 catalog_manager.cc:4872] TS 2537bbbd1a9947ae9facd87567e3ccb9 (127.17.60.193:33173): tablet 69c23627ab854000b360030c384dc966 (table test-workload [id=cdaf6d3a65a54ea08e8e2c0ecdfc9de7]) successfully deleted
I20250112 15:07:33.343891 21474 raft_consensus.cc:1212] T 5bcec60d3b974a9bb24aedfc738e1594 P a2a5bba4ac1145d5bab94a69c9f4df40 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.25->[1.26-1.26] Dedup: 1.26->[]
I20250112 15:07:33.440157 21754 raft_consensus.cc:1059] T 9601bcda24ed42e7b93ab9740f1e271e P 2537bbbd1a9947ae9facd87567e3ccb9: attempting to promote NON_VOTER a2a5bba4ac1145d5bab94a69c9f4df40 to VOTER
I20250112 15:07:33.440379 21754 consensus_queue.cc:237] T 9601bcda24ed42e7b93ab9740f1e271e P 2537bbbd1a9947ae9facd87567e3ccb9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 25, Committed index: 25, Last appended: 1.25, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 26 OBSOLETE_local: false peers { permanent_uuid: "2537bbbd1a9947ae9facd87567e3ccb9" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 33173 } } peers { permanent_uuid: "c561ec3374b94b95be0c155d80ff0bff" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 33627 } attrs { replace: true } } peers { permanent_uuid: "3710b74ed2044ce5995533df32e67351" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 38589 } } peers { permanent_uuid: "a2a5bba4ac1145d5bab94a69c9f4df40" member_type: VOTER last_known_addr { host: "127.17.60.196" port: 41807 } attrs { promote: false } }
I20250112 15:07:33.440869 21474 raft_consensus.cc:1270] T 9601bcda24ed42e7b93ab9740f1e271e P a2a5bba4ac1145d5bab94a69c9f4df40 [term 1 LEARNER]: Refusing update from remote peer 2537bbbd1a9947ae9facd87567e3ccb9: Log matching property violated. Preceding OpId in replica: term: 1 index: 25. Preceding OpId from leader: term: 1 index: 26. (index mismatch)
I20250112 15:07:33.440908 21222 raft_consensus.cc:1270] T 9601bcda24ed42e7b93ab9740f1e271e P 3710b74ed2044ce5995533df32e67351 [term 1 FOLLOWER]: Refusing update from remote peer 2537bbbd1a9947ae9facd87567e3ccb9: Log matching property violated. Preceding OpId in replica: term: 1 index: 25. Preceding OpId from leader: term: 1 index: 26. (index mismatch)
I20250112 15:07:33.441012 21147 raft_consensus.cc:1270] T 9601bcda24ed42e7b93ab9740f1e271e P c561ec3374b94b95be0c155d80ff0bff [term 1 FOLLOWER]: Refusing update from remote peer 2537bbbd1a9947ae9facd87567e3ccb9: Log matching property violated. Preceding OpId in replica: term: 1 index: 25. Preceding OpId from leader: term: 1 index: 26. (index mismatch)
I20250112 15:07:33.441053 21790 consensus_queue.cc:1035] T 9601bcda24ed42e7b93ab9740f1e271e P 2537bbbd1a9947ae9facd87567e3ccb9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2a5bba4ac1145d5bab94a69c9f4df40" member_type: VOTER last_known_addr { host: "127.17.60.196" port: 41807 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 26, Last known committed idx: 25, Time since last communication: 0.000s
I20250112 15:07:33.441326 21772 consensus_queue.cc:1035] T 9601bcda24ed42e7b93ab9740f1e271e P 2537bbbd1a9947ae9facd87567e3ccb9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c561ec3374b94b95be0c155d80ff0bff" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 33627 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 26, Last known committed idx: 25, Time since last communication: 0.000s
I20250112 15:07:33.441458 21716 consensus_queue.cc:1035] T 9601bcda24ed42e7b93ab9740f1e271e P 2537bbbd1a9947ae9facd87567e3ccb9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3710b74ed2044ce5995533df32e67351" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 38589 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 26, Last known committed idx: 25, Time since last communication: 0.000s
I20250112 15:07:33.442092 21716 raft_consensus.cc:2949] T 9601bcda24ed42e7b93ab9740f1e271e P 2537bbbd1a9947ae9facd87567e3ccb9 [term 1 LEADER]: Committing config change with OpId 1.26: config changed from index 25 to 26, a2a5bba4ac1145d5bab94a69c9f4df40 (127.17.60.196) changed from NON_VOTER to VOTER. New config: { opid_index: 26 OBSOLETE_local: false peers { permanent_uuid: "2537bbbd1a9947ae9facd87567e3ccb9" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 33173 } } peers { permanent_uuid: "c561ec3374b94b95be0c155d80ff0bff" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 33627 } attrs { replace: true } } peers { permanent_uuid: "3710b74ed2044ce5995533df32e67351" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 38589 } } peers { permanent_uuid: "a2a5bba4ac1145d5bab94a69c9f4df40" member_type: VOTER last_known_addr { host: "127.17.60.196" port: 41807 } attrs { promote: false } } }
I20250112 15:07:33.443013 21147 raft_consensus.cc:2949] T 9601bcda24ed42e7b93ab9740f1e271e P c561ec3374b94b95be0c155d80ff0bff [term 1 FOLLOWER]: Committing config change with OpId 1.26: config changed from index 25 to 26, a2a5bba4ac1145d5bab94a69c9f4df40 (127.17.60.196) changed from NON_VOTER to VOTER. New config: { opid_index: 26 OBSOLETE_local: false peers { permanent_uuid: "2537bbbd1a9947ae9facd87567e3ccb9" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 33173 } } peers { permanent_uuid: "c561ec3374b94b95be0c155d80ff0bff" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 33627 } attrs { replace: true } } peers { permanent_uuid: "3710b74ed2044ce5995533df32e67351" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 38589 } } peers { permanent_uuid: "a2a5bba4ac1145d5bab94a69c9f4df40" member_type: VOTER last_known_addr { host: "127.17.60.196" port: 41807 } attrs { promote: false } } }
I20250112 15:07:33.443025 21222 raft_consensus.cc:2949] T 9601bcda24ed42e7b93ab9740f1e271e P 3710b74ed2044ce5995533df32e67351 [term 1 FOLLOWER]: Committing config change with OpId 1.26: config changed from index 25 to 26, a2a5bba4ac1145d5bab94a69c9f4df40 (127.17.60.196) changed from NON_VOTER to VOTER. New config: { opid_index: 26 OBSOLETE_local: false peers { permanent_uuid: "2537bbbd1a9947ae9facd87567e3ccb9" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 33173 } } peers { permanent_uuid: "c561ec3374b94b95be0c155d80ff0bff" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 33627 } attrs { replace: true } } peers { permanent_uuid: "3710b74ed2044ce5995533df32e67351" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 38589 } } peers { permanent_uuid: "a2a5bba4ac1145d5bab94a69c9f4df40" member_type: VOTER last_known_addr { host: "127.17.60.196" port: 41807 } attrs { promote: false } } }
I20250112 15:07:33.443135 21474 raft_consensus.cc:2949] T 9601bcda24ed42e7b93ab9740f1e271e P a2a5bba4ac1145d5bab94a69c9f4df40 [term 1 FOLLOWER]: Committing config change with OpId 1.26: config changed from index 25 to 26, a2a5bba4ac1145d5bab94a69c9f4df40 (127.17.60.196) changed from NON_VOTER to VOTER. New config: { opid_index: 26 OBSOLETE_local: false peers { permanent_uuid: "2537bbbd1a9947ae9facd87567e3ccb9" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 33173 } } peers { permanent_uuid: "c561ec3374b94b95be0c155d80ff0bff" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 33627 } attrs { replace: true } } peers { permanent_uuid: "3710b74ed2044ce5995533df32e67351" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 38589 } } peers { permanent_uuid: "a2a5bba4ac1145d5bab94a69c9f4df40" member_type: VOTER last_known_addr { host: "127.17.60.196" port: 41807 } attrs { promote: false } } }
I20250112 15:07:33.443938 20970 catalog_manager.cc:5526] T 9601bcda24ed42e7b93ab9740f1e271e P 2537bbbd1a9947ae9facd87567e3ccb9 reported cstate change: config changed from index 25 to 26, a2a5bba4ac1145d5bab94a69c9f4df40 (127.17.60.196) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "2537bbbd1a9947ae9facd87567e3ccb9" committed_config { opid_index: 26 OBSOLETE_local: false peers { permanent_uuid: "2537bbbd1a9947ae9facd87567e3ccb9" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 33173 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c561ec3374b94b95be0c155d80ff0bff" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 33627 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3710b74ed2044ce5995533df32e67351" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 38589 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a2a5bba4ac1145d5bab94a69c9f4df40" member_type: VOTER last_known_addr { host: "127.17.60.196" port: 41807 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20250112 15:07:33.445294 21069 consensus_queue.cc:237] T 9601bcda24ed42e7b93ab9740f1e271e P 2537bbbd1a9947ae9facd87567e3ccb9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 26, Committed index: 26, Last appended: 1.26, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "2537bbbd1a9947ae9facd87567e3ccb9" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 33173 } } peers { permanent_uuid: "3710b74ed2044ce5995533df32e67351" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 38589 } } peers { permanent_uuid: "a2a5bba4ac1145d5bab94a69c9f4df40" member_type: VOTER last_known_addr { host: "127.17.60.196" port: 41807 } attrs { promote: false } }
I20250112 15:07:33.445626 21474 raft_consensus.cc:1270] T 9601bcda24ed42e7b93ab9740f1e271e P a2a5bba4ac1145d5bab94a69c9f4df40 [term 1 FOLLOWER]: Refusing update from remote peer 2537bbbd1a9947ae9facd87567e3ccb9: Log matching property violated. Preceding OpId in replica: term: 1 index: 26. Preceding OpId from leader: term: 1 index: 27. (index mismatch)
I20250112 15:07:33.445691 21222 raft_consensus.cc:1270] T 9601bcda24ed42e7b93ab9740f1e271e P 3710b74ed2044ce5995533df32e67351 [term 1 FOLLOWER]: Refusing update from remote peer 2537bbbd1a9947ae9facd87567e3ccb9: Log matching property violated. Preceding OpId in replica: term: 1 index: 26. Preceding OpId from leader: term: 1 index: 27. (index mismatch)
I20250112 15:07:33.445784 21790 consensus_queue.cc:1035] T 9601bcda24ed42e7b93ab9740f1e271e P 2537bbbd1a9947ae9facd87567e3ccb9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2a5bba4ac1145d5bab94a69c9f4df40" member_type: VOTER last_known_addr { host: "127.17.60.196" port: 41807 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 27, Last known committed idx: 26, Time since last communication: 0.000s
I20250112 15:07:33.445922 21772 consensus_queue.cc:1035] T 9601bcda24ed42e7b93ab9740f1e271e P 2537bbbd1a9947ae9facd87567e3ccb9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3710b74ed2044ce5995533df32e67351" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 38589 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 27, Last known committed idx: 26, Time since last communication: 0.000s
I20250112 15:07:33.446288 21790 raft_consensus.cc:2949] T 9601bcda24ed42e7b93ab9740f1e271e P 2537bbbd1a9947ae9facd87567e3ccb9 [term 1 LEADER]: Committing config change with OpId 1.27: config changed from index 26 to 27, VOTER c561ec3374b94b95be0c155d80ff0bff (127.17.60.194) evicted. New config: { opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "2537bbbd1a9947ae9facd87567e3ccb9" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 33173 } } peers { permanent_uuid: "3710b74ed2044ce5995533df32e67351" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 38589 } } peers { permanent_uuid: "a2a5bba4ac1145d5bab94a69c9f4df40" member_type: VOTER last_known_addr { host: "127.17.60.196" port: 41807 } attrs { promote: false } } }
I20250112 15:07:33.446404 21474 raft_consensus.cc:2949] T 9601bcda24ed42e7b93ab9740f1e271e P a2a5bba4ac1145d5bab94a69c9f4df40 [term 1 FOLLOWER]: Committing config change with OpId 1.27: config changed from index 26 to 27, VOTER c561ec3374b94b95be0c155d80ff0bff (127.17.60.194) evicted. New config: { opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "2537bbbd1a9947ae9facd87567e3ccb9" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 33173 } } peers { permanent_uuid: "3710b74ed2044ce5995533df32e67351" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 38589 } } peers { permanent_uuid: "a2a5bba4ac1145d5bab94a69c9f4df40" member_type: VOTER last_known_addr { host: "127.17.60.196" port: 41807 } attrs { promote: false } } }
I20250112 15:07:33.446695 21222 raft_consensus.cc:2949] T 9601bcda24ed42e7b93ab9740f1e271e P 3710b74ed2044ce5995533df32e67351 [term 1 FOLLOWER]: Committing config change with OpId 1.27: config changed from index 26 to 27, VOTER c561ec3374b94b95be0c155d80ff0bff (127.17.60.194) evicted. New config: { opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "2537bbbd1a9947ae9facd87567e3ccb9" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 33173 } } peers { permanent_uuid: "3710b74ed2044ce5995533df32e67351" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 38589 } } peers { permanent_uuid: "a2a5bba4ac1145d5bab94a69c9f4df40" member_type: VOTER last_known_addr { host: "127.17.60.196" port: 41807 } attrs { promote: false } } }
I20250112 15:07:33.447296 20966 catalog_manager.cc:5526] T 9601bcda24ed42e7b93ab9740f1e271e P a2a5bba4ac1145d5bab94a69c9f4df40 reported cstate change: config changed from index 26 to 27, VOTER c561ec3374b94b95be0c155d80ff0bff (127.17.60.194) evicted. New cstate: current_term: 1 leader_uuid: "2537bbbd1a9947ae9facd87567e3ccb9" committed_config { opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "2537bbbd1a9947ae9facd87567e3ccb9" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 33173 } } peers { permanent_uuid: "3710b74ed2044ce5995533df32e67351" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 38589 } } peers { permanent_uuid: "a2a5bba4ac1145d5bab94a69c9f4df40" member_type: VOTER last_known_addr { host: "127.17.60.196" port: 41807 } attrs { promote: false } } }
I20250112 15:07:33.447522 20956 catalog_manager.cc:5039] ChangeConfig:REMOVE_PEER RPC for tablet 9601bcda24ed42e7b93ab9740f1e271e with cas_config_opid_index 26: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250112 15:07:33.447913 21141 tablet_service.cc:1514] Processing DeleteTablet for tablet 9601bcda24ed42e7b93ab9740f1e271e with delete_type TABLET_DATA_TOMBSTONED (TS c561ec3374b94b95be0c155d80ff0bff not found in new config with opid_index 27) from {username='slave'} at 127.0.0.1:48072
I20250112 15:07:33.448151 21794 tablet_replica.cc:331] T 9601bcda24ed42e7b93ab9740f1e271e P c561ec3374b94b95be0c155d80ff0bff: stopping tablet replica
I20250112 15:07:33.448216 21794 raft_consensus.cc:2238] T 9601bcda24ed42e7b93ab9740f1e271e P c561ec3374b94b95be0c155d80ff0bff [term 1 FOLLOWER]: Raft consensus shutting down.
I20250112 15:07:33.448269 21794 raft_consensus.cc:2267] T 9601bcda24ed42e7b93ab9740f1e271e P c561ec3374b94b95be0c155d80ff0bff [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 15:07:33.448518 21794 ts_tablet_manager.cc:1905] T 9601bcda24ed42e7b93ab9740f1e271e P c561ec3374b94b95be0c155d80ff0bff: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250112 15:07:33.449739 21794 ts_tablet_manager.cc:1918] T 9601bcda24ed42e7b93ab9740f1e271e P c561ec3374b94b95be0c155d80ff0bff: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.26
I20250112 15:07:33.449788 21794 log.cc:1198] T 9601bcda24ed42e7b93ab9740f1e271e P c561ec3374b94b95be0c155d80ff0bff: Deleting WAL directory at /tmp/dist-test-task95dLSD/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1736694351070047-17651-0/minicluster-data/ts-1-root/wals/9601bcda24ed42e7b93ab9740f1e271e
I20250112 15:07:33.450096 20958 catalog_manager.cc:4872] TS c561ec3374b94b95be0c155d80ff0bff (127.17.60.194:33627): tablet 9601bcda24ed42e7b93ab9740f1e271e (table test-workload [id=cdaf6d3a65a54ea08e8e2c0ecdfc9de7]) successfully deleted
I20250112 15:07:33.505373 21716 raft_consensus.cc:1059] T 57e47b07fdcf434e96ca5dd0354e93b2 P 2537bbbd1a9947ae9facd87567e3ccb9: attempting to promote NON_VOTER b5e9178a7a83402e8c4a6943ab78589d to VOTER
I20250112 15:07:33.505590 21716 consensus_queue.cc:237] T 57e47b07fdcf434e96ca5dd0354e93b2 P 2537bbbd1a9947ae9facd87567e3ccb9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 27, Committed index: 27, Last appended: 2.27, Last appended by leader: 25, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "2537bbbd1a9947ae9facd87567e3ccb9" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 33173 } } peers { permanent_uuid: "c561ec3374b94b95be0c155d80ff0bff" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 33627 } } peers { permanent_uuid: "3710b74ed2044ce5995533df32e67351" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 38589 } attrs { replace: true } } peers { permanent_uuid: "b5e9178a7a83402e8c4a6943ab78589d" member_type: VOTER last_known_addr { host: "127.17.60.197" port: 36573 } attrs { promote: false } }
I20250112 15:07:33.506065 21549 raft_consensus.cc:1270] T 57e47b07fdcf434e96ca5dd0354e93b2 P b5e9178a7a83402e8c4a6943ab78589d [term 2 LEARNER]: Refusing update from remote peer 2537bbbd1a9947ae9facd87567e3ccb9: Log matching property violated. Preceding OpId in replica: term: 2 index: 27. Preceding OpId from leader: term: 2 index: 28. (index mismatch)
I20250112 15:07:33.506117 21147 raft_consensus.cc:1270] T 57e47b07fdcf434e96ca5dd0354e93b2 P c561ec3374b94b95be0c155d80ff0bff [term 2 FOLLOWER]: Refusing update from remote peer 2537bbbd1a9947ae9facd87567e3ccb9: Log matching property violated. Preceding OpId in replica: term: 2 index: 27. Preceding OpId from leader: term: 2 index: 28. (index mismatch)
I20250112 15:07:33.506065 21222 raft_consensus.cc:1270] T 57e47b07fdcf434e96ca5dd0354e93b2 P 3710b74ed2044ce5995533df32e67351 [term 2 FOLLOWER]: Refusing update from remote peer 2537bbbd1a9947ae9facd87567e3ccb9: Log matching property violated. Preceding OpId in replica: term: 2 index: 27. Preceding OpId from leader: term: 2 index: 28. (index mismatch)
I20250112 15:07:33.506276 21796 consensus_queue.cc:1035] T 57e47b07fdcf434e96ca5dd0354e93b2 P 2537bbbd1a9947ae9facd87567e3ccb9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b5e9178a7a83402e8c4a6943ab78589d" member_type: VOTER last_known_addr { host: "127.17.60.197" port: 36573 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 28, Last known committed idx: 27, Time since last communication: 0.000s
I20250112 15:07:33.506460 21716 consensus_queue.cc:1035] T 57e47b07fdcf434e96ca5dd0354e93b2 P 2537bbbd1a9947ae9facd87567e3ccb9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3710b74ed2044ce5995533df32e67351" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 38589 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 28, Last known committed idx: 27, Time since last communication: 0.000s
I20250112 15:07:33.506595 21790 consensus_queue.cc:1035] T 57e47b07fdcf434e96ca5dd0354e93b2 P 2537bbbd1a9947ae9facd87567e3ccb9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c561ec3374b94b95be0c155d80ff0bff" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 33627 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 28, Last known committed idx: 27, Time since last communication: 0.000s
I20250112 15:07:33.507300 21796 raft_consensus.cc:2949] T 57e47b07fdcf434e96ca5dd0354e93b2 P 2537bbbd1a9947ae9facd87567e3ccb9 [term 2 LEADER]: Committing config change with OpId 2.28: config changed from index 27 to 28, b5e9178a7a83402e8c4a6943ab78589d (127.17.60.197) changed from NON_VOTER to VOTER. New config: { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "2537bbbd1a9947ae9facd87567e3ccb9" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 33173 } } peers { permanent_uuid: "c561ec3374b94b95be0c155d80ff0bff" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 33627 } } peers { permanent_uuid: "3710b74ed2044ce5995533df32e67351" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 38589 } attrs { replace: true } } peers { permanent_uuid: "b5e9178a7a83402e8c4a6943ab78589d" member_type: VOTER last_known_addr { host: "127.17.60.197" port: 36573 } attrs { promote: false } } }
I20250112 15:07:33.507423 21549 raft_consensus.cc:2949] T 57e47b07fdcf434e96ca5dd0354e93b2 P b5e9178a7a83402e8c4a6943ab78589d [term 2 FOLLOWER]: Committing config change with OpId 2.28: config changed from index 27 to 28, b5e9178a7a83402e8c4a6943ab78589d (127.17.60.197) changed from NON_VOTER to VOTER. New config: { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "2537bbbd1a9947ae9facd87567e3ccb9" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 33173 } } peers { permanent_uuid: "c561ec3374b94b95be0c155d80ff0bff" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 33627 } } peers { permanent_uuid: "3710b74ed2044ce5995533df32e67351" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 38589 } attrs { replace: true } } peers { permanent_uuid: "b5e9178a7a83402e8c4a6943ab78589d" member_type: VOTER last_known_addr { host: "127.17.60.197" port: 36573 } attrs { promote: false } } }
I20250112 15:07:33.508503 21222 raft_consensus.cc:2949] T 57e47b07fdcf434e96ca5dd0354e93b2 P 3710b74ed2044ce5995533df32e67351 [term 2 FOLLOWER]: Committing config change with OpId 2.28: config changed from index 27 to 28, b5e9178a7a83402e8c4a6943ab78589d (127.17.60.197) changed from NON_VOTER to VOTER. New config: { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "2537bbbd1a9947ae9facd87567e3ccb9" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 33173 } } peers { permanent_uuid: "c561ec3374b94b95be0c155d80ff0bff" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 33627 } } peers { permanent_uuid: "3710b74ed2044ce5995533df32e67351" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 38589 } attrs { replace: true } } peers { permanent_uuid: "b5e9178a7a83402e8c4a6943ab78589d" member_type: VOTER last_known_addr { host: "127.17.60.197" port: 36573 } attrs { promote: false } } }
I20250112 15:07:33.508525 20964 catalog_manager.cc:5526] T 57e47b07fdcf434e96ca5dd0354e93b2 P b5e9178a7a83402e8c4a6943ab78589d reported cstate change: config changed from index 27 to 28, b5e9178a7a83402e8c4a6943ab78589d (127.17.60.197) changed from NON_VOTER to VOTER. New cstate: current_term: 2 leader_uuid: "2537bbbd1a9947ae9facd87567e3ccb9" committed_config { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "2537bbbd1a9947ae9facd87567e3ccb9" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 33173 } } peers { permanent_uuid: "c561ec3374b94b95be0c155d80ff0bff" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 33627 } } peers { permanent_uuid: "3710b74ed2044ce5995533df32e67351" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 38589 } attrs { replace: true } } peers { permanent_uuid: "b5e9178a7a83402e8c4a6943ab78589d" member_type: VOTER last_known_addr { host: "127.17.60.197" port: 36573 } attrs { promote: false } } }
I20250112 15:07:33.509037 21147 raft_consensus.cc:2949] T 57e47b07fdcf434e96ca5dd0354e93b2 P c561ec3374b94b95be0c155d80ff0bff [term 2 FOLLOWER]: Committing config change with OpId 2.28: config changed from index 27 to 28, b5e9178a7a83402e8c4a6943ab78589d (127.17.60.197) changed from NON_VOTER to VOTER. New config: { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "2537bbbd1a9947ae9facd87567e3ccb9" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 33173 } } peers { permanent_uuid: "c561ec3374b94b95be0c155d80ff0bff" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 33627 } } peers { permanent_uuid: "3710b74ed2044ce5995533df32e67351" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 38589 } attrs { replace: true } } peers { permanent_uuid: "b5e9178a7a83402e8c4a6943ab78589d" member_type: VOTER last_known_addr { host: "127.17.60.197" port: 36573 } attrs { promote: false } } }
I20250112 15:07:33.510535 21068 consensus_queue.cc:237] T 57e47b07fdcf434e96ca5dd0354e93b2 P 2537bbbd1a9947ae9facd87567e3ccb9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 28, Committed index: 28, Last appended: 2.28, Last appended by leader: 25, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "2537bbbd1a9947ae9facd87567e3ccb9" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 33173 } } peers { permanent_uuid: "c561ec3374b94b95be0c155d80ff0bff" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 33627 } } peers { permanent_uuid: "b5e9178a7a83402e8c4a6943ab78589d" member_type: VOTER last_known_addr { host: "127.17.60.197" port: 36573 } attrs { promote: false } }
I20250112 15:07:33.510898 21147 raft_consensus.cc:1270] T 57e47b07fdcf434e96ca5dd0354e93b2 P c561ec3374b94b95be0c155d80ff0bff [term 2 FOLLOWER]: Refusing update from remote peer 2537bbbd1a9947ae9facd87567e3ccb9: Log matching property violated. Preceding OpId in replica: term: 2 index: 28. Preceding OpId from leader: term: 2 index: 29. (index mismatch)
I20250112 15:07:33.511044 21549 raft_consensus.cc:1270] T 57e47b07fdcf434e96ca5dd0354e93b2 P b5e9178a7a83402e8c4a6943ab78589d [term 2 FOLLOWER]: Refusing update from remote peer 2537bbbd1a9947ae9facd87567e3ccb9: Log matching property violated. Preceding OpId in replica: term: 2 index: 28. Preceding OpId from leader: term: 2 index: 29. (index mismatch)
I20250112 15:07:33.511123 21796 consensus_queue.cc:1035] T 57e47b07fdcf434e96ca5dd0354e93b2 P 2537bbbd1a9947ae9facd87567e3ccb9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c561ec3374b94b95be0c155d80ff0bff" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 33627 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.000s
I20250112 15:07:33.511221 21796 consensus_queue.cc:1035] T 57e47b07fdcf434e96ca5dd0354e93b2 P 2537bbbd1a9947ae9facd87567e3ccb9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b5e9178a7a83402e8c4a6943ab78589d" member_type: VOTER last_known_addr { host: "127.17.60.197" port: 36573 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.000s
I20250112 15:07:33.511754 21754 raft_consensus.cc:2949] T 57e47b07fdcf434e96ca5dd0354e93b2 P 2537bbbd1a9947ae9facd87567e3ccb9 [term 2 LEADER]: Committing config change with OpId 2.29: config changed from index 28 to 29, VOTER 3710b74ed2044ce5995533df32e67351 (127.17.60.195) evicted. New config: { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "2537bbbd1a9947ae9facd87567e3ccb9" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 33173 } } peers { permanent_uuid: "c561ec3374b94b95be0c155d80ff0bff" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 33627 } } peers { permanent_uuid: "b5e9178a7a83402e8c4a6943ab78589d" member_type: VOTER last_known_addr { host: "127.17.60.197" port: 36573 } attrs { promote: false } } }
I20250112 15:07:33.511852 21549 raft_consensus.cc:2949] T 57e47b07fdcf434e96ca5dd0354e93b2 P b5e9178a7a83402e8c4a6943ab78589d [term 2 FOLLOWER]: Committing config change with OpId 2.29: config changed from index 28 to 29, VOTER 3710b74ed2044ce5995533df32e67351 (127.17.60.195) evicted. New config: { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "2537bbbd1a9947ae9facd87567e3ccb9" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 33173 } } peers { permanent_uuid: "c561ec3374b94b95be0c155d80ff0bff" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 33627 } } peers { permanent_uuid: "b5e9178a7a83402e8c4a6943ab78589d" member_type: VOTER last_known_addr { host: "127.17.60.197" port: 36573 } attrs { promote: false } } }
I20250112 15:07:33.511885 21147 raft_consensus.cc:2949] T 57e47b07fdcf434e96ca5dd0354e93b2 P c561ec3374b94b95be0c155d80ff0bff [term 2 FOLLOWER]: Committing config change with OpId 2.29: config changed from index 28 to 29, VOTER 3710b74ed2044ce5995533df32e67351 (127.17.60.195) evicted. New config: { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "2537bbbd1a9947ae9facd87567e3ccb9" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 33173 } } peers { permanent_uuid: "c561ec3374b94b95be0c155d80ff0bff" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 33627 } } peers { permanent_uuid: "b5e9178a7a83402e8c4a6943ab78589d" member_type: VOTER last_known_addr { host: "127.17.60.197" port: 36573 } attrs { promote: false } } }
I20250112 15:07:33.512746 20956 catalog_manager.cc:5039] ChangeConfig:REMOVE_PEER RPC for tablet 57e47b07fdcf434e96ca5dd0354e93b2 with cas_config_opid_index 28: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250112 15:07:33.512722 20966 catalog_manager.cc:5526] T 57e47b07fdcf434e96ca5dd0354e93b2 P b5e9178a7a83402e8c4a6943ab78589d reported cstate change: config changed from index 28 to 29, VOTER 3710b74ed2044ce5995533df32e67351 (127.17.60.195) evicted. New cstate: current_term: 2 leader_uuid: "2537bbbd1a9947ae9facd87567e3ccb9" committed_config { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "2537bbbd1a9947ae9facd87567e3ccb9" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 33173 } } peers { permanent_uuid: "c561ec3374b94b95be0c155d80ff0bff" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 33627 } } peers { permanent_uuid: "b5e9178a7a83402e8c4a6943ab78589d" member_type: VOTER last_known_addr { host: "127.17.60.197" port: 36573 } attrs { promote: false } } }
I20250112 15:07:33.513420 21212 tablet_service.cc:1514] Processing DeleteTablet for tablet 57e47b07fdcf434e96ca5dd0354e93b2 with delete_type TABLET_DATA_TOMBSTONED (TS 3710b74ed2044ce5995533df32e67351 not found in new config with opid_index 29) from {username='slave'} at 127.0.0.1:33192
I20250112 15:07:33.513660 21800 tablet_replica.cc:331] T 57e47b07fdcf434e96ca5dd0354e93b2 P 3710b74ed2044ce5995533df32e67351: stopping tablet replica
I20250112 15:07:33.513744 21800 raft_consensus.cc:2238] T 57e47b07fdcf434e96ca5dd0354e93b2 P 3710b74ed2044ce5995533df32e67351 [term 2 FOLLOWER]: Raft consensus shutting down.
I20250112 15:07:33.513847 21800 raft_consensus.cc:2267] T 57e47b07fdcf434e96ca5dd0354e93b2 P 3710b74ed2044ce5995533df32e67351 [term 2 FOLLOWER]: Raft consensus is shut down!
I20250112 15:07:33.514041 21800 ts_tablet_manager.cc:1905] T 57e47b07fdcf434e96ca5dd0354e93b2 P 3710b74ed2044ce5995533df32e67351: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250112 15:07:33.515431 21800 ts_tablet_manager.cc:1918] T 57e47b07fdcf434e96ca5dd0354e93b2 P 3710b74ed2044ce5995533df32e67351: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.28
I20250112 15:07:33.515491 21800 log.cc:1198] T 57e47b07fdcf434e96ca5dd0354e93b2 P 3710b74ed2044ce5995533df32e67351: Deleting WAL directory at /tmp/dist-test-task95dLSD/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1736694351070047-17651-0/minicluster-data/ts-2-root/wals/57e47b07fdcf434e96ca5dd0354e93b2
I20250112 15:07:33.515810 20957 catalog_manager.cc:4872] TS 3710b74ed2044ce5995533df32e67351 (127.17.60.195:38589): tablet 57e47b07fdcf434e96ca5dd0354e93b2 (table test-workload [id=cdaf6d3a65a54ea08e8e2c0ecdfc9de7]) successfully deleted
I20250112 15:07:33.927139 21716 raft_consensus.cc:1059] T 5bcec60d3b974a9bb24aedfc738e1594 P 2537bbbd1a9947ae9facd87567e3ccb9: attempting to promote NON_VOTER a2a5bba4ac1145d5bab94a69c9f4df40 to VOTER
I20250112 15:07:33.927362 21716 consensus_queue.cc:237] T 5bcec60d3b974a9bb24aedfc738e1594 P 2537bbbd1a9947ae9facd87567e3ccb9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 26, Committed index: 26, Last appended: 1.26, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "2537bbbd1a9947ae9facd87567e3ccb9" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 33173 } } peers { permanent_uuid: "c561ec3374b94b95be0c155d80ff0bff" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 33627 } attrs { replace: true } } peers { permanent_uuid: "3710b74ed2044ce5995533df32e67351" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 38589 } } peers { permanent_uuid: "a2a5bba4ac1145d5bab94a69c9f4df40" member_type: VOTER last_known_addr { host: "127.17.60.196" port: 41807 } attrs { promote: false } }
I20250112 15:07:33.927806 21222 raft_consensus.cc:1270] T 5bcec60d3b974a9bb24aedfc738e1594 P 3710b74ed2044ce5995533df32e67351 [term 1 FOLLOWER]: Refusing update from remote peer 2537bbbd1a9947ae9facd87567e3ccb9: Log matching property violated. Preceding OpId in replica: term: 1 index: 26. Preceding OpId from leader: term: 1 index: 27. (index mismatch)
I20250112 15:07:33.927953 21474 raft_consensus.cc:1270] T 5bcec60d3b974a9bb24aedfc738e1594 P a2a5bba4ac1145d5bab94a69c9f4df40 [term 1 LEARNER]: Refusing update from remote peer 2537bbbd1a9947ae9facd87567e3ccb9: Log matching property violated. Preceding OpId in replica: term: 1 index: 26. Preceding OpId from leader: term: 1 index: 27. (index mismatch)
I20250112 15:07:33.928001 21147 raft_consensus.cc:1270] T 5bcec60d3b974a9bb24aedfc738e1594 P c561ec3374b94b95be0c155d80ff0bff [term 1 FOLLOWER]: Refusing update from remote peer 2537bbbd1a9947ae9facd87567e3ccb9: Log matching property violated. Preceding OpId in replica: term: 1 index: 26. Preceding OpId from leader: term: 1 index: 27. (index mismatch)
I20250112 15:07:33.928129 21772 consensus_queue.cc:1035] T 5bcec60d3b974a9bb24aedfc738e1594 P 2537bbbd1a9947ae9facd87567e3ccb9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3710b74ed2044ce5995533df32e67351" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 38589 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 27, Last known committed idx: 26, Time since last communication: 0.000s
I20250112 15:07:33.928228 21790 consensus_queue.cc:1035] T 5bcec60d3b974a9bb24aedfc738e1594 P 2537bbbd1a9947ae9facd87567e3ccb9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c561ec3374b94b95be0c155d80ff0bff" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 33627 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 27, Last known committed idx: 26, Time since last communication: 0.000s
I20250112 15:07:33.928332 21716 consensus_queue.cc:1035] T 5bcec60d3b974a9bb24aedfc738e1594 P 2537bbbd1a9947ae9facd87567e3ccb9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2a5bba4ac1145d5bab94a69c9f4df40" member_type: VOTER last_known_addr { host: "127.17.60.196" port: 41807 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 27, Last known committed idx: 26, Time since last communication: 0.000s
I20250112 15:07:33.929172 21772 raft_consensus.cc:2949] T 5bcec60d3b974a9bb24aedfc738e1594 P 2537bbbd1a9947ae9facd87567e3ccb9 [term 1 LEADER]: Committing config change with OpId 1.27: config changed from index 26 to 27, a2a5bba4ac1145d5bab94a69c9f4df40 (127.17.60.196) changed from NON_VOTER to VOTER. New config: { opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "2537bbbd1a9947ae9facd87567e3ccb9" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 33173 } } peers { permanent_uuid: "c561ec3374b94b95be0c155d80ff0bff" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 33627 } attrs { replace: true } } peers { permanent_uuid: "3710b74ed2044ce5995533df32e67351" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 38589 } } peers { permanent_uuid: "a2a5bba4ac1145d5bab94a69c9f4df40" member_type: VOTER last_known_addr { host: "127.17.60.196" port: 41807 } attrs { promote: false } } }
I20250112 15:07:33.929251 21147 raft_consensus.cc:2949] T 5bcec60d3b974a9bb24aedfc738e1594 P c561ec3374b94b95be0c155d80ff0bff [term 1 FOLLOWER]: Committing config change with OpId 1.27: config changed from index 26 to 27, a2a5bba4ac1145d5bab94a69c9f4df40 (127.17.60.196) changed from NON_VOTER to VOTER. New config: { opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "2537bbbd1a9947ae9facd87567e3ccb9" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 33173 } } peers { permanent_uuid: "c561ec3374b94b95be0c155d80ff0bff" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 33627 } attrs { replace: true } } peers { permanent_uuid: "3710b74ed2044ce5995533df32e67351" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 38589 } } peers { permanent_uuid: "a2a5bba4ac1145d5bab94a69c9f4df40" member_type: VOTER last_known_addr { host: "127.17.60.196" port: 41807 } attrs { promote: false } } }
I20250112 15:07:33.929271 21474 raft_consensus.cc:2949] T 5bcec60d3b974a9bb24aedfc738e1594 P a2a5bba4ac1145d5bab94a69c9f4df40 [term 1 FOLLOWER]: Committing config change with OpId 1.27: config changed from index 26 to 27, a2a5bba4ac1145d5bab94a69c9f4df40 (127.17.60.196) changed from NON_VOTER to VOTER. New config: { opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "2537bbbd1a9947ae9facd87567e3ccb9" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 33173 } } peers { permanent_uuid: "c561ec3374b94b95be0c155d80ff0bff" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 33627 } attrs { replace: true } } peers { permanent_uuid: "3710b74ed2044ce5995533df32e67351" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 38589 } } peers { permanent_uuid: "a2a5bba4ac1145d5bab94a69c9f4df40" member_type: VOTER last_known_addr { host: "127.17.60.196" port: 41807 } attrs { promote: false } } }
I20250112 15:07:33.930318 21222 raft_consensus.cc:2949] T 5bcec60d3b974a9bb24aedfc738e1594 P 3710b74ed2044ce5995533df32e67351 [term 1 FOLLOWER]: Committing config change with OpId 1.27: config changed from index 26 to 27, a2a5bba4ac1145d5bab94a69c9f4df40 (127.17.60.196) changed from NON_VOTER to VOTER. New config: { opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "2537bbbd1a9947ae9facd87567e3ccb9" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 33173 } } peers { permanent_uuid: "c561ec3374b94b95be0c155d80ff0bff" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 33627 } attrs { replace: true } } peers { permanent_uuid: "3710b74ed2044ce5995533df32e67351" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 38589 } } peers { permanent_uuid: "a2a5bba4ac1145d5bab94a69c9f4df40" member_type: VOTER last_known_addr { host: "127.17.60.196" port: 41807 } attrs { promote: false } } }
I20250112 15:07:33.930384 20970 catalog_manager.cc:5526] T 5bcec60d3b974a9bb24aedfc738e1594 P c561ec3374b94b95be0c155d80ff0bff reported cstate change: config changed from index 26 to 27, a2a5bba4ac1145d5bab94a69c9f4df40 (127.17.60.196) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "2537bbbd1a9947ae9facd87567e3ccb9" committed_config { opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "2537bbbd1a9947ae9facd87567e3ccb9" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 33173 } } peers { permanent_uuid: "c561ec3374b94b95be0c155d80ff0bff" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 33627 } attrs { replace: true } } peers { permanent_uuid: "3710b74ed2044ce5995533df32e67351" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 38589 } } peers { permanent_uuid: "a2a5bba4ac1145d5bab94a69c9f4df40" member_type: VOTER last_known_addr { host: "127.17.60.196" port: 41807 } attrs { promote: false } } }
I20250112 15:07:33.931468 21069 consensus_queue.cc:237] T 5bcec60d3b974a9bb24aedfc738e1594 P 2537bbbd1a9947ae9facd87567e3ccb9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 27, Committed index: 27, Last appended: 1.27, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "2537bbbd1a9947ae9facd87567e3ccb9" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 33173 } } peers { permanent_uuid: "3710b74ed2044ce5995533df32e67351" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 38589 } } peers { permanent_uuid: "a2a5bba4ac1145d5bab94a69c9f4df40" member_type: VOTER last_known_addr { host: "127.17.60.196" port: 41807 } attrs { promote: false } }
I20250112 15:07:33.931789 21222 raft_consensus.cc:1270] T 5bcec60d3b974a9bb24aedfc738e1594 P 3710b74ed2044ce5995533df32e67351 [term 1 FOLLOWER]: Refusing update from remote peer 2537bbbd1a9947ae9facd87567e3ccb9: Log matching property violated. Preceding OpId in replica: term: 1 index: 27. Preceding OpId from leader: term: 1 index: 28. (index mismatch)
I20250112 15:07:33.931811 21474 raft_consensus.cc:1270] T 5bcec60d3b974a9bb24aedfc738e1594 P a2a5bba4ac1145d5bab94a69c9f4df40 [term 1 FOLLOWER]: Refusing update from remote peer 2537bbbd1a9947ae9facd87567e3ccb9: Log matching property violated. Preceding OpId in replica: term: 1 index: 27. Preceding OpId from leader: term: 1 index: 28. (index mismatch)
I20250112 15:07:33.931957 21790 consensus_queue.cc:1035] T 5bcec60d3b974a9bb24aedfc738e1594 P 2537bbbd1a9947ae9facd87567e3ccb9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3710b74ed2044ce5995533df32e67351" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 38589 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 28, Last known committed idx: 27, Time since last communication: 0.000s
I20250112 15:07:33.932037 21716 consensus_queue.cc:1035] T 5bcec60d3b974a9bb24aedfc738e1594 P 2537bbbd1a9947ae9facd87567e3ccb9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2a5bba4ac1145d5bab94a69c9f4df40" member_type: VOTER last_known_addr { host: "127.17.60.196" port: 41807 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 28, Last known committed idx: 27, Time since last communication: 0.000s
I20250112 15:07:33.932507 21716 raft_consensus.cc:2949] T 5bcec60d3b974a9bb24aedfc738e1594 P 2537bbbd1a9947ae9facd87567e3ccb9 [term 1 LEADER]: Committing config change with OpId 1.28: config changed from index 27 to 28, VOTER c561ec3374b94b95be0c155d80ff0bff (127.17.60.194) evicted. New config: { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "2537bbbd1a9947ae9facd87567e3ccb9" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 33173 } } peers { permanent_uuid: "3710b74ed2044ce5995533df32e67351" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 38589 } } peers { permanent_uuid: "a2a5bba4ac1145d5bab94a69c9f4df40" member_type: VOTER last_known_addr { host: "127.17.60.196" port: 41807 } attrs { promote: false } } }
I20250112 15:07:33.932786 21474 raft_consensus.cc:2949] T 5bcec60d3b974a9bb24aedfc738e1594 P a2a5bba4ac1145d5bab94a69c9f4df40 [term 1 FOLLOWER]: Committing config change with OpId 1.28: config changed from index 27 to 28, VOTER c561ec3374b94b95be0c155d80ff0bff (127.17.60.194) evicted. New config: { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "2537bbbd1a9947ae9facd87567e3ccb9" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 33173 } } peers { permanent_uuid: "3710b74ed2044ce5995533df32e67351" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 38589 } } peers { permanent_uuid: "a2a5bba4ac1145d5bab94a69c9f4df40" member_type: VOTER last_known_addr { host: "127.17.60.196" port: 41807 } attrs { promote: false } } }
I20250112 15:07:33.933395 20956 catalog_manager.cc:5039] ChangeConfig:REMOVE_PEER RPC for tablet 5bcec60d3b974a9bb24aedfc738e1594 with cas_config_opid_index 27: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250112 15:07:33.933362 21222 raft_consensus.cc:2949] T 5bcec60d3b974a9bb24aedfc738e1594 P 3710b74ed2044ce5995533df32e67351 [term 1 FOLLOWER]: Committing config change with OpId 1.28: config changed from index 27 to 28, VOTER c561ec3374b94b95be0c155d80ff0bff (127.17.60.194) evicted. New config: { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "2537bbbd1a9947ae9facd87567e3ccb9" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 33173 } } peers { permanent_uuid: "3710b74ed2044ce5995533df32e67351" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 38589 } } peers { permanent_uuid: "a2a5bba4ac1145d5bab94a69c9f4df40" member_type: VOTER last_known_addr { host: "127.17.60.196" port: 41807 } attrs { promote: false } } }
I20250112 15:07:33.933432 20966 catalog_manager.cc:5526] T 5bcec60d3b974a9bb24aedfc738e1594 P 2537bbbd1a9947ae9facd87567e3ccb9 reported cstate change: config changed from index 27 to 28, VOTER c561ec3374b94b95be0c155d80ff0bff (127.17.60.194) evicted. New cstate: current_term: 1 leader_uuid: "2537bbbd1a9947ae9facd87567e3ccb9" committed_config { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "2537bbbd1a9947ae9facd87567e3ccb9" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 33173 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3710b74ed2044ce5995533df32e67351" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 38589 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a2a5bba4ac1145d5bab94a69c9f4df40" member_type: VOTER last_known_addr { host: "127.17.60.196" port: 41807 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20250112 15:07:33.934077 21141 tablet_service.cc:1514] Processing DeleteTablet for tablet 5bcec60d3b974a9bb24aedfc738e1594 with delete_type TABLET_DATA_TOMBSTONED (TS c561ec3374b94b95be0c155d80ff0bff not found in new config with opid_index 28) from {username='slave'} at 127.0.0.1:48072
I20250112 15:07:33.934177 21794 tablet_replica.cc:331] T 5bcec60d3b974a9bb24aedfc738e1594 P c561ec3374b94b95be0c155d80ff0bff: stopping tablet replica
I20250112 15:07:33.934250 21794 raft_consensus.cc:2238] T 5bcec60d3b974a9bb24aedfc738e1594 P c561ec3374b94b95be0c155d80ff0bff [term 1 FOLLOWER]: Raft consensus shutting down.
I20250112 15:07:33.934325 21794 raft_consensus.cc:2267] T 5bcec60d3b974a9bb24aedfc738e1594 P c561ec3374b94b95be0c155d80ff0bff [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 15:07:33.934577 21794 ts_tablet_manager.cc:1905] T 5bcec60d3b974a9bb24aedfc738e1594 P c561ec3374b94b95be0c155d80ff0bff: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250112 15:07:33.935724 21794 ts_tablet_manager.cc:1918] T 5bcec60d3b974a9bb24aedfc738e1594 P c561ec3374b94b95be0c155d80ff0bff: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.27
I20250112 15:07:33.935768 21794 log.cc:1198] T 5bcec60d3b974a9bb24aedfc738e1594 P c561ec3374b94b95be0c155d80ff0bff: Deleting WAL directory at /tmp/dist-test-task95dLSD/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1736694351070047-17651-0/minicluster-data/ts-1-root/wals/5bcec60d3b974a9bb24aedfc738e1594
I20250112 15:07:33.936075 20958 catalog_manager.cc:4872] TS c561ec3374b94b95be0c155d80ff0bff (127.17.60.194:33627): tablet 5bcec60d3b974a9bb24aedfc738e1594 (table test-workload [id=cdaf6d3a65a54ea08e8e2c0ecdfc9de7]) successfully deleted
I20250112 15:07:34.082345 21026 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250112 15:07:34.085522 21222 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "02c33d457b6b47138dd712850819b619"
dest_uuid: "3710b74ed2044ce5995533df32e67351"
mode: GRACEFUL
new_leader_uuid: "2537bbbd1a9947ae9facd87567e3ccb9"
from {username='slave'} at 127.0.0.1:54970
I20250112 15:07:34.085613 21222 raft_consensus.cc:604] T 02c33d457b6b47138dd712850819b619 P 3710b74ed2044ce5995533df32e67351 [term 2 LEADER]: Received request to transfer leadership to TS 2537bbbd1a9947ae9facd87567e3ccb9
I20250112 15:07:34.086036 21068 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "57e47b07fdcf434e96ca5dd0354e93b2"
dest_uuid: "2537bbbd1a9947ae9facd87567e3ccb9"
mode: GRACEFUL
new_leader_uuid: "b5e9178a7a83402e8c4a6943ab78589d"
from {username='slave'} at 127.0.0.1:54414
I20250112 15:07:34.086103 21068 raft_consensus.cc:604] T 57e47b07fdcf434e96ca5dd0354e93b2 P 2537bbbd1a9947ae9facd87567e3ccb9 [term 2 LEADER]: Received request to transfer leadership to TS b5e9178a7a83402e8c4a6943ab78589d
I20250112 15:07:34.086371 21068 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "5bcec60d3b974a9bb24aedfc738e1594"
dest_uuid: "2537bbbd1a9947ae9facd87567e3ccb9"
mode: GRACEFUL
new_leader_uuid: "a2a5bba4ac1145d5bab94a69c9f4df40"
from {username='slave'} at 127.0.0.1:54414
I20250112 15:07:34.086416 21068 raft_consensus.cc:604] T 5bcec60d3b974a9bb24aedfc738e1594 P 2537bbbd1a9947ae9facd87567e3ccb9 [term 1 LEADER]: Received request to transfer leadership to TS a2a5bba4ac1145d5bab94a69c9f4df40
I20250112 15:07:34.086728 21222 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "8c7dbaf285a04da9b4eec92a29ab7d6c"
dest_uuid: "3710b74ed2044ce5995533df32e67351"
mode: GRACEFUL
new_leader_uuid: "2537bbbd1a9947ae9facd87567e3ccb9"
from {username='slave'} at 127.0.0.1:54970
I20250112 15:07:34.086781 21222 raft_consensus.cc:604] T 8c7dbaf285a04da9b4eec92a29ab7d6c P 3710b74ed2044ce5995533df32e67351 [term 1 LEADER]: Received request to transfer leadership to TS 2537bbbd1a9947ae9facd87567e3ccb9
I20250112 15:07:34.087080 21147 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "ef90b704208140a3983ffe6105939319"
dest_uuid: "c561ec3374b94b95be0c155d80ff0bff"
mode: GRACEFUL
new_leader_uuid: "3710b74ed2044ce5995533df32e67351"
from {username='slave'} at 127.0.0.1:48104
I20250112 15:07:34.087145 21147 raft_consensus.cc:604] T ef90b704208140a3983ffe6105939319 P c561ec3374b94b95be0c155d80ff0bff [term 2 LEADER]: Received request to transfer leadership to TS 3710b74ed2044ce5995533df32e67351
I20250112 15:07:34.087297 21026 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 5
I20250112 15:07:34.087359 21026 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20250112 15:07:34.133044 21676 raft_consensus.cc:988] T 8c7dbaf285a04da9b4eec92a29ab7d6c P 3710b74ed2044ce5995533df32e67351: : Instructing follower 2537bbbd1a9947ae9facd87567e3ccb9 to start an election
I20250112 15:07:34.133142 21676 raft_consensus.cc:1076] T 8c7dbaf285a04da9b4eec92a29ab7d6c P 3710b74ed2044ce5995533df32e67351 [term 1 LEADER]: Signalling peer 2537bbbd1a9947ae9facd87567e3ccb9 to start an election
I20250112 15:07:34.133424 21068 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "8c7dbaf285a04da9b4eec92a29ab7d6c"
dest_uuid: "2537bbbd1a9947ae9facd87567e3ccb9"
from {username='slave'} at 127.0.0.1:37204
I20250112 15:07:34.133528 21068 raft_consensus.cc:491] T 8c7dbaf285a04da9b4eec92a29ab7d6c P 2537bbbd1a9947ae9facd87567e3ccb9 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20250112 15:07:34.133553 21068 raft_consensus.cc:3054] T 8c7dbaf285a04da9b4eec92a29ab7d6c P 2537bbbd1a9947ae9facd87567e3ccb9 [term 1 FOLLOWER]: Advancing to term 2
I20250112 15:07:34.134378 21068 raft_consensus.cc:513] T 8c7dbaf285a04da9b4eec92a29ab7d6c P 2537bbbd1a9947ae9facd87567e3ccb9 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2537bbbd1a9947ae9facd87567e3ccb9" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 33173 } } peers { permanent_uuid: "c561ec3374b94b95be0c155d80ff0bff" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 33627 } } peers { permanent_uuid: "3710b74ed2044ce5995533df32e67351" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 38589 } }
I20250112 15:07:34.134529 21068 leader_election.cc:290] T 8c7dbaf285a04da9b4eec92a29ab7d6c P 2537bbbd1a9947ae9facd87567e3ccb9 [CANDIDATE]: Term 2 election: Requested vote from peers c561ec3374b94b95be0c155d80ff0bff (127.17.60.194:33627), 3710b74ed2044ce5995533df32e67351 (127.17.60.195:38589)
I20250112 15:07:34.134647 21147 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c7dbaf285a04da9b4eec92a29ab7d6c" candidate_uuid: "2537bbbd1a9947ae9facd87567e3ccb9" candidate_term: 2 candidate_status { last_received { term: 1 index: 25 } } ignore_live_leader: true dest_uuid: "c561ec3374b94b95be0c155d80ff0bff"
I20250112 15:07:34.134711 21147 raft_consensus.cc:3054] T 8c7dbaf285a04da9b4eec92a29ab7d6c P c561ec3374b94b95be0c155d80ff0bff [term 1 FOLLOWER]: Advancing to term 2
I20250112 15:07:34.134791 21222 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c7dbaf285a04da9b4eec92a29ab7d6c" candidate_uuid: "2537bbbd1a9947ae9facd87567e3ccb9" candidate_term: 2 candidate_status { last_received { term: 1 index: 25 } } ignore_live_leader: true dest_uuid: "3710b74ed2044ce5995533df32e67351"
I20250112 15:07:34.134877 21222 raft_consensus.cc:3049] T 8c7dbaf285a04da9b4eec92a29ab7d6c P 3710b74ed2044ce5995533df32e67351 [term 1 LEADER]: Stepping down as leader of term 1
I20250112 15:07:34.134909 21222 raft_consensus.cc:738] T 8c7dbaf285a04da9b4eec92a29ab7d6c P 3710b74ed2044ce5995533df32e67351 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 3710b74ed2044ce5995533df32e67351, State: Running, Role: LEADER
I20250112 15:07:34.134982 21222 consensus_queue.cc:260] T 8c7dbaf285a04da9b4eec92a29ab7d6c P 3710b74ed2044ce5995533df32e67351 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 25, Committed index: 25, Last appended: 1.25, Last appended by leader: 25, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2537bbbd1a9947ae9facd87567e3ccb9" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 33173 } } peers { permanent_uuid: "c561ec3374b94b95be0c155d80ff0bff" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 33627 } } peers { permanent_uuid: "3710b74ed2044ce5995533df32e67351" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 38589 } }
I20250112 15:07:34.135088 21222 raft_consensus.cc:3054] T 8c7dbaf285a04da9b4eec92a29ab7d6c P 3710b74ed2044ce5995533df32e67351 [term 1 FOLLOWER]: Advancing to term 2
I20250112 15:07:34.135440 21147 raft_consensus.cc:2463] T 8c7dbaf285a04da9b4eec92a29ab7d6c P c561ec3374b94b95be0c155d80ff0bff [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2537bbbd1a9947ae9facd87567e3ccb9 in term 2.
I20250112 15:07:34.135599 21040 leader_election.cc:304] T 8c7dbaf285a04da9b4eec92a29ab7d6c P 2537bbbd1a9947ae9facd87567e3ccb9 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2537bbbd1a9947ae9facd87567e3ccb9, c561ec3374b94b95be0c155d80ff0bff; no voters:
I20250112 15:07:34.135812 21222 raft_consensus.cc:2463] T 8c7dbaf285a04da9b4eec92a29ab7d6c P 3710b74ed2044ce5995533df32e67351 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2537bbbd1a9947ae9facd87567e3ccb9 in term 2.
I20250112 15:07:34.135977 21790 raft_consensus.cc:2798] T 8c7dbaf285a04da9b4eec92a29ab7d6c P 2537bbbd1a9947ae9facd87567e3ccb9 [term 2 FOLLOWER]: Leader election won for term 2
I20250112 15:07:34.136029 21790 raft_consensus.cc:695] T 8c7dbaf285a04da9b4eec92a29ab7d6c P 2537bbbd1a9947ae9facd87567e3ccb9 [term 2 LEADER]: Becoming Leader. State: Replica: 2537bbbd1a9947ae9facd87567e3ccb9, State: Running, Role: LEADER
I20250112 15:07:34.136080 21790 consensus_queue.cc:237] T 8c7dbaf285a04da9b4eec92a29ab7d6c P 2537bbbd1a9947ae9facd87567e3ccb9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 25, Committed index: 25, Last appended: 1.25, Last appended by leader: 25, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2537bbbd1a9947ae9facd87567e3ccb9" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 33173 } } peers { permanent_uuid: "c561ec3374b94b95be0c155d80ff0bff" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 33627 } } peers { permanent_uuid: "3710b74ed2044ce5995533df32e67351" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 38589 } }
I20250112 15:07:34.136667 20964 catalog_manager.cc:5526] T 8c7dbaf285a04da9b4eec92a29ab7d6c P 2537bbbd1a9947ae9facd87567e3ccb9 reported cstate change: term changed from 1 to 2, leader changed from 3710b74ed2044ce5995533df32e67351 (127.17.60.195) to 2537bbbd1a9947ae9facd87567e3ccb9 (127.17.60.193). New cstate: current_term: 2 leader_uuid: "2537bbbd1a9947ae9facd87567e3ccb9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2537bbbd1a9947ae9facd87567e3ccb9" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 33173 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c561ec3374b94b95be0c155d80ff0bff" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 33627 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3710b74ed2044ce5995533df32e67351" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 38589 } health_report { overall_health: UNKNOWN } } }
I20250112 15:07:34.159547 21775 raft_consensus.cc:988] T ef90b704208140a3983ffe6105939319 P c561ec3374b94b95be0c155d80ff0bff: : Instructing follower 3710b74ed2044ce5995533df32e67351 to start an election
I20250112 15:07:34.159641 21775 raft_consensus.cc:1076] T ef90b704208140a3983ffe6105939319 P c561ec3374b94b95be0c155d80ff0bff [term 2 LEADER]: Signalling peer 3710b74ed2044ce5995533df32e67351 to start an election
I20250112 15:07:34.159899 21222 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "ef90b704208140a3983ffe6105939319"
dest_uuid: "3710b74ed2044ce5995533df32e67351"
from {username='slave'} at 127.0.0.1:33202
I20250112 15:07:34.159982 21222 raft_consensus.cc:491] T ef90b704208140a3983ffe6105939319 P 3710b74ed2044ce5995533df32e67351 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20250112 15:07:34.160018 21222 raft_consensus.cc:3054] T ef90b704208140a3983ffe6105939319 P 3710b74ed2044ce5995533df32e67351 [term 2 FOLLOWER]: Advancing to term 3
I20250112 15:07:34.160919 21222 raft_consensus.cc:513] T ef90b704208140a3983ffe6105939319 P 3710b74ed2044ce5995533df32e67351 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2537bbbd1a9947ae9facd87567e3ccb9" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 33173 } } peers { permanent_uuid: "c561ec3374b94b95be0c155d80ff0bff" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 33627 } } peers { permanent_uuid: "3710b74ed2044ce5995533df32e67351" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 38589 } }
I20250112 15:07:34.161062 21222 leader_election.cc:290] T ef90b704208140a3983ffe6105939319 P 3710b74ed2044ce5995533df32e67351 [CANDIDATE]: Term 3 election: Requested vote from peers 2537bbbd1a9947ae9facd87567e3ccb9 (127.17.60.193:33173), c561ec3374b94b95be0c155d80ff0bff (127.17.60.194:33627)
I20250112 15:07:34.161193 21068 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef90b704208140a3983ffe6105939319" candidate_uuid: "3710b74ed2044ce5995533df32e67351" candidate_term: 3 candidate_status { last_received { term: 2 index: 26 } } ignore_live_leader: true dest_uuid: "2537bbbd1a9947ae9facd87567e3ccb9"
I20250112 15:07:34.161243 21147 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef90b704208140a3983ffe6105939319" candidate_uuid: "3710b74ed2044ce5995533df32e67351" candidate_term: 3 candidate_status { last_received { term: 2 index: 26 } } ignore_live_leader: true dest_uuid: "c561ec3374b94b95be0c155d80ff0bff"
I20250112 15:07:34.161268 21068 raft_consensus.cc:3054] T ef90b704208140a3983ffe6105939319 P 2537bbbd1a9947ae9facd87567e3ccb9 [term 2 FOLLOWER]: Advancing to term 3
I20250112 15:07:34.161293 21147 raft_consensus.cc:3049] T ef90b704208140a3983ffe6105939319 P c561ec3374b94b95be0c155d80ff0bff [term 2 LEADER]: Stepping down as leader of term 2
I20250112 15:07:34.161314 21147 raft_consensus.cc:738] T ef90b704208140a3983ffe6105939319 P c561ec3374b94b95be0c155d80ff0bff [term 2 LEADER]: Becoming Follower/Learner. State: Replica: c561ec3374b94b95be0c155d80ff0bff, State: Running, Role: LEADER
I20250112 15:07:34.161360 21147 consensus_queue.cc:260] T ef90b704208140a3983ffe6105939319 P c561ec3374b94b95be0c155d80ff0bff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 26, Committed index: 26, Last appended: 2.26, Last appended by leader: 26, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2537bbbd1a9947ae9facd87567e3ccb9" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 33173 } } peers { permanent_uuid: "c561ec3374b94b95be0c155d80ff0bff" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 33627 } } peers { permanent_uuid: "3710b74ed2044ce5995533df32e67351" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 38589 } }
I20250112 15:07:34.161427 21147 raft_consensus.cc:3054] T ef90b704208140a3983ffe6105939319 P c561ec3374b94b95be0c155d80ff0bff [term 2 FOLLOWER]: Advancing to term 3
I20250112 15:07:34.161957 21147 raft_consensus.cc:2463] T ef90b704208140a3983ffe6105939319 P c561ec3374b94b95be0c155d80ff0bff [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3710b74ed2044ce5995533df32e67351 in term 3.
I20250112 15:07:34.162119 21191 leader_election.cc:304] T ef90b704208140a3983ffe6105939319 P 3710b74ed2044ce5995533df32e67351 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3710b74ed2044ce5995533df32e67351, c561ec3374b94b95be0c155d80ff0bff; no voters:
I20250112 15:07:34.161957 21068 raft_consensus.cc:2463] T ef90b704208140a3983ffe6105939319 P 2537bbbd1a9947ae9facd87567e3ccb9 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3710b74ed2044ce5995533df32e67351 in term 3.
I20250112 15:07:34.162217 21807 raft_consensus.cc:2798] T ef90b704208140a3983ffe6105939319 P 3710b74ed2044ce5995533df32e67351 [term 3 FOLLOWER]: Leader election won for term 3
I20250112 15:07:34.162295 21807 raft_consensus.cc:695] T ef90b704208140a3983ffe6105939319 P 3710b74ed2044ce5995533df32e67351 [term 3 LEADER]: Becoming Leader. State: Replica: 3710b74ed2044ce5995533df32e67351, State: Running, Role: LEADER
I20250112 15:07:34.162349 21807 consensus_queue.cc:237] T ef90b704208140a3983ffe6105939319 P 3710b74ed2044ce5995533df32e67351 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 26, Committed index: 26, Last appended: 2.26, Last appended by leader: 26, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2537bbbd1a9947ae9facd87567e3ccb9" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 33173 } } peers { permanent_uuid: "c561ec3374b94b95be0c155d80ff0bff" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 33627 } } peers { permanent_uuid: "3710b74ed2044ce5995533df32e67351" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 38589 } }
I20250112 15:07:34.163059 20964 catalog_manager.cc:5526] T ef90b704208140a3983ffe6105939319 P 3710b74ed2044ce5995533df32e67351 reported cstate change: term changed from 2 to 3, leader changed from c561ec3374b94b95be0c155d80ff0bff (127.17.60.194) to 3710b74ed2044ce5995533df32e67351 (127.17.60.195). New cstate: current_term: 3 leader_uuid: "3710b74ed2044ce5995533df32e67351" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2537bbbd1a9947ae9facd87567e3ccb9" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 33173 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c561ec3374b94b95be0c155d80ff0bff" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 33627 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3710b74ed2044ce5995533df32e67351" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 38589 } health_report { overall_health: HEALTHY } } }
I20250112 15:07:34.185176 21676 raft_consensus.cc:988] T 02c33d457b6b47138dd712850819b619 P 3710b74ed2044ce5995533df32e67351: : Instructing follower 2537bbbd1a9947ae9facd87567e3ccb9 to start an election
I20250112 15:07:34.185266 21676 raft_consensus.cc:1076] T 02c33d457b6b47138dd712850819b619 P 3710b74ed2044ce5995533df32e67351 [term 2 LEADER]: Signalling peer 2537bbbd1a9947ae9facd87567e3ccb9 to start an election
I20250112 15:07:34.185439 21068 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "02c33d457b6b47138dd712850819b619"
dest_uuid: "2537bbbd1a9947ae9facd87567e3ccb9"
from {username='slave'} at 127.0.0.1:37204
I20250112 15:07:34.185523 21068 raft_consensus.cc:491] T 02c33d457b6b47138dd712850819b619 P 2537bbbd1a9947ae9facd87567e3ccb9 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20250112 15:07:34.185556 21068 raft_consensus.cc:3054] T 02c33d457b6b47138dd712850819b619 P 2537bbbd1a9947ae9facd87567e3ccb9 [term 2 FOLLOWER]: Advancing to term 3
I20250112 15:07:34.186151 21068 raft_consensus.cc:513] T 02c33d457b6b47138dd712850819b619 P 2537bbbd1a9947ae9facd87567e3ccb9 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3710b74ed2044ce5995533df32e67351" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 38589 } } peers { permanent_uuid: "2537bbbd1a9947ae9facd87567e3ccb9" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 33173 } } peers { permanent_uuid: "c561ec3374b94b95be0c155d80ff0bff" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 33627 } }
I20250112 15:07:34.186309 21068 leader_election.cc:290] T 02c33d457b6b47138dd712850819b619 P 2537bbbd1a9947ae9facd87567e3ccb9 [CANDIDATE]: Term 3 election: Requested vote from peers 3710b74ed2044ce5995533df32e67351 (127.17.60.195:38589), c561ec3374b94b95be0c155d80ff0bff (127.17.60.194:33627)
I20250112 15:07:34.186450 21222 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "02c33d457b6b47138dd712850819b619" candidate_uuid: "2537bbbd1a9947ae9facd87567e3ccb9" candidate_term: 3 candidate_status { last_received { term: 2 index: 26 } } ignore_live_leader: true dest_uuid: "3710b74ed2044ce5995533df32e67351"
I20250112 15:07:34.186508 21147 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "02c33d457b6b47138dd712850819b619" candidate_uuid: "2537bbbd1a9947ae9facd87567e3ccb9" candidate_term: 3 candidate_status { last_received { term: 2 index: 26 } } ignore_live_leader: true dest_uuid: "c561ec3374b94b95be0c155d80ff0bff"
I20250112 15:07:34.186522 21222 raft_consensus.cc:3049] T 02c33d457b6b47138dd712850819b619 P 3710b74ed2044ce5995533df32e67351 [term 2 LEADER]: Stepping down as leader of term 2
I20250112 15:07:34.186601 21147 raft_consensus.cc:3054] T 02c33d457b6b47138dd712850819b619 P c561ec3374b94b95be0c155d80ff0bff [term 2 FOLLOWER]: Advancing to term 3
I20250112 15:07:34.186609 21222 raft_consensus.cc:738] T 02c33d457b6b47138dd712850819b619 P 3710b74ed2044ce5995533df32e67351 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 3710b74ed2044ce5995533df32e67351, State: Running, Role: LEADER
I20250112 15:07:34.186693 21222 consensus_queue.cc:260] T 02c33d457b6b47138dd712850819b619 P 3710b74ed2044ce5995533df32e67351 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 26, Committed index: 26, Last appended: 2.26, Last appended by leader: 26, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3710b74ed2044ce5995533df32e67351" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 38589 } } peers { permanent_uuid: "2537bbbd1a9947ae9facd87567e3ccb9" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 33173 } } peers { permanent_uuid: "c561ec3374b94b95be0c155d80ff0bff" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 33627 } }
I20250112 15:07:34.186800 21222 raft_consensus.cc:3054] T 02c33d457b6b47138dd712850819b619 P 3710b74ed2044ce5995533df32e67351 [term 2 FOLLOWER]: Advancing to term 3
I20250112 15:07:34.187239 21147 raft_consensus.cc:2463] T 02c33d457b6b47138dd712850819b619 P c561ec3374b94b95be0c155d80ff0bff [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2537bbbd1a9947ae9facd87567e3ccb9 in term 3.
I20250112 15:07:34.187407 21222 raft_consensus.cc:2463] T 02c33d457b6b47138dd712850819b619 P 3710b74ed2044ce5995533df32e67351 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2537bbbd1a9947ae9facd87567e3ccb9 in term 3.
I20250112 15:07:34.187462 21040 leader_election.cc:304] T 02c33d457b6b47138dd712850819b619 P 2537bbbd1a9947ae9facd87567e3ccb9 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2537bbbd1a9947ae9facd87567e3ccb9, c561ec3374b94b95be0c155d80ff0bff; no voters:
I20250112 15:07:34.187551 21790 raft_consensus.cc:2798] T 02c33d457b6b47138dd712850819b619 P 2537bbbd1a9947ae9facd87567e3ccb9 [term 3 FOLLOWER]: Leader election won for term 3
I20250112 15:07:34.187604 21790 raft_consensus.cc:695] T 02c33d457b6b47138dd712850819b619 P 2537bbbd1a9947ae9facd87567e3ccb9 [term 3 LEADER]: Becoming Leader. State: Replica: 2537bbbd1a9947ae9facd87567e3ccb9, State: Running, Role: LEADER
I20250112 15:07:34.187657 21790 consensus_queue.cc:237] T 02c33d457b6b47138dd712850819b619 P 2537bbbd1a9947ae9facd87567e3ccb9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 26, Committed index: 26, Last appended: 2.26, Last appended by leader: 26, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3710b74ed2044ce5995533df32e67351" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 38589 } } peers { permanent_uuid: "2537bbbd1a9947ae9facd87567e3ccb9" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 33173 } } peers { permanent_uuid: "c561ec3374b94b95be0c155d80ff0bff" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 33627 } }
I20250112 15:07:34.188236 20964 catalog_manager.cc:5526] T 02c33d457b6b47138dd712850819b619 P 2537bbbd1a9947ae9facd87567e3ccb9 reported cstate change: term changed from 2 to 3, leader changed from 3710b74ed2044ce5995533df32e67351 (127.17.60.195) to 2537bbbd1a9947ae9facd87567e3ccb9 (127.17.60.193). New cstate: current_term: 3 leader_uuid: "2537bbbd1a9947ae9facd87567e3ccb9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3710b74ed2044ce5995533df32e67351" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 38589 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2537bbbd1a9947ae9facd87567e3ccb9" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 33173 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c561ec3374b94b95be0c155d80ff0bff" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 33627 } health_report { overall_health: UNKNOWN } } }
I20250112 15:07:34.451211 21796 raft_consensus.cc:988] T 5bcec60d3b974a9bb24aedfc738e1594 P 2537bbbd1a9947ae9facd87567e3ccb9: : Instructing follower a2a5bba4ac1145d5bab94a69c9f4df40 to start an election
I20250112 15:07:34.451289 21796 raft_consensus.cc:1076] T 5bcec60d3b974a9bb24aedfc738e1594 P 2537bbbd1a9947ae9facd87567e3ccb9 [term 1 LEADER]: Signalling peer a2a5bba4ac1145d5bab94a69c9f4df40 to start an election
I20250112 15:07:34.451460 21474 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "5bcec60d3b974a9bb24aedfc738e1594"
dest_uuid: "a2a5bba4ac1145d5bab94a69c9f4df40"
from {username='slave'} at 127.0.0.1:43268
I20250112 15:07:34.451552 21474 raft_consensus.cc:491] T 5bcec60d3b974a9bb24aedfc738e1594 P a2a5bba4ac1145d5bab94a69c9f4df40 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20250112 15:07:34.451573 21474 raft_consensus.cc:3054] T 5bcec60d3b974a9bb24aedfc738e1594 P a2a5bba4ac1145d5bab94a69c9f4df40 [term 1 FOLLOWER]: Advancing to term 2
I20250112 15:07:34.453549 21474 raft_consensus.cc:513] T 5bcec60d3b974a9bb24aedfc738e1594 P a2a5bba4ac1145d5bab94a69c9f4df40 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "2537bbbd1a9947ae9facd87567e3ccb9" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 33173 } } peers { permanent_uuid: "3710b74ed2044ce5995533df32e67351" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 38589 } } peers { permanent_uuid: "a2a5bba4ac1145d5bab94a69c9f4df40" member_type: VOTER last_known_addr { host: "127.17.60.196" port: 41807 } attrs { promote: false } }
I20250112 15:07:34.453790 21474 leader_election.cc:290] T 5bcec60d3b974a9bb24aedfc738e1594 P a2a5bba4ac1145d5bab94a69c9f4df40 [CANDIDATE]: Term 2 election: Requested vote from peers 2537bbbd1a9947ae9facd87567e3ccb9 (127.17.60.193:33173), 3710b74ed2044ce5995533df32e67351 (127.17.60.195:38589)
I20250112 15:07:34.453899 21068 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5bcec60d3b974a9bb24aedfc738e1594" candidate_uuid: "a2a5bba4ac1145d5bab94a69c9f4df40" candidate_term: 2 candidate_status { last_received { term: 1 index: 28 } } ignore_live_leader: true dest_uuid: "2537bbbd1a9947ae9facd87567e3ccb9"
I20250112 15:07:34.453984 21068 raft_consensus.cc:3049] T 5bcec60d3b974a9bb24aedfc738e1594 P 2537bbbd1a9947ae9facd87567e3ccb9 [term 1 LEADER]: Stepping down as leader of term 1
I20250112 15:07:34.454018 21068 raft_consensus.cc:738] T 5bcec60d3b974a9bb24aedfc738e1594 P 2537bbbd1a9947ae9facd87567e3ccb9 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 2537bbbd1a9947ae9facd87567e3ccb9, State: Running, Role: LEADER
I20250112 15:07:34.454077 21068 consensus_queue.cc:260] T 5bcec60d3b974a9bb24aedfc738e1594 P 2537bbbd1a9947ae9facd87567e3ccb9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 28, Committed index: 28, Last appended: 1.28, Last appended by leader: 28, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "2537bbbd1a9947ae9facd87567e3ccb9" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 33173 } } peers { permanent_uuid: "3710b74ed2044ce5995533df32e67351" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 38589 } } peers { permanent_uuid: "a2a5bba4ac1145d5bab94a69c9f4df40" member_type: VOTER last_known_addr { host: "127.17.60.196" port: 41807 } attrs { promote: false } }
I20250112 15:07:34.454193 21068 raft_consensus.cc:3054] T 5bcec60d3b974a9bb24aedfc738e1594 P 2537bbbd1a9947ae9facd87567e3ccb9 [term 1 FOLLOWER]: Advancing to term 2
I20250112 15:07:34.454871 21068 raft_consensus.cc:2463] T 5bcec60d3b974a9bb24aedfc738e1594 P 2537bbbd1a9947ae9facd87567e3ccb9 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2a5bba4ac1145d5bab94a69c9f4df40 in term 2.
I20250112 15:07:34.455061 21437 leader_election.cc:304] T 5bcec60d3b974a9bb24aedfc738e1594 P a2a5bba4ac1145d5bab94a69c9f4df40 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2537bbbd1a9947ae9facd87567e3ccb9, a2a5bba4ac1145d5bab94a69c9f4df40; no voters:
I20250112 15:07:34.455210 21813 raft_consensus.cc:2798] T 5bcec60d3b974a9bb24aedfc738e1594 P a2a5bba4ac1145d5bab94a69c9f4df40 [term 2 FOLLOWER]: Leader election won for term 2
I20250112 15:07:34.455324 21813 raft_consensus.cc:695] T 5bcec60d3b974a9bb24aedfc738e1594 P a2a5bba4ac1145d5bab94a69c9f4df40 [term 2 LEADER]: Becoming Leader. State: Replica: a2a5bba4ac1145d5bab94a69c9f4df40, State: Running, Role: LEADER
I20250112 15:07:34.455401 21813 consensus_queue.cc:237] T 5bcec60d3b974a9bb24aedfc738e1594 P a2a5bba4ac1145d5bab94a69c9f4df40 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 28, Committed index: 28, Last appended: 1.28, Last appended by leader: 28, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "2537bbbd1a9947ae9facd87567e3ccb9" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 33173 } } peers { permanent_uuid: "3710b74ed2044ce5995533df32e67351" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 38589 } } peers { permanent_uuid: "a2a5bba4ac1145d5bab94a69c9f4df40" member_type: VOTER last_known_addr { host: "127.17.60.196" port: 41807 } attrs { promote: false } }
I20250112 15:07:34.455876 20964 catalog_manager.cc:5526] T 5bcec60d3b974a9bb24aedfc738e1594 P a2a5bba4ac1145d5bab94a69c9f4df40 reported cstate change: term changed from 1 to 2, leader changed from 2537bbbd1a9947ae9facd87567e3ccb9 (127.17.60.193) to a2a5bba4ac1145d5bab94a69c9f4df40 (127.17.60.196). New cstate: current_term: 2 leader_uuid: "a2a5bba4ac1145d5bab94a69c9f4df40" committed_config { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "2537bbbd1a9947ae9facd87567e3ccb9" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 33173 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3710b74ed2044ce5995533df32e67351" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 38589 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a2a5bba4ac1145d5bab94a69c9f4df40" member_type: VOTER last_known_addr { host: "127.17.60.196" port: 41807 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20250112 15:07:34.456699 21226 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5bcec60d3b974a9bb24aedfc738e1594" candidate_uuid: "a2a5bba4ac1145d5bab94a69c9f4df40" candidate_term: 2 candidate_status { last_received { term: 1 index: 28 } } ignore_live_leader: true dest_uuid: "3710b74ed2044ce5995533df32e67351"
I20250112 15:07:34.456755 21226 raft_consensus.cc:3054] T 5bcec60d3b974a9bb24aedfc738e1594 P 3710b74ed2044ce5995533df32e67351 [term 1 FOLLOWER]: Advancing to term 2
I20250112 15:07:34.457234 21226 raft_consensus.cc:2463] T 5bcec60d3b974a9bb24aedfc738e1594 P 3710b74ed2044ce5995533df32e67351 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2a5bba4ac1145d5bab94a69c9f4df40 in term 2.
I20250112 15:07:34.512665 21147 raft_consensus.cc:1270] T 8c7dbaf285a04da9b4eec92a29ab7d6c P c561ec3374b94b95be0c155d80ff0bff [term 2 FOLLOWER]: Refusing update from remote peer 2537bbbd1a9947ae9facd87567e3ccb9: Log matching property violated. Preceding OpId in replica: term: 1 index: 25. Preceding OpId from leader: term: 2 index: 26. (index mismatch)
I20250112 15:07:34.512976 21796 consensus_queue.cc:1035] T 8c7dbaf285a04da9b4eec92a29ab7d6c P 2537bbbd1a9947ae9facd87567e3ccb9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c561ec3374b94b95be0c155d80ff0bff" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 33627 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 26, Last known committed idx: 25, Time since last communication: 0.000s
I20250112 15:07:34.514356 21226 raft_consensus.cc:1270] T 8c7dbaf285a04da9b4eec92a29ab7d6c P 3710b74ed2044ce5995533df32e67351 [term 2 FOLLOWER]: Refusing update from remote peer 2537bbbd1a9947ae9facd87567e3ccb9: Log matching property violated. Preceding OpId in replica: term: 1 index: 25. Preceding OpId from leader: term: 2 index: 26. (index mismatch)
I20250112 15:07:34.514552 21796 consensus_queue.cc:1035] T 8c7dbaf285a04da9b4eec92a29ab7d6c P 2537bbbd1a9947ae9facd87567e3ccb9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3710b74ed2044ce5995533df32e67351" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 38589 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 26, Last known committed idx: 25, Time since last communication: 0.000s
I20250112 15:07:34.597195 21068 raft_consensus.cc:1270] T ef90b704208140a3983ffe6105939319 P 2537bbbd1a9947ae9facd87567e3ccb9 [term 3 FOLLOWER]: Refusing update from remote peer 3710b74ed2044ce5995533df32e67351: Log matching property violated. Preceding OpId in replica: term: 2 index: 26. Preceding OpId from leader: term: 3 index: 27. (index mismatch)
I20250112 15:07:34.597455 21676 consensus_queue.cc:1035] T ef90b704208140a3983ffe6105939319 P 3710b74ed2044ce5995533df32e67351 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2537bbbd1a9947ae9facd87567e3ccb9" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 33173 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 27, Last known committed idx: 26, Time since last communication: 0.000s
I20250112 15:07:34.598451 21147 raft_consensus.cc:1270] T ef90b704208140a3983ffe6105939319 P c561ec3374b94b95be0c155d80ff0bff [term 3 FOLLOWER]: Refusing update from remote peer 3710b74ed2044ce5995533df32e67351: Log matching property violated. Preceding OpId in replica: term: 2 index: 26. Preceding OpId from leader: term: 3 index: 27. (index mismatch)
I20250112 15:07:34.598632 21807 consensus_queue.cc:1035] T ef90b704208140a3983ffe6105939319 P 3710b74ed2044ce5995533df32e67351 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c561ec3374b94b95be0c155d80ff0bff" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 33627 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 27, Last known committed idx: 26, Time since last communication: 0.000s
I20250112 15:07:34.602521 21226 raft_consensus.cc:1270] T 02c33d457b6b47138dd712850819b619 P 3710b74ed2044ce5995533df32e67351 [term 3 FOLLOWER]: Refusing update from remote peer 2537bbbd1a9947ae9facd87567e3ccb9: Log matching property violated. Preceding OpId in replica: term: 2 index: 26. Preceding OpId from leader: term: 3 index: 27. (index mismatch)
I20250112 15:07:34.602686 21796 consensus_queue.cc:1035] T 02c33d457b6b47138dd712850819b619 P 2537bbbd1a9947ae9facd87567e3ccb9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3710b74ed2044ce5995533df32e67351" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 38589 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 27, Last known committed idx: 26, Time since last communication: 0.000s
I20250112 15:07:34.603545 21147 raft_consensus.cc:1270] T 02c33d457b6b47138dd712850819b619 P c561ec3374b94b95be0c155d80ff0bff [term 3 FOLLOWER]: Refusing update from remote peer 2537bbbd1a9947ae9facd87567e3ccb9: Log matching property violated. Preceding OpId in replica: term: 2 index: 26. Preceding OpId from leader: term: 3 index: 27. (index mismatch)
I20250112 15:07:34.603859 21823 consensus_queue.cc:1035] T 02c33d457b6b47138dd712850819b619 P 2537bbbd1a9947ae9facd87567e3ccb9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c561ec3374b94b95be0c155d80ff0bff" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 33627 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 27, Last known committed idx: 26, Time since last communication: 0.000s
I20250112 15:07:34.641670 21796 raft_consensus.cc:988] T 57e47b07fdcf434e96ca5dd0354e93b2 P 2537bbbd1a9947ae9facd87567e3ccb9: : Instructing follower b5e9178a7a83402e8c4a6943ab78589d to start an election
I20250112 15:07:34.641779 21796 raft_consensus.cc:1076] T 57e47b07fdcf434e96ca5dd0354e93b2 P 2537bbbd1a9947ae9facd87567e3ccb9 [term 2 LEADER]: Signalling peer b5e9178a7a83402e8c4a6943ab78589d to start an election
I20250112 15:07:34.641917 21549 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "57e47b07fdcf434e96ca5dd0354e93b2"
dest_uuid: "b5e9178a7a83402e8c4a6943ab78589d"
from {username='slave'} at 127.0.0.1:44694
I20250112 15:07:34.641975 21549 raft_consensus.cc:491] T 57e47b07fdcf434e96ca5dd0354e93b2 P b5e9178a7a83402e8c4a6943ab78589d [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20250112 15:07:34.641994 21549 raft_consensus.cc:3054] T 57e47b07fdcf434e96ca5dd0354e93b2 P b5e9178a7a83402e8c4a6943ab78589d [term 2 FOLLOWER]: Advancing to term 3
I20250112 15:07:34.642642 21549 raft_consensus.cc:513] T 57e47b07fdcf434e96ca5dd0354e93b2 P b5e9178a7a83402e8c4a6943ab78589d [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "2537bbbd1a9947ae9facd87567e3ccb9" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 33173 } } peers { permanent_uuid: "c561ec3374b94b95be0c155d80ff0bff" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 33627 } } peers { permanent_uuid: "b5e9178a7a83402e8c4a6943ab78589d" member_type: VOTER last_known_addr { host: "127.17.60.197" port: 36573 } attrs { promote: false } }
I20250112 15:07:34.642813 21549 leader_election.cc:290] T 57e47b07fdcf434e96ca5dd0354e93b2 P b5e9178a7a83402e8c4a6943ab78589d [CANDIDATE]: Term 3 election: Requested vote from peers 2537bbbd1a9947ae9facd87567e3ccb9 (127.17.60.193:33173), c561ec3374b94b95be0c155d80ff0bff (127.17.60.194:33627)
I20250112 15:07:34.642982 21068 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "57e47b07fdcf434e96ca5dd0354e93b2" candidate_uuid: "b5e9178a7a83402e8c4a6943ab78589d" candidate_term: 3 candidate_status { last_received { term: 2 index: 29 } } ignore_live_leader: true dest_uuid: "2537bbbd1a9947ae9facd87567e3ccb9"
I20250112 15:07:34.642978 21147 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "57e47b07fdcf434e96ca5dd0354e93b2" candidate_uuid: "b5e9178a7a83402e8c4a6943ab78589d" candidate_term: 3 candidate_status { last_received { term: 2 index: 29 } } ignore_live_leader: true dest_uuid: "c561ec3374b94b95be0c155d80ff0bff"
I20250112 15:07:34.643095 21068 raft_consensus.cc:3049] T 57e47b07fdcf434e96ca5dd0354e93b2 P 2537bbbd1a9947ae9facd87567e3ccb9 [term 2 LEADER]: Stepping down as leader of term 2
I20250112 15:07:34.643114 21147 raft_consensus.cc:3054] T 57e47b07fdcf434e96ca5dd0354e93b2 P c561ec3374b94b95be0c155d80ff0bff [term 2 FOLLOWER]: Advancing to term 3
I20250112 15:07:34.643127 21068 raft_consensus.cc:738] T 57e47b07fdcf434e96ca5dd0354e93b2 P 2537bbbd1a9947ae9facd87567e3ccb9 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 2537bbbd1a9947ae9facd87567e3ccb9, State: Running, Role: LEADER
I20250112 15:07:34.643189 21068 consensus_queue.cc:260] T 57e47b07fdcf434e96ca5dd0354e93b2 P 2537bbbd1a9947ae9facd87567e3ccb9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 29, Committed index: 29, Last appended: 2.29, Last appended by leader: 29, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "2537bbbd1a9947ae9facd87567e3ccb9" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 33173 } } peers { permanent_uuid: "c561ec3374b94b95be0c155d80ff0bff" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 33627 } } peers { permanent_uuid: "b5e9178a7a83402e8c4a6943ab78589d" member_type: VOTER last_known_addr { host: "127.17.60.197" port: 36573 } attrs { promote: false } }
I20250112 15:07:34.643296 21068 raft_consensus.cc:3054] T 57e47b07fdcf434e96ca5dd0354e93b2 P 2537bbbd1a9947ae9facd87567e3ccb9 [term 2 FOLLOWER]: Advancing to term 3
I20250112 15:07:34.643896 21147 raft_consensus.cc:2463] T 57e47b07fdcf434e96ca5dd0354e93b2 P c561ec3374b94b95be0c155d80ff0bff [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b5e9178a7a83402e8c4a6943ab78589d in term 3.
I20250112 15:07:34.643958 21068 raft_consensus.cc:2463] T 57e47b07fdcf434e96ca5dd0354e93b2 P 2537bbbd1a9947ae9facd87567e3ccb9 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b5e9178a7a83402e8c4a6943ab78589d in term 3.
I20250112 15:07:34.644088 21514 leader_election.cc:304] T 57e47b07fdcf434e96ca5dd0354e93b2 P b5e9178a7a83402e8c4a6943ab78589d [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b5e9178a7a83402e8c4a6943ab78589d, c561ec3374b94b95be0c155d80ff0bff; no voters:
I20250112 15:07:34.644286 21825 raft_consensus.cc:2798] T 57e47b07fdcf434e96ca5dd0354e93b2 P b5e9178a7a83402e8c4a6943ab78589d [term 3 FOLLOWER]: Leader election won for term 3
I20250112 15:07:34.644467 21825 raft_consensus.cc:695] T 57e47b07fdcf434e96ca5dd0354e93b2 P b5e9178a7a83402e8c4a6943ab78589d [term 3 LEADER]: Becoming Leader. State: Replica: b5e9178a7a83402e8c4a6943ab78589d, State: Running, Role: LEADER
I20250112 15:07:34.644527 21825 consensus_queue.cc:237] T 57e47b07fdcf434e96ca5dd0354e93b2 P b5e9178a7a83402e8c4a6943ab78589d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 29, Committed index: 29, Last appended: 2.29, Last appended by leader: 29, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "2537bbbd1a9947ae9facd87567e3ccb9" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 33173 } } peers { permanent_uuid: "c561ec3374b94b95be0c155d80ff0bff" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 33627 } } peers { permanent_uuid: "b5e9178a7a83402e8c4a6943ab78589d" member_type: VOTER last_known_addr { host: "127.17.60.197" port: 36573 } attrs { promote: false } }
I20250112 15:07:34.645047 20964 catalog_manager.cc:5526] T 57e47b07fdcf434e96ca5dd0354e93b2 P b5e9178a7a83402e8c4a6943ab78589d reported cstate change: term changed from 2 to 3, leader changed from 2537bbbd1a9947ae9facd87567e3ccb9 (127.17.60.193) to b5e9178a7a83402e8c4a6943ab78589d (127.17.60.197). New cstate: current_term: 3 leader_uuid: "b5e9178a7a83402e8c4a6943ab78589d" committed_config { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "2537bbbd1a9947ae9facd87567e3ccb9" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 33173 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c561ec3374b94b95be0c155d80ff0bff" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 33627 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b5e9178a7a83402e8c4a6943ab78589d" member_type: VOTER last_known_addr { host: "127.17.60.197" port: 36573 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20250112 15:07:34.913906 21068 raft_consensus.cc:1270] T 5bcec60d3b974a9bb24aedfc738e1594 P 2537bbbd1a9947ae9facd87567e3ccb9 [term 2 FOLLOWER]: Refusing update from remote peer a2a5bba4ac1145d5bab94a69c9f4df40: Log matching property violated. Preceding OpId in replica: term: 1 index: 28. Preceding OpId from leader: term: 2 index: 29. (index mismatch)
I20250112 15:07:34.914201 21813 consensus_queue.cc:1035] T 5bcec60d3b974a9bb24aedfc738e1594 P a2a5bba4ac1145d5bab94a69c9f4df40 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2537bbbd1a9947ae9facd87567e3ccb9" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 33173 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.000s
I20250112 15:07:34.915218 21226 raft_consensus.cc:1270] T 5bcec60d3b974a9bb24aedfc738e1594 P 3710b74ed2044ce5995533df32e67351 [term 2 FOLLOWER]: Refusing update from remote peer a2a5bba4ac1145d5bab94a69c9f4df40: Log matching property violated. Preceding OpId in replica: term: 1 index: 28. Preceding OpId from leader: term: 2 index: 29. (index mismatch)
I20250112 15:07:34.915442 21813 consensus_queue.cc:1035] T 5bcec60d3b974a9bb24aedfc738e1594 P a2a5bba4ac1145d5bab94a69c9f4df40 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3710b74ed2044ce5995533df32e67351" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 38589 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.000s
I20250112 15:07:34.937072 21474 consensus_queue.cc:237] T 5bcec60d3b974a9bb24aedfc738e1594 P a2a5bba4ac1145d5bab94a69c9f4df40 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 29, Committed index: 29, Last appended: 2.29, Last appended by leader: 28, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "2537bbbd1a9947ae9facd87567e3ccb9" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 33173 } attrs { replace: true } } peers { permanent_uuid: "3710b74ed2044ce5995533df32e67351" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 38589 } } peers { permanent_uuid: "a2a5bba4ac1145d5bab94a69c9f4df40" member_type: VOTER last_known_addr { host: "127.17.60.196" port: 41807 } attrs { promote: false } } peers { permanent_uuid: "b5e9178a7a83402e8c4a6943ab78589d" member_type: NON_VOTER last_known_addr { host: "127.17.60.197" port: 36573 } attrs { promote: true } }
I20250112 15:07:34.937659 21226 raft_consensus.cc:1270] T 5bcec60d3b974a9bb24aedfc738e1594 P 3710b74ed2044ce5995533df32e67351 [term 2 FOLLOWER]: Refusing update from remote peer a2a5bba4ac1145d5bab94a69c9f4df40: Log matching property violated. Preceding OpId in replica: term: 2 index: 29. Preceding OpId from leader: term: 2 index: 30. (index mismatch)
I20250112 15:07:34.937774 21068 raft_consensus.cc:1270] T 5bcec60d3b974a9bb24aedfc738e1594 P 2537bbbd1a9947ae9facd87567e3ccb9 [term 2 FOLLOWER]: Refusing update from remote peer a2a5bba4ac1145d5bab94a69c9f4df40: Log matching property violated. Preceding OpId in replica: term: 2 index: 29. Preceding OpId from leader: term: 2 index: 30. (index mismatch)
I20250112 15:07:34.937886 21831 consensus_queue.cc:1035] T 5bcec60d3b974a9bb24aedfc738e1594 P a2a5bba4ac1145d5bab94a69c9f4df40 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3710b74ed2044ce5995533df32e67351" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 38589 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 30, Last known committed idx: 29, Time since last communication: 0.000s
I20250112 15:07:34.938006 21814 consensus_queue.cc:1035] T 5bcec60d3b974a9bb24aedfc738e1594 P a2a5bba4ac1145d5bab94a69c9f4df40 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2537bbbd1a9947ae9facd87567e3ccb9" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 33173 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 30, Last known committed idx: 29, Time since last communication: 0.000s
I20250112 15:07:34.938524 21831 raft_consensus.cc:2949] T 5bcec60d3b974a9bb24aedfc738e1594 P a2a5bba4ac1145d5bab94a69c9f4df40 [term 2 LEADER]: Committing config change with OpId 2.30: config changed from index 28 to 30, NON_VOTER b5e9178a7a83402e8c4a6943ab78589d (127.17.60.197) added. New config: { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "2537bbbd1a9947ae9facd87567e3ccb9" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 33173 } attrs { replace: true } } peers { permanent_uuid: "3710b74ed2044ce5995533df32e67351" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 38589 } } peers { permanent_uuid: "a2a5bba4ac1145d5bab94a69c9f4df40" member_type: VOTER last_known_addr { host: "127.17.60.196" port: 41807 } attrs { promote: false } } peers { permanent_uuid: "b5e9178a7a83402e8c4a6943ab78589d" member_type: NON_VOTER last_known_addr { host: "127.17.60.197" port: 36573 } attrs { promote: true } } }
I20250112 15:07:34.938664 21068 raft_consensus.cc:2949] T 5bcec60d3b974a9bb24aedfc738e1594 P 2537bbbd1a9947ae9facd87567e3ccb9 [term 2 FOLLOWER]: Committing config change with OpId 2.30: config changed from index 28 to 30, NON_VOTER b5e9178a7a83402e8c4a6943ab78589d (127.17.60.197) added. New config: { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "2537bbbd1a9947ae9facd87567e3ccb9" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 33173 } attrs { replace: true } } peers { permanent_uuid: "3710b74ed2044ce5995533df32e67351" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 38589 } } peers { permanent_uuid: "a2a5bba4ac1145d5bab94a69c9f4df40" member_type: VOTER last_known_addr { host: "127.17.60.196" port: 41807 } attrs { promote: false } } peers { permanent_uuid: "b5e9178a7a83402e8c4a6943ab78589d" member_type: NON_VOTER last_known_addr { host: "127.17.60.197" port: 36573 } attrs { promote: true } } }
I20250112 15:07:34.938773 21226 raft_consensus.cc:2949] T 5bcec60d3b974a9bb24aedfc738e1594 P 3710b74ed2044ce5995533df32e67351 [term 2 FOLLOWER]: Committing config change with OpId 2.30: config changed from index 28 to 30, NON_VOTER b5e9178a7a83402e8c4a6943ab78589d (127.17.60.197) added. New config: { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "2537bbbd1a9947ae9facd87567e3ccb9" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 33173 } attrs { replace: true } } peers { permanent_uuid: "3710b74ed2044ce5995533df32e67351" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 38589 } } peers { permanent_uuid: "a2a5bba4ac1145d5bab94a69c9f4df40" member_type: VOTER last_known_addr { host: "127.17.60.196" port: 41807 } attrs { promote: false } } peers { permanent_uuid: "b5e9178a7a83402e8c4a6943ab78589d" member_type: NON_VOTER last_known_addr { host: "127.17.60.197" port: 36573 } attrs { promote: true } } }
I20250112 15:07:34.940061 20964 catalog_manager.cc:5526] T 5bcec60d3b974a9bb24aedfc738e1594 P 3710b74ed2044ce5995533df32e67351 reported cstate change: config changed from index 28 to 30, NON_VOTER b5e9178a7a83402e8c4a6943ab78589d (127.17.60.197) added. New cstate: current_term: 2 leader_uuid: "a2a5bba4ac1145d5bab94a69c9f4df40" committed_config { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "2537bbbd1a9947ae9facd87567e3ccb9" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 33173 } attrs { replace: true } } peers { permanent_uuid: "3710b74ed2044ce5995533df32e67351" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 38589 } } peers { permanent_uuid: "a2a5bba4ac1145d5bab94a69c9f4df40" member_type: VOTER last_known_addr { host: "127.17.60.196" port: 41807 } attrs { promote: false } } peers { permanent_uuid: "b5e9178a7a83402e8c4a6943ab78589d" member_type: NON_VOTER last_known_addr { host: "127.17.60.197" port: 36573 } attrs { promote: true } } }
I20250112 15:07:34.940106 21147 consensus_queue.cc:237] T 69c23627ab854000b360030c384dc966 P c561ec3374b94b95be0c155d80ff0bff [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 27, Committed index: 27, Last appended: 1.27, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "c561ec3374b94b95be0c155d80ff0bff" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 33627 } } peers { permanent_uuid: "3710b74ed2044ce5995533df32e67351" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 38589 } attrs { replace: true } } peers { permanent_uuid: "b5e9178a7a83402e8c4a6943ab78589d" member_type: VOTER last_known_addr { host: "127.17.60.197" port: 36573 } attrs { promote: false } } peers { permanent_uuid: "a2a5bba4ac1145d5bab94a69c9f4df40" member_type: NON_VOTER last_known_addr { host: "127.17.60.196" port: 41807 } attrs { promote: true } }
I20250112 15:07:34.940773 21226 raft_consensus.cc:1270] T 69c23627ab854000b360030c384dc966 P 3710b74ed2044ce5995533df32e67351 [term 1 FOLLOWER]: Refusing update from remote peer c561ec3374b94b95be0c155d80ff0bff: Log matching property violated. Preceding OpId in replica: term: 1 index: 27. Preceding OpId from leader: term: 1 index: 28. (index mismatch)
I20250112 15:07:34.940999 21549 raft_consensus.cc:1270] T 69c23627ab854000b360030c384dc966 P b5e9178a7a83402e8c4a6943ab78589d [term 1 FOLLOWER]: Refusing update from remote peer c561ec3374b94b95be0c155d80ff0bff: Log matching property violated. Preceding OpId in replica: term: 1 index: 27. Preceding OpId from leader: term: 1 index: 28. (index mismatch)
I20250112 15:07:34.941195 21835 consensus_queue.cc:1035] T 69c23627ab854000b360030c384dc966 P c561ec3374b94b95be0c155d80ff0bff [LEADER]: Connected to new peer: Peer: permanent_uuid: "3710b74ed2044ce5995533df32e67351" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 38589 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 28, Last known committed idx: 27, Time since last communication: 0.000s
I20250112 15:07:34.941366 21775 consensus_queue.cc:1035] T 69c23627ab854000b360030c384dc966 P c561ec3374b94b95be0c155d80ff0bff [LEADER]: Connected to new peer: Peer: permanent_uuid: "b5e9178a7a83402e8c4a6943ab78589d" member_type: VOTER last_known_addr { host: "127.17.60.197" port: 36573 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 28, Last known committed idx: 27, Time since last communication: 0.000s
W20250112 15:07:34.941447 21437 consensus_peers.cc:487] T 5bcec60d3b974a9bb24aedfc738e1594 P a2a5bba4ac1145d5bab94a69c9f4df40 -> Peer b5e9178a7a83402e8c4a6943ab78589d (127.17.60.197:36573): Couldn't send request to peer b5e9178a7a83402e8c4a6943ab78589d. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 5bcec60d3b974a9bb24aedfc738e1594. This is attempt 1: this message will repeat every 5th retry.
I20250112 15:07:34.942245 21549 raft_consensus.cc:2949] T 69c23627ab854000b360030c384dc966 P b5e9178a7a83402e8c4a6943ab78589d [term 1 FOLLOWER]: Committing config change with OpId 1.28: config changed from index 27 to 28, NON_VOTER a2a5bba4ac1145d5bab94a69c9f4df40 (127.17.60.196) added. New config: { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "c561ec3374b94b95be0c155d80ff0bff" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 33627 } } peers { permanent_uuid: "3710b74ed2044ce5995533df32e67351" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 38589 } attrs { replace: true } } peers { permanent_uuid: "b5e9178a7a83402e8c4a6943ab78589d" member_type: VOTER last_known_addr { host: "127.17.60.197" port: 36573 } attrs { promote: false } } peers { permanent_uuid: "a2a5bba4ac1145d5bab94a69c9f4df40" member_type: NON_VOTER last_known_addr { host: "127.17.60.196" port: 41807 } attrs { promote: true } } }
I20250112 15:07:34.942281 21226 raft_consensus.cc:2949] T 69c23627ab854000b360030c384dc966 P 3710b74ed2044ce5995533df32e67351 [term 1 FOLLOWER]: Committing config change with OpId 1.28: config changed from index 27 to 28, NON_VOTER a2a5bba4ac1145d5bab94a69c9f4df40 (127.17.60.196) added. New config: { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "c561ec3374b94b95be0c155d80ff0bff" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 33627 } } peers { permanent_uuid: "3710b74ed2044ce5995533df32e67351" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 38589 } attrs { replace: true } } peers { permanent_uuid: "b5e9178a7a83402e8c4a6943ab78589d" member_type: VOTER last_known_addr { host: "127.17.60.197" port: 36573 } attrs { promote: false } } peers { permanent_uuid: "a2a5bba4ac1145d5bab94a69c9f4df40" member_type: NON_VOTER last_known_addr { host: "127.17.60.196" port: 41807 } attrs { promote: true } } }
I20250112 15:07:34.942144 21835 raft_consensus.cc:2949] T 69c23627ab854000b360030c384dc966 P c561ec3374b94b95be0c155d80ff0bff [term 1 LEADER]: Committing config change with OpId 1.28: config changed from index 27 to 28, NON_VOTER a2a5bba4ac1145d5bab94a69c9f4df40 (127.17.60.196) added. New config: { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "c561ec3374b94b95be0c155d80ff0bff" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 33627 } } peers { permanent_uuid: "3710b74ed2044ce5995533df32e67351" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 38589 } attrs { replace: true } } peers { permanent_uuid: "b5e9178a7a83402e8c4a6943ab78589d" member_type: VOTER last_known_addr { host: "127.17.60.197" port: 36573 } attrs { promote: false } } peers { permanent_uuid: "a2a5bba4ac1145d5bab94a69c9f4df40" member_type: NON_VOTER last_known_addr { host: "127.17.60.196" port: 41807 } attrs { promote: true } } }
I20250112 15:07:34.943166 20970 catalog_manager.cc:5526] T 69c23627ab854000b360030c384dc966 P 3710b74ed2044ce5995533df32e67351 reported cstate change: config changed from index 27 to 28, NON_VOTER a2a5bba4ac1145d5bab94a69c9f4df40 (127.17.60.196) added. New cstate: current_term: 1 leader_uuid: "c561ec3374b94b95be0c155d80ff0bff" committed_config { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "c561ec3374b94b95be0c155d80ff0bff" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 33627 } } peers { permanent_uuid: "3710b74ed2044ce5995533df32e67351" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 38589 } attrs { replace: true } } peers { permanent_uuid: "b5e9178a7a83402e8c4a6943ab78589d" member_type: VOTER last_known_addr { host: "127.17.60.197" port: 36573 } attrs { promote: false } } peers { permanent_uuid: "a2a5bba4ac1145d5bab94a69c9f4df40" member_type: NON_VOTER last_known_addr { host: "127.17.60.196" port: 41807 } attrs { promote: true } } }
W20250112 15:07:34.944622 21116 consensus_peers.cc:487] T 69c23627ab854000b360030c384dc966 P c561ec3374b94b95be0c155d80ff0bff -> Peer a2a5bba4ac1145d5bab94a69c9f4df40 (127.17.60.196:41807): Couldn't send request to peer a2a5bba4ac1145d5bab94a69c9f4df40. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 69c23627ab854000b360030c384dc966. This is attempt 1: this message will repeat every 5th retry.
W20250112 15:07:34.946738 21021 auto_rebalancer.cc:253] failed to send replica move request: Illegal state: Leader has not yet committed an operation in its own term
W20250112 15:07:34.947605 21021 auto_rebalancer.cc:663] Could not move replica: Incomplete: tablet 57e47b07fdcf434e96ca5dd0354e93b2, TS c561ec3374b94b95be0c155d80ff0bff -> TS 25286c485fa04c86a5d07c8f88643a47 move failed, destination replica disappeared from tablet's Raft config: opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "2537bbbd1a9947ae9facd87567e3ccb9" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 33173 } } peers { permanent_uuid: "c561ec3374b94b95be0c155d80ff0bff" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 33627 } } peers { permanent_uuid: "b5e9178a7a83402e8c4a6943ab78589d" member_type: VOTER last_known_addr { host: "127.17.60.197" port: 36573 } attrs { promote: false } }
W20250112 15:07:34.947645 21021 auto_rebalancer.cc:263] scheduled replica move failed to complete: Incomplete: tablet 57e47b07fdcf434e96ca5dd0354e93b2, TS c561ec3374b94b95be0c155d80ff0bff -> TS 25286c485fa04c86a5d07c8f88643a47 move failed, destination replica disappeared from tablet's Raft config: opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "2537bbbd1a9947ae9facd87567e3ccb9" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 33173 } } peers { permanent_uuid: "c561ec3374b94b95be0c155d80ff0bff" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 33627 } } peers { permanent_uuid: "b5e9178a7a83402e8c4a6943ab78589d" member_type: VOTER last_known_addr { host: "127.17.60.197" port: 36573 } attrs { promote: false } }
W20250112 15:07:34.948372 21021 auto_rebalancer.cc:663] Could not move replica: Incomplete: tablet 16f16fcd5cf347e5853189dc4da55c7f, TS c561ec3374b94b95be0c155d80ff0bff -> TS b5e9178a7a83402e8c4a6943ab78589d move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2537bbbd1a9947ae9facd87567e3ccb9" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 33173 } } peers { permanent_uuid: "c561ec3374b94b95be0c155d80ff0bff" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 33627 } } peers { permanent_uuid: "3710b74ed2044ce5995533df32e67351" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 38589 } }
W20250112 15:07:34.948406 21021 auto_rebalancer.cc:263] scheduled replica move failed to complete: Incomplete: tablet 16f16fcd5cf347e5853189dc4da55c7f, TS c561ec3374b94b95be0c155d80ff0bff -> TS b5e9178a7a83402e8c4a6943ab78589d move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2537bbbd1a9947ae9facd87567e3ccb9" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 33173 } } peers { permanent_uuid: "c561ec3374b94b95be0c155d80ff0bff" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 33627 } } peers { permanent_uuid: "3710b74ed2044ce5995533df32e67351" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 38589 } }
W20250112 15:07:34.949251 21021 auto_rebalancer.cc:663] Could not move replica: Incomplete: tablet 02c33d457b6b47138dd712850819b619, TS 3710b74ed2044ce5995533df32e67351 -> TS a2a5bba4ac1145d5bab94a69c9f4df40 move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3710b74ed2044ce5995533df32e67351" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 38589 } } peers { permanent_uuid: "2537bbbd1a9947ae9facd87567e3ccb9" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 33173 } } peers { permanent_uuid: "c561ec3374b94b95be0c155d80ff0bff" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 33627 } }
W20250112 15:07:34.949298 21021 auto_rebalancer.cc:263] scheduled replica move failed to complete: Incomplete: tablet 02c33d457b6b47138dd712850819b619, TS 3710b74ed2044ce5995533df32e67351 -> TS a2a5bba4ac1145d5bab94a69c9f4df40 move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3710b74ed2044ce5995533df32e67351" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 38589 } } peers { permanent_uuid: "2537bbbd1a9947ae9facd87567e3ccb9" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 33173 } } peers { permanent_uuid: "c561ec3374b94b95be0c155d80ff0bff" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 33627 } }
W20250112 15:07:34.950095 21021 auto_rebalancer.cc:663] Could not move replica: Incomplete: tablet bfaeae584ec04772b2d6fcd088022117, TS 2537bbbd1a9947ae9facd87567e3ccb9 -> TS 25286c485fa04c86a5d07c8f88643a47 move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2537bbbd1a9947ae9facd87567e3ccb9" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 33173 } } peers { permanent_uuid: "c561ec3374b94b95be0c155d80ff0bff" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 33627 } } peers { permanent_uuid: "3710b74ed2044ce5995533df32e67351" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 38589 } }
W20250112 15:07:34.950130 21021 auto_rebalancer.cc:263] scheduled replica move failed to complete: Incomplete: tablet bfaeae584ec04772b2d6fcd088022117, TS 2537bbbd1a9947ae9facd87567e3ccb9 -> TS 25286c485fa04c86a5d07c8f88643a47 move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2537bbbd1a9947ae9facd87567e3ccb9" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 33173 } } peers { permanent_uuid: "c561ec3374b94b95be0c155d80ff0bff" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 33627 } } peers { permanent_uuid: "3710b74ed2044ce5995533df32e67351" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 38589 } }
I20250112 15:07:35.020953 21068 raft_consensus.cc:1270] T 57e47b07fdcf434e96ca5dd0354e93b2 P 2537bbbd1a9947ae9facd87567e3ccb9 [term 3 FOLLOWER]: Refusing update from remote peer b5e9178a7a83402e8c4a6943ab78589d: Log matching property violated. Preceding OpId in replica: term: 2 index: 29. Preceding OpId from leader: term: 3 index: 30. (index mismatch)
I20250112 15:07:35.021159 21825 consensus_queue.cc:1035] T 57e47b07fdcf434e96ca5dd0354e93b2 P b5e9178a7a83402e8c4a6943ab78589d [LEADER]: Connected to new peer: Peer: permanent_uuid: "2537bbbd1a9947ae9facd87567e3ccb9" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 33173 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 30, Last known committed idx: 29, Time since last communication: 0.000s
I20250112 15:07:35.022061 21149 raft_consensus.cc:1270] T 57e47b07fdcf434e96ca5dd0354e93b2 P c561ec3374b94b95be0c155d80ff0bff [term 3 FOLLOWER]: Refusing update from remote peer b5e9178a7a83402e8c4a6943ab78589d: Log matching property violated. Preceding OpId in replica: term: 2 index: 29. Preceding OpId from leader: term: 3 index: 30. (index mismatch)
I20250112 15:07:35.022256 21825 consensus_queue.cc:1035] T 57e47b07fdcf434e96ca5dd0354e93b2 P b5e9178a7a83402e8c4a6943ab78589d [LEADER]: Connected to new peer: Peer: permanent_uuid: "c561ec3374b94b95be0c155d80ff0bff" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 33627 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 30, Last known committed idx: 29, Time since last communication: 0.000s
I20250112 15:07:35.425720 21845 ts_tablet_manager.cc:927] T 5bcec60d3b974a9bb24aedfc738e1594 P b5e9178a7a83402e8c4a6943ab78589d: Initiating tablet copy from peer a2a5bba4ac1145d5bab94a69c9f4df40 (127.17.60.196:41807)
I20250112 15:07:35.425930 21845 tablet_copy_client.cc:323] T 5bcec60d3b974a9bb24aedfc738e1594 P b5e9178a7a83402e8c4a6943ab78589d: tablet copy: Beginning tablet copy session from remote peer at address 127.17.60.196:41807
I20250112 15:07:35.429059 21484 tablet_copy_service.cc:140] P a2a5bba4ac1145d5bab94a69c9f4df40: Received BeginTabletCopySession request for tablet 5bcec60d3b974a9bb24aedfc738e1594 from peer b5e9178a7a83402e8c4a6943ab78589d ({username='slave'} at 127.0.0.1:43302)
I20250112 15:07:35.429128 21484 tablet_copy_service.cc:161] P a2a5bba4ac1145d5bab94a69c9f4df40: Beginning new tablet copy session on tablet 5bcec60d3b974a9bb24aedfc738e1594 from peer b5e9178a7a83402e8c4a6943ab78589d at {username='slave'} at 127.0.0.1:43302: session id = b5e9178a7a83402e8c4a6943ab78589d-5bcec60d3b974a9bb24aedfc738e1594
I20250112 15:07:35.429610 21484 tablet_copy_source_session.cc:215] T 5bcec60d3b974a9bb24aedfc738e1594 P a2a5bba4ac1145d5bab94a69c9f4df40: Tablet Copy: opened 0 blocks and 1 log segments
I20250112 15:07:35.429956 21845 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5bcec60d3b974a9bb24aedfc738e1594. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 15:07:35.431386 21845 tablet_copy_client.cc:806] T 5bcec60d3b974a9bb24aedfc738e1594 P b5e9178a7a83402e8c4a6943ab78589d: tablet copy: Starting download of 0 data blocks...
I20250112 15:07:35.431507 21845 tablet_copy_client.cc:670] T 5bcec60d3b974a9bb24aedfc738e1594 P b5e9178a7a83402e8c4a6943ab78589d: tablet copy: Starting download of 1 WAL segments...
I20250112 15:07:35.432022 21846 tablet_copy_client.cc:1025] T 5bcec60d3b974a9bb24aedfc738e1594 P b5e9178a7a83402e8c4a6943ab78589d: tablet copy: Injecting latency into file download: 2000
I20250112 15:07:35.517437 21848 ts_tablet_manager.cc:927] T 69c23627ab854000b360030c384dc966 P a2a5bba4ac1145d5bab94a69c9f4df40: Initiating tablet copy from peer c561ec3374b94b95be0c155d80ff0bff (127.17.60.194:33627)
I20250112 15:07:35.517609 21848 tablet_copy_client.cc:323] T 69c23627ab854000b360030c384dc966 P a2a5bba4ac1145d5bab94a69c9f4df40: tablet copy: Beginning tablet copy session from remote peer at address 127.17.60.194:33627
I20250112 15:07:35.520258 21161 tablet_copy_service.cc:140] P c561ec3374b94b95be0c155d80ff0bff: Received BeginTabletCopySession request for tablet 69c23627ab854000b360030c384dc966 from peer a2a5bba4ac1145d5bab94a69c9f4df40 ({username='slave'} at 127.0.0.1:58522)
I20250112 15:07:35.520341 21161 tablet_copy_service.cc:161] P c561ec3374b94b95be0c155d80ff0bff: Beginning new tablet copy session on tablet 69c23627ab854000b360030c384dc966 from peer a2a5bba4ac1145d5bab94a69c9f4df40 at {username='slave'} at 127.0.0.1:58522: session id = a2a5bba4ac1145d5bab94a69c9f4df40-69c23627ab854000b360030c384dc966
I20250112 15:07:35.520975 21161 tablet_copy_source_session.cc:215] T 69c23627ab854000b360030c384dc966 P c561ec3374b94b95be0c155d80ff0bff: Tablet Copy: opened 0 blocks and 1 log segments
I20250112 15:07:35.521286 21848 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 69c23627ab854000b360030c384dc966. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 15:07:35.522496 21848 tablet_copy_client.cc:806] T 69c23627ab854000b360030c384dc966 P a2a5bba4ac1145d5bab94a69c9f4df40: tablet copy: Starting download of 0 data blocks...
I20250112 15:07:35.522611 21848 tablet_copy_client.cc:670] T 69c23627ab854000b360030c384dc966 P a2a5bba4ac1145d5bab94a69c9f4df40: tablet copy: Starting download of 1 WAL segments...
I20250112 15:07:35.523181 21849 tablet_copy_client.cc:1025] T 69c23627ab854000b360030c384dc966 P a2a5bba4ac1145d5bab94a69c9f4df40: tablet copy: Injecting latency into file download: 2000
I20250112 15:07:36.087494 21026 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250112 15:07:36.087908 21226 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "16f16fcd5cf347e5853189dc4da55c7f"
dest_uuid: "3710b74ed2044ce5995533df32e67351"
mode: GRACEFUL
new_leader_uuid: "2537bbbd1a9947ae9facd87567e3ccb9"
from {username='slave'} at 127.0.0.1:54970
I20250112 15:07:36.088003 21226 raft_consensus.cc:604] T 16f16fcd5cf347e5853189dc4da55c7f P 3710b74ed2044ce5995533df32e67351 [term 1 LEADER]: Received request to transfer leadership to TS 2537bbbd1a9947ae9facd87567e3ccb9
I20250112 15:07:36.088346 21068 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "8c7dbaf285a04da9b4eec92a29ab7d6c"
dest_uuid: "2537bbbd1a9947ae9facd87567e3ccb9"
mode: GRACEFUL
new_leader_uuid: "c561ec3374b94b95be0c155d80ff0bff"
from {username='slave'} at 127.0.0.1:54414
I20250112 15:07:36.088438 21068 raft_consensus.cc:604] T 8c7dbaf285a04da9b4eec92a29ab7d6c P 2537bbbd1a9947ae9facd87567e3ccb9 [term 2 LEADER]: Received request to transfer leadership to TS c561ec3374b94b95be0c155d80ff0bff
I20250112 15:07:36.088769 21068 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "9601bcda24ed42e7b93ab9740f1e271e"
dest_uuid: "2537bbbd1a9947ae9facd87567e3ccb9"
mode: GRACEFUL
new_leader_uuid: "3710b74ed2044ce5995533df32e67351"
from {username='slave'} at 127.0.0.1:54414
I20250112 15:07:36.088845 21068 raft_consensus.cc:604] T 9601bcda24ed42e7b93ab9740f1e271e P 2537bbbd1a9947ae9facd87567e3ccb9 [term 1 LEADER]: Received request to transfer leadership to TS 3710b74ed2044ce5995533df32e67351
I20250112 15:07:36.089012 21026 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 3
I20250112 15:07:36.089107 21026 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20250112 15:07:36.149241 21843 raft_consensus.cc:988] T 16f16fcd5cf347e5853189dc4da55c7f P 3710b74ed2044ce5995533df32e67351: : Instructing follower 2537bbbd1a9947ae9facd87567e3ccb9 to start an election
I20250112 15:07:36.149350 21843 raft_consensus.cc:1076] T 16f16fcd5cf347e5853189dc4da55c7f P 3710b74ed2044ce5995533df32e67351 [term 1 LEADER]: Signalling peer 2537bbbd1a9947ae9facd87567e3ccb9 to start an election
I20250112 15:07:36.149495 21068 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "16f16fcd5cf347e5853189dc4da55c7f"
dest_uuid: "2537bbbd1a9947ae9facd87567e3ccb9"
from {username='slave'} at 127.0.0.1:37204
I20250112 15:07:36.149561 21068 raft_consensus.cc:491] T 16f16fcd5cf347e5853189dc4da55c7f P 2537bbbd1a9947ae9facd87567e3ccb9 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20250112 15:07:36.149600 21068 raft_consensus.cc:3054] T 16f16fcd5cf347e5853189dc4da55c7f P 2537bbbd1a9947ae9facd87567e3ccb9 [term 1 FOLLOWER]: Advancing to term 2
I20250112 15:07:36.150416 21068 raft_consensus.cc:513] T 16f16fcd5cf347e5853189dc4da55c7f P 2537bbbd1a9947ae9facd87567e3ccb9 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2537bbbd1a9947ae9facd87567e3ccb9" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 33173 } } peers { permanent_uuid: "c561ec3374b94b95be0c155d80ff0bff" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 33627 } } peers { permanent_uuid: "3710b74ed2044ce5995533df32e67351" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 38589 } }
I20250112 15:07:36.150553 21068 leader_election.cc:290] T 16f16fcd5cf347e5853189dc4da55c7f P 2537bbbd1a9947ae9facd87567e3ccb9 [CANDIDATE]: Term 2 election: Requested vote from peers c561ec3374b94b95be0c155d80ff0bff (127.17.60.194:33627), 3710b74ed2044ce5995533df32e67351 (127.17.60.195:38589)
I20250112 15:07:36.150666 21149 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "16f16fcd5cf347e5853189dc4da55c7f" candidate_uuid: "2537bbbd1a9947ae9facd87567e3ccb9" candidate_term: 2 candidate_status { last_received { term: 1 index: 25 } } ignore_live_leader: true dest_uuid: "c561ec3374b94b95be0c155d80ff0bff"
I20250112 15:07:36.150724 21226 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "16f16fcd5cf347e5853189dc4da55c7f" candidate_uuid: "2537bbbd1a9947ae9facd87567e3ccb9" candidate_term: 2 candidate_status { last_received { term: 1 index: 25 } } ignore_live_leader: true dest_uuid: "3710b74ed2044ce5995533df32e67351"
I20250112 15:07:36.150749 21149 raft_consensus.cc:3054] T 16f16fcd5cf347e5853189dc4da55c7f P c561ec3374b94b95be0c155d80ff0bff [term 1 FOLLOWER]: Advancing to term 2
I20250112 15:07:36.150774 21226 raft_consensus.cc:3049] T 16f16fcd5cf347e5853189dc4da55c7f P 3710b74ed2044ce5995533df32e67351 [term 1 LEADER]: Stepping down as leader of term 1
I20250112 15:07:36.150789 21226 raft_consensus.cc:738] T 16f16fcd5cf347e5853189dc4da55c7f P 3710b74ed2044ce5995533df32e67351 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 3710b74ed2044ce5995533df32e67351, State: Running, Role: LEADER
I20250112 15:07:36.150831 21226 consensus_queue.cc:260] T 16f16fcd5cf347e5853189dc4da55c7f P 3710b74ed2044ce5995533df32e67351 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 25, Committed index: 25, Last appended: 1.25, Last appended by leader: 25, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2537bbbd1a9947ae9facd87567e3ccb9" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 33173 } } peers { permanent_uuid: "c561ec3374b94b95be0c155d80ff0bff" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 33627 } } peers { permanent_uuid: "3710b74ed2044ce5995533df32e67351" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 38589 } }
I20250112 15:07:36.150894 21226 raft_consensus.cc:3054] T 16f16fcd5cf347e5853189dc4da55c7f P 3710b74ed2044ce5995533df32e67351 [term 1 FOLLOWER]: Advancing to term 2
I20250112 15:07:36.151290 21149 raft_consensus.cc:2463] T 16f16fcd5cf347e5853189dc4da55c7f P c561ec3374b94b95be0c155d80ff0bff [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2537bbbd1a9947ae9facd87567e3ccb9 in term 2.
I20250112 15:07:36.151355 21226 raft_consensus.cc:2463] T 16f16fcd5cf347e5853189dc4da55c7f P 3710b74ed2044ce5995533df32e67351 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2537bbbd1a9947ae9facd87567e3ccb9 in term 2.
I20250112 15:07:36.151443 21040 leader_election.cc:304] T 16f16fcd5cf347e5853189dc4da55c7f P 2537bbbd1a9947ae9facd87567e3ccb9 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2537bbbd1a9947ae9facd87567e3ccb9, c561ec3374b94b95be0c155d80ff0bff; no voters:
I20250112 15:07:36.151515 21790 raft_consensus.cc:2798] T 16f16fcd5cf347e5853189dc4da55c7f P 2537bbbd1a9947ae9facd87567e3ccb9 [term 2 FOLLOWER]: Leader election won for term 2
I20250112 15:07:36.151592 21790 raft_consensus.cc:695] T 16f16fcd5cf347e5853189dc4da55c7f P 2537bbbd1a9947ae9facd87567e3ccb9 [term 2 LEADER]: Becoming Leader. State: Replica: 2537bbbd1a9947ae9facd87567e3ccb9, State: Running, Role: LEADER
I20250112 15:07:36.151643 21790 consensus_queue.cc:237] T 16f16fcd5cf347e5853189dc4da55c7f P 2537bbbd1a9947ae9facd87567e3ccb9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 25, Committed index: 25, Last appended: 1.25, Last appended by leader: 25, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2537bbbd1a9947ae9facd87567e3ccb9" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 33173 } } peers { permanent_uuid: "c561ec3374b94b95be0c155d80ff0bff" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 33627 } } peers { permanent_uuid: "3710b74ed2044ce5995533df32e67351" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 38589 } }
I20250112 15:07:36.152143 20966 catalog_manager.cc:5526] T 16f16fcd5cf347e5853189dc4da55c7f P 2537bbbd1a9947ae9facd87567e3ccb9 reported cstate change: term changed from 1 to 2, leader changed from 3710b74ed2044ce5995533df32e67351 (127.17.60.195) to 2537bbbd1a9947ae9facd87567e3ccb9 (127.17.60.193). New cstate: current_term: 2 leader_uuid: "2537bbbd1a9947ae9facd87567e3ccb9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2537bbbd1a9947ae9facd87567e3ccb9" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 33173 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c561ec3374b94b95be0c155d80ff0bff" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 33627 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3710b74ed2044ce5995533df32e67351" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 38589 } health_report { overall_health: UNKNOWN } } }
I20250112 15:07:36.310247 21790 raft_consensus.cc:988] T 8c7dbaf285a04da9b4eec92a29ab7d6c P 2537bbbd1a9947ae9facd87567e3ccb9: : Instructing follower c561ec3374b94b95be0c155d80ff0bff to start an election
I20250112 15:07:36.310330 21790 raft_consensus.cc:1076] T 8c7dbaf285a04da9b4eec92a29ab7d6c P 2537bbbd1a9947ae9facd87567e3ccb9 [term 2 LEADER]: Signalling peer c561ec3374b94b95be0c155d80ff0bff to start an election
I20250112 15:07:36.310490 21149 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "8c7dbaf285a04da9b4eec92a29ab7d6c"
dest_uuid: "c561ec3374b94b95be0c155d80ff0bff"
from {username='slave'} at 127.0.0.1:48082
I20250112 15:07:36.310552 21149 raft_consensus.cc:491] T 8c7dbaf285a04da9b4eec92a29ab7d6c P c561ec3374b94b95be0c155d80ff0bff [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20250112 15:07:36.310570 21149 raft_consensus.cc:3054] T 8c7dbaf285a04da9b4eec92a29ab7d6c P c561ec3374b94b95be0c155d80ff0bff [term 2 FOLLOWER]: Advancing to term 3
I20250112 15:07:36.311156 21149 raft_consensus.cc:513] T 8c7dbaf285a04da9b4eec92a29ab7d6c P c561ec3374b94b95be0c155d80ff0bff [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2537bbbd1a9947ae9facd87567e3ccb9" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 33173 } } peers { permanent_uuid: "c561ec3374b94b95be0c155d80ff0bff" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 33627 } } peers { permanent_uuid: "3710b74ed2044ce5995533df32e67351" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 38589 } }
I20250112 15:07:36.311290 21149 leader_election.cc:290] T 8c7dbaf285a04da9b4eec92a29ab7d6c P c561ec3374b94b95be0c155d80ff0bff [CANDIDATE]: Term 3 election: Requested vote from peers 2537bbbd1a9947ae9facd87567e3ccb9 (127.17.60.193:33173), 3710b74ed2044ce5995533df32e67351 (127.17.60.195:38589)
I20250112 15:07:36.311421 21226 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c7dbaf285a04da9b4eec92a29ab7d6c" candidate_uuid: "c561ec3374b94b95be0c155d80ff0bff" candidate_term: 3 candidate_status { last_received { term: 2 index: 26 } } ignore_live_leader: true dest_uuid: "3710b74ed2044ce5995533df32e67351"
I20250112 15:07:36.311515 21226 raft_consensus.cc:3054] T 8c7dbaf285a04da9b4eec92a29ab7d6c P 3710b74ed2044ce5995533df32e67351 [term 2 FOLLOWER]: Advancing to term 3
I20250112 15:07:36.312211 21226 raft_consensus.cc:2463] T 8c7dbaf285a04da9b4eec92a29ab7d6c P 3710b74ed2044ce5995533df32e67351 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c561ec3374b94b95be0c155d80ff0bff in term 3.
I20250112 15:07:36.312330 21068 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c7dbaf285a04da9b4eec92a29ab7d6c" candidate_uuid: "c561ec3374b94b95be0c155d80ff0bff" candidate_term: 3 candidate_status { last_received { term: 2 index: 26 } } ignore_live_leader: true dest_uuid: "2537bbbd1a9947ae9facd87567e3ccb9"
I20250112 15:07:36.312376 21115 leader_election.cc:304] T 8c7dbaf285a04da9b4eec92a29ab7d6c P c561ec3374b94b95be0c155d80ff0bff [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3710b74ed2044ce5995533df32e67351, c561ec3374b94b95be0c155d80ff0bff; no voters:
I20250112 15:07:36.312393 21068 raft_consensus.cc:3049] T 8c7dbaf285a04da9b4eec92a29ab7d6c P 2537bbbd1a9947ae9facd87567e3ccb9 [term 2 LEADER]: Stepping down as leader of term 2
I20250112 15:07:36.312417 21068 raft_consensus.cc:738] T 8c7dbaf285a04da9b4eec92a29ab7d6c P 2537bbbd1a9947ae9facd87567e3ccb9 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 2537bbbd1a9947ae9facd87567e3ccb9, State: Running, Role: LEADER
I20250112 15:07:36.312469 21068 consensus_queue.cc:260] T 8c7dbaf285a04da9b4eec92a29ab7d6c P 2537bbbd1a9947ae9facd87567e3ccb9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 26, Committed index: 26, Last appended: 2.26, Last appended by leader: 26, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2537bbbd1a9947ae9facd87567e3ccb9" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 33173 } } peers { permanent_uuid: "c561ec3374b94b95be0c155d80ff0bff" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 33627 } } peers { permanent_uuid: "3710b74ed2044ce5995533df32e67351" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 38589 } }
I20250112 15:07:36.312554 21068 raft_consensus.cc:3054] T 8c7dbaf285a04da9b4eec92a29ab7d6c P 2537bbbd1a9947ae9facd87567e3ccb9 [term 2 FOLLOWER]: Advancing to term 3
I20250112 15:07:36.313237 21068 raft_consensus.cc:2463] T 8c7dbaf285a04da9b4eec92a29ab7d6c P 2537bbbd1a9947ae9facd87567e3ccb9 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c561ec3374b94b95be0c155d80ff0bff in term 3.
I20250112 15:07:36.313329 21854 raft_consensus.cc:2798] T 8c7dbaf285a04da9b4eec92a29ab7d6c P c561ec3374b94b95be0c155d80ff0bff [term 3 FOLLOWER]: Leader election won for term 3
I20250112 15:07:36.313374 21854 raft_consensus.cc:695] T 8c7dbaf285a04da9b4eec92a29ab7d6c P c561ec3374b94b95be0c155d80ff0bff [term 3 LEADER]: Becoming Leader. State: Replica: c561ec3374b94b95be0c155d80ff0bff, State: Running, Role: LEADER
I20250112 15:07:36.313423 21854 consensus_queue.cc:237] T 8c7dbaf285a04da9b4eec92a29ab7d6c P c561ec3374b94b95be0c155d80ff0bff [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 26, Committed index: 26, Last appended: 2.26, Last appended by leader: 26, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2537bbbd1a9947ae9facd87567e3ccb9" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 33173 } } peers { permanent_uuid: "c561ec3374b94b95be0c155d80ff0bff" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 33627 } } peers { permanent_uuid: "3710b74ed2044ce5995533df32e67351" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 38589 } }
I20250112 15:07:36.313936 20966 catalog_manager.cc:5526] T 8c7dbaf285a04da9b4eec92a29ab7d6c P c561ec3374b94b95be0c155d80ff0bff reported cstate change: term changed from 2 to 3, leader changed from 2537bbbd1a9947ae9facd87567e3ccb9 (127.17.60.193) to c561ec3374b94b95be0c155d80ff0bff (127.17.60.194). New cstate: current_term: 3 leader_uuid: "c561ec3374b94b95be0c155d80ff0bff" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2537bbbd1a9947ae9facd87567e3ccb9" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 33173 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c561ec3374b94b95be0c155d80ff0bff" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 33627 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3710b74ed2044ce5995533df32e67351" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 38589 } health_report { overall_health: UNKNOWN } } }
I20250112 15:07:36.400260 21790 raft_consensus.cc:988] T 9601bcda24ed42e7b93ab9740f1e271e P 2537bbbd1a9947ae9facd87567e3ccb9: : Instructing follower 3710b74ed2044ce5995533df32e67351 to start an election
I20250112 15:07:36.400360 21790 raft_consensus.cc:1076] T 9601bcda24ed42e7b93ab9740f1e271e P 2537bbbd1a9947ae9facd87567e3ccb9 [term 1 LEADER]: Signalling peer 3710b74ed2044ce5995533df32e67351 to start an election
I20250112 15:07:36.400497 21226 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "9601bcda24ed42e7b93ab9740f1e271e"
dest_uuid: "3710b74ed2044ce5995533df32e67351"
from {username='slave'} at 127.0.0.1:33194
I20250112 15:07:36.400596 21226 raft_consensus.cc:491] T 9601bcda24ed42e7b93ab9740f1e271e P 3710b74ed2044ce5995533df32e67351 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20250112 15:07:36.400632 21226 raft_consensus.cc:3054] T 9601bcda24ed42e7b93ab9740f1e271e P 3710b74ed2044ce5995533df32e67351 [term 1 FOLLOWER]: Advancing to term 2
I20250112 15:07:36.401504 21226 raft_consensus.cc:513] T 9601bcda24ed42e7b93ab9740f1e271e P 3710b74ed2044ce5995533df32e67351 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "2537bbbd1a9947ae9facd87567e3ccb9" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 33173 } } peers { permanent_uuid: "3710b74ed2044ce5995533df32e67351" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 38589 } } peers { permanent_uuid: "a2a5bba4ac1145d5bab94a69c9f4df40" member_type: VOTER last_known_addr { host: "127.17.60.196" port: 41807 } attrs { promote: false } }
I20250112 15:07:36.401682 21226 leader_election.cc:290] T 9601bcda24ed42e7b93ab9740f1e271e P 3710b74ed2044ce5995533df32e67351 [CANDIDATE]: Term 2 election: Requested vote from peers 2537bbbd1a9947ae9facd87567e3ccb9 (127.17.60.193:33173), a2a5bba4ac1145d5bab94a69c9f4df40 (127.17.60.196:41807)
I20250112 15:07:36.401780 21068 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9601bcda24ed42e7b93ab9740f1e271e" candidate_uuid: "3710b74ed2044ce5995533df32e67351" candidate_term: 2 candidate_status { last_received { term: 1 index: 27 } } ignore_live_leader: true dest_uuid: "2537bbbd1a9947ae9facd87567e3ccb9"
I20250112 15:07:36.401846 21068 raft_consensus.cc:3049] T 9601bcda24ed42e7b93ab9740f1e271e P 2537bbbd1a9947ae9facd87567e3ccb9 [term 1 LEADER]: Stepping down as leader of term 1
I20250112 15:07:36.401875 21068 raft_consensus.cc:738] T 9601bcda24ed42e7b93ab9740f1e271e P 2537bbbd1a9947ae9facd87567e3ccb9 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 2537bbbd1a9947ae9facd87567e3ccb9, State: Running, Role: LEADER
I20250112 15:07:36.401926 21068 consensus_queue.cc:260] T 9601bcda24ed42e7b93ab9740f1e271e P 2537bbbd1a9947ae9facd87567e3ccb9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 27, Committed index: 27, Last appended: 1.27, Last appended by leader: 27, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "2537bbbd1a9947ae9facd87567e3ccb9" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 33173 } } peers { permanent_uuid: "3710b74ed2044ce5995533df32e67351" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 38589 } } peers { permanent_uuid: "a2a5bba4ac1145d5bab94a69c9f4df40" member_type: VOTER last_known_addr { host: "127.17.60.196" port: 41807 } attrs { promote: false } }
I20250112 15:07:36.402009 21068 raft_consensus.cc:3054] T 9601bcda24ed42e7b93ab9740f1e271e P 2537bbbd1a9947ae9facd87567e3ccb9 [term 1 FOLLOWER]: Advancing to term 2
I20250112 15:07:36.402707 21068 raft_consensus.cc:2463] T 9601bcda24ed42e7b93ab9740f1e271e P 2537bbbd1a9947ae9facd87567e3ccb9 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3710b74ed2044ce5995533df32e67351 in term 2.
I20250112 15:07:36.402829 21189 leader_election.cc:304] T 9601bcda24ed42e7b93ab9740f1e271e P 3710b74ed2044ce5995533df32e67351 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2537bbbd1a9947ae9facd87567e3ccb9, 3710b74ed2044ce5995533df32e67351; no voters:
I20250112 15:07:36.402894 21843 raft_consensus.cc:2798] T 9601bcda24ed42e7b93ab9740f1e271e P 3710b74ed2044ce5995533df32e67351 [term 2 FOLLOWER]: Leader election won for term 2
I20250112 15:07:36.402938 21843 raft_consensus.cc:695] T 9601bcda24ed42e7b93ab9740f1e271e P 3710b74ed2044ce5995533df32e67351 [term 2 LEADER]: Becoming Leader. State: Replica: 3710b74ed2044ce5995533df32e67351, State: Running, Role: LEADER
I20250112 15:07:36.402983 21843 consensus_queue.cc:237] T 9601bcda24ed42e7b93ab9740f1e271e P 3710b74ed2044ce5995533df32e67351 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 27, Committed index: 27, Last appended: 1.27, Last appended by leader: 27, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "2537bbbd1a9947ae9facd87567e3ccb9" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 33173 } } peers { permanent_uuid: "3710b74ed2044ce5995533df32e67351" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 38589 } } peers { permanent_uuid: "a2a5bba4ac1145d5bab94a69c9f4df40" member_type: VOTER last_known_addr { host: "127.17.60.196" port: 41807 } attrs { promote: false } }
I20250112 15:07:36.403532 20966 catalog_manager.cc:5526] T 9601bcda24ed42e7b93ab9740f1e271e P 3710b74ed2044ce5995533df32e67351 reported cstate change: term changed from 1 to 2, leader changed from 2537bbbd1a9947ae9facd87567e3ccb9 (127.17.60.193) to 3710b74ed2044ce5995533df32e67351 (127.17.60.195). New cstate: current_term: 2 leader_uuid: "3710b74ed2044ce5995533df32e67351" committed_config { opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "2537bbbd1a9947ae9facd87567e3ccb9" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 33173 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3710b74ed2044ce5995533df32e67351" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 38589 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a2a5bba4ac1145d5bab94a69c9f4df40" member_type: VOTER last_known_addr { host: "127.17.60.196" port: 41807 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20250112 15:07:36.404959 21474 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9601bcda24ed42e7b93ab9740f1e271e" candidate_uuid: "3710b74ed2044ce5995533df32e67351" candidate_term: 2 candidate_status { last_received { term: 1 index: 27 } } ignore_live_leader: true dest_uuid: "a2a5bba4ac1145d5bab94a69c9f4df40"
I20250112 15:07:36.405037 21474 raft_consensus.cc:3054] T 9601bcda24ed42e7b93ab9740f1e271e P a2a5bba4ac1145d5bab94a69c9f4df40 [term 1 FOLLOWER]: Advancing to term 2
I20250112 15:07:36.405776 21474 raft_consensus.cc:2463] T 9601bcda24ed42e7b93ab9740f1e271e P a2a5bba4ac1145d5bab94a69c9f4df40 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3710b74ed2044ce5995533df32e67351 in term 2.
I20250112 15:07:36.568531 21226 raft_consensus.cc:1270] T 16f16fcd5cf347e5853189dc4da55c7f P 3710b74ed2044ce5995533df32e67351 [term 2 FOLLOWER]: Refusing update from remote peer 2537bbbd1a9947ae9facd87567e3ccb9: Log matching property violated. Preceding OpId in replica: term: 1 index: 25. Preceding OpId from leader: term: 2 index: 26. (index mismatch)
I20250112 15:07:36.568732 21790 consensus_queue.cc:1035] T 16f16fcd5cf347e5853189dc4da55c7f P 2537bbbd1a9947ae9facd87567e3ccb9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3710b74ed2044ce5995533df32e67351" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 38589 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 26, Last known committed idx: 25, Time since last communication: 0.000s
I20250112 15:07:36.569644 21147 raft_consensus.cc:1270] T 16f16fcd5cf347e5853189dc4da55c7f P c561ec3374b94b95be0c155d80ff0bff [term 2 FOLLOWER]: Refusing update from remote peer 2537bbbd1a9947ae9facd87567e3ccb9: Log matching property violated. Preceding OpId in replica: term: 1 index: 25. Preceding OpId from leader: term: 2 index: 26. (index mismatch)
I20250112 15:07:36.569864 21869 consensus_queue.cc:1035] T 16f16fcd5cf347e5853189dc4da55c7f P 2537bbbd1a9947ae9facd87567e3ccb9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c561ec3374b94b95be0c155d80ff0bff" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 33627 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 26, Last known committed idx: 25, Time since last communication: 0.000s
I20250112 15:07:36.689630 21068 raft_consensus.cc:1270] T 8c7dbaf285a04da9b4eec92a29ab7d6c P 2537bbbd1a9947ae9facd87567e3ccb9 [term 3 FOLLOWER]: Refusing update from remote peer c561ec3374b94b95be0c155d80ff0bff: Log matching property violated. Preceding OpId in replica: term: 2 index: 26. Preceding OpId from leader: term: 3 index: 27. (index mismatch)
I20250112 15:07:36.689877 21873 consensus_queue.cc:1035] T 8c7dbaf285a04da9b4eec92a29ab7d6c P c561ec3374b94b95be0c155d80ff0bff [LEADER]: Connected to new peer: Peer: permanent_uuid: "2537bbbd1a9947ae9facd87567e3ccb9" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 33173 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 27, Last known committed idx: 26, Time since last communication: 0.000s
I20250112 15:07:36.690908 21226 raft_consensus.cc:1270] T 8c7dbaf285a04da9b4eec92a29ab7d6c P 3710b74ed2044ce5995533df32e67351 [term 3 FOLLOWER]: Refusing update from remote peer c561ec3374b94b95be0c155d80ff0bff: Log matching property violated. Preceding OpId in replica: term: 2 index: 26. Preceding OpId from leader: term: 3 index: 27. (index mismatch)
I20250112 15:07:36.691133 21873 consensus_queue.cc:1035] T 8c7dbaf285a04da9b4eec92a29ab7d6c P c561ec3374b94b95be0c155d80ff0bff [LEADER]: Connected to new peer: Peer: permanent_uuid: "3710b74ed2044ce5995533df32e67351" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 38589 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 27, Last known committed idx: 26, Time since last communication: 0.000s
I20250112 15:07:36.819809 21474 raft_consensus.cc:1270] T 9601bcda24ed42e7b93ab9740f1e271e P a2a5bba4ac1145d5bab94a69c9f4df40 [term 2 FOLLOWER]: Refusing update from remote peer 3710b74ed2044ce5995533df32e67351: Log matching property violated. Preceding OpId in replica: term: 1 index: 27. Preceding OpId from leader: term: 2 index: 28. (index mismatch)
I20250112 15:07:36.820088 21843 consensus_queue.cc:1035] T 9601bcda24ed42e7b93ab9740f1e271e P 3710b74ed2044ce5995533df32e67351 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2a5bba4ac1145d5bab94a69c9f4df40" member_type: VOTER last_known_addr { host: "127.17.60.196" port: 41807 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 28, Last known committed idx: 27, Time since last communication: 0.000s
I20250112 15:07:36.821048 21068 raft_consensus.cc:1270] T 9601bcda24ed42e7b93ab9740f1e271e P 2537bbbd1a9947ae9facd87567e3ccb9 [term 2 FOLLOWER]: Refusing update from remote peer 3710b74ed2044ce5995533df32e67351: Log matching property violated. Preceding OpId in replica: term: 1 index: 27. Preceding OpId from leader: term: 2 index: 28. (index mismatch)
I20250112 15:07:36.821228 21855 consensus_queue.cc:1035] T 9601bcda24ed42e7b93ab9740f1e271e P 3710b74ed2044ce5995533df32e67351 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2537bbbd1a9947ae9facd87567e3ccb9" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 33173 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 28, Last known committed idx: 27, Time since last communication: 0.000s
I20250112 15:07:37.432216 21845 tablet_copy_client.cc:538] T 5bcec60d3b974a9bb24aedfc738e1594 P b5e9178a7a83402e8c4a6943ab78589d: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250112 15:07:37.433158 21845 tablet_bootstrap.cc:492] T 5bcec60d3b974a9bb24aedfc738e1594 P b5e9178a7a83402e8c4a6943ab78589d: Bootstrap starting.
I20250112 15:07:37.436714 21845 tablet_bootstrap.cc:492] T 5bcec60d3b974a9bb24aedfc738e1594 P b5e9178a7a83402e8c4a6943ab78589d: Bootstrap replayed 1/1 log segments. Stats: ops{read=30 overwritten=0 applied=30 ignored=0} inserts{seen=88 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250112 15:07:37.436902 21845 tablet_bootstrap.cc:492] T 5bcec60d3b974a9bb24aedfc738e1594 P b5e9178a7a83402e8c4a6943ab78589d: Bootstrap complete.
I20250112 15:07:37.437003 21845 ts_tablet_manager.cc:1397] T 5bcec60d3b974a9bb24aedfc738e1594 P b5e9178a7a83402e8c4a6943ab78589d: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.002s
I20250112 15:07:37.437140 21845 raft_consensus.cc:357] T 5bcec60d3b974a9bb24aedfc738e1594 P b5e9178a7a83402e8c4a6943ab78589d [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "2537bbbd1a9947ae9facd87567e3ccb9" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 33173 } attrs { replace: true } } peers { permanent_uuid: "3710b74ed2044ce5995533df32e67351" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 38589 } } peers { permanent_uuid: "a2a5bba4ac1145d5bab94a69c9f4df40" member_type: VOTER last_known_addr { host: "127.17.60.196" port: 41807 } attrs { promote: false } } peers { permanent_uuid: "b5e9178a7a83402e8c4a6943ab78589d" member_type: NON_VOTER last_known_addr { host: "127.17.60.197" port: 36573 } attrs { promote: true } }
I20250112 15:07:37.437213 21845 raft_consensus.cc:738] T 5bcec60d3b974a9bb24aedfc738e1594 P b5e9178a7a83402e8c4a6943ab78589d [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: b5e9178a7a83402e8c4a6943ab78589d, State: Initialized, Role: LEARNER
I20250112 15:07:37.437245 21845 consensus_queue.cc:260] T 5bcec60d3b974a9bb24aedfc738e1594 P b5e9178a7a83402e8c4a6943ab78589d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 30, Last appended: 2.30, Last appended by leader: 30, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "2537bbbd1a9947ae9facd87567e3ccb9" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 33173 } attrs { replace: true } } peers { permanent_uuid: "3710b74ed2044ce5995533df32e67351" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 38589 } } peers { permanent_uuid: "a2a5bba4ac1145d5bab94a69c9f4df40" member_type: VOTER last_known_addr { host: "127.17.60.196" port: 41807 } attrs { promote: false } } peers { permanent_uuid: "b5e9178a7a83402e8c4a6943ab78589d" member_type: NON_VOTER last_known_addr { host: "127.17.60.197" port: 36573 } attrs { promote: true } }
I20250112 15:07:37.437355 21845 ts_tablet_manager.cc:1428] T 5bcec60d3b974a9bb24aedfc738e1594 P b5e9178a7a83402e8c4a6943ab78589d: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20250112 15:07:37.437521 21484 tablet_copy_service.cc:342] P a2a5bba4ac1145d5bab94a69c9f4df40: Request end of tablet copy session b5e9178a7a83402e8c4a6943ab78589d-5bcec60d3b974a9bb24aedfc738e1594 received from {username='slave'} at 127.0.0.1:43302
I20250112 15:07:37.437567 21484 tablet_copy_service.cc:434] P a2a5bba4ac1145d5bab94a69c9f4df40: ending tablet copy session b5e9178a7a83402e8c4a6943ab78589d-5bcec60d3b974a9bb24aedfc738e1594 on tablet 5bcec60d3b974a9bb24aedfc738e1594 with peer b5e9178a7a83402e8c4a6943ab78589d
I20250112 15:07:37.523378 21848 tablet_copy_client.cc:538] T 69c23627ab854000b360030c384dc966 P a2a5bba4ac1145d5bab94a69c9f4df40: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250112 15:07:37.524097 21848 tablet_bootstrap.cc:492] T 69c23627ab854000b360030c384dc966 P a2a5bba4ac1145d5bab94a69c9f4df40: Bootstrap starting.
I20250112 15:07:37.527773 21848 tablet_bootstrap.cc:492] T 69c23627ab854000b360030c384dc966 P a2a5bba4ac1145d5bab94a69c9f4df40: Bootstrap replayed 1/1 log segments. Stats: ops{read=28 overwritten=0 applied=28 ignored=0} inserts{seen=103 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250112 15:07:37.527974 21848 tablet_bootstrap.cc:492] T 69c23627ab854000b360030c384dc966 P a2a5bba4ac1145d5bab94a69c9f4df40: Bootstrap complete.
I20250112 15:07:37.528084 21848 ts_tablet_manager.cc:1397] T 69c23627ab854000b360030c384dc966 P a2a5bba4ac1145d5bab94a69c9f4df40: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.002s
I20250112 15:07:37.528239 21848 raft_consensus.cc:357] T 69c23627ab854000b360030c384dc966 P a2a5bba4ac1145d5bab94a69c9f4df40 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "c561ec3374b94b95be0c155d80ff0bff" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 33627 } } peers { permanent_uuid: "3710b74ed2044ce5995533df32e67351" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 38589 } attrs { replace: true } } peers { permanent_uuid: "b5e9178a7a83402e8c4a6943ab78589d" member_type: VOTER last_known_addr { host: "127.17.60.197" port: 36573 } attrs { promote: false } } peers { permanent_uuid: "a2a5bba4ac1145d5bab94a69c9f4df40" member_type: NON_VOTER last_known_addr { host: "127.17.60.196" port: 41807 } attrs { promote: true } }
I20250112 15:07:37.528312 21848 raft_consensus.cc:738] T 69c23627ab854000b360030c384dc966 P a2a5bba4ac1145d5bab94a69c9f4df40 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: a2a5bba4ac1145d5bab94a69c9f4df40, State: Initialized, Role: LEARNER
I20250112 15:07:37.528340 21848 consensus_queue.cc:260] T 69c23627ab854000b360030c384dc966 P a2a5bba4ac1145d5bab94a69c9f4df40 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 28, Last appended: 1.28, Last appended by leader: 28, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "c561ec3374b94b95be0c155d80ff0bff" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 33627 } } peers { permanent_uuid: "3710b74ed2044ce5995533df32e67351" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 38589 } attrs { replace: true } } peers { permanent_uuid: "b5e9178a7a83402e8c4a6943ab78589d" member_type: VOTER last_known_addr { host: "127.17.60.197" port: 36573 } attrs { promote: false } } peers { permanent_uuid: "a2a5bba4ac1145d5bab94a69c9f4df40" member_type: NON_VOTER last_known_addr { host: "127.17.60.196" port: 41807 } attrs { promote: true } }
I20250112 15:07:37.528465 21848 ts_tablet_manager.cc:1428] T 69c23627ab854000b360030c384dc966 P a2a5bba4ac1145d5bab94a69c9f4df40: Time spent starting tablet: real 0.000s user 0.000s sys 0.003s
I20250112 15:07:37.528723 21161 tablet_copy_service.cc:342] P c561ec3374b94b95be0c155d80ff0bff: Request end of tablet copy session a2a5bba4ac1145d5bab94a69c9f4df40-69c23627ab854000b360030c384dc966 received from {username='slave'} at 127.0.0.1:58522
I20250112 15:07:37.528776 21161 tablet_copy_service.cc:434] P c561ec3374b94b95be0c155d80ff0bff: ending tablet copy session a2a5bba4ac1145d5bab94a69c9f4df40-69c23627ab854000b360030c384dc966 on tablet 69c23627ab854000b360030c384dc966 with peer a2a5bba4ac1145d5bab94a69c9f4df40
I20250112 15:07:37.789251 21473 raft_consensus.cc:1212] T 69c23627ab854000b360030c384dc966 P a2a5bba4ac1145d5bab94a69c9f4df40 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.27->[1.28-1.28] Dedup: 1.28->[]
I20250112 15:07:37.958858 21549 raft_consensus.cc:1212] T 5bcec60d3b974a9bb24aedfc738e1594 P b5e9178a7a83402e8c4a6943ab78589d [term 2 LEARNER]: Deduplicated request from leader. Original: 2.29->[2.30-2.30] Dedup: 2.30->[]
I20250112 15:07:38.089231 21026 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250112 15:07:38.089681 21069 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "02c33d457b6b47138dd712850819b619"
dest_uuid: "2537bbbd1a9947ae9facd87567e3ccb9"
mode: GRACEFUL
new_leader_uuid: "3710b74ed2044ce5995533df32e67351"
from {username='slave'} at 127.0.0.1:54414
I20250112 15:07:38.089776 21069 raft_consensus.cc:604] T 02c33d457b6b47138dd712850819b619 P 2537bbbd1a9947ae9facd87567e3ccb9 [term 3 LEADER]: Received request to transfer leadership to TS 3710b74ed2044ce5995533df32e67351
I20250112 15:07:38.090055 21147 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "69c23627ab854000b360030c384dc966"
dest_uuid: "c561ec3374b94b95be0c155d80ff0bff"
mode: GRACEFUL
new_leader_uuid: "3710b74ed2044ce5995533df32e67351"
from {username='slave'} at 127.0.0.1:48104
I20250112 15:07:38.090135 21147 raft_consensus.cc:604] T 69c23627ab854000b360030c384dc966 P c561ec3374b94b95be0c155d80ff0bff [term 1 LEADER]: Received request to transfer leadership to TS 3710b74ed2044ce5995533df32e67351
I20250112 15:07:38.090468 21226 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "bfaeae584ec04772b2d6fcd088022117"
dest_uuid: "3710b74ed2044ce5995533df32e67351"
mode: GRACEFUL
new_leader_uuid: "2537bbbd1a9947ae9facd87567e3ccb9"
from {username='slave'} at 127.0.0.1:54970
I20250112 15:07:38.090549 21226 raft_consensus.cc:604] T bfaeae584ec04772b2d6fcd088022117 P 3710b74ed2044ce5995533df32e67351 [term 1 LEADER]: Received request to transfer leadership to TS 2537bbbd1a9947ae9facd87567e3ccb9
I20250112 15:07:38.090672 21026 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 3
I20250112 15:07:38.090759 21026 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20250112 15:07:38.168195 21883 raft_consensus.cc:1059] T 69c23627ab854000b360030c384dc966 P c561ec3374b94b95be0c155d80ff0bff: attempting to promote NON_VOTER a2a5bba4ac1145d5bab94a69c9f4df40 to VOTER
W20250112 15:07:38.168282 21883 raft_consensus.cc:1062] T 69c23627ab854000b360030c384dc966 P c561ec3374b94b95be0c155d80ff0bff: Unable to promote non-voter a2a5bba4ac1145d5bab94a69c9f4df40: Service unavailable: leader transfer in progress
I20250112 15:07:38.323242 21855 raft_consensus.cc:988] T bfaeae584ec04772b2d6fcd088022117 P 3710b74ed2044ce5995533df32e67351: : Instructing follower 2537bbbd1a9947ae9facd87567e3ccb9 to start an election
I20250112 15:07:38.323349 21855 raft_consensus.cc:1076] T bfaeae584ec04772b2d6fcd088022117 P 3710b74ed2044ce5995533df32e67351 [term 1 LEADER]: Signalling peer 2537bbbd1a9947ae9facd87567e3ccb9 to start an election
I20250112 15:07:38.323524 21069 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "bfaeae584ec04772b2d6fcd088022117"
dest_uuid: "2537bbbd1a9947ae9facd87567e3ccb9"
from {username='slave'} at 127.0.0.1:37204
I20250112 15:07:38.323599 21069 raft_consensus.cc:491] T bfaeae584ec04772b2d6fcd088022117 P 2537bbbd1a9947ae9facd87567e3ccb9 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20250112 15:07:38.323627 21069 raft_consensus.cc:3054] T bfaeae584ec04772b2d6fcd088022117 P 2537bbbd1a9947ae9facd87567e3ccb9 [term 1 FOLLOWER]: Advancing to term 2
I20250112 15:07:38.324280 21069 raft_consensus.cc:513] T bfaeae584ec04772b2d6fcd088022117 P 2537bbbd1a9947ae9facd87567e3ccb9 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2537bbbd1a9947ae9facd87567e3ccb9" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 33173 } } peers { permanent_uuid: "c561ec3374b94b95be0c155d80ff0bff" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 33627 } } peers { permanent_uuid: "3710b74ed2044ce5995533df32e67351" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 38589 } }
I20250112 15:07:38.324427 21069 leader_election.cc:290] T bfaeae584ec04772b2d6fcd088022117 P 2537bbbd1a9947ae9facd87567e3ccb9 [CANDIDATE]: Term 2 election: Requested vote from peers c561ec3374b94b95be0c155d80ff0bff (127.17.60.194:33627), 3710b74ed2044ce5995533df32e67351 (127.17.60.195:38589)
I20250112 15:07:38.324532 21226 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bfaeae584ec04772b2d6fcd088022117" candidate_uuid: "2537bbbd1a9947ae9facd87567e3ccb9" candidate_term: 2 candidate_status { last_received { term: 1 index: 25 } } ignore_live_leader: true dest_uuid: "3710b74ed2044ce5995533df32e67351"
I20250112 15:07:38.324553 21149 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bfaeae584ec04772b2d6fcd088022117" candidate_uuid: "2537bbbd1a9947ae9facd87567e3ccb9" candidate_term: 2 candidate_status { last_received { term: 1 index: 25 } } ignore_live_leader: true dest_uuid: "c561ec3374b94b95be0c155d80ff0bff"
I20250112 15:07:38.324667 21226 raft_consensus.cc:3049] T bfaeae584ec04772b2d6fcd088022117 P 3710b74ed2044ce5995533df32e67351 [term 1 LEADER]: Stepping down as leader of term 1
I20250112 15:07:38.324674 21149 raft_consensus.cc:3054] T bfaeae584ec04772b2d6fcd088022117 P c561ec3374b94b95be0c155d80ff0bff [term 1 FOLLOWER]: Advancing to term 2
I20250112 15:07:38.324709 21226 raft_consensus.cc:738] T bfaeae584ec04772b2d6fcd088022117 P 3710b74ed2044ce5995533df32e67351 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 3710b74ed2044ce5995533df32e67351, State: Running, Role: LEADER
I20250112 15:07:38.324771 21226 consensus_queue.cc:260] T bfaeae584ec04772b2d6fcd088022117 P 3710b74ed2044ce5995533df32e67351 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 25, Committed index: 25, Last appended: 1.25, Last appended by leader: 25, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2537bbbd1a9947ae9facd87567e3ccb9" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 33173 } } peers { permanent_uuid: "c561ec3374b94b95be0c155d80ff0bff" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 33627 } } peers { permanent_uuid: "3710b74ed2044ce5995533df32e67351" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 38589 } }
I20250112 15:07:38.324856 21226 raft_consensus.cc:3054] T bfaeae584ec04772b2d6fcd088022117 P 3710b74ed2044ce5995533df32e67351 [term 1 FOLLOWER]: Advancing to term 2
I20250112 15:07:38.325443 21149 raft_consensus.cc:2463] T bfaeae584ec04772b2d6fcd088022117 P c561ec3374b94b95be0c155d80ff0bff [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2537bbbd1a9947ae9facd87567e3ccb9 in term 2.
I20250112 15:07:38.325495 21226 raft_consensus.cc:2463] T bfaeae584ec04772b2d6fcd088022117 P 3710b74ed2044ce5995533df32e67351 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2537bbbd1a9947ae9facd87567e3ccb9 in term 2.
I20250112 15:07:38.325619 21039 leader_election.cc:304] T bfaeae584ec04772b2d6fcd088022117 P 2537bbbd1a9947ae9facd87567e3ccb9 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2537bbbd1a9947ae9facd87567e3ccb9, 3710b74ed2044ce5995533df32e67351; no voters:
I20250112 15:07:38.325716 21856 raft_consensus.cc:2798] T bfaeae584ec04772b2d6fcd088022117 P 2537bbbd1a9947ae9facd87567e3ccb9 [term 2 FOLLOWER]: Leader election won for term 2
I20250112 15:07:38.325809 21856 raft_consensus.cc:695] T bfaeae584ec04772b2d6fcd088022117 P 2537bbbd1a9947ae9facd87567e3ccb9 [term 2 LEADER]: Becoming Leader. State: Replica: 2537bbbd1a9947ae9facd87567e3ccb9, State: Running, Role: LEADER
I20250112 15:07:38.325863 21856 consensus_queue.cc:237] T bfaeae584ec04772b2d6fcd088022117 P 2537bbbd1a9947ae9facd87567e3ccb9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 25, Committed index: 25, Last appended: 1.25, Last appended by leader: 25, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2537bbbd1a9947ae9facd87567e3ccb9" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 33173 } } peers { permanent_uuid: "c561ec3374b94b95be0c155d80ff0bff" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 33627 } } peers { permanent_uuid: "3710b74ed2044ce5995533df32e67351" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 38589 } }
I20250112 15:07:38.326364 20966 catalog_manager.cc:5526] T bfaeae584ec04772b2d6fcd088022117 P 2537bbbd1a9947ae9facd87567e3ccb9 reported cstate change: term changed from 1 to 2, leader changed from 3710b74ed2044ce5995533df32e67351 (127.17.60.195) to 2537bbbd1a9947ae9facd87567e3ccb9 (127.17.60.193). New cstate: current_term: 2 leader_uuid: "2537bbbd1a9947ae9facd87567e3ccb9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2537bbbd1a9947ae9facd87567e3ccb9" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 33173 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c561ec3374b94b95be0c155d80ff0bff" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 33627 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3710b74ed2044ce5995533df32e67351" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 38589 } health_report { overall_health: UNKNOWN } } }
I20250112 15:07:38.382441 21891 raft_consensus.cc:1059] T 5bcec60d3b974a9bb24aedfc738e1594 P a2a5bba4ac1145d5bab94a69c9f4df40: attempting to promote NON_VOTER b5e9178a7a83402e8c4a6943ab78589d to VOTER
I20250112 15:07:38.382642 21891 consensus_queue.cc:237] T 5bcec60d3b974a9bb24aedfc738e1594 P a2a5bba4ac1145d5bab94a69c9f4df40 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 30, Committed index: 30, Last appended: 2.30, Last appended by leader: 28, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "2537bbbd1a9947ae9facd87567e3ccb9" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 33173 } attrs { replace: true } } peers { permanent_uuid: "3710b74ed2044ce5995533df32e67351" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 38589 } } peers { permanent_uuid: "a2a5bba4ac1145d5bab94a69c9f4df40" member_type: VOTER last_known_addr { host: "127.17.60.196" port: 41807 } attrs { promote: false } } peers { permanent_uuid: "b5e9178a7a83402e8c4a6943ab78589d" member_type: VOTER last_known_addr { host: "127.17.60.197" port: 36573 } attrs { promote: false } }
I20250112 15:07:38.383149 21226 raft_consensus.cc:1270] T 5bcec60d3b974a9bb24aedfc738e1594 P 3710b74ed2044ce5995533df32e67351 [term 2 FOLLOWER]: Refusing update from remote peer a2a5bba4ac1145d5bab94a69c9f4df40: Log matching property violated. Preceding OpId in replica: term: 2 index: 30. Preceding OpId from leader: term: 2 index: 31. (index mismatch)
I20250112 15:07:38.383172 21549 raft_consensus.cc:1270] T 5bcec60d3b974a9bb24aedfc738e1594 P b5e9178a7a83402e8c4a6943ab78589d [term 2 LEARNER]: Refusing update from remote peer a2a5bba4ac1145d5bab94a69c9f4df40: Log matching property violated. Preceding OpId in replica: term: 2 index: 30. Preceding OpId from leader: term: 2 index: 31. (index mismatch)
I20250112 15:07:38.383201 21069 raft_consensus.cc:1270] T 5bcec60d3b974a9bb24aedfc738e1594 P 2537bbbd1a9947ae9facd87567e3ccb9 [term 2 FOLLOWER]: Refusing update from remote peer a2a5bba4ac1145d5bab94a69c9f4df40: Log matching property violated. Preceding OpId in replica: term: 2 index: 30. Preceding OpId from leader: term: 2 index: 31. (index mismatch)
I20250112 15:07:38.383421 21894 consensus_queue.cc:1035] T 5bcec60d3b974a9bb24aedfc738e1594 P a2a5bba4ac1145d5bab94a69c9f4df40 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3710b74ed2044ce5995533df32e67351" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 38589 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 31, Last known committed idx: 30, Time since last communication: 0.000s
I20250112 15:07:38.383610 21893 consensus_queue.cc:1035] T 5bcec60d3b974a9bb24aedfc738e1594 P a2a5bba4ac1145d5bab94a69c9f4df40 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b5e9178a7a83402e8c4a6943ab78589d" member_type: VOTER last_known_addr { host: "127.17.60.197" port: 36573 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 31, Last known committed idx: 30, Time since last communication: 0.000s
I20250112 15:07:38.383834 21891 consensus_queue.cc:1035] T 5bcec60d3b974a9bb24aedfc738e1594 P a2a5bba4ac1145d5bab94a69c9f4df40 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2537bbbd1a9947ae9facd87567e3ccb9" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 33173 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 31, Last known committed idx: 30, Time since last communication: 0.000s
I20250112 15:07:38.384750 21893 raft_consensus.cc:2949] T 5bcec60d3b974a9bb24aedfc738e1594 P a2a5bba4ac1145d5bab94a69c9f4df40 [term 2 LEADER]: Committing config change with OpId 2.31: config changed from index 30 to 31, b5e9178a7a83402e8c4a6943ab78589d (127.17.60.197) changed from NON_VOTER to VOTER. New config: { opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "2537bbbd1a9947ae9facd87567e3ccb9" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 33173 } attrs { replace: true } } peers { permanent_uuid: "3710b74ed2044ce5995533df32e67351" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 38589 } } peers { permanent_uuid: "a2a5bba4ac1145d5bab94a69c9f4df40" member_type: VOTER last_known_addr { host: "127.17.60.196" port: 41807 } attrs { promote: false } } peers { permanent_uuid: "b5e9178a7a83402e8c4a6943ab78589d" member_type: VOTER last_known_addr { host: "127.17.60.197" port: 36573 } attrs { promote: false } } }
I20250112 15:07:38.384951 21069 raft_consensus.cc:2949] T 5bcec60d3b974a9bb24aedfc738e1594 P 2537bbbd1a9947ae9facd87567e3ccb9 [term 2 FOLLOWER]: Committing config change with OpId 2.31: config changed from index 30 to 31, b5e9178a7a83402e8c4a6943ab78589d (127.17.60.197) changed from NON_VOTER to VOTER. New config: { opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "2537bbbd1a9947ae9facd87567e3ccb9" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 33173 } attrs { replace: true } } peers { permanent_uuid: "3710b74ed2044ce5995533df32e67351" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 38589 } } peers { permanent_uuid: "a2a5bba4ac1145d5bab94a69c9f4df40" member_type: VOTER last_known_addr { host: "127.17.60.196" port: 41807 } attrs { promote: false } } peers { permanent_uuid: "b5e9178a7a83402e8c4a6943ab78589d" member_type: VOTER last_known_addr { host: "127.17.60.197" port: 36573 } attrs { promote: false } } }
I20250112 15:07:38.386034 21549 raft_consensus.cc:2949] T 5bcec60d3b974a9bb24aedfc738e1594 P b5e9178a7a83402e8c4a6943ab78589d [term 2 FOLLOWER]: Committing config change with OpId 2.31: config changed from index 30 to 31, b5e9178a7a83402e8c4a6943ab78589d (127.17.60.197) changed from NON_VOTER to VOTER. New config: { opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "2537bbbd1a9947ae9facd87567e3ccb9" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 33173 } attrs { replace: true } } peers { permanent_uuid: "3710b74ed2044ce5995533df32e67351" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 38589 } } peers { permanent_uuid: "a2a5bba4ac1145d5bab94a69c9f4df40" member_type: VOTER last_known_addr { host: "127.17.60.196" port: 41807 } attrs { promote: false } } peers { permanent_uuid: "b5e9178a7a83402e8c4a6943ab78589d" member_type: VOTER last_known_addr { host: "127.17.60.197" port: 36573 } attrs { promote: false } } }
I20250112 15:07:38.386072 20966 catalog_manager.cc:5526] T 5bcec60d3b974a9bb24aedfc738e1594 P 2537bbbd1a9947ae9facd87567e3ccb9 reported cstate change: config changed from index 30 to 31, b5e9178a7a83402e8c4a6943ab78589d (127.17.60.197) changed from NON_VOTER to VOTER. New cstate: current_term: 2 leader_uuid: "a2a5bba4ac1145d5bab94a69c9f4df40" committed_config { opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "2537bbbd1a9947ae9facd87567e3ccb9" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 33173 } attrs { replace: true } } peers { permanent_uuid: "3710b74ed2044ce5995533df32e67351" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 38589 } } peers { permanent_uuid: "a2a5bba4ac1145d5bab94a69c9f4df40" member_type: VOTER last_known_addr { host: "127.17.60.196" port: 41807 } attrs { promote: false } } peers { permanent_uuid: "b5e9178a7a83402e8c4a6943ab78589d" member_type: VOTER last_known_addr { host: "127.17.60.197" port: 36573 } attrs { promote: false } } }
I20250112 15:07:38.387008 21226 raft_consensus.cc:2949] T 5bcec60d3b974a9bb24aedfc738e1594 P 3710b74ed2044ce5995533df32e67351 [term 2 FOLLOWER]: Committing config change with OpId 2.31: config changed from index 30 to 31, b5e9178a7a83402e8c4a6943ab78589d (127.17.60.197) changed from NON_VOTER to VOTER. New config: { opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "2537bbbd1a9947ae9facd87567e3ccb9" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 33173 } attrs { replace: true } } peers { permanent_uuid: "3710b74ed2044ce5995533df32e67351" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 38589 } } peers { permanent_uuid: "a2a5bba4ac1145d5bab94a69c9f4df40" member_type: VOTER last_known_addr { host: "127.17.60.196" port: 41807 } attrs { promote: false } } peers { permanent_uuid: "b5e9178a7a83402e8c4a6943ab78589d" member_type: VOTER last_known_addr { host: "127.17.60.197" port: 36573 } attrs { promote: false } } }
I20250112 15:07:38.388236 21474 consensus_queue.cc:237] T 5bcec60d3b974a9bb24aedfc738e1594 P a2a5bba4ac1145d5bab94a69c9f4df40 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 31, Committed index: 31, Last appended: 2.31, Last appended by leader: 28, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "3710b74ed2044ce5995533df32e67351" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 38589 } } peers { permanent_uuid: "a2a5bba4ac1145d5bab94a69c9f4df40" member_type: VOTER last_known_addr { host: "127.17.60.196" port: 41807 } attrs { promote: false } } peers { permanent_uuid: "b5e9178a7a83402e8c4a6943ab78589d" member_type: VOTER last_known_addr { host: "127.17.60.197" port: 36573 } attrs { promote: false } }
I20250112 15:07:38.388598 21226 raft_consensus.cc:1270] T 5bcec60d3b974a9bb24aedfc738e1594 P 3710b74ed2044ce5995533df32e67351 [term 2 FOLLOWER]: Refusing update from remote peer a2a5bba4ac1145d5bab94a69c9f4df40: Log matching property violated. Preceding OpId in replica: term: 2 index: 31. Preceding OpId from leader: term: 2 index: 32. (index mismatch)
I20250112 15:07:38.388614 21549 raft_consensus.cc:1270] T 5bcec60d3b974a9bb24aedfc738e1594 P b5e9178a7a83402e8c4a6943ab78589d [term 2 FOLLOWER]: Refusing update from remote peer a2a5bba4ac1145d5bab94a69c9f4df40: Log matching property violated. Preceding OpId in replica: term: 2 index: 31. Preceding OpId from leader: term: 2 index: 32. (index mismatch)
I20250112 15:07:38.388773 21894 consensus_queue.cc:1035] T 5bcec60d3b974a9bb24aedfc738e1594 P a2a5bba4ac1145d5bab94a69c9f4df40 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b5e9178a7a83402e8c4a6943ab78589d" member_type: VOTER last_known_addr { host: "127.17.60.197" port: 36573 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 32, Last known committed idx: 31, Time since last communication: 0.000s
I20250112 15:07:38.388857 21893 consensus_queue.cc:1035] T 5bcec60d3b974a9bb24aedfc738e1594 P a2a5bba4ac1145d5bab94a69c9f4df40 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3710b74ed2044ce5995533df32e67351" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 38589 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 32, Last known committed idx: 31, Time since last communication: 0.000s
I20250112 15:07:38.389436 21866 raft_consensus.cc:2949] T 5bcec60d3b974a9bb24aedfc738e1594 P a2a5bba4ac1145d5bab94a69c9f4df40 [term 2 LEADER]: Committing config change with OpId 2.32: config changed from index 31 to 32, VOTER 2537bbbd1a9947ae9facd87567e3ccb9 (127.17.60.193) evicted. New config: { opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "3710b74ed2044ce5995533df32e67351" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 38589 } } peers { permanent_uuid: "a2a5bba4ac1145d5bab94a69c9f4df40" member_type: VOTER last_known_addr { host: "127.17.60.196" port: 41807 } attrs { promote: false } } peers { permanent_uuid: "b5e9178a7a83402e8c4a6943ab78589d" member_type: VOTER last_known_addr { host: "127.17.60.197" port: 36573 } attrs { promote: false } } }
I20250112 15:07:38.389514 21549 raft_consensus.cc:2949] T 5bcec60d3b974a9bb24aedfc738e1594 P b5e9178a7a83402e8c4a6943ab78589d [term 2 FOLLOWER]: Committing config change with OpId 2.32: config changed from index 31 to 32, VOTER 2537bbbd1a9947ae9facd87567e3ccb9 (127.17.60.193) evicted. New config: { opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "3710b74ed2044ce5995533df32e67351" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 38589 } } peers { permanent_uuid: "a2a5bba4ac1145d5bab94a69c9f4df40" member_type: VOTER last_known_addr { host: "127.17.60.196" port: 41807 } attrs { promote: false } } peers { permanent_uuid: "b5e9178a7a83402e8c4a6943ab78589d" member_type: VOTER last_known_addr { host: "127.17.60.197" port: 36573 } attrs { promote: false } } }
I20250112 15:07:38.390353 21019 catalog_manager.cc:5039] ChangeConfig:REMOVE_PEER RPC for tablet 5bcec60d3b974a9bb24aedfc738e1594 with cas_config_opid_index 31: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250112 15:07:38.390403 20964 catalog_manager.cc:5526] T 5bcec60d3b974a9bb24aedfc738e1594 P b5e9178a7a83402e8c4a6943ab78589d reported cstate change: config changed from index 31 to 32, VOTER 2537bbbd1a9947ae9facd87567e3ccb9 (127.17.60.193) evicted. New cstate: current_term: 2 leader_uuid: "a2a5bba4ac1145d5bab94a69c9f4df40" committed_config { opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "3710b74ed2044ce5995533df32e67351" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 38589 } } peers { permanent_uuid: "a2a5bba4ac1145d5bab94a69c9f4df40" member_type: VOTER last_known_addr { host: "127.17.60.196" port: 41807 } attrs { promote: false } } peers { permanent_uuid: "b5e9178a7a83402e8c4a6943ab78589d" member_type: VOTER last_known_addr { host: "127.17.60.197" port: 36573 } attrs { promote: false } } }
I20250112 15:07:38.390483 21226 raft_consensus.cc:2949] T 5bcec60d3b974a9bb24aedfc738e1594 P 3710b74ed2044ce5995533df32e67351 [term 2 FOLLOWER]: Committing config change with OpId 2.32: config changed from index 31 to 32, VOTER 2537bbbd1a9947ae9facd87567e3ccb9 (127.17.60.193) evicted. New config: { opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "3710b74ed2044ce5995533df32e67351" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 38589 } } peers { permanent_uuid: "a2a5bba4ac1145d5bab94a69c9f4df40" member_type: VOTER last_known_addr { host: "127.17.60.196" port: 41807 } attrs { promote: false } } peers { permanent_uuid: "b5e9178a7a83402e8c4a6943ab78589d" member_type: VOTER last_known_addr { host: "127.17.60.197" port: 36573 } attrs { promote: false } } }
I20250112 15:07:38.391008 21065 tablet_service.cc:1514] Processing DeleteTablet for tablet 5bcec60d3b974a9bb24aedfc738e1594 with delete_type TABLET_DATA_TOMBSTONED (TS 2537bbbd1a9947ae9facd87567e3ccb9 not found in new config with opid_index 32) from {username='slave'} at 127.0.0.1:37188
I20250112 15:07:38.391562 21898 tablet_replica.cc:331] T 5bcec60d3b974a9bb24aedfc738e1594 P 2537bbbd1a9947ae9facd87567e3ccb9: stopping tablet replica
I20250112 15:07:38.391634 21898 raft_consensus.cc:2238] T 5bcec60d3b974a9bb24aedfc738e1594 P 2537bbbd1a9947ae9facd87567e3ccb9 [term 2 FOLLOWER]: Raft consensus shutting down.
I20250112 15:07:38.391696 21898 raft_consensus.cc:2267] T 5bcec60d3b974a9bb24aedfc738e1594 P 2537bbbd1a9947ae9facd87567e3ccb9 [term 2 FOLLOWER]: Raft consensus is shut down!
I20250112 15:07:38.391996 21898 ts_tablet_manager.cc:1905] T 5bcec60d3b974a9bb24aedfc738e1594 P 2537bbbd1a9947ae9facd87567e3ccb9: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250112 15:07:38.393476 21898 ts_tablet_manager.cc:1918] T 5bcec60d3b974a9bb24aedfc738e1594 P 2537bbbd1a9947ae9facd87567e3ccb9: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.31
I20250112 15:07:38.393536 21898 log.cc:1198] T 5bcec60d3b974a9bb24aedfc738e1594 P 2537bbbd1a9947ae9facd87567e3ccb9: Deleting WAL directory at /tmp/dist-test-task95dLSD/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1736694351070047-17651-0/minicluster-data/ts-0-root/wals/5bcec60d3b974a9bb24aedfc738e1594
I20250112 15:07:38.393850 20956 catalog_manager.cc:4872] TS 2537bbbd1a9947ae9facd87567e3ccb9 (127.17.60.193:33173): tablet 5bcec60d3b974a9bb24aedfc738e1594 (table test-workload [id=cdaf6d3a65a54ea08e8e2c0ecdfc9de7]) successfully deleted
I20250112 15:07:38.531484 21856 raft_consensus.cc:988] T 02c33d457b6b47138dd712850819b619 P 2537bbbd1a9947ae9facd87567e3ccb9: : Instructing follower 3710b74ed2044ce5995533df32e67351 to start an election
I20250112 15:07:38.531574 21856 raft_consensus.cc:1076] T 02c33d457b6b47138dd712850819b619 P 2537bbbd1a9947ae9facd87567e3ccb9 [term 3 LEADER]: Signalling peer 3710b74ed2044ce5995533df32e67351 to start an election
I20250112 15:07:38.531720 21226 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "02c33d457b6b47138dd712850819b619"
dest_uuid: "3710b74ed2044ce5995533df32e67351"
from {username='slave'} at 127.0.0.1:33194
I20250112 15:07:38.531791 21226 raft_consensus.cc:491] T 02c33d457b6b47138dd712850819b619 P 3710b74ed2044ce5995533df32e67351 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20250112 15:07:38.531810 21226 raft_consensus.cc:3054] T 02c33d457b6b47138dd712850819b619 P 3710b74ed2044ce5995533df32e67351 [term 3 FOLLOWER]: Advancing to term 4
I20250112 15:07:38.532634 21226 raft_consensus.cc:513] T 02c33d457b6b47138dd712850819b619 P 3710b74ed2044ce5995533df32e67351 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3710b74ed2044ce5995533df32e67351" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 38589 } } peers { permanent_uuid: "2537bbbd1a9947ae9facd87567e3ccb9" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 33173 } } peers { permanent_uuid: "c561ec3374b94b95be0c155d80ff0bff" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 33627 } }
I20250112 15:07:38.532769 21226 leader_election.cc:290] T 02c33d457b6b47138dd712850819b619 P 3710b74ed2044ce5995533df32e67351 [CANDIDATE]: Term 4 election: Requested vote from peers 2537bbbd1a9947ae9facd87567e3ccb9 (127.17.60.193:33173), c561ec3374b94b95be0c155d80ff0bff (127.17.60.194:33627)
I20250112 15:07:38.532876 21069 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "02c33d457b6b47138dd712850819b619" candidate_uuid: "3710b74ed2044ce5995533df32e67351" candidate_term: 4 candidate_status { last_received { term: 3 index: 27 } } ignore_live_leader: true dest_uuid: "2537bbbd1a9947ae9facd87567e3ccb9"
I20250112 15:07:38.532969 21069 raft_consensus.cc:3049] T 02c33d457b6b47138dd712850819b619 P 2537bbbd1a9947ae9facd87567e3ccb9 [term 3 LEADER]: Stepping down as leader of term 3
I20250112 15:07:38.532954 21147 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "02c33d457b6b47138dd712850819b619" candidate_uuid: "3710b74ed2044ce5995533df32e67351" candidate_term: 4 candidate_status { last_received { term: 3 index: 27 } } ignore_live_leader: true dest_uuid: "c561ec3374b94b95be0c155d80ff0bff"
I20250112 15:07:38.533026 21069 raft_consensus.cc:738] T 02c33d457b6b47138dd712850819b619 P 2537bbbd1a9947ae9facd87567e3ccb9 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 2537bbbd1a9947ae9facd87567e3ccb9, State: Running, Role: LEADER
I20250112 15:07:38.533042 21147 raft_consensus.cc:3054] T 02c33d457b6b47138dd712850819b619 P c561ec3374b94b95be0c155d80ff0bff [term 3 FOLLOWER]: Advancing to term 4
I20250112 15:07:38.533083 21069 consensus_queue.cc:260] T 02c33d457b6b47138dd712850819b619 P 2537bbbd1a9947ae9facd87567e3ccb9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 27, Committed index: 27, Last appended: 3.27, Last appended by leader: 27, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3710b74ed2044ce5995533df32e67351" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 38589 } } peers { permanent_uuid: "2537bbbd1a9947ae9facd87567e3ccb9" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 33173 } } peers { permanent_uuid: "c561ec3374b94b95be0c155d80ff0bff" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 33627 } }
I20250112 15:07:38.533165 21069 raft_consensus.cc:3054] T 02c33d457b6b47138dd712850819b619 P 2537bbbd1a9947ae9facd87567e3ccb9 [term 3 FOLLOWER]: Advancing to term 4
I20250112 15:07:38.533622 21147 raft_consensus.cc:2463] T 02c33d457b6b47138dd712850819b619 P c561ec3374b94b95be0c155d80ff0bff [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3710b74ed2044ce5995533df32e67351 in term 4.
I20250112 15:07:38.533710 21069 raft_consensus.cc:2463] T 02c33d457b6b47138dd712850819b619 P 2537bbbd1a9947ae9facd87567e3ccb9 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3710b74ed2044ce5995533df32e67351 in term 4.
I20250112 15:07:38.533782 21191 leader_election.cc:304] T 02c33d457b6b47138dd712850819b619 P 3710b74ed2044ce5995533df32e67351 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3710b74ed2044ce5995533df32e67351, c561ec3374b94b95be0c155d80ff0bff; no voters:
I20250112 15:07:38.533854 21855 raft_consensus.cc:2798] T 02c33d457b6b47138dd712850819b619 P 3710b74ed2044ce5995533df32e67351 [term 4 FOLLOWER]: Leader election won for term 4
I20250112 15:07:38.533900 21855 raft_consensus.cc:695] T 02c33d457b6b47138dd712850819b619 P 3710b74ed2044ce5995533df32e67351 [term 4 LEADER]: Becoming Leader. State: Replica: 3710b74ed2044ce5995533df32e67351, State: Running, Role: LEADER
I20250112 15:07:38.533957 21855 consensus_queue.cc:237] T 02c33d457b6b47138dd712850819b619 P 3710b74ed2044ce5995533df32e67351 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 27, Committed index: 27, Last appended: 3.27, Last appended by leader: 27, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3710b74ed2044ce5995533df32e67351" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 38589 } } peers { permanent_uuid: "2537bbbd1a9947ae9facd87567e3ccb9" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 33173 } } peers { permanent_uuid: "c561ec3374b94b95be0c155d80ff0bff" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 33627 } }
I20250112 15:07:38.534431 20964 catalog_manager.cc:5526] T 02c33d457b6b47138dd712850819b619 P 3710b74ed2044ce5995533df32e67351 reported cstate change: term changed from 3 to 4, leader changed from 2537bbbd1a9947ae9facd87567e3ccb9 (127.17.60.193) to 3710b74ed2044ce5995533df32e67351 (127.17.60.195). New cstate: current_term: 4 leader_uuid: "3710b74ed2044ce5995533df32e67351" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3710b74ed2044ce5995533df32e67351" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 38589 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2537bbbd1a9947ae9facd87567e3ccb9" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 33173 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c561ec3374b94b95be0c155d80ff0bff" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 33627 } health_report { overall_health: UNKNOWN } } }
I20250112 15:07:38.540176 21879 raft_consensus.cc:988] T 69c23627ab854000b360030c384dc966 P c561ec3374b94b95be0c155d80ff0bff: : Instructing follower 3710b74ed2044ce5995533df32e67351 to start an election
I20250112 15:07:38.540237 21879 raft_consensus.cc:1076] T 69c23627ab854000b360030c384dc966 P c561ec3374b94b95be0c155d80ff0bff [term 1 LEADER]: Signalling peer 3710b74ed2044ce5995533df32e67351 to start an election
I20250112 15:07:38.540369 21226 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "69c23627ab854000b360030c384dc966"
dest_uuid: "3710b74ed2044ce5995533df32e67351"
from {username='slave'} at 127.0.0.1:33202
I20250112 15:07:38.540423 21226 raft_consensus.cc:491] T 69c23627ab854000b360030c384dc966 P 3710b74ed2044ce5995533df32e67351 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20250112 15:07:38.540450 21226 raft_consensus.cc:3054] T 69c23627ab854000b360030c384dc966 P 3710b74ed2044ce5995533df32e67351 [term 1 FOLLOWER]: Advancing to term 2
I20250112 15:07:38.541132 21226 raft_consensus.cc:513] T 69c23627ab854000b360030c384dc966 P 3710b74ed2044ce5995533df32e67351 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "c561ec3374b94b95be0c155d80ff0bff" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 33627 } } peers { permanent_uuid: "3710b74ed2044ce5995533df32e67351" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 38589 } attrs { replace: true } } peers { permanent_uuid: "b5e9178a7a83402e8c4a6943ab78589d" member_type: VOTER last_known_addr { host: "127.17.60.197" port: 36573 } attrs { promote: false } } peers { permanent_uuid: "a2a5bba4ac1145d5bab94a69c9f4df40" member_type: NON_VOTER last_known_addr { host: "127.17.60.196" port: 41807 } attrs { promote: true } }
I20250112 15:07:38.541275 21226 leader_election.cc:290] T 69c23627ab854000b360030c384dc966 P 3710b74ed2044ce5995533df32e67351 [CANDIDATE]: Term 2 election: Requested vote from peers c561ec3374b94b95be0c155d80ff0bff (127.17.60.194:33627), b5e9178a7a83402e8c4a6943ab78589d (127.17.60.197:36573)
I20250112 15:07:38.541651 21149 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "69c23627ab854000b360030c384dc966" candidate_uuid: "3710b74ed2044ce5995533df32e67351" candidate_term: 2 candidate_status { last_received { term: 1 index: 28 } } ignore_live_leader: true dest_uuid: "c561ec3374b94b95be0c155d80ff0bff"
I20250112 15:07:38.541713 21149 raft_consensus.cc:3049] T 69c23627ab854000b360030c384dc966 P c561ec3374b94b95be0c155d80ff0bff [term 1 LEADER]: Stepping down as leader of term 1
I20250112 15:07:38.541736 21149 raft_consensus.cc:738] T 69c23627ab854000b360030c384dc966 P c561ec3374b94b95be0c155d80ff0bff [term 1 LEADER]: Becoming Follower/Learner. State: Replica: c561ec3374b94b95be0c155d80ff0bff, State: Running, Role: LEADER
I20250112 15:07:38.541780 21149 consensus_queue.cc:260] T 69c23627ab854000b360030c384dc966 P c561ec3374b94b95be0c155d80ff0bff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 28, Committed index: 28, Last appended: 1.28, Last appended by leader: 28, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "c561ec3374b94b95be0c155d80ff0bff" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 33627 } } peers { permanent_uuid: "3710b74ed2044ce5995533df32e67351" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 38589 } attrs { replace: true } } peers { permanent_uuid: "b5e9178a7a83402e8c4a6943ab78589d" member_type: VOTER last_known_addr { host: "127.17.60.197" port: 36573 } attrs { promote: false } } peers { permanent_uuid: "a2a5bba4ac1145d5bab94a69c9f4df40" member_type: NON_VOTER last_known_addr { host: "127.17.60.196" port: 41807 } attrs { promote: true } }
I20250112 15:07:38.541865 21149 raft_consensus.cc:3054] T 69c23627ab854000b360030c384dc966 P c561ec3374b94b95be0c155d80ff0bff [term 1 FOLLOWER]: Advancing to term 2
I20250112 15:07:38.542515 21149 raft_consensus.cc:2463] T 69c23627ab854000b360030c384dc966 P c561ec3374b94b95be0c155d80ff0bff [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3710b74ed2044ce5995533df32e67351 in term 2.
I20250112 15:07:38.542755 21191 leader_election.cc:304] T 69c23627ab854000b360030c384dc966 P 3710b74ed2044ce5995533df32e67351 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3710b74ed2044ce5995533df32e67351, c561ec3374b94b95be0c155d80ff0bff; no voters:
I20250112 15:07:38.542840 21855 raft_consensus.cc:2798] T 69c23627ab854000b360030c384dc966 P 3710b74ed2044ce5995533df32e67351 [term 2 FOLLOWER]: Leader election won for term 2
I20250112 15:07:38.542901 21855 raft_consensus.cc:695] T 69c23627ab854000b360030c384dc966 P 3710b74ed2044ce5995533df32e67351 [term 2 LEADER]: Becoming Leader. State: Replica: 3710b74ed2044ce5995533df32e67351, State: Running, Role: LEADER
I20250112 15:07:38.542960 21855 consensus_queue.cc:237] T 69c23627ab854000b360030c384dc966 P 3710b74ed2044ce5995533df32e67351 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 28, Committed index: 28, Last appended: 1.28, Last appended by leader: 28, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "c561ec3374b94b95be0c155d80ff0bff" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 33627 } } peers { permanent_uuid: "3710b74ed2044ce5995533df32e67351" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 38589 } attrs { replace: true } } peers { permanent_uuid: "b5e9178a7a83402e8c4a6943ab78589d" member_type: VOTER last_known_addr { host: "127.17.60.197" port: 36573 } attrs { promote: false } } peers { permanent_uuid: "a2a5bba4ac1145d5bab94a69c9f4df40" member_type: NON_VOTER last_known_addr { host: "127.17.60.196" port: 41807 } attrs { promote: true } }
I20250112 15:07:38.543506 20964 catalog_manager.cc:5526] T 69c23627ab854000b360030c384dc966 P 3710b74ed2044ce5995533df32e67351 reported cstate change: term changed from 1 to 2, leader changed from c561ec3374b94b95be0c155d80ff0bff (127.17.60.194) to 3710b74ed2044ce5995533df32e67351 (127.17.60.195). New cstate: current_term: 2 leader_uuid: "3710b74ed2044ce5995533df32e67351" committed_config { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "c561ec3374b94b95be0c155d80ff0bff" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 33627 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3710b74ed2044ce5995533df32e67351" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 38589 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b5e9178a7a83402e8c4a6943ab78589d" member_type: VOTER last_known_addr { host: "127.17.60.197" port: 36573 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a2a5bba4ac1145d5bab94a69c9f4df40" member_type: NON_VOTER last_known_addr { host: "127.17.60.196" port: 41807 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20250112 15:07:38.544220 21549 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "69c23627ab854000b360030c384dc966" candidate_uuid: "3710b74ed2044ce5995533df32e67351" candidate_term: 2 candidate_status { last_received { term: 1 index: 28 } } ignore_live_leader: true dest_uuid: "b5e9178a7a83402e8c4a6943ab78589d"
I20250112 15:07:38.544438 21549 raft_consensus.cc:3054] T 69c23627ab854000b360030c384dc966 P b5e9178a7a83402e8c4a6943ab78589d [term 1 FOLLOWER]: Advancing to term 2
I20250112 15:07:38.544675 21226 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "69c23627ab854000b360030c384dc966"
dest_uuid: "3710b74ed2044ce5995533df32e67351"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:33192
I20250112 15:07:38.544750 21226 raft_consensus.cc:604] T 69c23627ab854000b360030c384dc966 P 3710b74ed2044ce5995533df32e67351 [term 2 LEADER]: Received request to transfer leadership
I20250112 15:07:38.545162 21549 raft_consensus.cc:2463] T 69c23627ab854000b360030c384dc966 P b5e9178a7a83402e8c4a6943ab78589d [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3710b74ed2044ce5995533df32e67351 in term 2.
I20250112 15:07:38.545341 21226 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "69c23627ab854000b360030c384dc966"
dest_uuid: "3710b74ed2044ce5995533df32e67351"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:33192
I20250112 15:07:38.545405 21226 raft_consensus.cc:604] T 69c23627ab854000b360030c384dc966 P 3710b74ed2044ce5995533df32e67351 [term 2 LEADER]: Received request to transfer leadership
W20250112 15:07:38.545529 20957 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: leadership transfer for 69c23627ab854000b360030c384dc966 already in progress [suppressed 3 similar messages]
I20250112 15:07:38.546268 21226 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "69c23627ab854000b360030c384dc966"
dest_uuid: "3710b74ed2044ce5995533df32e67351"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:33192
I20250112 15:07:38.546337 21226 raft_consensus.cc:604] T 69c23627ab854000b360030c384dc966 P 3710b74ed2044ce5995533df32e67351 [term 2 LEADER]: Received request to transfer leadership
W20250112 15:07:38.546554 21021 auto_rebalancer.cc:663] Could not move replica: Remote error: Service unavailable: leadership transfer for 69c23627ab854000b360030c384dc966 already in progress
W20250112 15:07:38.546600 21021 auto_rebalancer.cc:263] scheduled replica move failed to complete: Remote error: Service unavailable: leadership transfer for 69c23627ab854000b360030c384dc966 already in progress
I20250112 15:07:38.865994 21147 raft_consensus.cc:1270] T bfaeae584ec04772b2d6fcd088022117 P c561ec3374b94b95be0c155d80ff0bff [term 2 FOLLOWER]: Refusing update from remote peer 2537bbbd1a9947ae9facd87567e3ccb9: Log matching property violated. Preceding OpId in replica: term: 1 index: 25. Preceding OpId from leader: term: 2 index: 26. (index mismatch)
I20250112 15:07:38.866276 21856 consensus_queue.cc:1035] T bfaeae584ec04772b2d6fcd088022117 P 2537bbbd1a9947ae9facd87567e3ccb9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c561ec3374b94b95be0c155d80ff0bff" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 33627 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 26, Last known committed idx: 25, Time since last communication: 0.000s
I20250112 15:07:38.867449 21226 raft_consensus.cc:1270] T bfaeae584ec04772b2d6fcd088022117 P 3710b74ed2044ce5995533df32e67351 [term 2 FOLLOWER]: Refusing update from remote peer 2537bbbd1a9947ae9facd87567e3ccb9: Log matching property violated. Preceding OpId in replica: term: 1 index: 25. Preceding OpId from leader: term: 2 index: 26. (index mismatch)
I20250112 15:07:38.867667 21856 consensus_queue.cc:1035] T bfaeae584ec04772b2d6fcd088022117 P 2537bbbd1a9947ae9facd87567e3ccb9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3710b74ed2044ce5995533df32e67351" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 38589 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 26, Last known committed idx: 25, Time since last communication: 0.000s
I20250112 15:07:39.025188 21474 raft_consensus.cc:3054] T 69c23627ab854000b360030c384dc966 P a2a5bba4ac1145d5bab94a69c9f4df40 [term 1 LEARNER]: Advancing to term 2
I20250112 15:07:39.025887 21474 raft_consensus.cc:1270] T 69c23627ab854000b360030c384dc966 P a2a5bba4ac1145d5bab94a69c9f4df40 [term 2 LEARNER]: Refusing update from remote peer 3710b74ed2044ce5995533df32e67351: Log matching property violated. Preceding OpId in replica: term: 1 index: 28. Preceding OpId from leader: term: 2 index: 29. (index mismatch)
I20250112 15:07:39.026118 21855 consensus_queue.cc:1035] T 69c23627ab854000b360030c384dc966 P 3710b74ed2044ce5995533df32e67351 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2a5bba4ac1145d5bab94a69c9f4df40" member_type: NON_VOTER last_known_addr { host: "127.17.60.196" port: 41807 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.000s
I20250112 15:07:39.026829 21855 raft_consensus.cc:1059] T 69c23627ab854000b360030c384dc966 P 3710b74ed2044ce5995533df32e67351: attempting to promote NON_VOTER a2a5bba4ac1145d5bab94a69c9f4df40 to VOTER
W20250112 15:07:39.026881 21855 raft_consensus.cc:1062] T 69c23627ab854000b360030c384dc966 P 3710b74ed2044ce5995533df32e67351: Unable to promote non-voter a2a5bba4ac1145d5bab94a69c9f4df40: Service unavailable: leader transfer in progress
I20250112 15:07:39.100819 21147 raft_consensus.cc:1270] T 02c33d457b6b47138dd712850819b619 P c561ec3374b94b95be0c155d80ff0bff [term 4 FOLLOWER]: Refusing update from remote peer 3710b74ed2044ce5995533df32e67351: Log matching property violated. Preceding OpId in replica: term: 3 index: 27. Preceding OpId from leader: term: 4 index: 28. (index mismatch)
I20250112 15:07:39.101030 21855 consensus_queue.cc:1035] T 02c33d457b6b47138dd712850819b619 P 3710b74ed2044ce5995533df32e67351 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c561ec3374b94b95be0c155d80ff0bff" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 33627 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 28, Last known committed idx: 27, Time since last communication: 0.000s
I20250112 15:07:39.101969 21069 raft_consensus.cc:1270] T 02c33d457b6b47138dd712850819b619 P 2537bbbd1a9947ae9facd87567e3ccb9 [term 4 FOLLOWER]: Refusing update from remote peer 3710b74ed2044ce5995533df32e67351: Log matching property violated. Preceding OpId in replica: term: 3 index: 27. Preceding OpId from leader: term: 4 index: 28. (index mismatch)
I20250112 15:07:39.102227 21855 consensus_queue.cc:1035] T 02c33d457b6b47138dd712850819b619 P 3710b74ed2044ce5995533df32e67351 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2537bbbd1a9947ae9facd87567e3ccb9" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 33173 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 28, Last known committed idx: 27, Time since last communication: 0.000s
I20250112 15:07:39.118028 21147 raft_consensus.cc:1270] T 69c23627ab854000b360030c384dc966 P c561ec3374b94b95be0c155d80ff0bff [term 2 FOLLOWER]: Refusing update from remote peer 3710b74ed2044ce5995533df32e67351: Log matching property violated. Preceding OpId in replica: term: 1 index: 28. Preceding OpId from leader: term: 2 index: 29. (index mismatch)
I20250112 15:07:39.118232 21855 consensus_queue.cc:1035] T 69c23627ab854000b360030c384dc966 P 3710b74ed2044ce5995533df32e67351 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c561ec3374b94b95be0c155d80ff0bff" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 33627 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.000s
I20250112 15:07:39.118878 21884 raft_consensus.cc:988] T 69c23627ab854000b360030c384dc966 P 3710b74ed2044ce5995533df32e67351: : Instructing follower c561ec3374b94b95be0c155d80ff0bff to start an election
I20250112 15:07:39.118963 21884 raft_consensus.cc:1076] T 69c23627ab854000b360030c384dc966 P 3710b74ed2044ce5995533df32e67351 [term 2 LEADER]: Signalling peer c561ec3374b94b95be0c155d80ff0bff to start an election
I20250112 15:07:39.119110 21149 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "69c23627ab854000b360030c384dc966"
dest_uuid: "c561ec3374b94b95be0c155d80ff0bff"
from {username='slave'} at 127.0.0.1:48092
I20250112 15:07:39.119184 21149 raft_consensus.cc:491] T 69c23627ab854000b360030c384dc966 P c561ec3374b94b95be0c155d80ff0bff [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20250112 15:07:39.119213 21149 raft_consensus.cc:3054] T 69c23627ab854000b360030c384dc966 P c561ec3374b94b95be0c155d80ff0bff [term 2 FOLLOWER]: Advancing to term 3
I20250112 15:07:39.119325 21549 raft_consensus.cc:1270] T 69c23627ab854000b360030c384dc966 P b5e9178a7a83402e8c4a6943ab78589d [term 2 FOLLOWER]: Refusing update from remote peer 3710b74ed2044ce5995533df32e67351: Log matching property violated. Preceding OpId in replica: term: 1 index: 28. Preceding OpId from leader: term: 2 index: 29. (index mismatch)
I20250112 15:07:39.119590 21884 consensus_queue.cc:1035] T 69c23627ab854000b360030c384dc966 P 3710b74ed2044ce5995533df32e67351 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b5e9178a7a83402e8c4a6943ab78589d" member_type: VOTER last_known_addr { host: "127.17.60.197" port: 36573 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.000s
I20250112 15:07:39.120011 21884 raft_consensus.cc:1059] T 69c23627ab854000b360030c384dc966 P 3710b74ed2044ce5995533df32e67351: attempting to promote NON_VOTER a2a5bba4ac1145d5bab94a69c9f4df40 to VOTER
W20250112 15:07:39.120059 21884 raft_consensus.cc:1062] T 69c23627ab854000b360030c384dc966 P 3710b74ed2044ce5995533df32e67351: Unable to promote non-voter a2a5bba4ac1145d5bab94a69c9f4df40: Service unavailable: leader transfer in progress
I20250112 15:07:39.120087 21149 raft_consensus.cc:513] T 69c23627ab854000b360030c384dc966 P c561ec3374b94b95be0c155d80ff0bff [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "c561ec3374b94b95be0c155d80ff0bff" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 33627 } } peers { permanent_uuid: "3710b74ed2044ce5995533df32e67351" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 38589 } attrs { replace: true } } peers { permanent_uuid: "b5e9178a7a83402e8c4a6943ab78589d" member_type: VOTER last_known_addr { host: "127.17.60.197" port: 36573 } attrs { promote: false } } peers { permanent_uuid: "a2a5bba4ac1145d5bab94a69c9f4df40" member_type: NON_VOTER last_known_addr { host: "127.17.60.196" port: 41807 } attrs { promote: true } }
I20250112 15:07:39.120260 21149 leader_election.cc:290] T 69c23627ab854000b360030c384dc966 P c561ec3374b94b95be0c155d80ff0bff [CANDIDATE]: Term 3 election: Requested vote from peers 3710b74ed2044ce5995533df32e67351 (127.17.60.195:38589), b5e9178a7a83402e8c4a6943ab78589d (127.17.60.197:36573)
I20250112 15:07:39.120359 21226 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "69c23627ab854000b360030c384dc966" candidate_uuid: "c561ec3374b94b95be0c155d80ff0bff" candidate_term: 3 candidate_status { last_received { term: 2 index: 29 } } ignore_live_leader: true dest_uuid: "3710b74ed2044ce5995533df32e67351"
I20250112 15:07:39.120419 21226 raft_consensus.cc:3049] T 69c23627ab854000b360030c384dc966 P 3710b74ed2044ce5995533df32e67351 [term 2 LEADER]: Stepping down as leader of term 2
I20250112 15:07:39.120443 21226 raft_consensus.cc:738] T 69c23627ab854000b360030c384dc966 P 3710b74ed2044ce5995533df32e67351 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 3710b74ed2044ce5995533df32e67351, State: Running, Role: LEADER
I20250112 15:07:39.120496 21226 consensus_queue.cc:260] T 69c23627ab854000b360030c384dc966 P 3710b74ed2044ce5995533df32e67351 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 29, Committed index: 29, Last appended: 2.29, Last appended by leader: 29, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "c561ec3374b94b95be0c155d80ff0bff" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 33627 } } peers { permanent_uuid: "3710b74ed2044ce5995533df32e67351" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 38589 } attrs { replace: true } } peers { permanent_uuid: "b5e9178a7a83402e8c4a6943ab78589d" member_type: VOTER last_known_addr { host: "127.17.60.197" port: 36573 } attrs { promote: false } } peers { permanent_uuid: "a2a5bba4ac1145d5bab94a69c9f4df40" member_type: NON_VOTER last_known_addr { host: "127.17.60.196" port: 41807 } attrs { promote: true } }
I20250112 15:07:39.120584 21226 raft_consensus.cc:3054] T 69c23627ab854000b360030c384dc966 P 3710b74ed2044ce5995533df32e67351 [term 2 FOLLOWER]: Advancing to term 3
I20250112 15:07:39.120625 21548 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "69c23627ab854000b360030c384dc966" candidate_uuid: "c561ec3374b94b95be0c155d80ff0bff" candidate_term: 3 candidate_status { last_received { term: 2 index: 29 } } ignore_live_leader: true dest_uuid: "b5e9178a7a83402e8c4a6943ab78589d"
I20250112 15:07:39.121142 21226 raft_consensus.cc:2463] T 69c23627ab854000b360030c384dc966 P 3710b74ed2044ce5995533df32e67351 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c561ec3374b94b95be0c155d80ff0bff in term 3.
I20250112 15:07:39.121277 21115 leader_election.cc:304] T 69c23627ab854000b360030c384dc966 P c561ec3374b94b95be0c155d80ff0bff [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3710b74ed2044ce5995533df32e67351, c561ec3374b94b95be0c155d80ff0bff; no voters: b5e9178a7a83402e8c4a6943ab78589d
I20250112 15:07:39.121346 21883 raft_consensus.cc:2798] T 69c23627ab854000b360030c384dc966 P c561ec3374b94b95be0c155d80ff0bff [term 3 FOLLOWER]: Leader election won for term 3
I20250112 15:07:39.121454 21883 raft_consensus.cc:695] T 69c23627ab854000b360030c384dc966 P c561ec3374b94b95be0c155d80ff0bff [term 3 LEADER]: Becoming Leader. State: Replica: c561ec3374b94b95be0c155d80ff0bff, State: Running, Role: LEADER
I20250112 15:07:39.121502 21883 consensus_queue.cc:237] T 69c23627ab854000b360030c384dc966 P c561ec3374b94b95be0c155d80ff0bff [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 29, Committed index: 29, Last appended: 2.29, Last appended by leader: 29, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "c561ec3374b94b95be0c155d80ff0bff" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 33627 } } peers { permanent_uuid: "3710b74ed2044ce5995533df32e67351" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 38589 } attrs { replace: true } } peers { permanent_uuid: "b5e9178a7a83402e8c4a6943ab78589d" member_type: VOTER last_known_addr { host: "127.17.60.197" port: 36573 } attrs { promote: false } } peers { permanent_uuid: "a2a5bba4ac1145d5bab94a69c9f4df40" member_type: NON_VOTER last_known_addr { host: "127.17.60.196" port: 41807 } attrs { promote: true } }
I20250112 15:07:39.121848 20964 catalog_manager.cc:5526] T 69c23627ab854000b360030c384dc966 P c561ec3374b94b95be0c155d80ff0bff reported cstate change: term changed from 2 to 3, leader changed from 3710b74ed2044ce5995533df32e67351 (127.17.60.195) to c561ec3374b94b95be0c155d80ff0bff (127.17.60.194). New cstate: current_term: 3 leader_uuid: "c561ec3374b94b95be0c155d80ff0bff" committed_config { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "c561ec3374b94b95be0c155d80ff0bff" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 33627 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3710b74ed2044ce5995533df32e67351" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 38589 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b5e9178a7a83402e8c4a6943ab78589d" member_type: VOTER last_known_addr { host: "127.17.60.197" port: 36573 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a2a5bba4ac1145d5bab94a69c9f4df40" member_type: NON_VOTER last_known_addr { host: "127.17.60.196" port: 41807 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20250112 15:07:39.547617 21549 consensus_queue.cc:237] T 57e47b07fdcf434e96ca5dd0354e93b2 P b5e9178a7a83402e8c4a6943ab78589d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 30, Committed index: 30, Last appended: 3.30, Last appended by leader: 29, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "2537bbbd1a9947ae9facd87567e3ccb9" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 33173 } } peers { permanent_uuid: "c561ec3374b94b95be0c155d80ff0bff" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 33627 } attrs { replace: true } } peers { permanent_uuid: "b5e9178a7a83402e8c4a6943ab78589d" member_type: VOTER last_known_addr { host: "127.17.60.197" port: 36573 } attrs { promote: false } } peers { permanent_uuid: "a2a5bba4ac1145d5bab94a69c9f4df40" member_type: NON_VOTER last_known_addr { host: "127.17.60.196" port: 41807 } attrs { promote: true } }
I20250112 15:07:39.548244 21069 raft_consensus.cc:1270] T 57e47b07fdcf434e96ca5dd0354e93b2 P 2537bbbd1a9947ae9facd87567e3ccb9 [term 3 FOLLOWER]: Refusing update from remote peer b5e9178a7a83402e8c4a6943ab78589d: Log matching property violated. Preceding OpId in replica: term: 3 index: 30. Preceding OpId from leader: term: 3 index: 31. (index mismatch)
I20250112 15:07:39.548302 21149 raft_consensus.cc:1270] T 57e47b07fdcf434e96ca5dd0354e93b2 P c561ec3374b94b95be0c155d80ff0bff [term 3 FOLLOWER]: Refusing update from remote peer b5e9178a7a83402e8c4a6943ab78589d: Log matching property violated. Preceding OpId in replica: term: 3 index: 30. Preceding OpId from leader: term: 3 index: 31. (index mismatch)
W20250112 15:07:39.548388 21514 consensus_peers.cc:487] T 57e47b07fdcf434e96ca5dd0354e93b2 P b5e9178a7a83402e8c4a6943ab78589d -> Peer a2a5bba4ac1145d5bab94a69c9f4df40 (127.17.60.196:41807): Couldn't send request to peer a2a5bba4ac1145d5bab94a69c9f4df40. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 57e47b07fdcf434e96ca5dd0354e93b2. This is attempt 1: this message will repeat every 5th retry.
I20250112 15:07:39.548452 21925 consensus_queue.cc:1035] T 57e47b07fdcf434e96ca5dd0354e93b2 P b5e9178a7a83402e8c4a6943ab78589d [LEADER]: Connected to new peer: Peer: permanent_uuid: "2537bbbd1a9947ae9facd87567e3ccb9" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 33173 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 31, Last known committed idx: 30, Time since last communication: 0.000s
I20250112 15:07:39.549474 21844 consensus_queue.cc:1035] T 57e47b07fdcf434e96ca5dd0354e93b2 P b5e9178a7a83402e8c4a6943ab78589d [LEADER]: Connected to new peer: Peer: permanent_uuid: "c561ec3374b94b95be0c155d80ff0bff" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 33627 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 31, Last known committed idx: 30, Time since last communication: 0.000s
I20250112 15:07:39.550238 21918 raft_consensus.cc:2949] T 57e47b07fdcf434e96ca5dd0354e93b2 P b5e9178a7a83402e8c4a6943ab78589d [term 3 LEADER]: Committing config change with OpId 3.31: config changed from index 29 to 31, NON_VOTER a2a5bba4ac1145d5bab94a69c9f4df40 (127.17.60.196) added. New config: { opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "2537bbbd1a9947ae9facd87567e3ccb9" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 33173 } } peers { permanent_uuid: "c561ec3374b94b95be0c155d80ff0bff" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 33627 } attrs { replace: true } } peers { permanent_uuid: "b5e9178a7a83402e8c4a6943ab78589d" member_type: VOTER last_known_addr { host: "127.17.60.197" port: 36573 } attrs { promote: false } } peers { permanent_uuid: "a2a5bba4ac1145d5bab94a69c9f4df40" member_type: NON_VOTER last_known_addr { host: "127.17.60.196" port: 41807 } attrs { promote: true } } }
I20250112 15:07:39.550369 21149 raft_consensus.cc:2949] T 57e47b07fdcf434e96ca5dd0354e93b2 P c561ec3374b94b95be0c155d80ff0bff [term 3 FOLLOWER]: Committing config change with OpId 3.31: config changed from index 29 to 31, NON_VOTER a2a5bba4ac1145d5bab94a69c9f4df40 (127.17.60.196) added. New config: { opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "2537bbbd1a9947ae9facd87567e3ccb9" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 33173 } } peers { permanent_uuid: "c561ec3374b94b95be0c155d80ff0bff" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 33627 } attrs { replace: true } } peers { permanent_uuid: "b5e9178a7a83402e8c4a6943ab78589d" member_type: VOTER last_known_addr { host: "127.17.60.197" port: 36573 } attrs { promote: false } } peers { permanent_uuid: "a2a5bba4ac1145d5bab94a69c9f4df40" member_type: NON_VOTER last_known_addr { host: "127.17.60.196" port: 41807 } attrs { promote: true } } }
I20250112 15:07:39.551244 21069 raft_consensus.cc:2949] T 57e47b07fdcf434e96ca5dd0354e93b2 P 2537bbbd1a9947ae9facd87567e3ccb9 [term 3 FOLLOWER]: Committing config change with OpId 3.31: config changed from index 29 to 31, NON_VOTER a2a5bba4ac1145d5bab94a69c9f4df40 (127.17.60.196) added. New config: { opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "2537bbbd1a9947ae9facd87567e3ccb9" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 33173 } } peers { permanent_uuid: "c561ec3374b94b95be0c155d80ff0bff" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 33627 } attrs { replace: true } } peers { permanent_uuid: "b5e9178a7a83402e8c4a6943ab78589d" member_type: VOTER last_known_addr { host: "127.17.60.197" port: 36573 } attrs { promote: false } } peers { permanent_uuid: "a2a5bba4ac1145d5bab94a69c9f4df40" member_type: NON_VOTER last_known_addr { host: "127.17.60.196" port: 41807 } attrs { promote: true } } }
I20250112 15:07:39.551426 20964 catalog_manager.cc:5526] T 57e47b07fdcf434e96ca5dd0354e93b2 P c561ec3374b94b95be0c155d80ff0bff reported cstate change: config changed from index 29 to 31, NON_VOTER a2a5bba4ac1145d5bab94a69c9f4df40 (127.17.60.196) added. New cstate: current_term: 3 leader_uuid: "b5e9178a7a83402e8c4a6943ab78589d" committed_config { opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "2537bbbd1a9947ae9facd87567e3ccb9" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 33173 } } peers { permanent_uuid: "c561ec3374b94b95be0c155d80ff0bff" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 33627 } attrs { replace: true } } peers { permanent_uuid: "b5e9178a7a83402e8c4a6943ab78589d" member_type: VOTER last_known_addr { host: "127.17.60.197" port: 36573 } attrs { promote: false } } peers { permanent_uuid: "a2a5bba4ac1145d5bab94a69c9f4df40" member_type: NON_VOTER last_known_addr { host: "127.17.60.196" port: 41807 } attrs { promote: true } } }
I20250112 15:07:39.551726 21068 consensus_queue.cc:237] T bfaeae584ec04772b2d6fcd088022117 P 2537bbbd1a9947ae9facd87567e3ccb9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 26, Committed index: 26, Last appended: 2.26, Last appended by leader: 25, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "2537bbbd1a9947ae9facd87567e3ccb9" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 33173 } } peers { permanent_uuid: "c561ec3374b94b95be0c155d80ff0bff" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 33627 } } peers { permanent_uuid: "3710b74ed2044ce5995533df32e67351" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 38589 } attrs { replace: true } } peers { permanent_uuid: "25286c485fa04c86a5d07c8f88643a47" member_type: NON_VOTER last_known_addr { host: "127.17.60.198" port: 37089 } attrs { promote: true } }
I20250112 15:07:39.552121 21149 raft_consensus.cc:1270] T bfaeae584ec04772b2d6fcd088022117 P c561ec3374b94b95be0c155d80ff0bff [term 2 FOLLOWER]: Refusing update from remote peer 2537bbbd1a9947ae9facd87567e3ccb9: Log matching property violated. Preceding OpId in replica: term: 2 index: 26. Preceding OpId from leader: term: 2 index: 27. (index mismatch)
I20250112 15:07:39.552198 21226 raft_consensus.cc:1270] T bfaeae584ec04772b2d6fcd088022117 P 3710b74ed2044ce5995533df32e67351 [term 2 FOLLOWER]: Refusing update from remote peer 2537bbbd1a9947ae9facd87567e3ccb9: Log matching property violated. Preceding OpId in replica: term: 2 index: 26. Preceding OpId from leader: term: 2 index: 27. (index mismatch)
I20250112 15:07:39.552595 21923 consensus_queue.cc:1035] T bfaeae584ec04772b2d6fcd088022117 P 2537bbbd1a9947ae9facd87567e3ccb9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c561ec3374b94b95be0c155d80ff0bff" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 33627 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 27, Last known committed idx: 26, Time since last communication: 0.000s
I20250112 15:07:39.552918 21928 consensus_queue.cc:1035] T bfaeae584ec04772b2d6fcd088022117 P 2537bbbd1a9947ae9facd87567e3ccb9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3710b74ed2044ce5995533df32e67351" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 38589 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 27, Last known committed idx: 26, Time since last communication: 0.000s
I20250112 15:07:39.553092 21856 raft_consensus.cc:2949] T bfaeae584ec04772b2d6fcd088022117 P 2537bbbd1a9947ae9facd87567e3ccb9 [term 2 LEADER]: Committing config change with OpId 2.27: config changed from index -1 to 27, NON_VOTER 25286c485fa04c86a5d07c8f88643a47 (127.17.60.198) added. New config: { opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "2537bbbd1a9947ae9facd87567e3ccb9" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 33173 } } peers { permanent_uuid: "c561ec3374b94b95be0c155d80ff0bff" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 33627 } } peers { permanent_uuid: "3710b74ed2044ce5995533df32e67351" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 38589 } attrs { replace: true } } peers { permanent_uuid: "25286c485fa04c86a5d07c8f88643a47" member_type: NON_VOTER last_known_addr { host: "127.17.60.198" port: 37089 } attrs { promote: true } } }
I20250112 15:07:39.553537 21226 raft_consensus.cc:2949] T bfaeae584ec04772b2d6fcd088022117 P 3710b74ed2044ce5995533df32e67351 [term 2 FOLLOWER]: Committing config change with OpId 2.27: config changed from index -1 to 27, NON_VOTER 25286c485fa04c86a5d07c8f88643a47 (127.17.60.198) added. New config: { opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "2537bbbd1a9947ae9facd87567e3ccb9" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 33173 } } peers { permanent_uuid: "c561ec3374b94b95be0c155d80ff0bff" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 33627 } } peers { permanent_uuid: "3710b74ed2044ce5995533df32e67351" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 38589 } attrs { replace: true } } peers { permanent_uuid: "25286c485fa04c86a5d07c8f88643a47" member_type: NON_VOTER last_known_addr { host: "127.17.60.198" port: 37089 } attrs { promote: true } } }
I20250112 15:07:39.554020 21149 raft_consensus.cc:2949] T bfaeae584ec04772b2d6fcd088022117 P c561ec3374b94b95be0c155d80ff0bff [term 2 FOLLOWER]: Committing config change with OpId 2.27: config changed from index -1 to 27, NON_VOTER 25286c485fa04c86a5d07c8f88643a47 (127.17.60.198) added. New config: { opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "2537bbbd1a9947ae9facd87567e3ccb9" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 33173 } } peers { permanent_uuid: "c561ec3374b94b95be0c155d80ff0bff" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 33627 } } peers { permanent_uuid: "3710b74ed2044ce5995533df32e67351" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 38589 } attrs { replace: true } } peers { permanent_uuid: "25286c485fa04c86a5d07c8f88643a47" member_type: NON_VOTER last_known_addr { host: "127.17.60.198" port: 37089 } attrs { promote: true } } }
I20250112 15:07:39.554104 20964 catalog_manager.cc:5526] T bfaeae584ec04772b2d6fcd088022117 P 2537bbbd1a9947ae9facd87567e3ccb9 reported cstate change: config changed from index -1 to 27, NON_VOTER 25286c485fa04c86a5d07c8f88643a47 (127.17.60.198) added. New cstate: current_term: 2 leader_uuid: "2537bbbd1a9947ae9facd87567e3ccb9" committed_config { opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "2537bbbd1a9947ae9facd87567e3ccb9" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 33173 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c561ec3374b94b95be0c155d80ff0bff" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 33627 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3710b74ed2044ce5995533df32e67351" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 38589 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "25286c485fa04c86a5d07c8f88643a47" member_type: NON_VOTER last_known_addr { host: "127.17.60.198" port: 37089 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20250112 15:07:39.555261 21149 consensus_queue.cc:237] T 8c7dbaf285a04da9b4eec92a29ab7d6c P c561ec3374b94b95be0c155d80ff0bff [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 27, Committed index: 27, Last appended: 3.27, Last appended by leader: 26, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "2537bbbd1a9947ae9facd87567e3ccb9" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 33173 } attrs { replace: true } } peers { permanent_uuid: "c561ec3374b94b95be0c155d80ff0bff" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 33627 } } peers { permanent_uuid: "3710b74ed2044ce5995533df32e67351" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 38589 } } peers { permanent_uuid: "a2a5bba4ac1145d5bab94a69c9f4df40" member_type: NON_VOTER last_known_addr { host: "127.17.60.196" port: 41807 } attrs { promote: true } }
I20250112 15:07:39.555763 21068 raft_consensus.cc:1270] T 8c7dbaf285a04da9b4eec92a29ab7d6c P 2537bbbd1a9947ae9facd87567e3ccb9 [term 3 FOLLOWER]: Refusing update from remote peer c561ec3374b94b95be0c155d80ff0bff: Log matching property violated. Preceding OpId in replica: term: 3 index: 27. Preceding OpId from leader: term: 3 index: 28. (index mismatch)
I20250112 15:07:39.555788 21226 raft_consensus.cc:1270] T 8c7dbaf285a04da9b4eec92a29ab7d6c P 3710b74ed2044ce5995533df32e67351 [term 3 FOLLOWER]: Refusing update from remote peer c561ec3374b94b95be0c155d80ff0bff: Log matching property violated. Preceding OpId in replica: term: 3 index: 27. Preceding OpId from leader: term: 3 index: 28. (index mismatch)
W20250112 15:07:39.555821 21116 consensus_peers.cc:487] T 8c7dbaf285a04da9b4eec92a29ab7d6c P c561ec3374b94b95be0c155d80ff0bff -> Peer a2a5bba4ac1145d5bab94a69c9f4df40 (127.17.60.196:41807): Couldn't send request to peer a2a5bba4ac1145d5bab94a69c9f4df40. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 8c7dbaf285a04da9b4eec92a29ab7d6c. This is attempt 1: this message will repeat every 5th retry.
W20250112 15:07:39.555943 21038 consensus_peers.cc:487] T bfaeae584ec04772b2d6fcd088022117 P 2537bbbd1a9947ae9facd87567e3ccb9 -> Peer 25286c485fa04c86a5d07c8f88643a47 (127.17.60.198:37089): Couldn't send request to peer 25286c485fa04c86a5d07c8f88643a47. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: bfaeae584ec04772b2d6fcd088022117. This is attempt 1: this message will repeat every 5th retry.
I20250112 15:07:39.555984 21932 consensus_queue.cc:1035] T 8c7dbaf285a04da9b4eec92a29ab7d6c P c561ec3374b94b95be0c155d80ff0bff [LEADER]: Connected to new peer: Peer: permanent_uuid: "2537bbbd1a9947ae9facd87567e3ccb9" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 33173 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 28, Last known committed idx: 27, Time since last communication: 0.000s
I20250112 15:07:39.556124 21919 consensus_queue.cc:1035] T 8c7dbaf285a04da9b4eec92a29ab7d6c P c561ec3374b94b95be0c155d80ff0bff [LEADER]: Connected to new peer: Peer: permanent_uuid: "3710b74ed2044ce5995533df32e67351" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 38589 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 28, Last known committed idx: 27, Time since last communication: 0.000s
I20250112 15:07:39.556761 21932 raft_consensus.cc:2949] T 8c7dbaf285a04da9b4eec92a29ab7d6c P c561ec3374b94b95be0c155d80ff0bff [term 3 LEADER]: Committing config change with OpId 3.28: config changed from index -1 to 28, NON_VOTER a2a5bba4ac1145d5bab94a69c9f4df40 (127.17.60.196) added. New config: { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "2537bbbd1a9947ae9facd87567e3ccb9" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 33173 } attrs { replace: true } } peers { permanent_uuid: "c561ec3374b94b95be0c155d80ff0bff" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 33627 } } peers { permanent_uuid: "3710b74ed2044ce5995533df32e67351" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 38589 } } peers { permanent_uuid: "a2a5bba4ac1145d5bab94a69c9f4df40" member_type: NON_VOTER last_known_addr { host: "127.17.60.196" port: 41807 } attrs { promote: true } } }
I20250112 15:07:39.556828 21068 raft_consensus.cc:2949] T 8c7dbaf285a04da9b4eec92a29ab7d6c P 2537bbbd1a9947ae9facd87567e3ccb9 [term 3 FOLLOWER]: Committing config change with OpId 3.28: config changed from index -1 to 28, NON_VOTER a2a5bba4ac1145d5bab94a69c9f4df40 (127.17.60.196) added. New config: { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "2537bbbd1a9947ae9facd87567e3ccb9" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 33173 } attrs { replace: true } } peers { permanent_uuid: "c561ec3374b94b95be0c155d80ff0bff" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 33627 } } peers { permanent_uuid: "3710b74ed2044ce5995533df32e67351" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 38589 } } peers { permanent_uuid: "a2a5bba4ac1145d5bab94a69c9f4df40" member_type: NON_VOTER last_known_addr { host: "127.17.60.196" port: 41807 } attrs { promote: true } } }
I20250112 15:07:39.556882 21226 raft_consensus.cc:2949] T 8c7dbaf285a04da9b4eec92a29ab7d6c P 3710b74ed2044ce5995533df32e67351 [term 3 FOLLOWER]: Committing config change with OpId 3.28: config changed from index -1 to 28, NON_VOTER a2a5bba4ac1145d5bab94a69c9f4df40 (127.17.60.196) added. New config: { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "2537bbbd1a9947ae9facd87567e3ccb9" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 33173 } attrs { replace: true } } peers { permanent_uuid: "c561ec3374b94b95be0c155d80ff0bff" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 33627 } } peers { permanent_uuid: "3710b74ed2044ce5995533df32e67351" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 38589 } } peers { permanent_uuid: "a2a5bba4ac1145d5bab94a69c9f4df40" member_type: NON_VOTER last_known_addr { host: "127.17.60.196" port: 41807 } attrs { promote: true } } }
I20250112 15:07:39.557667 20966 catalog_manager.cc:5526] T 8c7dbaf285a04da9b4eec92a29ab7d6c P 3710b74ed2044ce5995533df32e67351 reported cstate change: config changed from index -1 to 28, NON_VOTER a2a5bba4ac1145d5bab94a69c9f4df40 (127.17.60.196) added. New cstate: current_term: 3 leader_uuid: "c561ec3374b94b95be0c155d80ff0bff" committed_config { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "2537bbbd1a9947ae9facd87567e3ccb9" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 33173 } attrs { replace: true } } peers { permanent_uuid: "c561ec3374b94b95be0c155d80ff0bff" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 33627 } } peers { permanent_uuid: "3710b74ed2044ce5995533df32e67351" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 38589 } } peers { permanent_uuid: "a2a5bba4ac1145d5bab94a69c9f4df40" member_type: NON_VOTER last_known_addr { host: "127.17.60.196" port: 41807 } attrs { promote: true } } }
I20250112 15:07:39.557829 21068 consensus_queue.cc:237] T 9d4bbba799324d7fbbac082520c65686 P 2537bbbd1a9947ae9facd87567e3ccb9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 25, Committed index: 25, Last appended: 2.25, Last appended by leader: 24, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 26 OBSOLETE_local: false peers { permanent_uuid: "2537bbbd1a9947ae9facd87567e3ccb9" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 33173 } } peers { permanent_uuid: "c561ec3374b94b95be0c155d80ff0bff" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 33627 } } peers { permanent_uuid: "3710b74ed2044ce5995533df32e67351" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 38589 } attrs { replace: true } } peers { permanent_uuid: "b5e9178a7a83402e8c4a6943ab78589d" member_type: NON_VOTER last_known_addr { host: "127.17.60.197" port: 36573 } attrs { promote: true } }
I20250112 15:07:39.558389 21149 raft_consensus.cc:1270] T 9d4bbba799324d7fbbac082520c65686 P c561ec3374b94b95be0c155d80ff0bff [term 2 FOLLOWER]: Refusing update from remote peer 2537bbbd1a9947ae9facd87567e3ccb9: Log matching property violated. Preceding OpId in replica: term: 2 index: 25. Preceding OpId from leader: term: 2 index: 26. (index mismatch)
I20250112 15:07:39.558404 21226 raft_consensus.cc:1270] T 9d4bbba799324d7fbbac082520c65686 P 3710b74ed2044ce5995533df32e67351 [term 2 FOLLOWER]: Refusing update from remote peer 2537bbbd1a9947ae9facd87567e3ccb9: Log matching property violated. Preceding OpId in replica: term: 2 index: 25. Preceding OpId from leader: term: 2 index: 26. (index mismatch)
I20250112 15:07:39.558557 21923 consensus_queue.cc:1035] T 9d4bbba799324d7fbbac082520c65686 P 2537bbbd1a9947ae9facd87567e3ccb9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c561ec3374b94b95be0c155d80ff0bff" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 33627 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 26, Last known committed idx: 25, Time since last communication: 0.000s
W20250112 15:07:39.558633 21038 consensus_peers.cc:487] T 9d4bbba799324d7fbbac082520c65686 P 2537bbbd1a9947ae9facd87567e3ccb9 -> Peer b5e9178a7a83402e8c4a6943ab78589d (127.17.60.197:36573): Couldn't send request to peer b5e9178a7a83402e8c4a6943ab78589d. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 9d4bbba799324d7fbbac082520c65686. This is attempt 1: this message will repeat every 5th retry.
I20250112 15:07:39.558641 21928 consensus_queue.cc:1035] T 9d4bbba799324d7fbbac082520c65686 P 2537bbbd1a9947ae9facd87567e3ccb9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3710b74ed2044ce5995533df32e67351" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 38589 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 26, Last known committed idx: 25, Time since last communication: 0.000s
I20250112 15:07:39.559314 21928 raft_consensus.cc:2949] T 9d4bbba799324d7fbbac082520c65686 P 2537bbbd1a9947ae9facd87567e3ccb9 [term 2 LEADER]: Committing config change with OpId 2.26: config changed from index -1 to 26, NON_VOTER b5e9178a7a83402e8c4a6943ab78589d (127.17.60.197) added. New config: { opid_index: 26 OBSOLETE_local: false peers { permanent_uuid: "2537bbbd1a9947ae9facd87567e3ccb9" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 33173 } } peers { permanent_uuid: "c561ec3374b94b95be0c155d80ff0bff" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 33627 } } peers { permanent_uuid: "3710b74ed2044ce5995533df32e67351" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 38589 } attrs { replace: true } } peers { permanent_uuid: "b5e9178a7a83402e8c4a6943ab78589d" member_type: NON_VOTER last_known_addr { host: "127.17.60.197" port: 36573 } attrs { promote: true } } }
I20250112 15:07:39.559466 21149 raft_consensus.cc:2949] T 9d4bbba799324d7fbbac082520c65686 P c561ec3374b94b95be0c155d80ff0bff [term 2 FOLLOWER]: Committing config change with OpId 2.26: config changed from index -1 to 26, NON_VOTER b5e9178a7a83402e8c4a6943ab78589d (127.17.60.197) added. New config: { opid_index: 26 OBSOLETE_local: false peers { permanent_uuid: "2537bbbd1a9947ae9facd87567e3ccb9" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 33173 } } peers { permanent_uuid: "c561ec3374b94b95be0c155d80ff0bff" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 33627 } } peers { permanent_uuid: "3710b74ed2044ce5995533df32e67351" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 38589 } attrs { replace: true } } peers { permanent_uuid: "b5e9178a7a83402e8c4a6943ab78589d" member_type: NON_VOTER last_known_addr { host: "127.17.60.197" port: 36573 } attrs { promote: true } } }
I20250112 15:07:39.559501 21226 raft_consensus.cc:2949] T 9d4bbba799324d7fbbac082520c65686 P 3710b74ed2044ce5995533df32e67351 [term 2 FOLLOWER]: Committing config change with OpId 2.26: config changed from index -1 to 26, NON_VOTER b5e9178a7a83402e8c4a6943ab78589d (127.17.60.197) added. New config: { opid_index: 26 OBSOLETE_local: false peers { permanent_uuid: "2537bbbd1a9947ae9facd87567e3ccb9" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 33173 } } peers { permanent_uuid: "c561ec3374b94b95be0c155d80ff0bff" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 33627 } } peers { permanent_uuid: "3710b74ed2044ce5995533df32e67351" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 38589 } attrs { replace: true } } peers { permanent_uuid: "b5e9178a7a83402e8c4a6943ab78589d" member_type: NON_VOTER last_known_addr { host: "127.17.60.197" port: 36573 } attrs { promote: true } } }
I20250112 15:07:39.560434 20970 catalog_manager.cc:5526] T 9d4bbba799324d7fbbac082520c65686 P 3710b74ed2044ce5995533df32e67351 reported cstate change: config changed from index -1 to 26, NON_VOTER b5e9178a7a83402e8c4a6943ab78589d (127.17.60.197) added. New cstate: current_term: 2 leader_uuid: "2537bbbd1a9947ae9facd87567e3ccb9" committed_config { opid_index: 26 OBSOLETE_local: false peers { permanent_uuid: "2537bbbd1a9947ae9facd87567e3ccb9" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 33173 } } peers { permanent_uuid: "c561ec3374b94b95be0c155d80ff0bff" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 33627 } } peers { permanent_uuid: "3710b74ed2044ce5995533df32e67351" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 38589 } attrs { replace: true } } peers { permanent_uuid: "b5e9178a7a83402e8c4a6943ab78589d" member_type: NON_VOTER last_known_addr { host: "127.17.60.197" port: 36573 } attrs { promote: true } } }
I20250112 15:07:39.560506 21068 consensus_queue.cc:237] T 16f16fcd5cf347e5853189dc4da55c7f P 2537bbbd1a9947ae9facd87567e3ccb9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 26, Committed index: 26, Last appended: 2.26, Last appended by leader: 25, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "2537bbbd1a9947ae9facd87567e3ccb9" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 33173 } } peers { permanent_uuid: "c561ec3374b94b95be0c155d80ff0bff" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 33627 } attrs { replace: true } } peers { permanent_uuid: "3710b74ed2044ce5995533df32e67351" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 38589 } } peers { permanent_uuid: "25286c485fa04c86a5d07c8f88643a47" member_type: NON_VOTER last_known_addr { host: "127.17.60.198" port: 37089 } attrs { promote: true } }
I20250112 15:07:39.560941 21149 raft_consensus.cc:1270] T 16f16fcd5cf347e5853189dc4da55c7f P c561ec3374b94b95be0c155d80ff0bff [term 2 FOLLOWER]: Refusing update from remote peer 2537bbbd1a9947ae9facd87567e3ccb9: Log matching property violated. Preceding OpId in replica: term: 2 index: 26. Preceding OpId from leader: term: 2 index: 27. (index mismatch)
W20250112 15:07:39.561115 21038 consensus_peers.cc:487] T 16f16fcd5cf347e5853189dc4da55c7f P 2537bbbd1a9947ae9facd87567e3ccb9 -> Peer 25286c485fa04c86a5d07c8f88643a47 (127.17.60.198:37089): Couldn't send request to peer 25286c485fa04c86a5d07c8f88643a47. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 16f16fcd5cf347e5853189dc4da55c7f. This is attempt 1: this message will repeat every 5th retry.
I20250112 15:07:39.561133 21923 consensus_queue.cc:1035] T 16f16fcd5cf347e5853189dc4da55c7f P 2537bbbd1a9947ae9facd87567e3ccb9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c561ec3374b94b95be0c155d80ff0bff" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 33627 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 27, Last known committed idx: 26, Time since last communication: 0.000s
I20250112 15:07:39.561305 21226 raft_consensus.cc:1270] T 16f16fcd5cf347e5853189dc4da55c7f P 3710b74ed2044ce5995533df32e67351 [term 2 FOLLOWER]: Refusing update from remote peer 2537bbbd1a9947ae9facd87567e3ccb9: Log matching property violated. Preceding OpId in replica: term: 2 index: 26. Preceding OpId from leader: term: 2 index: 27. (index mismatch)
I20250112 15:07:39.561446 21923 consensus_queue.cc:1035] T 16f16fcd5cf347e5853189dc4da55c7f P 2537bbbd1a9947ae9facd87567e3ccb9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3710b74ed2044ce5995533df32e67351" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 38589 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 27, Last known committed idx: 26, Time since last communication: 0.000s
I20250112 15:07:39.561702 21928 raft_consensus.cc:2949] T 16f16fcd5cf347e5853189dc4da55c7f P 2537bbbd1a9947ae9facd87567e3ccb9 [term 2 LEADER]: Committing config change with OpId 2.27: config changed from index -1 to 27, NON_VOTER 25286c485fa04c86a5d07c8f88643a47 (127.17.60.198) added. New config: { opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "2537bbbd1a9947ae9facd87567e3ccb9" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 33173 } } peers { permanent_uuid: "c561ec3374b94b95be0c155d80ff0bff" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 33627 } attrs { replace: true } } peers { permanent_uuid: "3710b74ed2044ce5995533df32e67351" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 38589 } } peers { permanent_uuid: "25286c485fa04c86a5d07c8f88643a47" member_type: NON_VOTER last_known_addr { host: "127.17.60.198" port: 37089 } attrs { promote: true } } }
I20250112 15:07:39.561808 21149 raft_consensus.cc:2949] T 16f16fcd5cf347e5853189dc4da55c7f P c561ec3374b94b95be0c155d80ff0bff [term 2 FOLLOWER]: Committing config change with OpId 2.27: config changed from index -1 to 27, NON_VOTER 25286c485fa04c86a5d07c8f88643a47 (127.17.60.198) added. New config: { opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "2537bbbd1a9947ae9facd87567e3ccb9" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 33173 } } peers { permanent_uuid: "c561ec3374b94b95be0c155d80ff0bff" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 33627 } attrs { replace: true } } peers { permanent_uuid: "3710b74ed2044ce5995533df32e67351" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 38589 } } peers { permanent_uuid: "25286c485fa04c86a5d07c8f88643a47" member_type: NON_VOTER last_known_addr { host: "127.17.60.198" port: 37089 } attrs { promote: true } } }
I20250112 15:07:39.562180 21226 raft_consensus.cc:2949] T 16f16fcd5cf347e5853189dc4da55c7f P 3710b74ed2044ce5995533df32e67351 [term 2 FOLLOWER]: Committing config change with OpId 2.27: config changed from index -1 to 27, NON_VOTER 25286c485fa04c86a5d07c8f88643a47 (127.17.60.198) added. New config: { opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "2537bbbd1a9947ae9facd87567e3ccb9" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 33173 } } peers { permanent_uuid: "c561ec3374b94b95be0c155d80ff0bff" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 33627 } attrs { replace: true } } peers { permanent_uuid: "3710b74ed2044ce5995533df32e67351" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 38589 } } peers { permanent_uuid: "25286c485fa04c86a5d07c8f88643a47" member_type: NON_VOTER last_known_addr { host: "127.17.60.198" port: 37089 } attrs { promote: true } } }
I20250112 15:07:39.563002 20966 catalog_manager.cc:5526] T 16f16fcd5cf347e5853189dc4da55c7f P c561ec3374b94b95be0c155d80ff0bff reported cstate change: config changed from index -1 to 27, NON_VOTER 25286c485fa04c86a5d07c8f88643a47 (127.17.60.198) added. New cstate: current_term: 2 leader_uuid: "2537bbbd1a9947ae9facd87567e3ccb9" committed_config { opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "2537bbbd1a9947ae9facd87567e3ccb9" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 33173 } } peers { permanent_uuid: "c561ec3374b94b95be0c155d80ff0bff" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 33627 } attrs { replace: true } } peers { permanent_uuid: "3710b74ed2044ce5995533df32e67351" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 38589 } } peers { permanent_uuid: "25286c485fa04c86a5d07c8f88643a47" member_type: NON_VOTER last_known_addr { host: "127.17.60.198" port: 37089 } attrs { promote: true } } }
I20250112 15:07:39.565668 21226 consensus_queue.cc:237] T 9601bcda24ed42e7b93ab9740f1e271e P 3710b74ed2044ce5995533df32e67351 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 28, Committed index: 28, Last appended: 2.28, Last appended by leader: 27, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "2537bbbd1a9947ae9facd87567e3ccb9" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 33173 } attrs { replace: true } } peers { permanent_uuid: "3710b74ed2044ce5995533df32e67351" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 38589 } } peers { permanent_uuid: "a2a5bba4ac1145d5bab94a69c9f4df40" member_type: VOTER last_known_addr { host: "127.17.60.196" port: 41807 } attrs { promote: false } } peers { permanent_uuid: "25286c485fa04c86a5d07c8f88643a47" member_type: NON_VOTER last_known_addr { host: "127.17.60.198" port: 37089 } attrs { promote: true } }
I20250112 15:07:39.566115 21474 raft_consensus.cc:1270] T 9601bcda24ed42e7b93ab9740f1e271e P a2a5bba4ac1145d5bab94a69c9f4df40 [term 2 FOLLOWER]: Refusing update from remote peer 3710b74ed2044ce5995533df32e67351: Log matching property violated. Preceding OpId in replica: term: 2 index: 28. Preceding OpId from leader: term: 2 index: 29. (index mismatch)
I20250112 15:07:39.566331 21068 raft_consensus.cc:1270] T 9601bcda24ed42e7b93ab9740f1e271e P 2537bbbd1a9947ae9facd87567e3ccb9 [term 2 FOLLOWER]: Refusing update from remote peer 3710b74ed2044ce5995533df32e67351: Log matching property violated. Preceding OpId in replica: term: 2 index: 28. Preceding OpId from leader: term: 2 index: 29. (index mismatch)
I20250112 15:07:39.566387 21884 consensus_queue.cc:1035] T 9601bcda24ed42e7b93ab9740f1e271e P 3710b74ed2044ce5995533df32e67351 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2a5bba4ac1145d5bab94a69c9f4df40" member_type: VOTER last_known_addr { host: "127.17.60.196" port: 41807 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.000s
I20250112 15:07:39.566500 21855 consensus_queue.cc:1035] T 9601bcda24ed42e7b93ab9740f1e271e P 3710b74ed2044ce5995533df32e67351 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2537bbbd1a9947ae9facd87567e3ccb9" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 33173 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.000s
I20250112 15:07:39.567108 21884 raft_consensus.cc:2949] T 9601bcda24ed42e7b93ab9740f1e271e P 3710b74ed2044ce5995533df32e67351 [term 2 LEADER]: Committing config change with OpId 2.29: config changed from index 27 to 29, NON_VOTER 25286c485fa04c86a5d07c8f88643a47 (127.17.60.198) added. New config: { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "2537bbbd1a9947ae9facd87567e3ccb9" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 33173 } attrs { replace: true } } peers { permanent_uuid: "3710b74ed2044ce5995533df32e67351" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 38589 } } peers { permanent_uuid: "a2a5bba4ac1145d5bab94a69c9f4df40" member_type: VOTER last_known_addr { host: "127.17.60.196" port: 41807 } attrs { promote: false } } peers { permanent_uuid: "25286c485fa04c86a5d07c8f88643a47" member_type: NON_VOTER last_known_addr { host: "127.17.60.198" port: 37089 } attrs { promote: true } } }
I20250112 15:07:39.567209 21474 raft_consensus.cc:2949] T 9601bcda24ed42e7b93ab9740f1e271e P a2a5bba4ac1145d5bab94a69c9f4df40 [term 2 FOLLOWER]: Committing config change with OpId 2.29: config changed from index 27 to 29, NON_VOTER 25286c485fa04c86a5d07c8f88643a47 (127.17.60.198) added. New config: { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "2537bbbd1a9947ae9facd87567e3ccb9" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 33173 } attrs { replace: true } } peers { permanent_uuid: "3710b74ed2044ce5995533df32e67351" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 38589 } } peers { permanent_uuid: "a2a5bba4ac1145d5bab94a69c9f4df40" member_type: VOTER last_known_addr { host: "127.17.60.196" port: 41807 } attrs { promote: false } } peers { permanent_uuid: "25286c485fa04c86a5d07c8f88643a47" member_type: NON_VOTER last_known_addr { host: "127.17.60.198" port: 37089 } attrs { promote: true } } }
I20250112 15:07:39.568055 20964 catalog_manager.cc:5526] T 9601bcda24ed42e7b93ab9740f1e271e P a2a5bba4ac1145d5bab94a69c9f4df40 reported cstate change: config changed from index 27 to 29, NON_VOTER 25286c485fa04c86a5d07c8f88643a47 (127.17.60.198) added. New cstate: current_term: 2 leader_uuid: "3710b74ed2044ce5995533df32e67351" committed_config { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "2537bbbd1a9947ae9facd87567e3ccb9" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 33173 } attrs { replace: true } } peers { permanent_uuid: "3710b74ed2044ce5995533df32e67351" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 38589 } } peers { permanent_uuid: "a2a5bba4ac1145d5bab94a69c9f4df40" member_type: VOTER last_known_addr { host: "127.17.60.196" port: 41807 } attrs { promote: false } } peers { permanent_uuid: "25286c485fa04c86a5d07c8f88643a47" member_type: NON_VOTER last_known_addr { host: "127.17.60.198" port: 37089 } attrs { promote: true } } }
I20250112 15:07:39.568239 21068 raft_consensus.cc:2949] T 9601bcda24ed42e7b93ab9740f1e271e P 2537bbbd1a9947ae9facd87567e3ccb9 [term 2 FOLLOWER]: Committing config change with OpId 2.29: config changed from index 27 to 29, NON_VOTER 25286c485fa04c86a5d07c8f88643a47 (127.17.60.198) added. New config: { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "2537bbbd1a9947ae9facd87567e3ccb9" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 33173 } attrs { replace: true } } peers { permanent_uuid: "3710b74ed2044ce5995533df32e67351" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 38589 } } peers { permanent_uuid: "a2a5bba4ac1145d5bab94a69c9f4df40" member_type: VOTER last_known_addr { host: "127.17.60.196" port: 41807 } attrs { promote: false } } peers { permanent_uuid: "25286c485fa04c86a5d07c8f88643a47" member_type: NON_VOTER last_known_addr { host: "127.17.60.198" port: 37089 } attrs { promote: true } } }
W20250112 15:07:39.569558 21189 consensus_peers.cc:487] T 9601bcda24ed42e7b93ab9740f1e271e P 3710b74ed2044ce5995533df32e67351 -> Peer 25286c485fa04c86a5d07c8f88643a47 (127.17.60.198:37089): Couldn't send request to peer 25286c485fa04c86a5d07c8f88643a47. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 9601bcda24ed42e7b93ab9740f1e271e. This is attempt 1: this message will repeat every 5th retry.
I20250112 15:07:39.618763 21549 raft_consensus.cc:3054] T 69c23627ab854000b360030c384dc966 P b5e9178a7a83402e8c4a6943ab78589d [term 2 FOLLOWER]: Advancing to term 3
I20250112 15:07:39.619791 21549 raft_consensus.cc:1270] T 69c23627ab854000b360030c384dc966 P b5e9178a7a83402e8c4a6943ab78589d [term 3 FOLLOWER]: Refusing update from remote peer c561ec3374b94b95be0c155d80ff0bff: Log matching property violated. Preceding OpId in replica: term: 2 index: 29. Preceding OpId from leader: term: 3 index: 30. (index mismatch)
I20250112 15:07:39.620065 21935 consensus_queue.cc:1035] T 69c23627ab854000b360030c384dc966 P c561ec3374b94b95be0c155d80ff0bff [LEADER]: Connected to new peer: Peer: permanent_uuid: "b5e9178a7a83402e8c4a6943ab78589d" member_type: VOTER last_known_addr { host: "127.17.60.197" port: 36573 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 30, Last known committed idx: 29, Time since last communication: 0.000s
I20250112 15:07:39.621220 21226 raft_consensus.cc:1270] T 69c23627ab854000b360030c384dc966 P 3710b74ed2044ce5995533df32e67351 [term 3 FOLLOWER]: Refusing update from remote peer c561ec3374b94b95be0c155d80ff0bff: Log matching property violated. Preceding OpId in replica: term: 2 index: 29. Preceding OpId from leader: term: 3 index: 30. (index mismatch)
I20250112 15:07:39.621318 21474 raft_consensus.cc:3054] T 69c23627ab854000b360030c384dc966 P a2a5bba4ac1145d5bab94a69c9f4df40 [term 2 LEARNER]: Advancing to term 3
I20250112 15:07:39.621379 21919 consensus_queue.cc:1035] T 69c23627ab854000b360030c384dc966 P c561ec3374b94b95be0c155d80ff0bff [LEADER]: Connected to new peer: Peer: permanent_uuid: "3710b74ed2044ce5995533df32e67351" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 38589 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 30, Last known committed idx: 29, Time since last communication: 0.000s
I20250112 15:07:39.622037 21474 raft_consensus.cc:1270] T 69c23627ab854000b360030c384dc966 P a2a5bba4ac1145d5bab94a69c9f4df40 [term 3 LEARNER]: Refusing update from remote peer c561ec3374b94b95be0c155d80ff0bff: Log matching property violated. Preceding OpId in replica: term: 2 index: 29. Preceding OpId from leader: term: 3 index: 30. (index mismatch)
I20250112 15:07:39.622210 21932 consensus_queue.cc:1035] T 69c23627ab854000b360030c384dc966 P c561ec3374b94b95be0c155d80ff0bff [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2a5bba4ac1145d5bab94a69c9f4df40" member_type: NON_VOTER last_known_addr { host: "127.17.60.196" port: 41807 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 30, Last known committed idx: 29, Time since last communication: 0.000s
I20250112 15:07:39.622851 21919 raft_consensus.cc:1059] T 69c23627ab854000b360030c384dc966 P c561ec3374b94b95be0c155d80ff0bff: attempting to promote NON_VOTER a2a5bba4ac1145d5bab94a69c9f4df40 to VOTER
I20250112 15:07:39.623001 21919 consensus_queue.cc:237] T 69c23627ab854000b360030c384dc966 P c561ec3374b94b95be0c155d80ff0bff [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 30, Committed index: 30, Last appended: 3.30, Last appended by leader: 29, Current term: 3, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "c561ec3374b94b95be0c155d80ff0bff" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 33627 } } peers { permanent_uuid: "3710b74ed2044ce5995533df32e67351" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 38589 } attrs { replace: true } } peers { permanent_uuid: "b5e9178a7a83402e8c4a6943ab78589d" member_type: VOTER last_known_addr { host: "127.17.60.197" port: 36573 } attrs { promote: false } } peers { permanent_uuid: "a2a5bba4ac1145d5bab94a69c9f4df40" member_type: VOTER last_known_addr { host: "127.17.60.196" port: 41807 } attrs { promote: false } }
I20250112 15:07:39.623343 21474 raft_consensus.cc:1270] T 69c23627ab854000b360030c384dc966 P a2a5bba4ac1145d5bab94a69c9f4df40 [term 3 LEARNER]: Refusing update from remote peer c561ec3374b94b95be0c155d80ff0bff: Log matching property violated. Preceding OpId in replica: term: 3 index: 30. Preceding OpId from leader: term: 3 index: 31. (index mismatch)
I20250112 15:07:39.623456 21226 raft_consensus.cc:1270] T 69c23627ab854000b360030c384dc966 P 3710b74ed2044ce5995533df32e67351 [term 3 FOLLOWER]: Refusing update from remote peer c561ec3374b94b95be0c155d80ff0bff: Log matching property violated. Preceding OpId in replica: term: 3 index: 30. Preceding OpId from leader: term: 3 index: 31. (index mismatch)
I20250112 15:07:39.623492 21932 consensus_queue.cc:1035] T 69c23627ab854000b360030c384dc966 P c561ec3374b94b95be0c155d80ff0bff [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2a5bba4ac1145d5bab94a69c9f4df40" member_type: VOTER last_known_addr { host: "127.17.60.196" port: 41807 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 31, Last known committed idx: 30, Time since last communication: 0.000s
I20250112 15:07:39.623492 21548 raft_consensus.cc:1270] T 69c23627ab854000b360030c384dc966 P b5e9178a7a83402e8c4a6943ab78589d [term 3 FOLLOWER]: Refusing update from remote peer c561ec3374b94b95be0c155d80ff0bff: Log matching property violated. Preceding OpId in replica: term: 3 index: 30. Preceding OpId from leader: term: 3 index: 31. (index mismatch)
I20250112 15:07:39.623594 21883 consensus_queue.cc:1035] T 69c23627ab854000b360030c384dc966 P c561ec3374b94b95be0c155d80ff0bff [LEADER]: Connected to new peer: Peer: permanent_uuid: "3710b74ed2044ce5995533df32e67351" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 38589 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 31, Last known committed idx: 30, Time since last communication: 0.000s
I20250112 15:07:39.623685 21883 consensus_queue.cc:1035] T 69c23627ab854000b360030c384dc966 P c561ec3374b94b95be0c155d80ff0bff [LEADER]: Connected to new peer: Peer: permanent_uuid: "b5e9178a7a83402e8c4a6943ab78589d" member_type: VOTER last_known_addr { host: "127.17.60.197" port: 36573 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 31, Last known committed idx: 30, Time since last communication: 0.000s
I20250112 15:07:39.624209 21935 raft_consensus.cc:2949] T 69c23627ab854000b360030c384dc966 P c561ec3374b94b95be0c155d80ff0bff [term 3 LEADER]: Committing config change with OpId 3.31: config changed from index 28 to 31, a2a5bba4ac1145d5bab94a69c9f4df40 (127.17.60.196) changed from NON_VOTER to VOTER. New config: { opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "c561ec3374b94b95be0c155d80ff0bff" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 33627 } } peers { permanent_uuid: "3710b74ed2044ce5995533df32e67351" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 38589 } attrs { replace: true } } peers { permanent_uuid: "b5e9178a7a83402e8c4a6943ab78589d" member_type: VOTER last_known_addr { host: "127.17.60.197" port: 36573 } attrs { promote: false } } peers { permanent_uuid: "a2a5bba4ac1145d5bab94a69c9f4df40" member_type: VOTER last_known_addr { host: "127.17.60.196" port: 41807 } attrs { promote: false } } }
I20250112 15:07:39.624364 21222 raft_consensus.cc:2949] T 69c23627ab854000b360030c384dc966 P 3710b74ed2044ce5995533df32e67351 [term 3 FOLLOWER]: Committing config change with OpId 3.31: config changed from index 28 to 31, a2a5bba4ac1145d5bab94a69c9f4df40 (127.17.60.196) changed from NON_VOTER to VOTER. New config: { opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "c561ec3374b94b95be0c155d80ff0bff" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 33627 } } peers { permanent_uuid: "3710b74ed2044ce5995533df32e67351" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 38589 } attrs { replace: true } } peers { permanent_uuid: "b5e9178a7a83402e8c4a6943ab78589d" member_type: VOTER last_known_addr { host: "127.17.60.197" port: 36573 } attrs { promote: false } } peers { permanent_uuid: "a2a5bba4ac1145d5bab94a69c9f4df40" member_type: VOTER last_known_addr { host: "127.17.60.196" port: 41807 } attrs { promote: false } } }
I20250112 15:07:39.624368 21474 raft_consensus.cc:2949] T 69c23627ab854000b360030c384dc966 P a2a5bba4ac1145d5bab94a69c9f4df40 [term 3 FOLLOWER]: Committing config change with OpId 3.31: config changed from index 28 to 31, a2a5bba4ac1145d5bab94a69c9f4df40 (127.17.60.196) changed from NON_VOTER to VOTER. New config: { opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "c561ec3374b94b95be0c155d80ff0bff" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 33627 } } peers { permanent_uuid: "3710b74ed2044ce5995533df32e67351" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 38589 } attrs { replace: true } } peers { permanent_uuid: "b5e9178a7a83402e8c4a6943ab78589d" member_type: VOTER last_known_addr { host: "127.17.60.197" port: 36573 } attrs { promote: false } } peers { permanent_uuid: "a2a5bba4ac1145d5bab94a69c9f4df40" member_type: VOTER last_known_addr { host: "127.17.60.196" port: 41807 } attrs { promote: false } } }
I20250112 15:07:39.625319 21548 raft_consensus.cc:2949] T 69c23627ab854000b360030c384dc966 P b5e9178a7a83402e8c4a6943ab78589d [term 3 FOLLOWER]: Committing config change with OpId 3.31: config changed from index 28 to 31, a2a5bba4ac1145d5bab94a69c9f4df40 (127.17.60.196) changed from NON_VOTER to VOTER. New config: { opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "c561ec3374b94b95be0c155d80ff0bff" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 33627 } } peers { permanent_uuid: "3710b74ed2044ce5995533df32e67351" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 38589 } attrs { replace: true } } peers { permanent_uuid: "b5e9178a7a83402e8c4a6943ab78589d" member_type: VOTER last_known_addr { host: "127.17.60.197" port: 36573 } attrs { promote: false } } peers { permanent_uuid: "a2a5bba4ac1145d5bab94a69c9f4df40" member_type: VOTER last_known_addr { host: "127.17.60.196" port: 41807 } attrs { promote: false } } }
I20250112 15:07:39.625392 20970 catalog_manager.cc:5526] T 69c23627ab854000b360030c384dc966 P a2a5bba4ac1145d5bab94a69c9f4df40 reported cstate change: config changed from index 28 to 31, a2a5bba4ac1145d5bab94a69c9f4df40 (127.17.60.196) changed from NON_VOTER to VOTER. New cstate: current_term: 3 leader_uuid: "c561ec3374b94b95be0c155d80ff0bff" committed_config { opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "c561ec3374b94b95be0c155d80ff0bff" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 33627 } } peers { permanent_uuid: "3710b74ed2044ce5995533df32e67351" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 38589 } attrs { replace: true } } peers { permanent_uuid: "b5e9178a7a83402e8c4a6943ab78589d" member_type: VOTER last_known_addr { host: "127.17.60.197" port: 36573 } attrs { promote: false } } peers { permanent_uuid: "a2a5bba4ac1145d5bab94a69c9f4df40" member_type: VOTER last_known_addr { host: "127.17.60.196" port: 41807 } attrs { promote: false } } }
I20250112 15:07:39.626246 21149 consensus_queue.cc:237] T 69c23627ab854000b360030c384dc966 P c561ec3374b94b95be0c155d80ff0bff [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 31, Committed index: 31, Last appended: 3.31, Last appended by leader: 29, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "c561ec3374b94b95be0c155d80ff0bff" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 33627 } } peers { permanent_uuid: "b5e9178a7a83402e8c4a6943ab78589d" member_type: VOTER last_known_addr { host: "127.17.60.197" port: 36573 } attrs { promote: false } } peers { permanent_uuid: "a2a5bba4ac1145d5bab94a69c9f4df40" member_type: VOTER last_known_addr { host: "127.17.60.196" port: 41807 } attrs { promote: false } }
I20250112 15:07:39.626809 21474 raft_consensus.cc:1270] T 69c23627ab854000b360030c384dc966 P a2a5bba4ac1145d5bab94a69c9f4df40 [term 3 FOLLOWER]: Refusing update from remote peer c561ec3374b94b95be0c155d80ff0bff: Log matching property violated. Preceding OpId in replica: term: 3 index: 31. Preceding OpId from leader: term: 3 index: 32. (index mismatch)
I20250112 15:07:39.626977 21548 raft_consensus.cc:1270] T 69c23627ab854000b360030c384dc966 P b5e9178a7a83402e8c4a6943ab78589d [term 3 FOLLOWER]: Refusing update from remote peer c561ec3374b94b95be0c155d80ff0bff: Log matching property violated. Preceding OpId in replica: term: 3 index: 31. Preceding OpId from leader: term: 3 index: 32. (index mismatch)
I20250112 15:07:39.626998 21919 consensus_queue.cc:1035] T 69c23627ab854000b360030c384dc966 P c561ec3374b94b95be0c155d80ff0bff [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2a5bba4ac1145d5bab94a69c9f4df40" member_type: VOTER last_known_addr { host: "127.17.60.196" port: 41807 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 32, Last known committed idx: 31, Time since last communication: 0.000s
I20250112 15:07:39.627133 21883 consensus_queue.cc:1035] T 69c23627ab854000b360030c384dc966 P c561ec3374b94b95be0c155d80ff0bff [LEADER]: Connected to new peer: Peer: permanent_uuid: "b5e9178a7a83402e8c4a6943ab78589d" member_type: VOTER last_known_addr { host: "127.17.60.197" port: 36573 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 32, Last known committed idx: 31, Time since last communication: 0.000s
I20250112 15:07:39.627537 21919 raft_consensus.cc:2949] T 69c23627ab854000b360030c384dc966 P c561ec3374b94b95be0c155d80ff0bff [term 3 LEADER]: Committing config change with OpId 3.32: config changed from index 31 to 32, VOTER 3710b74ed2044ce5995533df32e67351 (127.17.60.195) evicted. New config: { opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "c561ec3374b94b95be0c155d80ff0bff" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 33627 } } peers { permanent_uuid: "b5e9178a7a83402e8c4a6943ab78589d" member_type: VOTER last_known_addr { host: "127.17.60.197" port: 36573 } attrs { promote: false } } peers { permanent_uuid: "a2a5bba4ac1145d5bab94a69c9f4df40" member_type: VOTER last_known_addr { host: "127.17.60.196" port: 41807 } attrs { promote: false } } }
I20250112 15:07:39.627642 21474 raft_consensus.cc:2949] T 69c23627ab854000b360030c384dc966 P a2a5bba4ac1145d5bab94a69c9f4df40 [term 3 FOLLOWER]: Committing config change with OpId 3.32: config changed from index 31 to 32, VOTER 3710b74ed2044ce5995533df32e67351 (127.17.60.195) evicted. New config: { opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "c561ec3374b94b95be0c155d80ff0bff" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 33627 } } peers { permanent_uuid: "b5e9178a7a83402e8c4a6943ab78589d" member_type: VOTER last_known_addr { host: "127.17.60.197" port: 36573 } attrs { promote: false } } peers { permanent_uuid: "a2a5bba4ac1145d5bab94a69c9f4df40" member_type: VOTER last_known_addr { host: "127.17.60.196" port: 41807 } attrs { promote: false } } }
I20250112 15:07:39.627739 21548 raft_consensus.cc:2949] T 69c23627ab854000b360030c384dc966 P b5e9178a7a83402e8c4a6943ab78589d [term 3 FOLLOWER]: Committing config change with OpId 3.32: config changed from index 31 to 32, VOTER 3710b74ed2044ce5995533df32e67351 (127.17.60.195) evicted. New config: { opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "c561ec3374b94b95be0c155d80ff0bff" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 33627 } } peers { permanent_uuid: "b5e9178a7a83402e8c4a6943ab78589d" member_type: VOTER last_known_addr { host: "127.17.60.197" port: 36573 } attrs { promote: false } } peers { permanent_uuid: "a2a5bba4ac1145d5bab94a69c9f4df40" member_type: VOTER last_known_addr { host: "127.17.60.196" port: 41807 } attrs { promote: false } } }
I20250112 15:07:39.628363 20958 catalog_manager.cc:5039] ChangeConfig:REMOVE_PEER RPC for tablet 69c23627ab854000b360030c384dc966 with cas_config_opid_index 31: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250112 15:07:39.628638 20964 catalog_manager.cc:5526] T 69c23627ab854000b360030c384dc966 P a2a5bba4ac1145d5bab94a69c9f4df40 reported cstate change: config changed from index 31 to 32, VOTER 3710b74ed2044ce5995533df32e67351 (127.17.60.195) evicted. New cstate: current_term: 3 leader_uuid: "c561ec3374b94b95be0c155d80ff0bff" committed_config { opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "c561ec3374b94b95be0c155d80ff0bff" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 33627 } } peers { permanent_uuid: "b5e9178a7a83402e8c4a6943ab78589d" member_type: VOTER last_known_addr { host: "127.17.60.197" port: 36573 } attrs { promote: false } } peers { permanent_uuid: "a2a5bba4ac1145d5bab94a69c9f4df40" member_type: VOTER last_known_addr { host: "127.17.60.196" port: 41807 } attrs { promote: false } } }
I20250112 15:07:39.629498 21212 tablet_service.cc:1514] Processing DeleteTablet for tablet 69c23627ab854000b360030c384dc966 with delete_type TABLET_DATA_TOMBSTONED (TS 3710b74ed2044ce5995533df32e67351 not found in new config with opid_index 32) from {username='slave'} at 127.0.0.1:33192
I20250112 15:07:39.629694 21953 tablet_replica.cc:331] T 69c23627ab854000b360030c384dc966 P 3710b74ed2044ce5995533df32e67351: stopping tablet replica
I20250112 15:07:39.629760 21953 raft_consensus.cc:2238] T 69c23627ab854000b360030c384dc966 P 3710b74ed2044ce5995533df32e67351 [term 3 FOLLOWER]: Raft consensus shutting down.
I20250112 15:07:39.629850 21953 raft_consensus.cc:2267] T 69c23627ab854000b360030c384dc966 P 3710b74ed2044ce5995533df32e67351 [term 3 FOLLOWER]: Raft consensus is shut down!
I20250112 15:07:39.630043 21953 ts_tablet_manager.cc:1905] T 69c23627ab854000b360030c384dc966 P 3710b74ed2044ce5995533df32e67351: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250112 15:07:39.631373 21953 ts_tablet_manager.cc:1918] T 69c23627ab854000b360030c384dc966 P 3710b74ed2044ce5995533df32e67351: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 3.31
I20250112 15:07:39.631430 21953 log.cc:1198] T 69c23627ab854000b360030c384dc966 P 3710b74ed2044ce5995533df32e67351: Deleting WAL directory at /tmp/dist-test-task95dLSD/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1736694351070047-17651-0/minicluster-data/ts-2-root/wals/69c23627ab854000b360030c384dc966
I20250112 15:07:39.631742 20957 catalog_manager.cc:4872] TS 3710b74ed2044ce5995533df32e67351 (127.17.60.195:38589): tablet 69c23627ab854000b360030c384dc966 (table test-workload [id=cdaf6d3a65a54ea08e8e2c0ecdfc9de7]) successfully deleted
I20250112 15:07:40.014964 21954 ts_tablet_manager.cc:927] T 8c7dbaf285a04da9b4eec92a29ab7d6c P a2a5bba4ac1145d5bab94a69c9f4df40: Initiating tablet copy from peer c561ec3374b94b95be0c155d80ff0bff (127.17.60.194:33627)
I20250112 15:07:40.015179 21954 tablet_copy_client.cc:323] T 8c7dbaf285a04da9b4eec92a29ab7d6c P a2a5bba4ac1145d5bab94a69c9f4df40: tablet copy: Beginning tablet copy session from remote peer at address 127.17.60.194:33627
I20250112 15:07:40.015385 21161 tablet_copy_service.cc:140] P c561ec3374b94b95be0c155d80ff0bff: Received BeginTabletCopySession request for tablet 8c7dbaf285a04da9b4eec92a29ab7d6c from peer a2a5bba4ac1145d5bab94a69c9f4df40 ({username='slave'} at 127.0.0.1:58522)
I20250112 15:07:40.015460 21161 tablet_copy_service.cc:161] P c561ec3374b94b95be0c155d80ff0bff: Beginning new tablet copy session on tablet 8c7dbaf285a04da9b4eec92a29ab7d6c from peer a2a5bba4ac1145d5bab94a69c9f4df40 at {username='slave'} at 127.0.0.1:58522: session id = a2a5bba4ac1145d5bab94a69c9f4df40-8c7dbaf285a04da9b4eec92a29ab7d6c
I20250112 15:07:40.016078 21161 tablet_copy_source_session.cc:215] T 8c7dbaf285a04da9b4eec92a29ab7d6c P c561ec3374b94b95be0c155d80ff0bff: Tablet Copy: opened 0 blocks and 1 log segments
I20250112 15:07:40.016554 21954 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8c7dbaf285a04da9b4eec92a29ab7d6c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 15:07:40.018049 21954 tablet_copy_client.cc:806] T 8c7dbaf285a04da9b4eec92a29ab7d6c P a2a5bba4ac1145d5bab94a69c9f4df40: tablet copy: Starting download of 0 data blocks...
I20250112 15:07:40.018227 21954 tablet_copy_client.cc:670] T 8c7dbaf285a04da9b4eec92a29ab7d6c P a2a5bba4ac1145d5bab94a69c9f4df40: tablet copy: Starting download of 1 WAL segments...
I20250112 15:07:40.019866 21955 tablet_copy_client.cc:1025] T 8c7dbaf285a04da9b4eec92a29ab7d6c P a2a5bba4ac1145d5bab94a69c9f4df40: tablet copy: Injecting latency into file download: 2000
I20250112 15:07:40.033521 21956 ts_tablet_manager.cc:927] T bfaeae584ec04772b2d6fcd088022117 P 25286c485fa04c86a5d07c8f88643a47: Initiating tablet copy from peer 2537bbbd1a9947ae9facd87567e3ccb9 (127.17.60.193:33173)
I20250112 15:07:40.033720 21956 tablet_copy_client.cc:323] T bfaeae584ec04772b2d6fcd088022117 P 25286c485fa04c86a5d07c8f88643a47: tablet copy: Beginning tablet copy session from remote peer at address 127.17.60.193:33173
I20250112 15:07:40.033910 21085 tablet_copy_service.cc:140] P 2537bbbd1a9947ae9facd87567e3ccb9: Received BeginTabletCopySession request for tablet bfaeae584ec04772b2d6fcd088022117 from peer 25286c485fa04c86a5d07c8f88643a47 ({username='slave'} at 127.0.0.1:54434)
I20250112 15:07:40.033957 21085 tablet_copy_service.cc:161] P 2537bbbd1a9947ae9facd87567e3ccb9: Beginning new tablet copy session on tablet bfaeae584ec04772b2d6fcd088022117 from peer 25286c485fa04c86a5d07c8f88643a47 at {username='slave'} at 127.0.0.1:54434: session id = 25286c485fa04c86a5d07c8f88643a47-bfaeae584ec04772b2d6fcd088022117
I20250112 15:07:40.034354 21085 tablet_copy_source_session.cc:215] T bfaeae584ec04772b2d6fcd088022117 P 2537bbbd1a9947ae9facd87567e3ccb9: Tablet Copy: opened 0 blocks and 1 log segments
I20250112 15:07:40.034723 21956 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bfaeae584ec04772b2d6fcd088022117. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 15:07:40.035930 21956 tablet_copy_client.cc:806] T bfaeae584ec04772b2d6fcd088022117 P 25286c485fa04c86a5d07c8f88643a47: tablet copy: Starting download of 0 data blocks...
I20250112 15:07:40.036108 21956 tablet_copy_client.cc:670] T bfaeae584ec04772b2d6fcd088022117 P 25286c485fa04c86a5d07c8f88643a47: tablet copy: Starting download of 1 WAL segments...
I20250112 15:07:40.036629 21957 tablet_copy_client.cc:1025] T bfaeae584ec04772b2d6fcd088022117 P 25286c485fa04c86a5d07c8f88643a47: tablet copy: Injecting latency into file download: 2000
I20250112 15:07:40.062764 21958 ts_tablet_manager.cc:927] T 16f16fcd5cf347e5853189dc4da55c7f P 25286c485fa04c86a5d07c8f88643a47: Initiating tablet copy from peer 2537bbbd1a9947ae9facd87567e3ccb9 (127.17.60.193:33173)
I20250112 15:07:40.062994 21958 tablet_copy_client.cc:323] T 16f16fcd5cf347e5853189dc4da55c7f P 25286c485fa04c86a5d07c8f88643a47: tablet copy: Beginning tablet copy session from remote peer at address 127.17.60.193:33173
I20250112 15:07:40.063171 21085 tablet_copy_service.cc:140] P 2537bbbd1a9947ae9facd87567e3ccb9: Received BeginTabletCopySession request for tablet 16f16fcd5cf347e5853189dc4da55c7f from peer 25286c485fa04c86a5d07c8f88643a47 ({username='slave'} at 127.0.0.1:54434)
I20250112 15:07:40.063221 21085 tablet_copy_service.cc:161] P 2537bbbd1a9947ae9facd87567e3ccb9: Beginning new tablet copy session on tablet 16f16fcd5cf347e5853189dc4da55c7f from peer 25286c485fa04c86a5d07c8f88643a47 at {username='slave'} at 127.0.0.1:54434: session id = 25286c485fa04c86a5d07c8f88643a47-16f16fcd5cf347e5853189dc4da55c7f
I20250112 15:07:40.063633 21085 tablet_copy_source_session.cc:215] T 16f16fcd5cf347e5853189dc4da55c7f P 2537bbbd1a9947ae9facd87567e3ccb9: Tablet Copy: opened 0 blocks and 1 log segments
I20250112 15:07:40.063930 21958 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 16f16fcd5cf347e5853189dc4da55c7f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 15:07:40.065505 21958 tablet_copy_client.cc:806] T 16f16fcd5cf347e5853189dc4da55c7f P 25286c485fa04c86a5d07c8f88643a47: tablet copy: Starting download of 0 data blocks...
I20250112 15:07:40.065701 21958 tablet_copy_client.cc:670] T 16f16fcd5cf347e5853189dc4da55c7f P 25286c485fa04c86a5d07c8f88643a47: tablet copy: Starting download of 1 WAL segments...
I20250112 15:07:40.066229 21959 tablet_copy_client.cc:1025] T 16f16fcd5cf347e5853189dc4da55c7f P 25286c485fa04c86a5d07c8f88643a47: tablet copy: Injecting latency into file download: 2000
I20250112 15:07:40.070430 21960 ts_tablet_manager.cc:927] T 9d4bbba799324d7fbbac082520c65686 P b5e9178a7a83402e8c4a6943ab78589d: Initiating tablet copy from peer 2537bbbd1a9947ae9facd87567e3ccb9 (127.17.60.193:33173)
I20250112 15:07:40.070549 21960 tablet_copy_client.cc:323] T 9d4bbba799324d7fbbac082520c65686 P b5e9178a7a83402e8c4a6943ab78589d: tablet copy: Beginning tablet copy session from remote peer at address 127.17.60.193:33173
I20250112 15:07:40.070703 21085 tablet_copy_service.cc:140] P 2537bbbd1a9947ae9facd87567e3ccb9: Received BeginTabletCopySession request for tablet 9d4bbba799324d7fbbac082520c65686 from peer b5e9178a7a83402e8c4a6943ab78589d ({username='slave'} at 127.0.0.1:54448)
I20250112 15:07:40.070756 21085 tablet_copy_service.cc:161] P 2537bbbd1a9947ae9facd87567e3ccb9: Beginning new tablet copy session on tablet 9d4bbba799324d7fbbac082520c65686 from peer b5e9178a7a83402e8c4a6943ab78589d at {username='slave'} at 127.0.0.1:54448: session id = b5e9178a7a83402e8c4a6943ab78589d-9d4bbba799324d7fbbac082520c65686
I20250112 15:07:40.071233 21085 tablet_copy_source_session.cc:215] T 9d4bbba799324d7fbbac082520c65686 P 2537bbbd1a9947ae9facd87567e3ccb9: Tablet Copy: opened 0 blocks and 1 log segments
I20250112 15:07:40.071516 21960 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9d4bbba799324d7fbbac082520c65686. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 15:07:40.072824 21960 tablet_copy_client.cc:806] T 9d4bbba799324d7fbbac082520c65686 P b5e9178a7a83402e8c4a6943ab78589d: tablet copy: Starting download of 0 data blocks...
I20250112 15:07:40.072963 21960 tablet_copy_client.cc:670] T 9d4bbba799324d7fbbac082520c65686 P b5e9178a7a83402e8c4a6943ab78589d: tablet copy: Starting download of 1 WAL segments...
I20250112 15:07:40.073441 21961 tablet_copy_client.cc:1025] T 9d4bbba799324d7fbbac082520c65686 P b5e9178a7a83402e8c4a6943ab78589d: tablet copy: Injecting latency into file download: 2000
I20250112 15:07:40.090883 21026 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250112 15:07:40.091250 21149 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "80cc505a04394c53a18c5beeded4655d"
dest_uuid: "c561ec3374b94b95be0c155d80ff0bff"
mode: GRACEFUL
new_leader_uuid: "3710b74ed2044ce5995533df32e67351"
from {username='slave'} at 127.0.0.1:48104
I20250112 15:07:40.091338 21149 raft_consensus.cc:604] T 80cc505a04394c53a18c5beeded4655d P c561ec3374b94b95be0c155d80ff0bff [term 1 LEADER]: Received request to transfer leadership to TS 3710b74ed2044ce5995533df32e67351
I20250112 15:07:40.091601 21068 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "9d4bbba799324d7fbbac082520c65686"
dest_uuid: "2537bbbd1a9947ae9facd87567e3ccb9"
mode: GRACEFUL
new_leader_uuid: "c561ec3374b94b95be0c155d80ff0bff"
from {username='slave'} at 127.0.0.1:54414
I20250112 15:07:40.091681 21068 raft_consensus.cc:604] T 9d4bbba799324d7fbbac082520c65686 P 2537bbbd1a9947ae9facd87567e3ccb9 [term 2 LEADER]: Received request to transfer leadership to TS c561ec3374b94b95be0c155d80ff0bff
I20250112 15:07:40.091903 21222 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "ef90b704208140a3983ffe6105939319"
dest_uuid: "3710b74ed2044ce5995533df32e67351"
mode: GRACEFUL
new_leader_uuid: "2537bbbd1a9947ae9facd87567e3ccb9"
from {username='slave'} at 127.0.0.1:54970
I20250112 15:07:40.091964 21222 raft_consensus.cc:604] T ef90b704208140a3983ffe6105939319 P 3710b74ed2044ce5995533df32e67351 [term 3 LEADER]: Received request to transfer leadership to TS 2537bbbd1a9947ae9facd87567e3ccb9
I20250112 15:07:40.092087 21026 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 3
I20250112 15:07:40.092142 21026 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20250112 15:07:40.121129 21962 ts_tablet_manager.cc:927] T 57e47b07fdcf434e96ca5dd0354e93b2 P a2a5bba4ac1145d5bab94a69c9f4df40: Initiating tablet copy from peer b5e9178a7a83402e8c4a6943ab78589d (127.17.60.197:36573)
I20250112 15:07:40.121378 21962 tablet_copy_client.cc:323] T 57e47b07fdcf434e96ca5dd0354e93b2 P a2a5bba4ac1145d5bab94a69c9f4df40: tablet copy: Beginning tablet copy session from remote peer at address 127.17.60.197:36573
I20250112 15:07:40.121553 21559 tablet_copy_service.cc:140] P b5e9178a7a83402e8c4a6943ab78589d: Received BeginTabletCopySession request for tablet 57e47b07fdcf434e96ca5dd0354e93b2 from peer a2a5bba4ac1145d5bab94a69c9f4df40 ({username='slave'} at 127.0.0.1:44706)
I20250112 15:07:40.121605 21559 tablet_copy_service.cc:161] P b5e9178a7a83402e8c4a6943ab78589d: Beginning new tablet copy session on tablet 57e47b07fdcf434e96ca5dd0354e93b2 from peer a2a5bba4ac1145d5bab94a69c9f4df40 at {username='slave'} at 127.0.0.1:44706: session id = a2a5bba4ac1145d5bab94a69c9f4df40-57e47b07fdcf434e96ca5dd0354e93b2
I20250112 15:07:40.122188 21559 tablet_copy_source_session.cc:215] T 57e47b07fdcf434e96ca5dd0354e93b2 P b5e9178a7a83402e8c4a6943ab78589d: Tablet Copy: opened 0 blocks and 1 log segments
I20250112 15:07:40.122462 21962 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 57e47b07fdcf434e96ca5dd0354e93b2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 15:07:40.123544 21962 tablet_copy_client.cc:806] T 57e47b07fdcf434e96ca5dd0354e93b2 P a2a5bba4ac1145d5bab94a69c9f4df40: tablet copy: Starting download of 0 data blocks...
I20250112 15:07:40.123684 21962 tablet_copy_client.cc:670] T 57e47b07fdcf434e96ca5dd0354e93b2 P a2a5bba4ac1145d5bab94a69c9f4df40: tablet copy: Starting download of 1 WAL segments...
I20250112 15:07:40.124148 21963 tablet_copy_client.cc:1025] T 57e47b07fdcf434e96ca5dd0354e93b2 P a2a5bba4ac1145d5bab94a69c9f4df40: tablet copy: Injecting latency into file download: 2000
I20250112 15:07:40.173717 21964 ts_tablet_manager.cc:927] T 9601bcda24ed42e7b93ab9740f1e271e P 25286c485fa04c86a5d07c8f88643a47: Initiating tablet copy from peer 3710b74ed2044ce5995533df32e67351 (127.17.60.195:38589)
I20250112 15:07:40.173883 21964 tablet_copy_client.cc:323] T 9601bcda24ed42e7b93ab9740f1e271e P 25286c485fa04c86a5d07c8f88643a47: tablet copy: Beginning tablet copy session from remote peer at address 127.17.60.195:38589
I20250112 15:07:40.177028 21236 tablet_copy_service.cc:140] P 3710b74ed2044ce5995533df32e67351: Received BeginTabletCopySession request for tablet 9601bcda24ed42e7b93ab9740f1e271e from peer 25286c485fa04c86a5d07c8f88643a47 ({username='slave'} at 127.0.0.1:52200)
I20250112 15:07:40.177090 21236 tablet_copy_service.cc:161] P 3710b74ed2044ce5995533df32e67351: Beginning new tablet copy session on tablet 9601bcda24ed42e7b93ab9740f1e271e from peer 25286c485fa04c86a5d07c8f88643a47 at {username='slave'} at 127.0.0.1:52200: session id = 25286c485fa04c86a5d07c8f88643a47-9601bcda24ed42e7b93ab9740f1e271e
I20250112 15:07:40.177794 21236 tablet_copy_source_session.cc:215] T 9601bcda24ed42e7b93ab9740f1e271e P 3710b74ed2044ce5995533df32e67351: Tablet Copy: opened 0 blocks and 1 log segments
I20250112 15:07:40.178067 21964 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9601bcda24ed42e7b93ab9740f1e271e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 15:07:40.179217 21964 tablet_copy_client.cc:806] T 9601bcda24ed42e7b93ab9740f1e271e P 25286c485fa04c86a5d07c8f88643a47: tablet copy: Starting download of 0 data blocks...
I20250112 15:07:40.179337 21964 tablet_copy_client.cc:670] T 9601bcda24ed42e7b93ab9740f1e271e P 25286c485fa04c86a5d07c8f88643a47: tablet copy: Starting download of 1 WAL segments...
I20250112 15:07:40.179787 21965 tablet_copy_client.cc:1025] T 9601bcda24ed42e7b93ab9740f1e271e P 25286c485fa04c86a5d07c8f88643a47: tablet copy: Injecting latency into file download: 2000
I20250112 15:07:40.248880 21914 raft_consensus.cc:988] T ef90b704208140a3983ffe6105939319 P 3710b74ed2044ce5995533df32e67351: : Instructing follower 2537bbbd1a9947ae9facd87567e3ccb9 to start an election
I20250112 15:07:40.248996 21914 raft_consensus.cc:1076] T ef90b704208140a3983ffe6105939319 P 3710b74ed2044ce5995533df32e67351 [term 3 LEADER]: Signalling peer 2537bbbd1a9947ae9facd87567e3ccb9 to start an election
I20250112 15:07:40.249135 21068 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "ef90b704208140a3983ffe6105939319"
dest_uuid: "2537bbbd1a9947ae9facd87567e3ccb9"
from {username='slave'} at 127.0.0.1:37204
I20250112 15:07:40.249220 21068 raft_consensus.cc:491] T ef90b704208140a3983ffe6105939319 P 2537bbbd1a9947ae9facd87567e3ccb9 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20250112 15:07:40.249253 21068 raft_consensus.cc:3054] T ef90b704208140a3983ffe6105939319 P 2537bbbd1a9947ae9facd87567e3ccb9 [term 3 FOLLOWER]: Advancing to term 4
I20250112 15:07:40.250113 21068 raft_consensus.cc:513] T ef90b704208140a3983ffe6105939319 P 2537bbbd1a9947ae9facd87567e3ccb9 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2537bbbd1a9947ae9facd87567e3ccb9" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 33173 } } peers { permanent_uuid: "c561ec3374b94b95be0c155d80ff0bff" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 33627 } } peers { permanent_uuid: "3710b74ed2044ce5995533df32e67351" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 38589 } }
I20250112 15:07:40.250279 21068 leader_election.cc:290] T ef90b704208140a3983ffe6105939319 P 2537bbbd1a9947ae9facd87567e3ccb9 [CANDIDATE]: Term 4 election: Requested vote from peers c561ec3374b94b95be0c155d80ff0bff (127.17.60.194:33627), 3710b74ed2044ce5995533df32e67351 (127.17.60.195:38589)
I20250112 15:07:40.250351 21149 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef90b704208140a3983ffe6105939319" candidate_uuid: "2537bbbd1a9947ae9facd87567e3ccb9" candidate_term: 4 candidate_status { last_received { term: 3 index: 27 } } ignore_live_leader: true dest_uuid: "c561ec3374b94b95be0c155d80ff0bff"
I20250112 15:07:40.250412 21149 raft_consensus.cc:3054] T ef90b704208140a3983ffe6105939319 P c561ec3374b94b95be0c155d80ff0bff [term 3 FOLLOWER]: Advancing to term 4
I20250112 15:07:40.250384 21222 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef90b704208140a3983ffe6105939319" candidate_uuid: "2537bbbd1a9947ae9facd87567e3ccb9" candidate_term: 4 candidate_status { last_received { term: 3 index: 27 } } ignore_live_leader: true dest_uuid: "3710b74ed2044ce5995533df32e67351"
I20250112 15:07:40.250468 21222 raft_consensus.cc:3049] T ef90b704208140a3983ffe6105939319 P 3710b74ed2044ce5995533df32e67351 [term 3 LEADER]: Stepping down as leader of term 3
I20250112 15:07:40.250497 21222 raft_consensus.cc:738] T ef90b704208140a3983ffe6105939319 P 3710b74ed2044ce5995533df32e67351 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 3710b74ed2044ce5995533df32e67351, State: Running, Role: LEADER
I20250112 15:07:40.250555 21222 consensus_queue.cc:260] T ef90b704208140a3983ffe6105939319 P 3710b74ed2044ce5995533df32e67351 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 27, Committed index: 27, Last appended: 3.27, Last appended by leader: 27, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2537bbbd1a9947ae9facd87567e3ccb9" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 33173 } } peers { permanent_uuid: "c561ec3374b94b95be0c155d80ff0bff" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 33627 } } peers { permanent_uuid: "3710b74ed2044ce5995533df32e67351" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 38589 } }
I20250112 15:07:40.250622 21222 raft_consensus.cc:3054] T ef90b704208140a3983ffe6105939319 P 3710b74ed2044ce5995533df32e67351 [term 3 FOLLOWER]: Advancing to term 4
I20250112 15:07:40.250994 21149 raft_consensus.cc:2463] T ef90b704208140a3983ffe6105939319 P c561ec3374b94b95be0c155d80ff0bff [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2537bbbd1a9947ae9facd87567e3ccb9 in term 4.
I20250112 15:07:40.251150 21222 raft_consensus.cc:2463] T ef90b704208140a3983ffe6105939319 P 3710b74ed2044ce5995533df32e67351 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2537bbbd1a9947ae9facd87567e3ccb9 in term 4.
I20250112 15:07:40.251137 21040 leader_election.cc:304] T ef90b704208140a3983ffe6105939319 P 2537bbbd1a9947ae9facd87567e3ccb9 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2537bbbd1a9947ae9facd87567e3ccb9, c561ec3374b94b95be0c155d80ff0bff; no voters:
I20250112 15:07:40.251252 21856 raft_consensus.cc:2798] T ef90b704208140a3983ffe6105939319 P 2537bbbd1a9947ae9facd87567e3ccb9 [term 4 FOLLOWER]: Leader election won for term 4
I20250112 15:07:40.251385 21856 raft_consensus.cc:695] T ef90b704208140a3983ffe6105939319 P 2537bbbd1a9947ae9facd87567e3ccb9 [term 4 LEADER]: Becoming Leader. State: Replica: 2537bbbd1a9947ae9facd87567e3ccb9, State: Running, Role: LEADER
I20250112 15:07:40.251437 21856 consensus_queue.cc:237] T ef90b704208140a3983ffe6105939319 P 2537bbbd1a9947ae9facd87567e3ccb9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 27, Committed index: 27, Last appended: 3.27, Last appended by leader: 27, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2537bbbd1a9947ae9facd87567e3ccb9" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 33173 } } peers { permanent_uuid: "c561ec3374b94b95be0c155d80ff0bff" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 33627 } } peers { permanent_uuid: "3710b74ed2044ce5995533df32e67351" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 38589 } }
I20250112 15:07:40.251886 20970 catalog_manager.cc:5526] T ef90b704208140a3983ffe6105939319 P 2537bbbd1a9947ae9facd87567e3ccb9 reported cstate change: term changed from 3 to 4, leader changed from 3710b74ed2044ce5995533df32e67351 (127.17.60.195) to 2537bbbd1a9947ae9facd87567e3ccb9 (127.17.60.193). New cstate: current_term: 4 leader_uuid: "2537bbbd1a9947ae9facd87567e3ccb9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2537bbbd1a9947ae9facd87567e3ccb9" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 33173 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c561ec3374b94b95be0c155d80ff0bff" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 33627 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3710b74ed2044ce5995533df32e67351" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 38589 } health_report { overall_health: UNKNOWN } } }
I20250112 15:07:40.362015 21935 raft_consensus.cc:988] T 80cc505a04394c53a18c5beeded4655d P c561ec3374b94b95be0c155d80ff0bff: : Instructing follower 3710b74ed2044ce5995533df32e67351 to start an election
I20250112 15:07:40.362116 21935 raft_consensus.cc:1076] T 80cc505a04394c53a18c5beeded4655d P c561ec3374b94b95be0c155d80ff0bff [term 1 LEADER]: Signalling peer 3710b74ed2044ce5995533df32e67351 to start an election
I20250112 15:07:40.362257 21226 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "80cc505a04394c53a18c5beeded4655d"
dest_uuid: "3710b74ed2044ce5995533df32e67351"
from {username='slave'} at 127.0.0.1:33202
I20250112 15:07:40.362337 21226 raft_consensus.cc:491] T 80cc505a04394c53a18c5beeded4655d P 3710b74ed2044ce5995533df32e67351 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20250112 15:07:40.362360 21226 raft_consensus.cc:3054] T 80cc505a04394c53a18c5beeded4655d P 3710b74ed2044ce5995533df32e67351 [term 1 FOLLOWER]: Advancing to term 2
I20250112 15:07:40.363101 21226 raft_consensus.cc:513] T 80cc505a04394c53a18c5beeded4655d P 3710b74ed2044ce5995533df32e67351 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "c561ec3374b94b95be0c155d80ff0bff" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 33627 } } peers { permanent_uuid: "3710b74ed2044ce5995533df32e67351" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 38589 } } peers { permanent_uuid: "25286c485fa04c86a5d07c8f88643a47" member_type: VOTER last_known_addr { host: "127.17.60.198" port: 37089 } attrs { promote: false } }
I20250112 15:07:40.363262 21226 leader_election.cc:290] T 80cc505a04394c53a18c5beeded4655d P 3710b74ed2044ce5995533df32e67351 [CANDIDATE]: Term 2 election: Requested vote from peers c561ec3374b94b95be0c155d80ff0bff (127.17.60.194:33627), 25286c485fa04c86a5d07c8f88643a47 (127.17.60.198:37089)
I20250112 15:07:40.363387 21149 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "80cc505a04394c53a18c5beeded4655d" candidate_uuid: "3710b74ed2044ce5995533df32e67351" candidate_term: 2 candidate_status { last_received { term: 1 index: 28 } } ignore_live_leader: true dest_uuid: "c561ec3374b94b95be0c155d80ff0bff"
I20250112 15:07:40.363404 21624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "80cc505a04394c53a18c5beeded4655d" candidate_uuid: "3710b74ed2044ce5995533df32e67351" candidate_term: 2 candidate_status { last_received { term: 1 index: 28 } } ignore_live_leader: true dest_uuid: "25286c485fa04c86a5d07c8f88643a47"
I20250112 15:07:40.363480 21149 raft_consensus.cc:3049] T 80cc505a04394c53a18c5beeded4655d P c561ec3374b94b95be0c155d80ff0bff [term 1 LEADER]: Stepping down as leader of term 1
I20250112 15:07:40.363515 21624 raft_consensus.cc:3054] T 80cc505a04394c53a18c5beeded4655d P 25286c485fa04c86a5d07c8f88643a47 [term 1 FOLLOWER]: Advancing to term 2
I20250112 15:07:40.363535 21149 raft_consensus.cc:738] T 80cc505a04394c53a18c5beeded4655d P c561ec3374b94b95be0c155d80ff0bff [term 1 LEADER]: Becoming Follower/Learner. State: Replica: c561ec3374b94b95be0c155d80ff0bff, State: Running, Role: LEADER
I20250112 15:07:40.363593 21149 consensus_queue.cc:260] T 80cc505a04394c53a18c5beeded4655d P c561ec3374b94b95be0c155d80ff0bff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 28, Committed index: 28, Last appended: 1.28, Last appended by leader: 28, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "c561ec3374b94b95be0c155d80ff0bff" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 33627 } } peers { permanent_uuid: "3710b74ed2044ce5995533df32e67351" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 38589 } } peers { permanent_uuid: "25286c485fa04c86a5d07c8f88643a47" member_type: VOTER last_known_addr { host: "127.17.60.198" port: 37089 } attrs { promote: false } }
I20250112 15:07:40.363659 21149 raft_consensus.cc:3054] T 80cc505a04394c53a18c5beeded4655d P c561ec3374b94b95be0c155d80ff0bff [term 1 FOLLOWER]: Advancing to term 2
I20250112 15:07:40.364060 21624 raft_consensus.cc:2463] T 80cc505a04394c53a18c5beeded4655d P 25286c485fa04c86a5d07c8f88643a47 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3710b74ed2044ce5995533df32e67351 in term 2.
I20250112 15:07:40.364192 21149 raft_consensus.cc:2463] T 80cc505a04394c53a18c5beeded4655d P c561ec3374b94b95be0c155d80ff0bff [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3710b74ed2044ce5995533df32e67351 in term 2.
I20250112 15:07:40.364233 21189 leader_election.cc:304] T 80cc505a04394c53a18c5beeded4655d P 3710b74ed2044ce5995533df32e67351 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 25286c485fa04c86a5d07c8f88643a47, 3710b74ed2044ce5995533df32e67351; no voters:
I20250112 15:07:40.364332 21968 raft_consensus.cc:2798] T 80cc505a04394c53a18c5beeded4655d P 3710b74ed2044ce5995533df32e67351 [term 2 FOLLOWER]: Leader election won for term 2
I20250112 15:07:40.364383 21968 raft_consensus.cc:695] T 80cc505a04394c53a18c5beeded4655d P 3710b74ed2044ce5995533df32e67351 [term 2 LEADER]: Becoming Leader. State: Replica: 3710b74ed2044ce5995533df32e67351, State: Running, Role: LEADER
I20250112 15:07:40.364436 21968 consensus_queue.cc:237] T 80cc505a04394c53a18c5beeded4655d P 3710b74ed2044ce5995533df32e67351 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 28, Committed index: 28, Last appended: 1.28, Last appended by leader: 28, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "c561ec3374b94b95be0c155d80ff0bff" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 33627 } } peers { permanent_uuid: "3710b74ed2044ce5995533df32e67351" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 38589 } } peers { permanent_uuid: "25286c485fa04c86a5d07c8f88643a47" member_type: VOTER last_known_addr { host: "127.17.60.198" port: 37089 } attrs { promote: false } }
I20250112 15:07:40.365001 20970 catalog_manager.cc:5526] T 80cc505a04394c53a18c5beeded4655d P 3710b74ed2044ce5995533df32e67351 reported cstate change: term changed from 1 to 2, leader changed from c561ec3374b94b95be0c155d80ff0bff (127.17.60.194) to 3710b74ed2044ce5995533df32e67351 (127.17.60.195). New cstate: current_term: 2 leader_uuid: "3710b74ed2044ce5995533df32e67351" committed_config { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "c561ec3374b94b95be0c155d80ff0bff" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 33627 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3710b74ed2044ce5995533df32e67351" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 38589 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "25286c485fa04c86a5d07c8f88643a47" member_type: VOTER last_known_addr { host: "127.17.60.198" port: 37089 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20250112 15:07:40.604475 21977 raft_consensus.cc:988] T 9d4bbba799324d7fbbac082520c65686 P 2537bbbd1a9947ae9facd87567e3ccb9: : Instructing follower c561ec3374b94b95be0c155d80ff0bff to start an election
I20250112 15:07:40.604650 21969 raft_consensus.cc:1076] T 9d4bbba799324d7fbbac082520c65686 P 2537bbbd1a9947ae9facd87567e3ccb9 [term 2 LEADER]: Signalling peer c561ec3374b94b95be0c155d80ff0bff to start an election
I20250112 15:07:40.604887 21147 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "9d4bbba799324d7fbbac082520c65686"
dest_uuid: "c561ec3374b94b95be0c155d80ff0bff"
from {username='slave'} at 127.0.0.1:48082
I20250112 15:07:40.604982 21147 raft_consensus.cc:491] T 9d4bbba799324d7fbbac082520c65686 P c561ec3374b94b95be0c155d80ff0bff [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20250112 15:07:40.605017 21147 raft_consensus.cc:3054] T 9d4bbba799324d7fbbac082520c65686 P c561ec3374b94b95be0c155d80ff0bff [term 2 FOLLOWER]: Advancing to term 3
I20250112 15:07:40.605912 21147 raft_consensus.cc:513] T 9d4bbba799324d7fbbac082520c65686 P c561ec3374b94b95be0c155d80ff0bff [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: 26 OBSOLETE_local: false peers { permanent_uuid: "2537bbbd1a9947ae9facd87567e3ccb9" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 33173 } } peers { permanent_uuid: "c561ec3374b94b95be0c155d80ff0bff" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 33627 } } peers { permanent_uuid: "3710b74ed2044ce5995533df32e67351" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 38589 } attrs { replace: true } } peers { permanent_uuid: "b5e9178a7a83402e8c4a6943ab78589d" member_type: NON_VOTER last_known_addr { host: "127.17.60.197" port: 36573 } attrs { promote: true } }
I20250112 15:07:40.606077 21147 leader_election.cc:290] T 9d4bbba799324d7fbbac082520c65686 P c561ec3374b94b95be0c155d80ff0bff [CANDIDATE]: Term 3 election: Requested vote from peers 2537bbbd1a9947ae9facd87567e3ccb9 (127.17.60.193:33173), 3710b74ed2044ce5995533df32e67351 (127.17.60.195:38589)
I20250112 15:07:40.606135 21069 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9d4bbba799324d7fbbac082520c65686" candidate_uuid: "c561ec3374b94b95be0c155d80ff0bff" candidate_term: 3 candidate_status { last_received { term: 2 index: 26 } } ignore_live_leader: true dest_uuid: "2537bbbd1a9947ae9facd87567e3ccb9"
I20250112 15:07:40.606207 21069 raft_consensus.cc:3049] T 9d4bbba799324d7fbbac082520c65686 P 2537bbbd1a9947ae9facd87567e3ccb9 [term 2 LEADER]: Stepping down as leader of term 2
I20250112 15:07:40.606232 21069 raft_consensus.cc:738] T 9d4bbba799324d7fbbac082520c65686 P 2537bbbd1a9947ae9facd87567e3ccb9 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 2537bbbd1a9947ae9facd87567e3ccb9, State: Running, Role: LEADER
I20250112 15:07:40.606287 21069 consensus_queue.cc:260] T 9d4bbba799324d7fbbac082520c65686 P 2537bbbd1a9947ae9facd87567e3ccb9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 26, Committed index: 26, Last appended: 2.26, Last appended by leader: 26, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 26 OBSOLETE_local: false peers { permanent_uuid: "2537bbbd1a9947ae9facd87567e3ccb9" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 33173 } } peers { permanent_uuid: "c561ec3374b94b95be0c155d80ff0bff" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 33627 } } peers { permanent_uuid: "3710b74ed2044ce5995533df32e67351" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 38589 } attrs { replace: true } } peers { permanent_uuid: "b5e9178a7a83402e8c4a6943ab78589d" member_type: NON_VOTER last_known_addr { host: "127.17.60.197" port: 36573 } attrs { promote: true } }
I20250112 15:07:40.606407 21069 raft_consensus.cc:3054] T 9d4bbba799324d7fbbac082520c65686 P 2537bbbd1a9947ae9facd87567e3ccb9 [term 2 FOLLOWER]: Advancing to term 3
I20250112 15:07:40.607165 21069 raft_consensus.cc:2463] T 9d4bbba799324d7fbbac082520c65686 P 2537bbbd1a9947ae9facd87567e3ccb9 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c561ec3374b94b95be0c155d80ff0bff in term 3.
I20250112 15:07:40.607273 21222 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9d4bbba799324d7fbbac082520c65686" candidate_uuid: "c561ec3374b94b95be0c155d80ff0bff" candidate_term: 3 candidate_status { last_received { term: 2 index: 26 } } ignore_live_leader: true dest_uuid: "3710b74ed2044ce5995533df32e67351"
I20250112 15:07:40.607328 21222 raft_consensus.cc:3054] T 9d4bbba799324d7fbbac082520c65686 P 3710b74ed2044ce5995533df32e67351 [term 2 FOLLOWER]: Advancing to term 3
I20250112 15:07:40.607832 21222 raft_consensus.cc:2463] T 9d4bbba799324d7fbbac082520c65686 P 3710b74ed2044ce5995533df32e67351 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c561ec3374b94b95be0c155d80ff0bff in term 3.
I20250112 15:07:40.607920 21114 leader_election.cc:304] T 9d4bbba799324d7fbbac082520c65686 P c561ec3374b94b95be0c155d80ff0bff [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2537bbbd1a9947ae9facd87567e3ccb9, c561ec3374b94b95be0c155d80ff0bff; no voters:
I20250112 15:07:40.608035 21976 raft_consensus.cc:2798] T 9d4bbba799324d7fbbac082520c65686 P c561ec3374b94b95be0c155d80ff0bff [term 3 FOLLOWER]: Leader election won for term 3
I20250112 15:07:40.608073 21976 raft_consensus.cc:695] T 9d4bbba799324d7fbbac082520c65686 P c561ec3374b94b95be0c155d80ff0bff [term 3 LEADER]: Becoming Leader. State: Replica: c561ec3374b94b95be0c155d80ff0bff, State: Running, Role: LEADER
I20250112 15:07:40.608135 21976 consensus_queue.cc:237] T 9d4bbba799324d7fbbac082520c65686 P c561ec3374b94b95be0c155d80ff0bff [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 26, Committed index: 26, Last appended: 2.26, Last appended by leader: 26, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 26 OBSOLETE_local: false peers { permanent_uuid: "2537bbbd1a9947ae9facd87567e3ccb9" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 33173 } } peers { permanent_uuid: "c561ec3374b94b95be0c155d80ff0bff" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 33627 } } peers { permanent_uuid: "3710b74ed2044ce5995533df32e67351" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 38589 } attrs { replace: true } } peers { permanent_uuid: "b5e9178a7a83402e8c4a6943ab78589d" member_type: NON_VOTER last_known_addr { host: "127.17.60.197" port: 36573 } attrs { promote: true } }
I20250112 15:07:40.608707 20970 catalog_manager.cc:5526] T 9d4bbba799324d7fbbac082520c65686 P c561ec3374b94b95be0c155d80ff0bff reported cstate change: term changed from 2 to 3, leader changed from 2537bbbd1a9947ae9facd87567e3ccb9 (127.17.60.193) to c561ec3374b94b95be0c155d80ff0bff (127.17.60.194). New cstate: current_term: 3 leader_uuid: "c561ec3374b94b95be0c155d80ff0bff" committed_config { opid_index: 26 OBSOLETE_local: false peers { permanent_uuid: "2537bbbd1a9947ae9facd87567e3ccb9" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 33173 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c561ec3374b94b95be0c155d80ff0bff" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 33627 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3710b74ed2044ce5995533df32e67351" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 38589 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b5e9178a7a83402e8c4a6943ab78589d" member_type: NON_VOTER last_known_addr { host: "127.17.60.197" port: 36573 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20250112 15:07:40.633111 21222 raft_consensus.cc:1270] T ef90b704208140a3983ffe6105939319 P 3710b74ed2044ce5995533df32e67351 [term 4 FOLLOWER]: Refusing update from remote peer 2537bbbd1a9947ae9facd87567e3ccb9: Log matching property violated. Preceding OpId in replica: term: 3 index: 27. Preceding OpId from leader: term: 4 index: 28. (index mismatch)
I20250112 15:07:40.633374 21977 consensus_queue.cc:1035] T ef90b704208140a3983ffe6105939319 P 2537bbbd1a9947ae9facd87567e3ccb9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3710b74ed2044ce5995533df32e67351" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 38589 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 28, Last known committed idx: 27, Time since last communication: 0.000s
I20250112 15:07:40.634577 21147 raft_consensus.cc:1270] T ef90b704208140a3983ffe6105939319 P c561ec3374b94b95be0c155d80ff0bff [term 4 FOLLOWER]: Refusing update from remote peer 2537bbbd1a9947ae9facd87567e3ccb9: Log matching property violated. Preceding OpId in replica: term: 3 index: 27. Preceding OpId from leader: term: 4 index: 28. (index mismatch)
I20250112 15:07:40.634790 21977 consensus_queue.cc:1035] T ef90b704208140a3983ffe6105939319 P 2537bbbd1a9947ae9facd87567e3ccb9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c561ec3374b94b95be0c155d80ff0bff" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 33627 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 28, Last known committed idx: 27, Time since last communication: 0.000s
I20250112 15:07:40.759639 21147 raft_consensus.cc:1270] T 80cc505a04394c53a18c5beeded4655d P c561ec3374b94b95be0c155d80ff0bff [term 2 FOLLOWER]: Refusing update from remote peer 3710b74ed2044ce5995533df32e67351: Log matching property violated. Preceding OpId in replica: term: 1 index: 28. Preceding OpId from leader: term: 2 index: 29. (index mismatch)
I20250112 15:07:40.759928 21979 consensus_queue.cc:1035] T 80cc505a04394c53a18c5beeded4655d P 3710b74ed2044ce5995533df32e67351 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c561ec3374b94b95be0c155d80ff0bff" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 33627 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.000s
I20250112 15:07:40.761119 21624 raft_consensus.cc:1270] T 80cc505a04394c53a18c5beeded4655d P 25286c485fa04c86a5d07c8f88643a47 [term 2 FOLLOWER]: Refusing update from remote peer 3710b74ed2044ce5995533df32e67351: Log matching property violated. Preceding OpId in replica: term: 1 index: 28. Preceding OpId from leader: term: 2 index: 29. (index mismatch)
I20250112 15:07:40.761274 21914 consensus_queue.cc:1035] T 80cc505a04394c53a18c5beeded4655d P 3710b74ed2044ce5995533df32e67351 [LEADER]: Connected to new peer: Peer: permanent_uuid: "25286c485fa04c86a5d07c8f88643a47" member_type: VOTER last_known_addr { host: "127.17.60.198" port: 37089 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.000s
I20250112 15:07:40.988256 21069 raft_consensus.cc:1270] T 9d4bbba799324d7fbbac082520c65686 P 2537bbbd1a9947ae9facd87567e3ccb9 [term 3 FOLLOWER]: Refusing update from remote peer c561ec3374b94b95be0c155d80ff0bff: Log matching property violated. Preceding OpId in replica: term: 2 index: 26. Preceding OpId from leader: term: 3 index: 27. (index mismatch)
I20250112 15:07:40.988461 21935 consensus_queue.cc:1035] T 9d4bbba799324d7fbbac082520c65686 P c561ec3374b94b95be0c155d80ff0bff [LEADER]: Connected to new peer: Peer: permanent_uuid: "2537bbbd1a9947ae9facd87567e3ccb9" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 33173 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 27, Last known committed idx: 26, Time since last communication: 0.000s
I20250112 15:07:40.989552 21226 raft_consensus.cc:1270] T 9d4bbba799324d7fbbac082520c65686 P 3710b74ed2044ce5995533df32e67351 [term 3 FOLLOWER]: Refusing update from remote peer c561ec3374b94b95be0c155d80ff0bff: Log matching property violated. Preceding OpId in replica: term: 2 index: 26. Preceding OpId from leader: term: 3 index: 27. (index mismatch)
I20250112 15:07:40.989779 21976 consensus_queue.cc:1035] T 9d4bbba799324d7fbbac082520c65686 P c561ec3374b94b95be0c155d80ff0bff [LEADER]: Connected to new peer: Peer: permanent_uuid: "3710b74ed2044ce5995533df32e67351" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 38589 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 27, Last known committed idx: 26, Time since last communication: 0.000s
W20250112 15:07:40.989955 21114 consensus_peers.cc:487] T 9d4bbba799324d7fbbac082520c65686 P c561ec3374b94b95be0c155d80ff0bff -> Peer b5e9178a7a83402e8c4a6943ab78589d (127.17.60.197:36573): Couldn't send request to peer b5e9178a7a83402e8c4a6943ab78589d. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20250112 15:07:42.020226 21954 tablet_copy_client.cc:538] T 8c7dbaf285a04da9b4eec92a29ab7d6c P a2a5bba4ac1145d5bab94a69c9f4df40: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250112 15:07:42.021219 21954 tablet_bootstrap.cc:492] T 8c7dbaf285a04da9b4eec92a29ab7d6c P a2a5bba4ac1145d5bab94a69c9f4df40: Bootstrap starting.
I20250112 15:07:42.024426 21954 tablet_bootstrap.cc:492] T 8c7dbaf285a04da9b4eec92a29ab7d6c P a2a5bba4ac1145d5bab94a69c9f4df40: Bootstrap replayed 1/1 log segments. Stats: ops{read=28 overwritten=0 applied=28 ignored=0} inserts{seen=105 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250112 15:07:42.024621 21954 tablet_bootstrap.cc:492] T 8c7dbaf285a04da9b4eec92a29ab7d6c P a2a5bba4ac1145d5bab94a69c9f4df40: Bootstrap complete.
I20250112 15:07:42.024755 21954 ts_tablet_manager.cc:1397] T 8c7dbaf285a04da9b4eec92a29ab7d6c P a2a5bba4ac1145d5bab94a69c9f4df40: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.002s
I20250112 15:07:42.024914 21954 raft_consensus.cc:357] T 8c7dbaf285a04da9b4eec92a29ab7d6c P a2a5bba4ac1145d5bab94a69c9f4df40 [term 3 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "2537bbbd1a9947ae9facd87567e3ccb9" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 33173 } attrs { replace: true } } peers { permanent_uuid: "c561ec3374b94b95be0c155d80ff0bff" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 33627 } } peers { permanent_uuid: "3710b74ed2044ce5995533df32e67351" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 38589 } } peers { permanent_uuid: "a2a5bba4ac1145d5bab94a69c9f4df40" member_type: NON_VOTER last_known_addr { host: "127.17.60.196" port: 41807 } attrs { promote: true } }
I20250112 15:07:42.024983 21954 raft_consensus.cc:738] T 8c7dbaf285a04da9b4eec92a29ab7d6c P a2a5bba4ac1145d5bab94a69c9f4df40 [term 3 LEARNER]: Becoming Follower/Learner. State: Replica: a2a5bba4ac1145d5bab94a69c9f4df40, State: Initialized, Role: LEARNER
I20250112 15:07:42.025017 21954 consensus_queue.cc:260] T 8c7dbaf285a04da9b4eec92a29ab7d6c P a2a5bba4ac1145d5bab94a69c9f4df40 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 28, Last appended: 3.28, Last appended by leader: 28, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "2537bbbd1a9947ae9facd87567e3ccb9" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 33173 } attrs { replace: true } } peers { permanent_uuid: "c561ec3374b94b95be0c155d80ff0bff" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 33627 } } peers { permanent_uuid: "3710b74ed2044ce5995533df32e67351" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 38589 } } peers { permanent_uuid: "a2a5bba4ac1145d5bab94a69c9f4df40" member_type: NON_VOTER last_known_addr { host: "127.17.60.196" port: 41807 } attrs { promote: true } }
I20250112 15:07:42.025259 21954 ts_tablet_manager.cc:1428] T 8c7dbaf285a04da9b4eec92a29ab7d6c P a2a5bba4ac1145d5bab94a69c9f4df40: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20250112 15:07:42.025442 21161 tablet_copy_service.cc:342] P c561ec3374b94b95be0c155d80ff0bff: Request end of tablet copy session a2a5bba4ac1145d5bab94a69c9f4df40-8c7dbaf285a04da9b4eec92a29ab7d6c received from {username='slave'} at 127.0.0.1:58522
I20250112 15:07:42.025514 21161 tablet_copy_service.cc:434] P c561ec3374b94b95be0c155d80ff0bff: ending tablet copy session a2a5bba4ac1145d5bab94a69c9f4df40-8c7dbaf285a04da9b4eec92a29ab7d6c on tablet 8c7dbaf285a04da9b4eec92a29ab7d6c with peer a2a5bba4ac1145d5bab94a69c9f4df40
I20250112 15:07:42.036835 21956 tablet_copy_client.cc:538] T bfaeae584ec04772b2d6fcd088022117 P 25286c485fa04c86a5d07c8f88643a47: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250112 15:07:42.037634 21956 tablet_bootstrap.cc:492] T bfaeae584ec04772b2d6fcd088022117 P 25286c485fa04c86a5d07c8f88643a47: Bootstrap starting.
I20250112 15:07:42.040938 21956 tablet_bootstrap.cc:492] T bfaeae584ec04772b2d6fcd088022117 P 25286c485fa04c86a5d07c8f88643a47: Bootstrap replayed 1/1 log segments. Stats: ops{read=27 overwritten=0 applied=27 ignored=0} inserts{seen=100 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250112 15:07:42.041107 21956 tablet_bootstrap.cc:492] T bfaeae584ec04772b2d6fcd088022117 P 25286c485fa04c86a5d07c8f88643a47: Bootstrap complete.
I20250112 15:07:42.041196 21956 ts_tablet_manager.cc:1397] T bfaeae584ec04772b2d6fcd088022117 P 25286c485fa04c86a5d07c8f88643a47: Time spent bootstrapping tablet: real 0.004s user 0.004s sys 0.000s
I20250112 15:07:42.041344 21956 raft_consensus.cc:357] T bfaeae584ec04772b2d6fcd088022117 P 25286c485fa04c86a5d07c8f88643a47 [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "2537bbbd1a9947ae9facd87567e3ccb9" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 33173 } } peers { permanent_uuid: "c561ec3374b94b95be0c155d80ff0bff" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 33627 } } peers { permanent_uuid: "3710b74ed2044ce5995533df32e67351" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 38589 } attrs { replace: true } } peers { permanent_uuid: "25286c485fa04c86a5d07c8f88643a47" member_type: NON_VOTER last_known_addr { host: "127.17.60.198" port: 37089 } attrs { promote: true } }
I20250112 15:07:42.041410 21956 raft_consensus.cc:738] T bfaeae584ec04772b2d6fcd088022117 P 25286c485fa04c86a5d07c8f88643a47 [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: 25286c485fa04c86a5d07c8f88643a47, State: Initialized, Role: LEARNER
I20250112 15:07:42.041440 21956 consensus_queue.cc:260] T bfaeae584ec04772b2d6fcd088022117 P 25286c485fa04c86a5d07c8f88643a47 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 27, Last appended: 2.27, Last appended by leader: 27, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "2537bbbd1a9947ae9facd87567e3ccb9" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 33173 } } peers { permanent_uuid: "c561ec3374b94b95be0c155d80ff0bff" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 33627 } } peers { permanent_uuid: "3710b74ed2044ce5995533df32e67351" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 38589 } attrs { replace: true } } peers { permanent_uuid: "25286c485fa04c86a5d07c8f88643a47" member_type: NON_VOTER last_known_addr { host: "127.17.60.198" port: 37089 } attrs { promote: true } }
I20250112 15:07:42.041546 21956 ts_tablet_manager.cc:1428] T bfaeae584ec04772b2d6fcd088022117 P 25286c485fa04c86a5d07c8f88643a47: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20250112 15:07:42.041751 21085 tablet_copy_service.cc:342] P 2537bbbd1a9947ae9facd87567e3ccb9: Request end of tablet copy session 25286c485fa04c86a5d07c8f88643a47-bfaeae584ec04772b2d6fcd088022117 received from {username='slave'} at 127.0.0.1:54434
I20250112 15:07:42.041793 21085 tablet_copy_service.cc:434] P 2537bbbd1a9947ae9facd87567e3ccb9: ending tablet copy session 25286c485fa04c86a5d07c8f88643a47-bfaeae584ec04772b2d6fcd088022117 on tablet bfaeae584ec04772b2d6fcd088022117 with peer 25286c485fa04c86a5d07c8f88643a47
I20250112 15:07:42.066434 21958 tablet_copy_client.cc:538] T 16f16fcd5cf347e5853189dc4da55c7f P 25286c485fa04c86a5d07c8f88643a47: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250112 15:07:42.067224 21958 tablet_bootstrap.cc:492] T 16f16fcd5cf347e5853189dc4da55c7f P 25286c485fa04c86a5d07c8f88643a47: Bootstrap starting.
I20250112 15:07:42.070504 21958 tablet_bootstrap.cc:492] T 16f16fcd5cf347e5853189dc4da55c7f P 25286c485fa04c86a5d07c8f88643a47: Bootstrap replayed 1/1 log segments. Stats: ops{read=27 overwritten=0 applied=27 ignored=0} inserts{seen=120 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250112 15:07:42.070735 21958 tablet_bootstrap.cc:492] T 16f16fcd5cf347e5853189dc4da55c7f P 25286c485fa04c86a5d07c8f88643a47: Bootstrap complete.
I20250112 15:07:42.070849 21958 ts_tablet_manager.cc:1397] T 16f16fcd5cf347e5853189dc4da55c7f P 25286c485fa04c86a5d07c8f88643a47: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.002s
I20250112 15:07:42.071000 21958 raft_consensus.cc:357] T 16f16fcd5cf347e5853189dc4da55c7f P 25286c485fa04c86a5d07c8f88643a47 [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "2537bbbd1a9947ae9facd87567e3ccb9" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 33173 } } peers { permanent_uuid: "c561ec3374b94b95be0c155d80ff0bff" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 33627 } attrs { replace: true } } peers { permanent_uuid: "3710b74ed2044ce5995533df32e67351" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 38589 } } peers { permanent_uuid: "25286c485fa04c86a5d07c8f88643a47" member_type: NON_VOTER last_known_addr { host: "127.17.60.198" port: 37089 } attrs { promote: true } }
I20250112 15:07:42.071089 21958 raft_consensus.cc:738] T 16f16fcd5cf347e5853189dc4da55c7f P 25286c485fa04c86a5d07c8f88643a47 [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: 25286c485fa04c86a5d07c8f88643a47, State: Initialized, Role: LEARNER
I20250112 15:07:42.071146 21958 consensus_queue.cc:260] T 16f16fcd5cf347e5853189dc4da55c7f P 25286c485fa04c86a5d07c8f88643a47 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 27, Last appended: 2.27, Last appended by leader: 27, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "2537bbbd1a9947ae9facd87567e3ccb9" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 33173 } } peers { permanent_uuid: "c561ec3374b94b95be0c155d80ff0bff" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 33627 } attrs { replace: true } } peers { permanent_uuid: "3710b74ed2044ce5995533df32e67351" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 38589 } } peers { permanent_uuid: "25286c485fa04c86a5d07c8f88643a47" member_type: NON_VOTER last_known_addr { host: "127.17.60.198" port: 37089 } attrs { promote: true } }
I20250112 15:07:42.071406 21958 ts_tablet_manager.cc:1428] T 16f16fcd5cf347e5853189dc4da55c7f P 25286c485fa04c86a5d07c8f88643a47: Time spent starting tablet: real 0.001s user 0.003s sys 0.000s
I20250112 15:07:42.071566 21085 tablet_copy_service.cc:342] P 2537bbbd1a9947ae9facd87567e3ccb9: Request end of tablet copy session 25286c485fa04c86a5d07c8f88643a47-16f16fcd5cf347e5853189dc4da55c7f received from {username='slave'} at 127.0.0.1:54434
I20250112 15:07:42.071614 21085 tablet_copy_service.cc:434] P 2537bbbd1a9947ae9facd87567e3ccb9: ending tablet copy session 25286c485fa04c86a5d07c8f88643a47-16f16fcd5cf347e5853189dc4da55c7f on tablet 16f16fcd5cf347e5853189dc4da55c7f with peer 25286c485fa04c86a5d07c8f88643a47
I20250112 15:07:42.073612 21960 tablet_copy_client.cc:538] T 9d4bbba799324d7fbbac082520c65686 P b5e9178a7a83402e8c4a6943ab78589d: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250112 15:07:42.074262 21960 tablet_bootstrap.cc:492] T 9d4bbba799324d7fbbac082520c65686 P b5e9178a7a83402e8c4a6943ab78589d: Bootstrap starting.
I20250112 15:07:42.077638 21960 tablet_bootstrap.cc:492] T 9d4bbba799324d7fbbac082520c65686 P b5e9178a7a83402e8c4a6943ab78589d: Bootstrap replayed 1/1 log segments. Stats: ops{read=26 overwritten=0 applied=26 ignored=0} inserts{seen=87 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250112 15:07:42.077823 21960 tablet_bootstrap.cc:492] T 9d4bbba799324d7fbbac082520c65686 P b5e9178a7a83402e8c4a6943ab78589d: Bootstrap complete.
I20250112 15:07:42.077965 21960 ts_tablet_manager.cc:1397] T 9d4bbba799324d7fbbac082520c65686 P b5e9178a7a83402e8c4a6943ab78589d: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.002s
I20250112 15:07:42.078146 21960 raft_consensus.cc:357] T 9d4bbba799324d7fbbac082520c65686 P b5e9178a7a83402e8c4a6943ab78589d [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 26 OBSOLETE_local: false peers { permanent_uuid: "2537bbbd1a9947ae9facd87567e3ccb9" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 33173 } } peers { permanent_uuid: "c561ec3374b94b95be0c155d80ff0bff" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 33627 } } peers { permanent_uuid: "3710b74ed2044ce5995533df32e67351" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 38589 } attrs { replace: true } } peers { permanent_uuid: "b5e9178a7a83402e8c4a6943ab78589d" member_type: NON_VOTER last_known_addr { host: "127.17.60.197" port: 36573 } attrs { promote: true } }
I20250112 15:07:42.078230 21960 raft_consensus.cc:738] T 9d4bbba799324d7fbbac082520c65686 P b5e9178a7a83402e8c4a6943ab78589d [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: b5e9178a7a83402e8c4a6943ab78589d, State: Initialized, Role: LEARNER
I20250112 15:07:42.078269 21960 consensus_queue.cc:260] T 9d4bbba799324d7fbbac082520c65686 P b5e9178a7a83402e8c4a6943ab78589d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 26, Last appended: 2.26, Last appended by leader: 26, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 26 OBSOLETE_local: false peers { permanent_uuid: "2537bbbd1a9947ae9facd87567e3ccb9" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 33173 } } peers { permanent_uuid: "c561ec3374b94b95be0c155d80ff0bff" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 33627 } } peers { permanent_uuid: "3710b74ed2044ce5995533df32e67351" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 38589 } attrs { replace: true } } peers { permanent_uuid: "b5e9178a7a83402e8c4a6943ab78589d" member_type: NON_VOTER last_known_addr { host: "127.17.60.197" port: 36573 } attrs { promote: true } }
I20250112 15:07:42.078356 21960 ts_tablet_manager.cc:1428] T 9d4bbba799324d7fbbac082520c65686 P b5e9178a7a83402e8c4a6943ab78589d: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20250112 15:07:42.078522 21085 tablet_copy_service.cc:342] P 2537bbbd1a9947ae9facd87567e3ccb9: Request end of tablet copy session b5e9178a7a83402e8c4a6943ab78589d-9d4bbba799324d7fbbac082520c65686 received from {username='slave'} at 127.0.0.1:54448
I20250112 15:07:42.078567 21085 tablet_copy_service.cc:434] P 2537bbbd1a9947ae9facd87567e3ccb9: ending tablet copy session b5e9178a7a83402e8c4a6943ab78589d-9d4bbba799324d7fbbac082520c65686 on tablet 9d4bbba799324d7fbbac082520c65686 with peer b5e9178a7a83402e8c4a6943ab78589d
I20250112 15:07:42.080143 21549 raft_consensus.cc:3054] T 9d4bbba799324d7fbbac082520c65686 P b5e9178a7a83402e8c4a6943ab78589d [term 2 LEARNER]: Advancing to term 3
I20250112 15:07:42.092270 21026 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250112 15:07:42.092636 21147 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "8c7dbaf285a04da9b4eec92a29ab7d6c"
dest_uuid: "c561ec3374b94b95be0c155d80ff0bff"
mode: GRACEFUL
new_leader_uuid: "3710b74ed2044ce5995533df32e67351"
from {username='slave'} at 127.0.0.1:48104
I20250112 15:07:42.092715 21147 raft_consensus.cc:604] T 8c7dbaf285a04da9b4eec92a29ab7d6c P c561ec3374b94b95be0c155d80ff0bff [term 3 LEADER]: Received request to transfer leadership to TS 3710b74ed2044ce5995533df32e67351
I20250112 15:07:42.092980 21226 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "9601bcda24ed42e7b93ab9740f1e271e"
dest_uuid: "3710b74ed2044ce5995533df32e67351"
mode: GRACEFUL
new_leader_uuid: "2537bbbd1a9947ae9facd87567e3ccb9"
from {username='slave'} at 127.0.0.1:54970
I20250112 15:07:42.093055 21226 raft_consensus.cc:604] T 9601bcda24ed42e7b93ab9740f1e271e P 3710b74ed2044ce5995533df32e67351 [term 2 LEADER]: Received request to transfer leadership to TS 2537bbbd1a9947ae9facd87567e3ccb9
I20250112 15:07:42.093312 21069 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "bfaeae584ec04772b2d6fcd088022117"
dest_uuid: "2537bbbd1a9947ae9facd87567e3ccb9"
mode: GRACEFUL
new_leader_uuid: "c561ec3374b94b95be0c155d80ff0bff"
from {username='slave'} at 127.0.0.1:54414
I20250112 15:07:42.093366 21069 raft_consensus.cc:604] T bfaeae584ec04772b2d6fcd088022117 P 2537bbbd1a9947ae9facd87567e3ccb9 [term 2 LEADER]: Received request to transfer leadership to TS c561ec3374b94b95be0c155d80ff0bff
I20250112 15:07:42.093495 21026 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 3
I20250112 15:07:42.093550 21026 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20250112 15:07:42.124364 21962 tablet_copy_client.cc:538] T 57e47b07fdcf434e96ca5dd0354e93b2 P a2a5bba4ac1145d5bab94a69c9f4df40: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250112 15:07:42.125389 21962 tablet_bootstrap.cc:492] T 57e47b07fdcf434e96ca5dd0354e93b2 P a2a5bba4ac1145d5bab94a69c9f4df40: Bootstrap starting.
I20250112 15:07:42.129746 21962 tablet_bootstrap.cc:492] T 57e47b07fdcf434e96ca5dd0354e93b2 P a2a5bba4ac1145d5bab94a69c9f4df40: Bootstrap replayed 1/1 log segments. Stats: ops{read=31 overwritten=0 applied=31 ignored=0} inserts{seen=83 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250112 15:07:42.129966 21962 tablet_bootstrap.cc:492] T 57e47b07fdcf434e96ca5dd0354e93b2 P a2a5bba4ac1145d5bab94a69c9f4df40: Bootstrap complete.
I20250112 15:07:42.130090 21962 ts_tablet_manager.cc:1397] T 57e47b07fdcf434e96ca5dd0354e93b2 P a2a5bba4ac1145d5bab94a69c9f4df40: Time spent bootstrapping tablet: real 0.005s user 0.005s sys 0.000s
I20250112 15:07:42.130250 21962 raft_consensus.cc:357] T 57e47b07fdcf434e96ca5dd0354e93b2 P a2a5bba4ac1145d5bab94a69c9f4df40 [term 3 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "2537bbbd1a9947ae9facd87567e3ccb9" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 33173 } } peers { permanent_uuid: "c561ec3374b94b95be0c155d80ff0bff" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 33627 } attrs { replace: true } } peers { permanent_uuid: "b5e9178a7a83402e8c4a6943ab78589d" member_type: VOTER last_known_addr { host: "127.17.60.197" port: 36573 } attrs { promote: false } } peers { permanent_uuid: "a2a5bba4ac1145d5bab94a69c9f4df40" member_type: NON_VOTER last_known_addr { host: "127.17.60.196" port: 41807 } attrs { promote: true } }
I20250112 15:07:42.130365 21962 raft_consensus.cc:738] T 57e47b07fdcf434e96ca5dd0354e93b2 P a2a5bba4ac1145d5bab94a69c9f4df40 [term 3 LEARNER]: Becoming Follower/Learner. State: Replica: a2a5bba4ac1145d5bab94a69c9f4df40, State: Initialized, Role: LEARNER
I20250112 15:07:42.130409 21962 consensus_queue.cc:260] T 57e47b07fdcf434e96ca5dd0354e93b2 P a2a5bba4ac1145d5bab94a69c9f4df40 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 31, Last appended: 3.31, Last appended by leader: 31, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "2537bbbd1a9947ae9facd87567e3ccb9" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 33173 } } peers { permanent_uuid: "c561ec3374b94b95be0c155d80ff0bff" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 33627 } attrs { replace: true } } peers { permanent_uuid: "b5e9178a7a83402e8c4a6943ab78589d" member_type: VOTER last_known_addr { host: "127.17.60.197" port: 36573 } attrs { promote: false } } peers { permanent_uuid: "a2a5bba4ac1145d5bab94a69c9f4df40" member_type: NON_VOTER last_known_addr { host: "127.17.60.196" port: 41807 } attrs { promote: true } }
I20250112 15:07:42.130517 21962 ts_tablet_manager.cc:1428] T 57e47b07fdcf434e96ca5dd0354e93b2 P a2a5bba4ac1145d5bab94a69c9f4df40: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20250112 15:07:42.130678 21559 tablet_copy_service.cc:342] P b5e9178a7a83402e8c4a6943ab78589d: Request end of tablet copy session a2a5bba4ac1145d5bab94a69c9f4df40-57e47b07fdcf434e96ca5dd0354e93b2 received from {username='slave'} at 127.0.0.1:44706
I20250112 15:07:42.130722 21559 tablet_copy_service.cc:434] P b5e9178a7a83402e8c4a6943ab78589d: ending tablet copy session a2a5bba4ac1145d5bab94a69c9f4df40-57e47b07fdcf434e96ca5dd0354e93b2 on tablet 57e47b07fdcf434e96ca5dd0354e93b2 with peer a2a5bba4ac1145d5bab94a69c9f4df40
I20250112 15:07:42.168957 21624 raft_consensus.cc:1212] T bfaeae584ec04772b2d6fcd088022117 P 25286c485fa04c86a5d07c8f88643a47 [term 2 LEARNER]: Deduplicated request from leader. Original: 2.26->[2.27-2.27] Dedup: 2.27->[]
I20250112 15:07:42.179970 21964 tablet_copy_client.cc:538] T 9601bcda24ed42e7b93ab9740f1e271e P 25286c485fa04c86a5d07c8f88643a47: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250112 15:07:42.181049 21964 tablet_bootstrap.cc:492] T 9601bcda24ed42e7b93ab9740f1e271e P 25286c485fa04c86a5d07c8f88643a47: Bootstrap starting.
I20250112 15:07:42.184967 21964 tablet_bootstrap.cc:492] T 9601bcda24ed42e7b93ab9740f1e271e P 25286c485fa04c86a5d07c8f88643a47: Bootstrap replayed 1/1 log segments. Stats: ops{read=29 overwritten=0 applied=29 ignored=0} inserts{seen=96 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250112 15:07:42.185199 21964 tablet_bootstrap.cc:492] T 9601bcda24ed42e7b93ab9740f1e271e P 25286c485fa04c86a5d07c8f88643a47: Bootstrap complete.
I20250112 15:07:42.185313 21964 ts_tablet_manager.cc:1397] T 9601bcda24ed42e7b93ab9740f1e271e P 25286c485fa04c86a5d07c8f88643a47: Time spent bootstrapping tablet: real 0.004s user 0.004s sys 0.000s
I20250112 15:07:42.185503 21964 raft_consensus.cc:357] T 9601bcda24ed42e7b93ab9740f1e271e P 25286c485fa04c86a5d07c8f88643a47 [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "2537bbbd1a9947ae9facd87567e3ccb9" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 33173 } attrs { replace: true } } peers { permanent_uuid: "3710b74ed2044ce5995533df32e67351" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 38589 } } peers { permanent_uuid: "a2a5bba4ac1145d5bab94a69c9f4df40" member_type: VOTER last_known_addr { host: "127.17.60.196" port: 41807 } attrs { promote: false } } peers { permanent_uuid: "25286c485fa04c86a5d07c8f88643a47" member_type: NON_VOTER last_known_addr { host: "127.17.60.198" port: 37089 } attrs { promote: true } }
I20250112 15:07:42.185593 21964 raft_consensus.cc:738] T 9601bcda24ed42e7b93ab9740f1e271e P 25286c485fa04c86a5d07c8f88643a47 [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: 25286c485fa04c86a5d07c8f88643a47, State: Initialized, Role: LEARNER
I20250112 15:07:42.185623 21964 consensus_queue.cc:260] T 9601bcda24ed42e7b93ab9740f1e271e P 25286c485fa04c86a5d07c8f88643a47 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 29, Last appended: 2.29, Last appended by leader: 29, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "2537bbbd1a9947ae9facd87567e3ccb9" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 33173 } attrs { replace: true } } peers { permanent_uuid: "3710b74ed2044ce5995533df32e67351" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 38589 } } peers { permanent_uuid: "a2a5bba4ac1145d5bab94a69c9f4df40" member_type: VOTER last_known_addr { host: "127.17.60.196" port: 41807 } attrs { promote: false } } peers { permanent_uuid: "25286c485fa04c86a5d07c8f88643a47" member_type: NON_VOTER last_known_addr { host: "127.17.60.198" port: 37089 } attrs { promote: true } }
I20250112 15:07:42.185741 21964 ts_tablet_manager.cc:1428] T 9601bcda24ed42e7b93ab9740f1e271e P 25286c485fa04c86a5d07c8f88643a47: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20250112 15:07:42.185949 21236 tablet_copy_service.cc:342] P 3710b74ed2044ce5995533df32e67351: Request end of tablet copy session 25286c485fa04c86a5d07c8f88643a47-9601bcda24ed42e7b93ab9740f1e271e received from {username='slave'} at 127.0.0.1:52200
I20250112 15:07:42.185995 21236 tablet_copy_service.cc:434] P 3710b74ed2044ce5995533df32e67351: ending tablet copy session 25286c485fa04c86a5d07c8f88643a47-9601bcda24ed42e7b93ab9740f1e271e on tablet 9601bcda24ed42e7b93ab9740f1e271e with peer 25286c485fa04c86a5d07c8f88643a47
I20250112 15:07:42.267644 21624 raft_consensus.cc:1212] T 9601bcda24ed42e7b93ab9740f1e271e P 25286c485fa04c86a5d07c8f88643a47 [term 2 LEARNER]: Deduplicated request from leader. Original: 2.28->[2.29-2.29] Dedup: 2.29->[]
I20250112 15:07:42.336033 21624 raft_consensus.cc:1212] T 16f16fcd5cf347e5853189dc4da55c7f P 25286c485fa04c86a5d07c8f88643a47 [term 2 LEARNER]: Deduplicated request from leader. Original: 2.26->[2.27-2.27] Dedup: 2.27->[]
I20250112 15:07:42.369326 21474 raft_consensus.cc:1212] T 57e47b07fdcf434e96ca5dd0354e93b2 P a2a5bba4ac1145d5bab94a69c9f4df40 [term 3 LEARNER]: Deduplicated request from leader. Original: 3.30->[3.31-3.31] Dedup: 3.31->[]
I20250112 15:07:42.378461 21474 raft_consensus.cc:1212] T 8c7dbaf285a04da9b4eec92a29ab7d6c P a2a5bba4ac1145d5bab94a69c9f4df40 [term 3 LEARNER]: Deduplicated request from leader. Original: 3.27->[3.28-3.28] Dedup: 3.28->[]
I20250112 15:07:42.496908 21935 raft_consensus.cc:988] T 8c7dbaf285a04da9b4eec92a29ab7d6c P c561ec3374b94b95be0c155d80ff0bff: : Instructing follower 3710b74ed2044ce5995533df32e67351 to start an election
I20250112 15:07:42.497000 21935 raft_consensus.cc:1076] T 8c7dbaf285a04da9b4eec92a29ab7d6c P c561ec3374b94b95be0c155d80ff0bff [term 3 LEADER]: Signalling peer 3710b74ed2044ce5995533df32e67351 to start an election
I20250112 15:07:42.497246 21226 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "8c7dbaf285a04da9b4eec92a29ab7d6c"
dest_uuid: "3710b74ed2044ce5995533df32e67351"
from {username='slave'} at 127.0.0.1:33202
I20250112 15:07:42.497334 21226 raft_consensus.cc:491] T 8c7dbaf285a04da9b4eec92a29ab7d6c P 3710b74ed2044ce5995533df32e67351 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20250112 15:07:42.497364 21226 raft_consensus.cc:3054] T 8c7dbaf285a04da9b4eec92a29ab7d6c P 3710b74ed2044ce5995533df32e67351 [term 3 FOLLOWER]: Advancing to term 4
I20250112 15:07:42.497973 21226 raft_consensus.cc:513] T 8c7dbaf285a04da9b4eec92a29ab7d6c P 3710b74ed2044ce5995533df32e67351 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "2537bbbd1a9947ae9facd87567e3ccb9" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 33173 } attrs { replace: true } } peers { permanent_uuid: "c561ec3374b94b95be0c155d80ff0bff" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 33627 } } peers { permanent_uuid: "3710b74ed2044ce5995533df32e67351" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 38589 } } peers { permanent_uuid: "a2a5bba4ac1145d5bab94a69c9f4df40" member_type: NON_VOTER last_known_addr { host: "127.17.60.196" port: 41807 } attrs { promote: true } }
I20250112 15:07:42.498122 21226 leader_election.cc:290] T 8c7dbaf285a04da9b4eec92a29ab7d6c P 3710b74ed2044ce5995533df32e67351 [CANDIDATE]: Term 4 election: Requested vote from peers 2537bbbd1a9947ae9facd87567e3ccb9 (127.17.60.193:33173), c561ec3374b94b95be0c155d80ff0bff (127.17.60.194:33627)
I20250112 15:07:42.498359 21069 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c7dbaf285a04da9b4eec92a29ab7d6c" candidate_uuid: "3710b74ed2044ce5995533df32e67351" candidate_term: 4 candidate_status { last_received { term: 3 index: 28 } } ignore_live_leader: true dest_uuid: "2537bbbd1a9947ae9facd87567e3ccb9"
I20250112 15:07:42.498452 21069 raft_consensus.cc:3054] T 8c7dbaf285a04da9b4eec92a29ab7d6c P 2537bbbd1a9947ae9facd87567e3ccb9 [term 3 FOLLOWER]: Advancing to term 4
I20250112 15:07:42.498359 21149 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c7dbaf285a04da9b4eec92a29ab7d6c" candidate_uuid: "3710b74ed2044ce5995533df32e67351" candidate_term: 4 candidate_status { last_received { term: 3 index: 28 } } ignore_live_leader: true dest_uuid: "c561ec3374b94b95be0c155d80ff0bff"
I20250112 15:07:42.498495 21149 raft_consensus.cc:3049] T 8c7dbaf285a04da9b4eec92a29ab7d6c P c561ec3374b94b95be0c155d80ff0bff [term 3 LEADER]: Stepping down as leader of term 3
I20250112 15:07:42.498519 21149 raft_consensus.cc:738] T 8c7dbaf285a04da9b4eec92a29ab7d6c P c561ec3374b94b95be0c155d80ff0bff [term 3 LEADER]: Becoming Follower/Learner. State: Replica: c561ec3374b94b95be0c155d80ff0bff, State: Running, Role: LEADER
I20250112 15:07:42.498570 21149 consensus_queue.cc:260] T 8c7dbaf285a04da9b4eec92a29ab7d6c P c561ec3374b94b95be0c155d80ff0bff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 28, Committed index: 28, Last appended: 3.28, Last appended by leader: 28, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "2537bbbd1a9947ae9facd87567e3ccb9" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 33173 } attrs { replace: true } } peers { permanent_uuid: "c561ec3374b94b95be0c155d80ff0bff" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 33627 } } peers { permanent_uuid: "3710b74ed2044ce5995533df32e67351" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 38589 } } peers { permanent_uuid: "a2a5bba4ac1145d5bab94a69c9f4df40" member_type: NON_VOTER last_known_addr { host: "127.17.60.196" port: 41807 } attrs { promote: true } }
I20250112 15:07:42.498669 21149 raft_consensus.cc:3054] T 8c7dbaf285a04da9b4eec92a29ab7d6c P c561ec3374b94b95be0c155d80ff0bff [term 3 FOLLOWER]: Advancing to term 4
I20250112 15:07:42.499164 21069 raft_consensus.cc:2463] T 8c7dbaf285a04da9b4eec92a29ab7d6c P 2537bbbd1a9947ae9facd87567e3ccb9 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3710b74ed2044ce5995533df32e67351 in term 4.
I20250112 15:07:42.499300 21149 raft_consensus.cc:2463] T 8c7dbaf285a04da9b4eec92a29ab7d6c P c561ec3374b94b95be0c155d80ff0bff [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3710b74ed2044ce5995533df32e67351 in term 4.
I20250112 15:07:42.499351 21189 leader_election.cc:304] T 8c7dbaf285a04da9b4eec92a29ab7d6c P 3710b74ed2044ce5995533df32e67351 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2537bbbd1a9947ae9facd87567e3ccb9, 3710b74ed2044ce5995533df32e67351; no voters:
I20250112 15:07:42.499430 21914 raft_consensus.cc:2798] T 8c7dbaf285a04da9b4eec92a29ab7d6c P 3710b74ed2044ce5995533df32e67351 [term 4 FOLLOWER]: Leader election won for term 4
I20250112 15:07:42.499473 21914 raft_consensus.cc:695] T 8c7dbaf285a04da9b4eec92a29ab7d6c P 3710b74ed2044ce5995533df32e67351 [term 4 LEADER]: Becoming Leader. State: Replica: 3710b74ed2044ce5995533df32e67351, State: Running, Role: LEADER
I20250112 15:07:42.499527 21914 consensus_queue.cc:237] T 8c7dbaf285a04da9b4eec92a29ab7d6c P 3710b74ed2044ce5995533df32e67351 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 28, Committed index: 28, Last appended: 3.28, Last appended by leader: 28, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "2537bbbd1a9947ae9facd87567e3ccb9" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 33173 } attrs { replace: true } } peers { permanent_uuid: "c561ec3374b94b95be0c155d80ff0bff" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 33627 } } peers { permanent_uuid: "3710b74ed2044ce5995533df32e67351" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 38589 } } peers { permanent_uuid: "a2a5bba4ac1145d5bab94a69c9f4df40" member_type: NON_VOTER last_known_addr { host: "127.17.60.196" port: 41807 } attrs { promote: true } }
I20250112 15:07:42.500013 20966 catalog_manager.cc:5526] T 8c7dbaf285a04da9b4eec92a29ab7d6c P 3710b74ed2044ce5995533df32e67351 reported cstate change: term changed from 3 to 4, leader changed from c561ec3374b94b95be0c155d80ff0bff (127.17.60.194) to 3710b74ed2044ce5995533df32e67351 (127.17.60.195). New cstate: current_term: 4 leader_uuid: "3710b74ed2044ce5995533df32e67351" committed_config { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "2537bbbd1a9947ae9facd87567e3ccb9" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 33173 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c561ec3374b94b95be0c155d80ff0bff" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 33627 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3710b74ed2044ce5995533df32e67351" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 38589 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a2a5bba4ac1145d5bab94a69c9f4df40" member_type: NON_VOTER last_known_addr { host: "127.17.60.196" port: 41807 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20250112 15:07:42.546682 21969 raft_consensus.cc:988] T bfaeae584ec04772b2d6fcd088022117 P 2537bbbd1a9947ae9facd87567e3ccb9: : Instructing follower c561ec3374b94b95be0c155d80ff0bff to start an election
I20250112 15:07:42.546764 21969 raft_consensus.cc:1076] T bfaeae584ec04772b2d6fcd088022117 P 2537bbbd1a9947ae9facd87567e3ccb9 [term 2 LEADER]: Signalling peer c561ec3374b94b95be0c155d80ff0bff to start an election
I20250112 15:07:42.546921 21147 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "bfaeae584ec04772b2d6fcd088022117"
dest_uuid: "c561ec3374b94b95be0c155d80ff0bff"
from {username='slave'} at 127.0.0.1:48082
I20250112 15:07:42.546996 21147 raft_consensus.cc:491] T bfaeae584ec04772b2d6fcd088022117 P c561ec3374b94b95be0c155d80ff0bff [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20250112 15:07:42.547019 21147 raft_consensus.cc:3054] T bfaeae584ec04772b2d6fcd088022117 P c561ec3374b94b95be0c155d80ff0bff [term 2 FOLLOWER]: Advancing to term 3
I20250112 15:07:42.547821 21147 raft_consensus.cc:513] T bfaeae584ec04772b2d6fcd088022117 P c561ec3374b94b95be0c155d80ff0bff [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "2537bbbd1a9947ae9facd87567e3ccb9" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 33173 } } peers { permanent_uuid: "c561ec3374b94b95be0c155d80ff0bff" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 33627 } } peers { permanent_uuid: "3710b74ed2044ce5995533df32e67351" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 38589 } attrs { replace: true } } peers { permanent_uuid: "25286c485fa04c86a5d07c8f88643a47" member_type: NON_VOTER last_known_addr { host: "127.17.60.198" port: 37089 } attrs { promote: true } }
I20250112 15:07:42.547963 21147 leader_election.cc:290] T bfaeae584ec04772b2d6fcd088022117 P c561ec3374b94b95be0c155d80ff0bff [CANDIDATE]: Term 3 election: Requested vote from peers 2537bbbd1a9947ae9facd87567e3ccb9 (127.17.60.193:33173), 3710b74ed2044ce5995533df32e67351 (127.17.60.195:38589)
I20250112 15:07:42.548055 21069 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bfaeae584ec04772b2d6fcd088022117" candidate_uuid: "c561ec3374b94b95be0c155d80ff0bff" candidate_term: 3 candidate_status { last_received { term: 2 index: 27 } } ignore_live_leader: true dest_uuid: "2537bbbd1a9947ae9facd87567e3ccb9"
I20250112 15:07:42.548142 21069 raft_consensus.cc:3049] T bfaeae584ec04772b2d6fcd088022117 P 2537bbbd1a9947ae9facd87567e3ccb9 [term 2 LEADER]: Stepping down as leader of term 2
I20250112 15:07:42.548174 21069 raft_consensus.cc:738] T bfaeae584ec04772b2d6fcd088022117 P 2537bbbd1a9947ae9facd87567e3ccb9 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 2537bbbd1a9947ae9facd87567e3ccb9, State: Running, Role: LEADER
I20250112 15:07:42.548198 21222 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bfaeae584ec04772b2d6fcd088022117" candidate_uuid: "c561ec3374b94b95be0c155d80ff0bff" candidate_term: 3 candidate_status { last_received { term: 2 index: 27 } } ignore_live_leader: true dest_uuid: "3710b74ed2044ce5995533df32e67351"
I20250112 15:07:42.548310 21222 raft_consensus.cc:3054] T bfaeae584ec04772b2d6fcd088022117 P 3710b74ed2044ce5995533df32e67351 [term 2 FOLLOWER]: Advancing to term 3
I20250112 15:07:42.548285 21069 consensus_queue.cc:260] T bfaeae584ec04772b2d6fcd088022117 P 2537bbbd1a9947ae9facd87567e3ccb9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 27, Committed index: 27, Last appended: 2.27, Last appended by leader: 27, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "2537bbbd1a9947ae9facd87567e3ccb9" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 33173 } } peers { permanent_uuid: "c561ec3374b94b95be0c155d80ff0bff" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 33627 } } peers { permanent_uuid: "3710b74ed2044ce5995533df32e67351" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 38589 } attrs { replace: true } } peers { permanent_uuid: "25286c485fa04c86a5d07c8f88643a47" member_type: NON_VOTER last_known_addr { host: "127.17.60.198" port: 37089 } attrs { promote: true } }
I20250112 15:07:42.548456 21069 raft_consensus.cc:3054] T bfaeae584ec04772b2d6fcd088022117 P 2537bbbd1a9947ae9facd87567e3ccb9 [term 2 FOLLOWER]: Advancing to term 3
I20250112 15:07:42.548910 21222 raft_consensus.cc:2463] T bfaeae584ec04772b2d6fcd088022117 P 3710b74ed2044ce5995533df32e67351 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c561ec3374b94b95be0c155d80ff0bff in term 3.
I20250112 15:07:42.549005 21069 raft_consensus.cc:2463] T bfaeae584ec04772b2d6fcd088022117 P 2537bbbd1a9947ae9facd87567e3ccb9 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c561ec3374b94b95be0c155d80ff0bff in term 3.
I20250112 15:07:42.549063 21115 leader_election.cc:304] T bfaeae584ec04772b2d6fcd088022117 P c561ec3374b94b95be0c155d80ff0bff [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3710b74ed2044ce5995533df32e67351, c561ec3374b94b95be0c155d80ff0bff; no voters:
I20250112 15:07:42.549127 21935 raft_consensus.cc:2798] T bfaeae584ec04772b2d6fcd088022117 P c561ec3374b94b95be0c155d80ff0bff [term 3 FOLLOWER]: Leader election won for term 3
I20250112 15:07:42.549165 21935 raft_consensus.cc:695] T bfaeae584ec04772b2d6fcd088022117 P c561ec3374b94b95be0c155d80ff0bff [term 3 LEADER]: Becoming Leader. State: Replica: c561ec3374b94b95be0c155d80ff0bff, State: Running, Role: LEADER
I20250112 15:07:42.549213 21935 consensus_queue.cc:237] T bfaeae584ec04772b2d6fcd088022117 P c561ec3374b94b95be0c155d80ff0bff [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 27, Committed index: 27, Last appended: 2.27, Last appended by leader: 27, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "2537bbbd1a9947ae9facd87567e3ccb9" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 33173 } } peers { permanent_uuid: "c561ec3374b94b95be0c155d80ff0bff" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 33627 } } peers { permanent_uuid: "3710b74ed2044ce5995533df32e67351" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 38589 } attrs { replace: true } } peers { permanent_uuid: "25286c485fa04c86a5d07c8f88643a47" member_type: NON_VOTER last_known_addr { host: "127.17.60.198" port: 37089 } attrs { promote: true } }
I20250112 15:07:42.549856 20966 catalog_manager.cc:5526] T bfaeae584ec04772b2d6fcd088022117 P c561ec3374b94b95be0c155d80ff0bff reported cstate change: term changed from 2 to 3, leader changed from 2537bbbd1a9947ae9facd87567e3ccb9 (127.17.60.193) to c561ec3374b94b95be0c155d80ff0bff (127.17.60.194). New cstate: current_term: 3 leader_uuid: "c561ec3374b94b95be0c155d80ff0bff" committed_config { opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "2537bbbd1a9947ae9facd87567e3ccb9" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 33173 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c561ec3374b94b95be0c155d80ff0bff" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 33627 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3710b74ed2044ce5995533df32e67351" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 38589 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "25286c485fa04c86a5d07c8f88643a47" member_type: NON_VOTER last_known_addr { host: "127.17.60.198" port: 37089 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20250112 15:07:42.550089 21914 raft_consensus.cc:988] T 9601bcda24ed42e7b93ab9740f1e271e P 3710b74ed2044ce5995533df32e67351: : Instructing follower 2537bbbd1a9947ae9facd87567e3ccb9 to start an election
I20250112 15:07:42.550136 21914 raft_consensus.cc:1076] T 9601bcda24ed42e7b93ab9740f1e271e P 3710b74ed2044ce5995533df32e67351 [term 2 LEADER]: Signalling peer 2537bbbd1a9947ae9facd87567e3ccb9 to start an election
I20250112 15:07:42.550310 21069 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "9601bcda24ed42e7b93ab9740f1e271e"
dest_uuid: "2537bbbd1a9947ae9facd87567e3ccb9"
from {username='slave'} at 127.0.0.1:37204
I20250112 15:07:42.550387 21069 raft_consensus.cc:491] T 9601bcda24ed42e7b93ab9740f1e271e P 2537bbbd1a9947ae9facd87567e3ccb9 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20250112 15:07:42.550411 21069 raft_consensus.cc:3054] T 9601bcda24ed42e7b93ab9740f1e271e P 2537bbbd1a9947ae9facd87567e3ccb9 [term 2 FOLLOWER]: Advancing to term 3
I20250112 15:07:42.551044 21069 raft_consensus.cc:513] T 9601bcda24ed42e7b93ab9740f1e271e P 2537bbbd1a9947ae9facd87567e3ccb9 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "2537bbbd1a9947ae9facd87567e3ccb9" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 33173 } attrs { replace: true } } peers { permanent_uuid: "3710b74ed2044ce5995533df32e67351" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 38589 } } peers { permanent_uuid: "a2a5bba4ac1145d5bab94a69c9f4df40" member_type: VOTER last_known_addr { host: "127.17.60.196" port: 41807 } attrs { promote: false } } peers { permanent_uuid: "25286c485fa04c86a5d07c8f88643a47" member_type: NON_VOTER last_known_addr { host: "127.17.60.198" port: 37089 } attrs { promote: true } }
I20250112 15:07:42.551155 21069 leader_election.cc:290] T 9601bcda24ed42e7b93ab9740f1e271e P 2537bbbd1a9947ae9facd87567e3ccb9 [CANDIDATE]: Term 3 election: Requested vote from peers 3710b74ed2044ce5995533df32e67351 (127.17.60.195:38589), a2a5bba4ac1145d5bab94a69c9f4df40 (127.17.60.196:41807)
I20250112 15:07:42.551265 21474 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9601bcda24ed42e7b93ab9740f1e271e" candidate_uuid: "2537bbbd1a9947ae9facd87567e3ccb9" candidate_term: 3 candidate_status { last_received { term: 2 index: 29 } } ignore_live_leader: true dest_uuid: "a2a5bba4ac1145d5bab94a69c9f4df40"
I20250112 15:07:42.551258 21226 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9601bcda24ed42e7b93ab9740f1e271e" candidate_uuid: "2537bbbd1a9947ae9facd87567e3ccb9" candidate_term: 3 candidate_status { last_received { term: 2 index: 29 } } ignore_live_leader: true dest_uuid: "3710b74ed2044ce5995533df32e67351"
I20250112 15:07:42.551355 21474 raft_consensus.cc:3054] T 9601bcda24ed42e7b93ab9740f1e271e P a2a5bba4ac1145d5bab94a69c9f4df40 [term 2 FOLLOWER]: Advancing to term 3
I20250112 15:07:42.551364 21226 raft_consensus.cc:3049] T 9601bcda24ed42e7b93ab9740f1e271e P 3710b74ed2044ce5995533df32e67351 [term 2 LEADER]: Stepping down as leader of term 2
I20250112 15:07:42.551400 21226 raft_consensus.cc:738] T 9601bcda24ed42e7b93ab9740f1e271e P 3710b74ed2044ce5995533df32e67351 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 3710b74ed2044ce5995533df32e67351, State: Running, Role: LEADER
I20250112 15:07:42.551474 21226 consensus_queue.cc:260] T 9601bcda24ed42e7b93ab9740f1e271e P 3710b74ed2044ce5995533df32e67351 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 29, Committed index: 29, Last appended: 2.29, Last appended by leader: 29, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "2537bbbd1a9947ae9facd87567e3ccb9" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 33173 } attrs { replace: true } } peers { permanent_uuid: "3710b74ed2044ce5995533df32e67351" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 38589 } } peers { permanent_uuid: "a2a5bba4ac1145d5bab94a69c9f4df40" member_type: VOTER last_known_addr { host: "127.17.60.196" port: 41807 } attrs { promote: false } } peers { permanent_uuid: "25286c485fa04c86a5d07c8f88643a47" member_type: NON_VOTER last_known_addr { host: "127.17.60.198" port: 37089 } attrs { promote: true } }
I20250112 15:07:42.551550 21226 raft_consensus.cc:3054] T 9601bcda24ed42e7b93ab9740f1e271e P 3710b74ed2044ce5995533df32e67351 [term 2 FOLLOWER]: Advancing to term 3
I20250112 15:07:42.551864 21474 raft_consensus.cc:2463] T 9601bcda24ed42e7b93ab9740f1e271e P a2a5bba4ac1145d5bab94a69c9f4df40 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2537bbbd1a9947ae9facd87567e3ccb9 in term 3.
I20250112 15:07:42.552027 21226 raft_consensus.cc:2463] T 9601bcda24ed42e7b93ab9740f1e271e P 3710b74ed2044ce5995533df32e67351 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2537bbbd1a9947ae9facd87567e3ccb9 in term 3.
I20250112 15:07:42.552031 21040 leader_election.cc:304] T 9601bcda24ed42e7b93ab9740f1e271e P 2537bbbd1a9947ae9facd87567e3ccb9 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2537bbbd1a9947ae9facd87567e3ccb9, a2a5bba4ac1145d5bab94a69c9f4df40; no voters:
I20250112 15:07:42.552170 21969 raft_consensus.cc:2798] T 9601bcda24ed42e7b93ab9740f1e271e P 2537bbbd1a9947ae9facd87567e3ccb9 [term 3 FOLLOWER]: Leader election won for term 3
I20250112 15:07:42.552208 21969 raft_consensus.cc:695] T 9601bcda24ed42e7b93ab9740f1e271e P 2537bbbd1a9947ae9facd87567e3ccb9 [term 3 LEADER]: Becoming Leader. State: Replica: 2537bbbd1a9947ae9facd87567e3ccb9, State: Running, Role: LEADER
I20250112 15:07:42.552253 21969 consensus_queue.cc:237] T 9601bcda24ed42e7b93ab9740f1e271e P 2537bbbd1a9947ae9facd87567e3ccb9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 29, Committed index: 29, Last appended: 2.29, Last appended by leader: 29, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "2537bbbd1a9947ae9facd87567e3ccb9" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 33173 } attrs { replace: true } } peers { permanent_uuid: "3710b74ed2044ce5995533df32e67351" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 38589 } } peers { permanent_uuid: "a2a5bba4ac1145d5bab94a69c9f4df40" member_type: VOTER last_known_addr { host: "127.17.60.196" port: 41807 } attrs { promote: false } } peers { permanent_uuid: "25286c485fa04c86a5d07c8f88643a47" member_type: NON_VOTER last_known_addr { host: "127.17.60.198" port: 37089 } attrs { promote: true } }
I20250112 15:07:42.552796 20966 catalog_manager.cc:5526] T 9601bcda24ed42e7b93ab9740f1e271e P 2537bbbd1a9947ae9facd87567e3ccb9 reported cstate change: term changed from 2 to 3, leader changed from 3710b74ed2044ce5995533df32e67351 (127.17.60.195) to 2537bbbd1a9947ae9facd87567e3ccb9 (127.17.60.193). New cstate: current_term: 3 leader_uuid: "2537bbbd1a9947ae9facd87567e3ccb9" committed_config { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "2537bbbd1a9947ae9facd87567e3ccb9" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 33173 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3710b74ed2044ce5995533df32e67351" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 38589 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a2a5bba4ac1145d5bab94a69c9f4df40" member_type: VOTER last_known_addr { host: "127.17.60.196" port: 41807 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "25286c485fa04c86a5d07c8f88643a47" member_type: NON_VOTER last_known_addr { host: "127.17.60.198" port: 37089 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20250112 15:07:42.553651 21069 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "9601bcda24ed42e7b93ab9740f1e271e"
dest_uuid: "2537bbbd1a9947ae9facd87567e3ccb9"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:37188
I20250112 15:07:42.553700 21069 raft_consensus.cc:604] T 9601bcda24ed42e7b93ab9740f1e271e P 2537bbbd1a9947ae9facd87567e3ccb9 [term 3 LEADER]: Received request to transfer leadership
I20250112 15:07:42.555150 21069 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "9601bcda24ed42e7b93ab9740f1e271e"
dest_uuid: "2537bbbd1a9947ae9facd87567e3ccb9"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:37188
I20250112 15:07:42.555207 21069 raft_consensus.cc:604] T 9601bcda24ed42e7b93ab9740f1e271e P 2537bbbd1a9947ae9facd87567e3ccb9 [term 3 LEADER]: Received request to transfer leadership
I20250112 15:07:42.555548 21069 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "9601bcda24ed42e7b93ab9740f1e271e"
dest_uuid: "2537bbbd1a9947ae9facd87567e3ccb9"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:37188
I20250112 15:07:42.555595 21069 raft_consensus.cc:604] T 9601bcda24ed42e7b93ab9740f1e271e P 2537bbbd1a9947ae9facd87567e3ccb9 [term 3 LEADER]: Received request to transfer leadership
W20250112 15:07:42.555717 21021 auto_rebalancer.cc:663] Could not move replica: Remote error: Service unavailable: leadership transfer for 9601bcda24ed42e7b93ab9740f1e271e already in progress
W20250112 15:07:42.555760 21021 auto_rebalancer.cc:263] scheduled replica move failed to complete: Remote error: Service unavailable: leadership transfer for 9601bcda24ed42e7b93ab9740f1e271e already in progress
I20250112 15:07:42.581427 22010 raft_consensus.cc:1059] T 9d4bbba799324d7fbbac082520c65686 P c561ec3374b94b95be0c155d80ff0bff: attempting to promote NON_VOTER b5e9178a7a83402e8c4a6943ab78589d to VOTER
I20250112 15:07:42.581650 22010 consensus_queue.cc:237] T 9d4bbba799324d7fbbac082520c65686 P c561ec3374b94b95be0c155d80ff0bff [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 27, Committed index: 27, Last appended: 3.27, Last appended by leader: 26, Current term: 3, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "2537bbbd1a9947ae9facd87567e3ccb9" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 33173 } } peers { permanent_uuid: "c561ec3374b94b95be0c155d80ff0bff" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 33627 } } peers { permanent_uuid: "3710b74ed2044ce5995533df32e67351" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 38589 } attrs { replace: true } } peers { permanent_uuid: "b5e9178a7a83402e8c4a6943ab78589d" member_type: VOTER last_known_addr { host: "127.17.60.197" port: 36573 } attrs { promote: false } }
I20250112 15:07:42.582111 21222 raft_consensus.cc:1270] T 9d4bbba799324d7fbbac082520c65686 P 3710b74ed2044ce5995533df32e67351 [term 3 FOLLOWER]: Refusing update from remote peer c561ec3374b94b95be0c155d80ff0bff: Log matching property violated. Preceding OpId in replica: term: 3 index: 27. Preceding OpId from leader: term: 3 index: 28. (index mismatch)
I20250112 15:07:42.582145 21069 raft_consensus.cc:1270] T 9d4bbba799324d7fbbac082520c65686 P 2537bbbd1a9947ae9facd87567e3ccb9 [term 3 FOLLOWER]: Refusing update from remote peer c561ec3374b94b95be0c155d80ff0bff: Log matching property violated. Preceding OpId in replica: term: 3 index: 27. Preceding OpId from leader: term: 3 index: 28. (index mismatch)
I20250112 15:07:42.582211 21549 raft_consensus.cc:1270] T 9d4bbba799324d7fbbac082520c65686 P b5e9178a7a83402e8c4a6943ab78589d [term 3 LEARNER]: Refusing update from remote peer c561ec3374b94b95be0c155d80ff0bff: Log matching property violated. Preceding OpId in replica: term: 3 index: 27. Preceding OpId from leader: term: 3 index: 28. (index mismatch)
I20250112 15:07:42.582289 22010 consensus_queue.cc:1035] T 9d4bbba799324d7fbbac082520c65686 P c561ec3374b94b95be0c155d80ff0bff [LEADER]: Connected to new peer: Peer: permanent_uuid: "3710b74ed2044ce5995533df32e67351" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 38589 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 28, Last known committed idx: 27, Time since last communication: 0.000s
I20250112 15:07:42.582403 21935 consensus_queue.cc:1035] T 9d4bbba799324d7fbbac082520c65686 P c561ec3374b94b95be0c155d80ff0bff [LEADER]: Connected to new peer: Peer: permanent_uuid: "b5e9178a7a83402e8c4a6943ab78589d" member_type: VOTER last_known_addr { host: "127.17.60.197" port: 36573 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 28, Last known committed idx: 27, Time since last communication: 0.000s
I20250112 15:07:42.582538 21935 consensus_queue.cc:1035] T 9d4bbba799324d7fbbac082520c65686 P c561ec3374b94b95be0c155d80ff0bff [LEADER]: Connected to new peer: Peer: permanent_uuid: "2537bbbd1a9947ae9facd87567e3ccb9" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 33173 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 28, Last known committed idx: 27, Time since last communication: 0.000s
I20250112 15:07:42.583374 21935 raft_consensus.cc:2949] T 9d4bbba799324d7fbbac082520c65686 P c561ec3374b94b95be0c155d80ff0bff [term 3 LEADER]: Committing config change with OpId 3.28: config changed from index 26 to 28, b5e9178a7a83402e8c4a6943ab78589d (127.17.60.197) changed from NON_VOTER to VOTER. New config: { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "2537bbbd1a9947ae9facd87567e3ccb9" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 33173 } } peers { permanent_uuid: "c561ec3374b94b95be0c155d80ff0bff" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 33627 } } peers { permanent_uuid: "3710b74ed2044ce5995533df32e67351" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 38589 } attrs { replace: true } } peers { permanent_uuid: "b5e9178a7a83402e8c4a6943ab78589d" member_type: VOTER last_known_addr { host: "127.17.60.197" port: 36573 } attrs { promote: false } } }
I20250112 15:07:42.583495 21068 raft_consensus.cc:2949] T 9d4bbba799324d7fbbac082520c65686 P 2537bbbd1a9947ae9facd87567e3ccb9 [term 3 FOLLOWER]: Committing config change with OpId 3.28: config changed from index 26 to 28, b5e9178a7a83402e8c4a6943ab78589d (127.17.60.197) changed from NON_VOTER to VOTER. New config: { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "2537bbbd1a9947ae9facd87567e3ccb9" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 33173 } } peers { permanent_uuid: "c561ec3374b94b95be0c155d80ff0bff" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 33627 } } peers { permanent_uuid: "3710b74ed2044ce5995533df32e67351" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 38589 } attrs { replace: true } } peers { permanent_uuid: "b5e9178a7a83402e8c4a6943ab78589d" member_type: VOTER last_known_addr { host: "127.17.60.197" port: 36573 } attrs { promote: false } } }
I20250112 15:07:42.584271 21222 raft_consensus.cc:2949] T 9d4bbba799324d7fbbac082520c65686 P 3710b74ed2044ce5995533df32e67351 [term 3 FOLLOWER]: Committing config change with OpId 3.28: config changed from index 26 to 28, b5e9178a7a83402e8c4a6943ab78589d (127.17.60.197) changed from NON_VOTER to VOTER. New config: { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "2537bbbd1a9947ae9facd87567e3ccb9" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 33173 } } peers { permanent_uuid: "c561ec3374b94b95be0c155d80ff0bff" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 33627 } } peers { permanent_uuid: "3710b74ed2044ce5995533df32e67351" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 38589 } attrs { replace: true } } peers { permanent_uuid: "b5e9178a7a83402e8c4a6943ab78589d" member_type: VOTER last_known_addr { host: "127.17.60.197" port: 36573 } attrs { promote: false } } }
I20250112 15:07:42.584268 21549 raft_consensus.cc:2949] T 9d4bbba799324d7fbbac082520c65686 P b5e9178a7a83402e8c4a6943ab78589d [term 3 FOLLOWER]: Committing config change with OpId 3.28: config changed from index 26 to 28, b5e9178a7a83402e8c4a6943ab78589d (127.17.60.197) changed from NON_VOTER to VOTER. New config: { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "2537bbbd1a9947ae9facd87567e3ccb9" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 33173 } } peers { permanent_uuid: "c561ec3374b94b95be0c155d80ff0bff" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 33627 } } peers { permanent_uuid: "3710b74ed2044ce5995533df32e67351" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 38589 } attrs { replace: true } } peers { permanent_uuid: "b5e9178a7a83402e8c4a6943ab78589d" member_type: VOTER last_known_addr { host: "127.17.60.197" port: 36573 } attrs { promote: false } } }
I20250112 15:07:42.584522 20966 catalog_manager.cc:5526] T 9d4bbba799324d7fbbac082520c65686 P 2537bbbd1a9947ae9facd87567e3ccb9 reported cstate change: config changed from index 26 to 28, b5e9178a7a83402e8c4a6943ab78589d (127.17.60.197) changed from NON_VOTER to VOTER. New cstate: current_term: 3 leader_uuid: "c561ec3374b94b95be0c155d80ff0bff" committed_config { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "2537bbbd1a9947ae9facd87567e3ccb9" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 33173 } } peers { permanent_uuid: "c561ec3374b94b95be0c155d80ff0bff" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 33627 } } peers { permanent_uuid: "3710b74ed2044ce5995533df32e67351" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 38589 } attrs { replace: true } } peers { permanent_uuid: "b5e9178a7a83402e8c4a6943ab78589d" member_type: VOTER last_known_addr { host: "127.17.60.197" port: 36573 } attrs { promote: false } } }
I20250112 15:07:42.585588 21149 consensus_queue.cc:237] T 9d4bbba799324d7fbbac082520c65686 P c561ec3374b94b95be0c155d80ff0bff [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 28, Committed index: 28, Last appended: 3.28, Last appended by leader: 26, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "2537bbbd1a9947ae9facd87567e3ccb9" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 33173 } } peers { permanent_uuid: "c561ec3374b94b95be0c155d80ff0bff" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 33627 } } peers { permanent_uuid: "b5e9178a7a83402e8c4a6943ab78589d" member_type: VOTER last_known_addr { host: "127.17.60.197" port: 36573 } attrs { promote: false } }
I20250112 15:07:42.585924 21548 raft_consensus.cc:1270] T 9d4bbba799324d7fbbac082520c65686 P b5e9178a7a83402e8c4a6943ab78589d [term 3 FOLLOWER]: Refusing update from remote peer c561ec3374b94b95be0c155d80ff0bff: Log matching property violated. Preceding OpId in replica: term: 3 index: 28. Preceding OpId from leader: term: 3 index: 29. (index mismatch)
I20250112 15:07:42.586115 22015 consensus_queue.cc:1035] T 9d4bbba799324d7fbbac082520c65686 P c561ec3374b94b95be0c155d80ff0bff [LEADER]: Connected to new peer: Peer: permanent_uuid: "b5e9178a7a83402e8c4a6943ab78589d" member_type: VOTER last_known_addr { host: "127.17.60.197" port: 36573 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.000s
I20250112 15:07:42.586277 21069 raft_consensus.cc:1270] T 9d4bbba799324d7fbbac082520c65686 P 2537bbbd1a9947ae9facd87567e3ccb9 [term 3 FOLLOWER]: Refusing update from remote peer c561ec3374b94b95be0c155d80ff0bff: Log matching property violated. Preceding OpId in replica: term: 3 index: 28. Preceding OpId from leader: term: 3 index: 29. (index mismatch)
I20250112 15:07:42.586529 22015 consensus_queue.cc:1035] T 9d4bbba799324d7fbbac082520c65686 P c561ec3374b94b95be0c155d80ff0bff [LEADER]: Connected to new peer: Peer: permanent_uuid: "2537bbbd1a9947ae9facd87567e3ccb9" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 33173 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.000s
I20250112 15:07:42.586676 22015 raft_consensus.cc:2949] T 9d4bbba799324d7fbbac082520c65686 P c561ec3374b94b95be0c155d80ff0bff [term 3 LEADER]: Committing config change with OpId 3.29: config changed from index 28 to 29, VOTER 3710b74ed2044ce5995533df32e67351 (127.17.60.195) evicted. New config: { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "2537bbbd1a9947ae9facd87567e3ccb9" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 33173 } } peers { permanent_uuid: "c561ec3374b94b95be0c155d80ff0bff" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 33627 } } peers { permanent_uuid: "b5e9178a7a83402e8c4a6943ab78589d" member_type: VOTER last_known_addr { host: "127.17.60.197" port: 36573 } attrs { promote: false } } }
I20250112 15:07:42.587188 21069 raft_consensus.cc:2949] T 9d4bbba799324d7fbbac082520c65686 P 2537bbbd1a9947ae9facd87567e3ccb9 [term 3 FOLLOWER]: Committing config change with OpId 3.29: config changed from index 28 to 29, VOTER 3710b74ed2044ce5995533df32e67351 (127.17.60.195) evicted. New config: { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "2537bbbd1a9947ae9facd87567e3ccb9" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 33173 } } peers { permanent_uuid: "c561ec3374b94b95be0c155d80ff0bff" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 33627 } } peers { permanent_uuid: "b5e9178a7a83402e8c4a6943ab78589d" member_type: VOTER last_known_addr { host: "127.17.60.197" port: 36573 } attrs { promote: false } } }
I20250112 15:07:42.587680 20958 catalog_manager.cc:5039] ChangeConfig:REMOVE_PEER RPC for tablet 9d4bbba799324d7fbbac082520c65686 with cas_config_opid_index 28: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250112 15:07:42.587643 21548 raft_consensus.cc:2949] T 9d4bbba799324d7fbbac082520c65686 P b5e9178a7a83402e8c4a6943ab78589d [term 3 FOLLOWER]: Committing config change with OpId 3.29: config changed from index 28 to 29, VOTER 3710b74ed2044ce5995533df32e67351 (127.17.60.195) evicted. New config: { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "2537bbbd1a9947ae9facd87567e3ccb9" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 33173 } } peers { permanent_uuid: "c561ec3374b94b95be0c155d80ff0bff" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 33627 } } peers { permanent_uuid: "b5e9178a7a83402e8c4a6943ab78589d" member_type: VOTER last_known_addr { host: "127.17.60.197" port: 36573 } attrs { promote: false } } }
I20250112 15:07:42.587731 20964 catalog_manager.cc:5526] T 9d4bbba799324d7fbbac082520c65686 P c561ec3374b94b95be0c155d80ff0bff reported cstate change: config changed from index 28 to 29, VOTER 3710b74ed2044ce5995533df32e67351 (127.17.60.195) evicted. New cstate: current_term: 3 leader_uuid: "c561ec3374b94b95be0c155d80ff0bff" committed_config { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "2537bbbd1a9947ae9facd87567e3ccb9" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 33173 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c561ec3374b94b95be0c155d80ff0bff" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 33627 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b5e9178a7a83402e8c4a6943ab78589d" member_type: VOTER last_known_addr { host: "127.17.60.197" port: 36573 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20250112 15:07:42.588383 21212 tablet_service.cc:1514] Processing DeleteTablet for tablet 9d4bbba799324d7fbbac082520c65686 with delete_type TABLET_DATA_TOMBSTONED (TS 3710b74ed2044ce5995533df32e67351 not found in new config with opid_index 29) from {username='slave'} at 127.0.0.1:33192
I20250112 15:07:42.588583 22018 tablet_replica.cc:331] T 9d4bbba799324d7fbbac082520c65686 P 3710b74ed2044ce5995533df32e67351: stopping tablet replica
I20250112 15:07:42.588656 22018 raft_consensus.cc:2238] T 9d4bbba799324d7fbbac082520c65686 P 3710b74ed2044ce5995533df32e67351 [term 3 FOLLOWER]: Raft consensus shutting down.
I20250112 15:07:42.588809 22018 raft_consensus.cc:2267] T 9d4bbba799324d7fbbac082520c65686 P 3710b74ed2044ce5995533df32e67351 [term 3 FOLLOWER]: Raft consensus is shut down!
I20250112 15:07:42.589147 22018 ts_tablet_manager.cc:1905] T 9d4bbba799324d7fbbac082520c65686 P 3710b74ed2044ce5995533df32e67351: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250112 15:07:42.590415 22018 ts_tablet_manager.cc:1918] T 9d4bbba799324d7fbbac082520c65686 P 3710b74ed2044ce5995533df32e67351: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 3.28
I20250112 15:07:42.590472 22018 log.cc:1198] T 9d4bbba799324d7fbbac082520c65686 P 3710b74ed2044ce5995533df32e67351: Deleting WAL directory at /tmp/dist-test-task95dLSD/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1736694351070047-17651-0/minicluster-data/ts-2-root/wals/9d4bbba799324d7fbbac082520c65686
I20250112 15:07:42.590855 20957 catalog_manager.cc:4872] TS 3710b74ed2044ce5995533df32e67351 (127.17.60.195:38589): tablet 9d4bbba799324d7fbbac082520c65686 (table test-workload [id=cdaf6d3a65a54ea08e8e2c0ecdfc9de7]) successfully deleted
I20250112 15:07:42.827996 21969 raft_consensus.cc:1059] T 16f16fcd5cf347e5853189dc4da55c7f P 2537bbbd1a9947ae9facd87567e3ccb9: attempting to promote NON_VOTER 25286c485fa04c86a5d07c8f88643a47 to VOTER
I20250112 15:07:42.828246 21969 consensus_queue.cc:237] T 16f16fcd5cf347e5853189dc4da55c7f P 2537bbbd1a9947ae9facd87567e3ccb9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 27, Committed index: 27, Last appended: 2.27, Last appended by leader: 25, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "2537bbbd1a9947ae9facd87567e3ccb9" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 33173 } } peers { permanent_uuid: "c561ec3374b94b95be0c155d80ff0bff" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 33627 } attrs { replace: true } } peers { permanent_uuid: "3710b74ed2044ce5995533df32e67351" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 38589 } } peers { permanent_uuid: "25286c485fa04c86a5d07c8f88643a47" member_type: VOTER last_known_addr { host: "127.17.60.198" port: 37089 } attrs { promote: false } }
I20250112 15:07:42.828758 21624 raft_consensus.cc:1270] T 16f16fcd5cf347e5853189dc4da55c7f P 25286c485fa04c86a5d07c8f88643a47 [term 2 LEARNER]: Refusing update from remote peer 2537bbbd1a9947ae9facd87567e3ccb9: Log matching property violated. Preceding OpId in replica: term: 2 index: 27. Preceding OpId from leader: term: 2 index: 28. (index mismatch)
I20250112 15:07:42.828804 21222 raft_consensus.cc:1270] T 16f16fcd5cf347e5853189dc4da55c7f P 3710b74ed2044ce5995533df32e67351 [term 2 FOLLOWER]: Refusing update from remote peer 2537bbbd1a9947ae9facd87567e3ccb9: Log matching property violated. Preceding OpId in replica: term: 2 index: 27. Preceding OpId from leader: term: 2 index: 28. (index mismatch)
I20250112 15:07:42.828826 21147 raft_consensus.cc:1270] T 16f16fcd5cf347e5853189dc4da55c7f P c561ec3374b94b95be0c155d80ff0bff [term 2 FOLLOWER]: Refusing update from remote peer 2537bbbd1a9947ae9facd87567e3ccb9: Log matching property violated. Preceding OpId in replica: term: 2 index: 27. Preceding OpId from leader: term: 2 index: 28. (index mismatch)
I20250112 15:07:42.829006 22019 consensus_queue.cc:1035] T 16f16fcd5cf347e5853189dc4da55c7f P 2537bbbd1a9947ae9facd87567e3ccb9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3710b74ed2044ce5995533df32e67351" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 38589 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 28, Last known committed idx: 27, Time since last communication: 0.000s
I20250112 15:07:42.829109 22009 consensus_queue.cc:1035] T 16f16fcd5cf347e5853189dc4da55c7f P 2537bbbd1a9947ae9facd87567e3ccb9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c561ec3374b94b95be0c155d80ff0bff" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 33627 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 28, Last known committed idx: 27, Time since last communication: 0.000s
I20250112 15:07:42.829242 22021 consensus_queue.cc:1035] T 16f16fcd5cf347e5853189dc4da55c7f P 2537bbbd1a9947ae9facd87567e3ccb9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "25286c485fa04c86a5d07c8f88643a47" member_type: VOTER last_known_addr { host: "127.17.60.198" port: 37089 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 28, Last known committed idx: 27, Time since last communication: 0.000s
I20250112 15:07:42.829957 21969 raft_consensus.cc:2949] T 16f16fcd5cf347e5853189dc4da55c7f P 2537bbbd1a9947ae9facd87567e3ccb9 [term 2 LEADER]: Committing config change with OpId 2.28: config changed from index 27 to 28, 25286c485fa04c86a5d07c8f88643a47 (127.17.60.198) changed from NON_VOTER to VOTER. New config: { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "2537bbbd1a9947ae9facd87567e3ccb9" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 33173 } } peers { permanent_uuid: "c561ec3374b94b95be0c155d80ff0bff" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 33627 } attrs { replace: true } } peers { permanent_uuid: "3710b74ed2044ce5995533df32e67351" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 38589 } } peers { permanent_uuid: "25286c485fa04c86a5d07c8f88643a47" member_type: VOTER last_known_addr { host: "127.17.60.198" port: 37089 } attrs { promote: false } } }
I20250112 15:07:42.830060 21624 raft_consensus.cc:2949] T 16f16fcd5cf347e5853189dc4da55c7f P 25286c485fa04c86a5d07c8f88643a47 [term 2 FOLLOWER]: Committing config change with OpId 2.28: config changed from index 27 to 28, 25286c485fa04c86a5d07c8f88643a47 (127.17.60.198) changed from NON_VOTER to VOTER. New config: { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "2537bbbd1a9947ae9facd87567e3ccb9" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 33173 } } peers { permanent_uuid: "c561ec3374b94b95be0c155d80ff0bff" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 33627 } attrs { replace: true } } peers { permanent_uuid: "3710b74ed2044ce5995533df32e67351" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 38589 } } peers { permanent_uuid: "25286c485fa04c86a5d07c8f88643a47" member_type: VOTER last_known_addr { host: "127.17.60.198" port: 37089 } attrs { promote: false } } }
I20250112 15:07:42.830101 21222 raft_consensus.cc:2949] T 16f16fcd5cf347e5853189dc4da55c7f P 3710b74ed2044ce5995533df32e67351 [term 2 FOLLOWER]: Committing config change with OpId 2.28: config changed from index 27 to 28, 25286c485fa04c86a5d07c8f88643a47 (127.17.60.198) changed from NON_VOTER to VOTER. New config: { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "2537bbbd1a9947ae9facd87567e3ccb9" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 33173 } } peers { permanent_uuid: "c561ec3374b94b95be0c155d80ff0bff" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 33627 } attrs { replace: true } } peers { permanent_uuid: "3710b74ed2044ce5995533df32e67351" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 38589 } } peers { permanent_uuid: "25286c485fa04c86a5d07c8f88643a47" member_type: VOTER last_known_addr { host: "127.17.60.198" port: 37089 } attrs { promote: false } } }
I20250112 15:07:42.831069 20966 catalog_manager.cc:5526] T 16f16fcd5cf347e5853189dc4da55c7f P 25286c485fa04c86a5d07c8f88643a47 reported cstate change: config changed from index 27 to 28, 25286c485fa04c86a5d07c8f88643a47 (127.17.60.198) changed from NON_VOTER to VOTER. New cstate: current_term: 2 leader_uuid: "2537bbbd1a9947ae9facd87567e3ccb9" committed_config { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "2537bbbd1a9947ae9facd87567e3ccb9" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 33173 } } peers { permanent_uuid: "c561ec3374b94b95be0c155d80ff0bff" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 33627 } attrs { replace: true } } peers { permanent_uuid: "3710b74ed2044ce5995533df32e67351" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 38589 } } peers { permanent_uuid: "25286c485fa04c86a5d07c8f88643a47" member_type: VOTER last_known_addr { host: "127.17.60.198" port: 37089 } attrs { promote: false } } }
I20250112 15:07:42.831458 21147 raft_consensus.cc:2949] T 16f16fcd5cf347e5853189dc4da55c7f P c561ec3374b94b95be0c155d80ff0bff [term 2 FOLLOWER]: Committing config change with OpId 2.28: config changed from index 27 to 28, 25286c485fa04c86a5d07c8f88643a47 (127.17.60.198) changed from NON_VOTER to VOTER. New config: { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "2537bbbd1a9947ae9facd87567e3ccb9" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 33173 } } peers { permanent_uuid: "c561ec3374b94b95be0c155d80ff0bff" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 33627 } attrs { replace: true } } peers { permanent_uuid: "3710b74ed2044ce5995533df32e67351" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 38589 } } peers { permanent_uuid: "25286c485fa04c86a5d07c8f88643a47" member_type: VOTER last_known_addr { host: "127.17.60.198" port: 37089 } attrs { promote: false } } }
I20250112 15:07:42.831960 21069 consensus_queue.cc:237] T 16f16fcd5cf347e5853189dc4da55c7f P 2537bbbd1a9947ae9facd87567e3ccb9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 28, Committed index: 28, Last appended: 2.28, Last appended by leader: 25, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "2537bbbd1a9947ae9facd87567e3ccb9" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 33173 } } peers { permanent_uuid: "3710b74ed2044ce5995533df32e67351" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 38589 } } peers { permanent_uuid: "25286c485fa04c86a5d07c8f88643a47" member_type: VOTER last_known_addr { host: "127.17.60.198" port: 37089 } attrs { promote: false } }
I20250112 15:07:42.832301 21226 raft_consensus.cc:1270] T 16f16fcd5cf347e5853189dc4da55c7f P 3710b74ed2044ce5995533df32e67351 [term 2 FOLLOWER]: Refusing update from remote peer 2537bbbd1a9947ae9facd87567e3ccb9: Log matching property violated. Preceding OpId in replica: term: 2 index: 28. Preceding OpId from leader: term: 2 index: 29. (index mismatch)
I20250112 15:07:42.832615 22019 consensus_queue.cc:1035] T 16f16fcd5cf347e5853189dc4da55c7f P 2537bbbd1a9947ae9facd87567e3ccb9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3710b74ed2044ce5995533df32e67351" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 38589 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.000s
I20250112 15:07:42.832648 21624 raft_consensus.cc:1270] T 16f16fcd5cf347e5853189dc4da55c7f P 25286c485fa04c86a5d07c8f88643a47 [term 2 FOLLOWER]: Refusing update from remote peer 2537bbbd1a9947ae9facd87567e3ccb9: Log matching property violated. Preceding OpId in replica: term: 2 index: 28. Preceding OpId from leader: term: 2 index: 29. (index mismatch)
I20250112 15:07:42.832859 22019 consensus_queue.cc:1035] T 16f16fcd5cf347e5853189dc4da55c7f P 2537bbbd1a9947ae9facd87567e3ccb9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "25286c485fa04c86a5d07c8f88643a47" member_type: VOTER last_known_addr { host: "127.17.60.198" port: 37089 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.000s
I20250112 15:07:42.833273 22021 raft_consensus.cc:2949] T 16f16fcd5cf347e5853189dc4da55c7f P 2537bbbd1a9947ae9facd87567e3ccb9 [term 2 LEADER]: Committing config change with OpId 2.29: config changed from index 28 to 29, VOTER c561ec3374b94b95be0c155d80ff0bff (127.17.60.194) evicted. New config: { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "2537bbbd1a9947ae9facd87567e3ccb9" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 33173 } } peers { permanent_uuid: "3710b74ed2044ce5995533df32e67351" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 38589 } } peers { permanent_uuid: "25286c485fa04c86a5d07c8f88643a47" member_type: VOTER last_known_addr { host: "127.17.60.198" port: 37089 } attrs { promote: false } } }
I20250112 15:07:42.833382 21222 raft_consensus.cc:2949] T 16f16fcd5cf347e5853189dc4da55c7f P 3710b74ed2044ce5995533df32e67351 [term 2 FOLLOWER]: Committing config change with OpId 2.29: config changed from index 28 to 29, VOTER c561ec3374b94b95be0c155d80ff0bff (127.17.60.194) evicted. New config: { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "2537bbbd1a9947ae9facd87567e3ccb9" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 33173 } } peers { permanent_uuid: "3710b74ed2044ce5995533df32e67351" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 38589 } } peers { permanent_uuid: "25286c485fa04c86a5d07c8f88643a47" member_type: VOTER last_known_addr { host: "127.17.60.198" port: 37089 } attrs { promote: false } } }
I20250112 15:07:42.834153 21624 raft_consensus.cc:2949] T 16f16fcd5cf347e5853189dc4da55c7f P 25286c485fa04c86a5d07c8f88643a47 [term 2 FOLLOWER]: Committing config change with OpId 2.29: config changed from index 28 to 29, VOTER c561ec3374b94b95be0c155d80ff0bff (127.17.60.194) evicted. New config: { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "2537bbbd1a9947ae9facd87567e3ccb9" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 33173 } } peers { permanent_uuid: "3710b74ed2044ce5995533df32e67351" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 38589 } } peers { permanent_uuid: "25286c485fa04c86a5d07c8f88643a47" member_type: VOTER last_known_addr { host: "127.17.60.198" port: 37089 } attrs { promote: false } } }
I20250112 15:07:42.834208 20970 catalog_manager.cc:5526] T 16f16fcd5cf347e5853189dc4da55c7f P 3710b74ed2044ce5995533df32e67351 reported cstate change: config changed from index 28 to 29, VOTER c561ec3374b94b95be0c155d80ff0bff (127.17.60.194) evicted. New cstate: current_term: 2 leader_uuid: "2537bbbd1a9947ae9facd87567e3ccb9" committed_config { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "2537bbbd1a9947ae9facd87567e3ccb9" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 33173 } } peers { permanent_uuid: "3710b74ed2044ce5995533df32e67351" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 38589 } } peers { permanent_uuid: "25286c485fa04c86a5d07c8f88643a47" member_type: VOTER last_known_addr { host: "127.17.60.198" port: 37089 } attrs { promote: false } } }
I20250112 15:07:42.834882 21141 tablet_service.cc:1514] Processing DeleteTablet for tablet 16f16fcd5cf347e5853189dc4da55c7f with delete_type TABLET_DATA_TOMBSTONED (TS c561ec3374b94b95be0c155d80ff0bff not found in new config with opid_index 29) from {username='slave'} at 127.0.0.1:48072
I20250112 15:07:42.835147 20956 catalog_manager.cc:5039] ChangeConfig:REMOVE_PEER RPC for tablet 16f16fcd5cf347e5853189dc4da55c7f with cas_config_opid_index 28: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250112 15:07:42.835354 22025 tablet_replica.cc:331] T 16f16fcd5cf347e5853189dc4da55c7f P c561ec3374b94b95be0c155d80ff0bff: stopping tablet replica
I20250112 15:07:42.835419 22025 raft_consensus.cc:2238] T 16f16fcd5cf347e5853189dc4da55c7f P c561ec3374b94b95be0c155d80ff0bff [term 2 FOLLOWER]: Raft consensus shutting down.
I20250112 15:07:42.835491 22025 raft_consensus.cc:2267] T 16f16fcd5cf347e5853189dc4da55c7f P c561ec3374b94b95be0c155d80ff0bff [term 2 FOLLOWER]: Raft consensus is shut down!
I20250112 15:07:42.835727 22025 ts_tablet_manager.cc:1905] T 16f16fcd5cf347e5853189dc4da55c7f P c561ec3374b94b95be0c155d80ff0bff: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250112 15:07:42.837224 22025 ts_tablet_manager.cc:1918] T 16f16fcd5cf347e5853189dc4da55c7f P c561ec3374b94b95be0c155d80ff0bff: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.28
I20250112 15:07:42.837284 22025 log.cc:1198] T 16f16fcd5cf347e5853189dc4da55c7f P c561ec3374b94b95be0c155d80ff0bff: Deleting WAL directory at /tmp/dist-test-task95dLSD/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1736694351070047-17651-0/minicluster-data/ts-1-root/wals/16f16fcd5cf347e5853189dc4da55c7f
I20250112 15:07:42.837615 20958 catalog_manager.cc:4872] TS c561ec3374b94b95be0c155d80ff0bff (127.17.60.194:33627): tablet 16f16fcd5cf347e5853189dc4da55c7f (table test-workload [id=cdaf6d3a65a54ea08e8e2c0ecdfc9de7]) successfully deleted
I20250112 15:07:42.851823 22026 raft_consensus.cc:1059] T 57e47b07fdcf434e96ca5dd0354e93b2 P b5e9178a7a83402e8c4a6943ab78589d: attempting to promote NON_VOTER a2a5bba4ac1145d5bab94a69c9f4df40 to VOTER
I20250112 15:07:42.852048 22026 consensus_queue.cc:237] T 57e47b07fdcf434e96ca5dd0354e93b2 P b5e9178a7a83402e8c4a6943ab78589d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 31, Committed index: 31, Last appended: 3.31, Last appended by leader: 29, Current term: 3, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "2537bbbd1a9947ae9facd87567e3ccb9" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 33173 } } peers { permanent_uuid: "c561ec3374b94b95be0c155d80ff0bff" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 33627 } attrs { replace: true } } peers { permanent_uuid: "b5e9178a7a83402e8c4a6943ab78589d" member_type: VOTER last_known_addr { host: "127.17.60.197" port: 36573 } attrs { promote: false } } peers { permanent_uuid: "a2a5bba4ac1145d5bab94a69c9f4df40" member_type: VOTER last_known_addr { host: "127.17.60.196" port: 41807 } attrs { promote: false } }
I20250112 15:07:42.852692 21147 raft_consensus.cc:1270] T 57e47b07fdcf434e96ca5dd0354e93b2 P c561ec3374b94b95be0c155d80ff0bff [term 3 FOLLOWER]: Refusing update from remote peer b5e9178a7a83402e8c4a6943ab78589d: Log matching property violated. Preceding OpId in replica: term: 3 index: 31. Preceding OpId from leader: term: 3 index: 32. (index mismatch)
I20250112 15:07:42.852710 21069 raft_consensus.cc:1270] T 57e47b07fdcf434e96ca5dd0354e93b2 P 2537bbbd1a9947ae9facd87567e3ccb9 [term 3 FOLLOWER]: Refusing update from remote peer b5e9178a7a83402e8c4a6943ab78589d: Log matching property violated. Preceding OpId in replica: term: 3 index: 31. Preceding OpId from leader: term: 3 index: 32. (index mismatch)
I20250112 15:07:42.852681 21474 raft_consensus.cc:1270] T 57e47b07fdcf434e96ca5dd0354e93b2 P a2a5bba4ac1145d5bab94a69c9f4df40 [term 3 LEARNER]: Refusing update from remote peer b5e9178a7a83402e8c4a6943ab78589d: Log matching property violated. Preceding OpId in replica: term: 3 index: 31. Preceding OpId from leader: term: 3 index: 32. (index mismatch)
I20250112 15:07:42.852939 22027 consensus_queue.cc:1035] T 57e47b07fdcf434e96ca5dd0354e93b2 P b5e9178a7a83402e8c4a6943ab78589d [LEADER]: Connected to new peer: Peer: permanent_uuid: "2537bbbd1a9947ae9facd87567e3ccb9" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 33173 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 32, Last known committed idx: 31, Time since last communication: 0.000s
I20250112 15:07:42.853049 22026 consensus_queue.cc:1035] T 57e47b07fdcf434e96ca5dd0354e93b2 P b5e9178a7a83402e8c4a6943ab78589d [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2a5bba4ac1145d5bab94a69c9f4df40" member_type: VOTER last_known_addr { host: "127.17.60.196" port: 41807 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 32, Last known committed idx: 31, Time since last communication: 0.000s
I20250112 15:07:42.853160 22029 consensus_queue.cc:1035] T 57e47b07fdcf434e96ca5dd0354e93b2 P b5e9178a7a83402e8c4a6943ab78589d [LEADER]: Connected to new peer: Peer: permanent_uuid: "c561ec3374b94b95be0c155d80ff0bff" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 33627 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 32, Last known committed idx: 31, Time since last communication: 0.000s
I20250112 15:07:42.854039 22026 raft_consensus.cc:2949] T 57e47b07fdcf434e96ca5dd0354e93b2 P b5e9178a7a83402e8c4a6943ab78589d [term 3 LEADER]: Committing config change with OpId 3.32: config changed from index 31 to 32, a2a5bba4ac1145d5bab94a69c9f4df40 (127.17.60.196) changed from NON_VOTER to VOTER. New config: { opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "2537bbbd1a9947ae9facd87567e3ccb9" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 33173 } } peers { permanent_uuid: "c561ec3374b94b95be0c155d80ff0bff" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 33627 } attrs { replace: true } } peers { permanent_uuid: "b5e9178a7a83402e8c4a6943ab78589d" member_type: VOTER last_known_addr { host: "127.17.60.197" port: 36573 } attrs { promote: false } } peers { permanent_uuid: "a2a5bba4ac1145d5bab94a69c9f4df40" member_type: VOTER last_known_addr { host: "127.17.60.196" port: 41807 } attrs { promote: false } } }
I20250112 15:07:42.854262 21069 raft_consensus.cc:2949] T 57e47b07fdcf434e96ca5dd0354e93b2 P 2537bbbd1a9947ae9facd87567e3ccb9 [term 3 FOLLOWER]: Committing config change with OpId 3.32: config changed from index 31 to 32, a2a5bba4ac1145d5bab94a69c9f4df40 (127.17.60.196) changed from NON_VOTER to VOTER. New config: { opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "2537bbbd1a9947ae9facd87567e3ccb9" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 33173 } } peers { permanent_uuid: "c561ec3374b94b95be0c155d80ff0bff" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 33627 } attrs { replace: true } } peers { permanent_uuid: "b5e9178a7a83402e8c4a6943ab78589d" member_type: VOTER last_known_addr { host: "127.17.60.197" port: 36573 } attrs { promote: false } } peers { permanent_uuid: "a2a5bba4ac1145d5bab94a69c9f4df40" member_type: VOTER last_known_addr { host: "127.17.60.196" port: 41807 } attrs { promote: false } } }
I20250112 15:07:42.854336 21147 raft_consensus.cc:2949] T 57e47b07fdcf434e96ca5dd0354e93b2 P c561ec3374b94b95be0c155d80ff0bff [term 3 FOLLOWER]: Committing config change with OpId 3.32: config changed from index 31 to 32, a2a5bba4ac1145d5bab94a69c9f4df40 (127.17.60.196) changed from NON_VOTER to VOTER. New config: { opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "2537bbbd1a9947ae9facd87567e3ccb9" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 33173 } } peers { permanent_uuid: "c561ec3374b94b95be0c155d80ff0bff" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 33627 } attrs { replace: true } } peers { permanent_uuid: "b5e9178a7a83402e8c4a6943ab78589d" member_type: VOTER last_known_addr { host: "127.17.60.197" port: 36573 } attrs { promote: false } } peers { permanent_uuid: "a2a5bba4ac1145d5bab94a69c9f4df40" member_type: VOTER last_known_addr { host: "127.17.60.196" port: 41807 } attrs { promote: false } } }
I20250112 15:07:42.855348 20970 catalog_manager.cc:5526] T 57e47b07fdcf434e96ca5dd0354e93b2 P c561ec3374b94b95be0c155d80ff0bff reported cstate change: config changed from index 31 to 32, a2a5bba4ac1145d5bab94a69c9f4df40 (127.17.60.196) changed from NON_VOTER to VOTER. New cstate: current_term: 3 leader_uuid: "b5e9178a7a83402e8c4a6943ab78589d" committed_config { opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "2537bbbd1a9947ae9facd87567e3ccb9" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 33173 } } peers { permanent_uuid: "c561ec3374b94b95be0c155d80ff0bff" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 33627 } attrs { replace: true } } peers { permanent_uuid: "b5e9178a7a83402e8c4a6943ab78589d" member_type: VOTER last_known_addr { host: "127.17.60.197" port: 36573 } attrs { promote: false } } peers { permanent_uuid: "a2a5bba4ac1145d5bab94a69c9f4df40" member_type: VOTER last_known_addr { host: "127.17.60.196" port: 41807 } attrs { promote: false } } }
I20250112 15:07:42.855633 21474 raft_consensus.cc:2949] T 57e47b07fdcf434e96ca5dd0354e93b2 P a2a5bba4ac1145d5bab94a69c9f4df40 [term 3 FOLLOWER]: Committing config change with OpId 3.32: config changed from index 31 to 32, a2a5bba4ac1145d5bab94a69c9f4df40 (127.17.60.196) changed from NON_VOTER to VOTER. New config: { opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "2537bbbd1a9947ae9facd87567e3ccb9" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 33173 } } peers { permanent_uuid: "c561ec3374b94b95be0c155d80ff0bff" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 33627 } attrs { replace: true } } peers { permanent_uuid: "b5e9178a7a83402e8c4a6943ab78589d" member_type: VOTER last_known_addr { host: "127.17.60.197" port: 36573 } attrs { promote: false } } peers { permanent_uuid: "a2a5bba4ac1145d5bab94a69c9f4df40" member_type: VOTER last_known_addr { host: "127.17.60.196" port: 41807 } attrs { promote: false } } }
I20250112 15:07:42.856227 21548 consensus_queue.cc:237] T 57e47b07fdcf434e96ca5dd0354e93b2 P b5e9178a7a83402e8c4a6943ab78589d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 32, Committed index: 32, Last appended: 3.32, Last appended by leader: 29, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 33 OBSOLETE_local: false peers { permanent_uuid: "2537bbbd1a9947ae9facd87567e3ccb9" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 33173 } } peers { permanent_uuid: "b5e9178a7a83402e8c4a6943ab78589d" member_type: VOTER last_known_addr { host: "127.17.60.197" port: 36573 } attrs { promote: false } } peers { permanent_uuid: "a2a5bba4ac1145d5bab94a69c9f4df40" member_type: VOTER last_known_addr { host: "127.17.60.196" port: 41807 } attrs { promote: false } }
I20250112 15:07:42.856680 21474 raft_consensus.cc:1270] T 57e47b07fdcf434e96ca5dd0354e93b2 P a2a5bba4ac1145d5bab94a69c9f4df40 [term 3 FOLLOWER]: Refusing update from remote peer b5e9178a7a83402e8c4a6943ab78589d: Log matching property violated. Preceding OpId in replica: term: 3 index: 32. Preceding OpId from leader: term: 3 index: 33. (index mismatch)
I20250112 15:07:42.856729 21069 raft_consensus.cc:1270] T 57e47b07fdcf434e96ca5dd0354e93b2 P 2537bbbd1a9947ae9facd87567e3ccb9 [term 3 FOLLOWER]: Refusing update from remote peer b5e9178a7a83402e8c4a6943ab78589d: Log matching property violated. Preceding OpId in replica: term: 3 index: 32. Preceding OpId from leader: term: 3 index: 33. (index mismatch)
I20250112 15:07:42.856848 22026 consensus_queue.cc:1035] T 57e47b07fdcf434e96ca5dd0354e93b2 P b5e9178a7a83402e8c4a6943ab78589d [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2a5bba4ac1145d5bab94a69c9f4df40" member_type: VOTER last_known_addr { host: "127.17.60.196" port: 41807 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 33, Last known committed idx: 32, Time since last communication: 0.000s
I20250112 15:07:42.857036 22026 consensus_queue.cc:1035] T 57e47b07fdcf434e96ca5dd0354e93b2 P b5e9178a7a83402e8c4a6943ab78589d [LEADER]: Connected to new peer: Peer: permanent_uuid: "2537bbbd1a9947ae9facd87567e3ccb9" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 33173 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 33, Last known committed idx: 32, Time since last communication: 0.000s
I20250112 15:07:42.857535 22027 raft_consensus.cc:2949] T 57e47b07fdcf434e96ca5dd0354e93b2 P b5e9178a7a83402e8c4a6943ab78589d [term 3 LEADER]: Committing config change with OpId 3.33: config changed from index 32 to 33, VOTER c561ec3374b94b95be0c155d80ff0bff (127.17.60.194) evicted. New config: { opid_index: 33 OBSOLETE_local: false peers { permanent_uuid: "2537bbbd1a9947ae9facd87567e3ccb9" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 33173 } } peers { permanent_uuid: "b5e9178a7a83402e8c4a6943ab78589d" member_type: VOTER last_known_addr { host: "127.17.60.197" port: 36573 } attrs { promote: false } } peers { permanent_uuid: "a2a5bba4ac1145d5bab94a69c9f4df40" member_type: VOTER last_known_addr { host: "127.17.60.196" port: 41807 } attrs { promote: false } } }
I20250112 15:07:42.857640 21069 raft_consensus.cc:2949] T 57e47b07fdcf434e96ca5dd0354e93b2 P 2537bbbd1a9947ae9facd87567e3ccb9 [term 3 FOLLOWER]: Committing config change with OpId 3.33: config changed from index 32 to 33, VOTER c561ec3374b94b95be0c155d80ff0bff (127.17.60.194) evicted. New config: { opid_index: 33 OBSOLETE_local: false peers { permanent_uuid: "2537bbbd1a9947ae9facd87567e3ccb9" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 33173 } } peers { permanent_uuid: "b5e9178a7a83402e8c4a6943ab78589d" member_type: VOTER last_known_addr { host: "127.17.60.197" port: 36573 } attrs { promote: false } } peers { permanent_uuid: "a2a5bba4ac1145d5bab94a69c9f4df40" member_type: VOTER last_known_addr { host: "127.17.60.196" port: 41807 } attrs { promote: false } } }
I20250112 15:07:42.858558 20966 catalog_manager.cc:5526] T 57e47b07fdcf434e96ca5dd0354e93b2 P 2537bbbd1a9947ae9facd87567e3ccb9 reported cstate change: config changed from index 32 to 33, VOTER c561ec3374b94b95be0c155d80ff0bff (127.17.60.194) evicted. New cstate: current_term: 3 leader_uuid: "b5e9178a7a83402e8c4a6943ab78589d" committed_config { opid_index: 33 OBSOLETE_local: false peers { permanent_uuid: "2537bbbd1a9947ae9facd87567e3ccb9" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 33173 } } peers { permanent_uuid: "b5e9178a7a83402e8c4a6943ab78589d" member_type: VOTER last_known_addr { host: "127.17.60.197" port: 36573 } attrs { promote: false } } peers { permanent_uuid: "a2a5bba4ac1145d5bab94a69c9f4df40" member_type: VOTER last_known_addr { host: "127.17.60.196" port: 41807 } attrs { promote: false } } }
I20250112 15:07:42.858646 21474 raft_consensus.cc:2949] T 57e47b07fdcf434e96ca5dd0354e93b2 P a2a5bba4ac1145d5bab94a69c9f4df40 [term 3 FOLLOWER]: Committing config change with OpId 3.33: config changed from index 32 to 33, VOTER c561ec3374b94b95be0c155d80ff0bff (127.17.60.194) evicted. New config: { opid_index: 33 OBSOLETE_local: false peers { permanent_uuid: "2537bbbd1a9947ae9facd87567e3ccb9" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 33173 } } peers { permanent_uuid: "b5e9178a7a83402e8c4a6943ab78589d" member_type: VOTER last_known_addr { host: "127.17.60.197" port: 36573 } attrs { promote: false } } peers { permanent_uuid: "a2a5bba4ac1145d5bab94a69c9f4df40" member_type: VOTER last_known_addr { host: "127.17.60.196" port: 41807 } attrs { promote: false } } }
I20250112 15:07:42.858783 21015 catalog_manager.cc:5039] ChangeConfig:REMOVE_PEER RPC for tablet 57e47b07fdcf434e96ca5dd0354e93b2 with cas_config_opid_index 32: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250112 15:07:42.859311 21141 tablet_service.cc:1514] Processing DeleteTablet for tablet 57e47b07fdcf434e96ca5dd0354e93b2 with delete_type TABLET_DATA_TOMBSTONED (TS c561ec3374b94b95be0c155d80ff0bff not found in new config with opid_index 33) from {username='slave'} at 127.0.0.1:48072
I20250112 15:07:42.859442 22025 tablet_replica.cc:331] T 57e47b07fdcf434e96ca5dd0354e93b2 P c561ec3374b94b95be0c155d80ff0bff: stopping tablet replica
I20250112 15:07:42.859514 22025 raft_consensus.cc:2238] T 57e47b07fdcf434e96ca5dd0354e93b2 P c561ec3374b94b95be0c155d80ff0bff [term 3 FOLLOWER]: Raft consensus shutting down.
I20250112 15:07:42.859596 22025 raft_consensus.cc:2267] T 57e47b07fdcf434e96ca5dd0354e93b2 P c561ec3374b94b95be0c155d80ff0bff [term 3 FOLLOWER]: Raft consensus is shut down!
I20250112 15:07:42.859941 22025 ts_tablet_manager.cc:1905] T 57e47b07fdcf434e96ca5dd0354e93b2 P c561ec3374b94b95be0c155d80ff0bff: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250112 15:07:42.861351 22025 ts_tablet_manager.cc:1918] T 57e47b07fdcf434e96ca5dd0354e93b2 P c561ec3374b94b95be0c155d80ff0bff: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 3.32
I20250112 15:07:42.861394 22025 log.cc:1198] T 57e47b07fdcf434e96ca5dd0354e93b2 P c561ec3374b94b95be0c155d80ff0bff: Deleting WAL directory at /tmp/dist-test-task95dLSD/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1736694351070047-17651-0/minicluster-data/ts-1-root/wals/57e47b07fdcf434e96ca5dd0354e93b2
I20250112 15:07:42.861686 20958 catalog_manager.cc:4872] TS c561ec3374b94b95be0c155d80ff0bff (127.17.60.194:33627): tablet 57e47b07fdcf434e96ca5dd0354e93b2 (table test-workload [id=cdaf6d3a65a54ea08e8e2c0ecdfc9de7]) successfully deleted
I20250112 15:07:42.887058 21147 raft_consensus.cc:1270] T 8c7dbaf285a04da9b4eec92a29ab7d6c P c561ec3374b94b95be0c155d80ff0bff [term 4 FOLLOWER]: Refusing update from remote peer 3710b74ed2044ce5995533df32e67351: Log matching property violated. Preceding OpId in replica: term: 3 index: 28. Preceding OpId from leader: term: 4 index: 29. (index mismatch)
I20250112 15:07:42.887277 21914 consensus_queue.cc:1035] T 8c7dbaf285a04da9b4eec92a29ab7d6c P 3710b74ed2044ce5995533df32e67351 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c561ec3374b94b95be0c155d80ff0bff" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 33627 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.000s
I20250112 15:07:42.888438 21069 raft_consensus.cc:1270] T 8c7dbaf285a04da9b4eec92a29ab7d6c P 2537bbbd1a9947ae9facd87567e3ccb9 [term 4 FOLLOWER]: Refusing update from remote peer 3710b74ed2044ce5995533df32e67351: Log matching property violated. Preceding OpId in replica: term: 3 index: 28. Preceding OpId from leader: term: 4 index: 29. (index mismatch)
I20250112 15:07:42.888414 21474 raft_consensus.cc:3054] T 8c7dbaf285a04da9b4eec92a29ab7d6c P a2a5bba4ac1145d5bab94a69c9f4df40 [term 3 LEARNER]: Advancing to term 4
I20250112 15:07:42.888748 22035 consensus_queue.cc:1035] T 8c7dbaf285a04da9b4eec92a29ab7d6c P 3710b74ed2044ce5995533df32e67351 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2537bbbd1a9947ae9facd87567e3ccb9" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 33173 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.000s
I20250112 15:07:42.889488 21474 raft_consensus.cc:1270] T 8c7dbaf285a04da9b4eec92a29ab7d6c P a2a5bba4ac1145d5bab94a69c9f4df40 [term 4 LEARNER]: Refusing update from remote peer 3710b74ed2044ce5995533df32e67351: Log matching property violated. Preceding OpId in replica: term: 3 index: 28. Preceding OpId from leader: term: 4 index: 29. (index mismatch)
I20250112 15:07:42.889704 22035 consensus_queue.cc:1035] T 8c7dbaf285a04da9b4eec92a29ab7d6c P 3710b74ed2044ce5995533df32e67351 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2a5bba4ac1145d5bab94a69c9f4df40" member_type: NON_VOTER last_known_addr { host: "127.17.60.196" port: 41807 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.000s
I20250112 15:07:42.890420 21914 raft_consensus.cc:1059] T 8c7dbaf285a04da9b4eec92a29ab7d6c P 3710b74ed2044ce5995533df32e67351: attempting to promote NON_VOTER a2a5bba4ac1145d5bab94a69c9f4df40 to VOTER
I20250112 15:07:42.890568 21914 consensus_queue.cc:237] T 8c7dbaf285a04da9b4eec92a29ab7d6c P 3710b74ed2044ce5995533df32e67351 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 29, Committed index: 29, Last appended: 4.29, Last appended by leader: 28, Current term: 4, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "2537bbbd1a9947ae9facd87567e3ccb9" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 33173 } attrs { replace: true } } peers { permanent_uuid: "c561ec3374b94b95be0c155d80ff0bff" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 33627 } } peers { permanent_uuid: "3710b74ed2044ce5995533df32e67351" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 38589 } } peers { permanent_uuid: "a2a5bba4ac1145d5bab94a69c9f4df40" member_type: VOTER last_known_addr { host: "127.17.60.196" port: 41807 } attrs { promote: false } }
I20250112 15:07:42.890983 21069 raft_consensus.cc:1270] T 8c7dbaf285a04da9b4eec92a29ab7d6c P 2537bbbd1a9947ae9facd87567e3ccb9 [term 4 FOLLOWER]: Refusing update from remote peer 3710b74ed2044ce5995533df32e67351: Log matching property violated. Preceding OpId in replica: term: 4 index: 29. Preceding OpId from leader: term: 4 index: 30. (index mismatch)
I20250112 15:07:42.890997 21474 raft_consensus.cc:1270] T 8c7dbaf285a04da9b4eec92a29ab7d6c P a2a5bba4ac1145d5bab94a69c9f4df40 [term 4 LEARNER]: Refusing update from remote peer 3710b74ed2044ce5995533df32e67351: Log matching property violated. Preceding OpId in replica: term: 4 index: 29. Preceding OpId from leader: term: 4 index: 30. (index mismatch)
I20250112 15:07:42.891256 22042 consensus_queue.cc:1035] T 8c7dbaf285a04da9b4eec92a29ab7d6c P 3710b74ed2044ce5995533df32e67351 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2a5bba4ac1145d5bab94a69c9f4df40" member_type: VOTER last_known_addr { host: "127.17.60.196" port: 41807 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 30, Last known committed idx: 29, Time since last communication: 0.000s
I20250112 15:07:42.890983 21147 raft_consensus.cc:1270] T 8c7dbaf285a04da9b4eec92a29ab7d6c P c561ec3374b94b95be0c155d80ff0bff [term 4 FOLLOWER]: Refusing update from remote peer 3710b74ed2044ce5995533df32e67351: Log matching property violated. Preceding OpId in replica: term: 4 index: 29. Preceding OpId from leader: term: 4 index: 30. (index mismatch)
I20250112 15:07:42.891333 21914 consensus_queue.cc:1035] T 8c7dbaf285a04da9b4eec92a29ab7d6c P 3710b74ed2044ce5995533df32e67351 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2537bbbd1a9947ae9facd87567e3ccb9" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 33173 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 30, Last known committed idx: 29, Time since last communication: 0.000s
I20250112 15:07:42.891665 22042 consensus_queue.cc:1035] T 8c7dbaf285a04da9b4eec92a29ab7d6c P 3710b74ed2044ce5995533df32e67351 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c561ec3374b94b95be0c155d80ff0bff" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 33627 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 30, Last known committed idx: 29, Time since last communication: 0.000s
I20250112 15:07:42.892095 21914 raft_consensus.cc:2949] T 8c7dbaf285a04da9b4eec92a29ab7d6c P 3710b74ed2044ce5995533df32e67351 [term 4 LEADER]: Committing config change with OpId 4.30: config changed from index 28 to 30, a2a5bba4ac1145d5bab94a69c9f4df40 (127.17.60.196) changed from NON_VOTER to VOTER. New config: { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "2537bbbd1a9947ae9facd87567e3ccb9" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 33173 } attrs { replace: true } } peers { permanent_uuid: "c561ec3374b94b95be0c155d80ff0bff" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 33627 } } peers { permanent_uuid: "3710b74ed2044ce5995533df32e67351" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 38589 } } peers { permanent_uuid: "a2a5bba4ac1145d5bab94a69c9f4df40" member_type: VOTER last_known_addr { host: "127.17.60.196" port: 41807 } attrs { promote: false } } }
I20250112 15:07:42.892234 21149 raft_consensus.cc:2949] T 8c7dbaf285a04da9b4eec92a29ab7d6c P c561ec3374b94b95be0c155d80ff0bff [term 4 FOLLOWER]: Committing config change with OpId 4.30: config changed from index 28 to 30, a2a5bba4ac1145d5bab94a69c9f4df40 (127.17.60.196) changed from NON_VOTER to VOTER. New config: { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "2537bbbd1a9947ae9facd87567e3ccb9" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 33173 } attrs { replace: true } } peers { permanent_uuid: "c561ec3374b94b95be0c155d80ff0bff" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 33627 } } peers { permanent_uuid: "3710b74ed2044ce5995533df32e67351" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 38589 } } peers { permanent_uuid: "a2a5bba4ac1145d5bab94a69c9f4df40" member_type: VOTER last_known_addr { host: "127.17.60.196" port: 41807 } attrs { promote: false } } }
I20250112 15:07:42.892688 21474 raft_consensus.cc:2949] T 8c7dbaf285a04da9b4eec92a29ab7d6c P a2a5bba4ac1145d5bab94a69c9f4df40 [term 4 FOLLOWER]: Committing config change with OpId 4.30: config changed from index 28 to 30, a2a5bba4ac1145d5bab94a69c9f4df40 (127.17.60.196) changed from NON_VOTER to VOTER. New config: { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "2537bbbd1a9947ae9facd87567e3ccb9" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 33173 } attrs { replace: true } } peers { permanent_uuid: "c561ec3374b94b95be0c155d80ff0bff" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 33627 } } peers { permanent_uuid: "3710b74ed2044ce5995533df32e67351" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 38589 } } peers { permanent_uuid: "a2a5bba4ac1145d5bab94a69c9f4df40" member_type: VOTER last_known_addr { host: "127.17.60.196" port: 41807 } attrs { promote: false } } }
I20250112 15:07:42.893136 21069 raft_consensus.cc:2949] T 8c7dbaf285a04da9b4eec92a29ab7d6c P 2537bbbd1a9947ae9facd87567e3ccb9 [term 4 FOLLOWER]: Committing config change with OpId 4.30: config changed from index 28 to 30, a2a5bba4ac1145d5bab94a69c9f4df40 (127.17.60.196) changed from NON_VOTER to VOTER. New config: { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "2537bbbd1a9947ae9facd87567e3ccb9" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 33173 } attrs { replace: true } } peers { permanent_uuid: "c561ec3374b94b95be0c155d80ff0bff" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 33627 } } peers { permanent_uuid: "3710b74ed2044ce5995533df32e67351" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 38589 } } peers { permanent_uuid: "a2a5bba4ac1145d5bab94a69c9f4df40" member_type: VOTER last_known_addr { host: "127.17.60.196" port: 41807 } attrs { promote: false } } }
I20250112 15:07:42.893484 20966 catalog_manager.cc:5526] T 8c7dbaf285a04da9b4eec92a29ab7d6c P c561ec3374b94b95be0c155d80ff0bff reported cstate change: config changed from index 28 to 30, a2a5bba4ac1145d5bab94a69c9f4df40 (127.17.60.196) changed from NON_VOTER to VOTER. New cstate: current_term: 4 leader_uuid: "3710b74ed2044ce5995533df32e67351" committed_config { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "2537bbbd1a9947ae9facd87567e3ccb9" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 33173 } attrs { replace: true } } peers { permanent_uuid: "c561ec3374b94b95be0c155d80ff0bff" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 33627 } } peers { permanent_uuid: "3710b74ed2044ce5995533df32e67351" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 38589 } } peers { permanent_uuid: "a2a5bba4ac1145d5bab94a69c9f4df40" member_type: VOTER last_known_addr { host: "127.17.60.196" port: 41807 } attrs { promote: false } } }
I20250112 15:07:42.894539 21222 consensus_queue.cc:237] T 8c7dbaf285a04da9b4eec92a29ab7d6c P 3710b74ed2044ce5995533df32e67351 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 30, Committed index: 30, Last appended: 4.30, Last appended by leader: 28, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "c561ec3374b94b95be0c155d80ff0bff" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 33627 } } peers { permanent_uuid: "3710b74ed2044ce5995533df32e67351" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 38589 } } peers { permanent_uuid: "a2a5bba4ac1145d5bab94a69c9f4df40" member_type: VOTER last_known_addr { host: "127.17.60.196" port: 41807 } attrs { promote: false } }
I20250112 15:07:42.894889 21474 raft_consensus.cc:1270] T 8c7dbaf285a04da9b4eec92a29ab7d6c P a2a5bba4ac1145d5bab94a69c9f4df40 [term 4 FOLLOWER]: Refusing update from remote peer 3710b74ed2044ce5995533df32e67351: Log matching property violated. Preceding OpId in replica: term: 4 index: 30. Preceding OpId from leader: term: 4 index: 31. (index mismatch)
I20250112 15:07:42.894969 21149 raft_consensus.cc:1270] T 8c7dbaf285a04da9b4eec92a29ab7d6c P c561ec3374b94b95be0c155d80ff0bff [term 4 FOLLOWER]: Refusing update from remote peer 3710b74ed2044ce5995533df32e67351: Log matching property violated. Preceding OpId in replica: term: 4 index: 30. Preceding OpId from leader: term: 4 index: 31. (index mismatch)
I20250112 15:07:42.895073 22042 consensus_queue.cc:1035] T 8c7dbaf285a04da9b4eec92a29ab7d6c P 3710b74ed2044ce5995533df32e67351 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2a5bba4ac1145d5bab94a69c9f4df40" member_type: VOTER last_known_addr { host: "127.17.60.196" port: 41807 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 31, Last known committed idx: 30, Time since last communication: 0.000s
I20250112 15:07:42.895263 22035 consensus_queue.cc:1035] T 8c7dbaf285a04da9b4eec92a29ab7d6c P 3710b74ed2044ce5995533df32e67351 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c561ec3374b94b95be0c155d80ff0bff" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 33627 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 31, Last known committed idx: 30, Time since last communication: 0.000s
I20250112 15:07:42.895835 21914 raft_consensus.cc:2949] T 8c7dbaf285a04da9b4eec92a29ab7d6c P 3710b74ed2044ce5995533df32e67351 [term 4 LEADER]: Committing config change with OpId 4.31: config changed from index 30 to 31, VOTER 2537bbbd1a9947ae9facd87567e3ccb9 (127.17.60.193) evicted. New config: { opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "c561ec3374b94b95be0c155d80ff0bff" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 33627 } } peers { permanent_uuid: "3710b74ed2044ce5995533df32e67351" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 38589 } } peers { permanent_uuid: "a2a5bba4ac1145d5bab94a69c9f4df40" member_type: VOTER last_known_addr { host: "127.17.60.196" port: 41807 } attrs { promote: false } } }
I20250112 15:07:42.895967 21147 raft_consensus.cc:2949] T 8c7dbaf285a04da9b4eec92a29ab7d6c P c561ec3374b94b95be0c155d80ff0bff [term 4 FOLLOWER]: Committing config change with OpId 4.31: config changed from index 30 to 31, VOTER 2537bbbd1a9947ae9facd87567e3ccb9 (127.17.60.193) evicted. New config: { opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "c561ec3374b94b95be0c155d80ff0bff" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 33627 } } peers { permanent_uuid: "3710b74ed2044ce5995533df32e67351" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 38589 } } peers { permanent_uuid: "a2a5bba4ac1145d5bab94a69c9f4df40" member_type: VOTER last_known_addr { host: "127.17.60.196" port: 41807 } attrs { promote: false } } }
I20250112 15:07:42.895967 21474 raft_consensus.cc:2949] T 8c7dbaf285a04da9b4eec92a29ab7d6c P a2a5bba4ac1145d5bab94a69c9f4df40 [term 4 FOLLOWER]: Committing config change with OpId 4.31: config changed from index 30 to 31, VOTER 2537bbbd1a9947ae9facd87567e3ccb9 (127.17.60.193) evicted. New config: { opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "c561ec3374b94b95be0c155d80ff0bff" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 33627 } } peers { permanent_uuid: "3710b74ed2044ce5995533df32e67351" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 38589 } } peers { permanent_uuid: "a2a5bba4ac1145d5bab94a69c9f4df40" member_type: VOTER last_known_addr { host: "127.17.60.196" port: 41807 } attrs { promote: false } } }
I20250112 15:07:42.896646 20957 catalog_manager.cc:5039] ChangeConfig:REMOVE_PEER RPC for tablet 8c7dbaf285a04da9b4eec92a29ab7d6c with cas_config_opid_index 30: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250112 15:07:42.896745 20964 catalog_manager.cc:5526] T 8c7dbaf285a04da9b4eec92a29ab7d6c P 3710b74ed2044ce5995533df32e67351 reported cstate change: config changed from index 30 to 31, VOTER 2537bbbd1a9947ae9facd87567e3ccb9 (127.17.60.193) evicted. New cstate: current_term: 4 leader_uuid: "3710b74ed2044ce5995533df32e67351" committed_config { opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "c561ec3374b94b95be0c155d80ff0bff" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 33627 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3710b74ed2044ce5995533df32e67351" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 38589 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a2a5bba4ac1145d5bab94a69c9f4df40" member_type: VOTER last_known_addr { host: "127.17.60.196" port: 41807 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20250112 15:07:42.897460 21065 tablet_service.cc:1514] Processing DeleteTablet for tablet 8c7dbaf285a04da9b4eec92a29ab7d6c with delete_type TABLET_DATA_TOMBSTONED (TS 2537bbbd1a9947ae9facd87567e3ccb9 not found in new config with opid_index 31) from {username='slave'} at 127.0.0.1:37188
I20250112 15:07:42.897688 22043 tablet_replica.cc:331] T 8c7dbaf285a04da9b4eec92a29ab7d6c P 2537bbbd1a9947ae9facd87567e3ccb9: stopping tablet replica
I20250112 15:07:42.897763 22043 raft_consensus.cc:2238] T 8c7dbaf285a04da9b4eec92a29ab7d6c P 2537bbbd1a9947ae9facd87567e3ccb9 [term 4 FOLLOWER]: Raft consensus shutting down.
I20250112 15:07:42.897820 22043 raft_consensus.cc:2267] T 8c7dbaf285a04da9b4eec92a29ab7d6c P 2537bbbd1a9947ae9facd87567e3ccb9 [term 4 FOLLOWER]: Raft consensus is shut down!
I20250112 15:07:42.898051 22043 ts_tablet_manager.cc:1905] T 8c7dbaf285a04da9b4eec92a29ab7d6c P 2537bbbd1a9947ae9facd87567e3ccb9: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250112 15:07:42.899494 22043 ts_tablet_manager.cc:1918] T 8c7dbaf285a04da9b4eec92a29ab7d6c P 2537bbbd1a9947ae9facd87567e3ccb9: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 4.30
I20250112 15:07:42.899554 22043 log.cc:1198] T 8c7dbaf285a04da9b4eec92a29ab7d6c P 2537bbbd1a9947ae9facd87567e3ccb9: Deleting WAL directory at /tmp/dist-test-task95dLSD/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1736694351070047-17651-0/minicluster-data/ts-0-root/wals/8c7dbaf285a04da9b4eec92a29ab7d6c
I20250112 15:07:42.899904 20956 catalog_manager.cc:4872] TS 2537bbbd1a9947ae9facd87567e3ccb9 (127.17.60.193:33173): tablet 8c7dbaf285a04da9b4eec92a29ab7d6c (table test-workload [id=cdaf6d3a65a54ea08e8e2c0ecdfc9de7]) successfully deleted
I20250112 15:07:42.990339 21624 raft_consensus.cc:3054] T 9601bcda24ed42e7b93ab9740f1e271e P 25286c485fa04c86a5d07c8f88643a47 [term 2 LEARNER]: Advancing to term 3
I20250112 15:07:42.991109 21624 raft_consensus.cc:1270] T 9601bcda24ed42e7b93ab9740f1e271e P 25286c485fa04c86a5d07c8f88643a47 [term 3 LEARNER]: Refusing update from remote peer 2537bbbd1a9947ae9facd87567e3ccb9: Log matching property violated. Preceding OpId in replica: term: 2 index: 29. Preceding OpId from leader: term: 3 index: 30. (index mismatch)
I20250112 15:07:42.991379 21969 consensus_queue.cc:1035] T 9601bcda24ed42e7b93ab9740f1e271e P 2537bbbd1a9947ae9facd87567e3ccb9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "25286c485fa04c86a5d07c8f88643a47" member_type: NON_VOTER last_known_addr { host: "127.17.60.198" port: 37089 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 30, Last known committed idx: 29, Time since last communication: 0.000s
I20250112 15:07:42.992161 22009 raft_consensus.cc:1059] T 9601bcda24ed42e7b93ab9740f1e271e P 2537bbbd1a9947ae9facd87567e3ccb9: attempting to promote NON_VOTER 25286c485fa04c86a5d07c8f88643a47 to VOTER
W20250112 15:07:42.992221 22009 raft_consensus.cc:1062] T 9601bcda24ed42e7b93ab9740f1e271e P 2537bbbd1a9947ae9facd87567e3ccb9: Unable to promote non-voter 25286c485fa04c86a5d07c8f88643a47: Service unavailable: leader transfer in progress
I20250112 15:07:42.999922 21069 raft_consensus.cc:1270] T bfaeae584ec04772b2d6fcd088022117 P 2537bbbd1a9947ae9facd87567e3ccb9 [term 3 FOLLOWER]: Refusing update from remote peer c561ec3374b94b95be0c155d80ff0bff: Log matching property violated. Preceding OpId in replica: term: 2 index: 27. Preceding OpId from leader: term: 3 index: 28. (index mismatch)
I20250112 15:07:43.000223 22010 consensus_queue.cc:1035] T bfaeae584ec04772b2d6fcd088022117 P c561ec3374b94b95be0c155d80ff0bff [LEADER]: Connected to new peer: Peer: permanent_uuid: "2537bbbd1a9947ae9facd87567e3ccb9" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 33173 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 28, Last known committed idx: 27, Time since last communication: 0.000s
I20250112 15:07:43.001318 21222 raft_consensus.cc:1270] T bfaeae584ec04772b2d6fcd088022117 P 3710b74ed2044ce5995533df32e67351 [term 3 FOLLOWER]: Refusing update from remote peer c561ec3374b94b95be0c155d80ff0bff: Log matching property violated. Preceding OpId in replica: term: 2 index: 27. Preceding OpId from leader: term: 3 index: 28. (index mismatch)
I20250112 15:07:43.001400 21624 raft_consensus.cc:3054] T bfaeae584ec04772b2d6fcd088022117 P 25286c485fa04c86a5d07c8f88643a47 [term 2 LEARNER]: Advancing to term 3
I20250112 15:07:43.002128 21624 raft_consensus.cc:1270] T bfaeae584ec04772b2d6fcd088022117 P 25286c485fa04c86a5d07c8f88643a47 [term 3 LEARNER]: Refusing update from remote peer c561ec3374b94b95be0c155d80ff0bff: Log matching property violated. Preceding OpId in replica: term: 2 index: 27. Preceding OpId from leader: term: 3 index: 28. (index mismatch)
I20250112 15:07:43.002388 22003 consensus_queue.cc:1035] T bfaeae584ec04772b2d6fcd088022117 P c561ec3374b94b95be0c155d80ff0bff [LEADER]: Connected to new peer: Peer: permanent_uuid: "25286c485fa04c86a5d07c8f88643a47" member_type: NON_VOTER last_known_addr { host: "127.17.60.198" port: 37089 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 28, Last known committed idx: 27, Time since last communication: 0.000s
I20250112 15:07:43.002511 22003 consensus_queue.cc:1035] T bfaeae584ec04772b2d6fcd088022117 P c561ec3374b94b95be0c155d80ff0bff [LEADER]: Connected to new peer: Peer: permanent_uuid: "3710b74ed2044ce5995533df32e67351" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 38589 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 28, Last known committed idx: 27, Time since last communication: 0.000s
I20250112 15:07:43.003140 22003 raft_consensus.cc:1059] T bfaeae584ec04772b2d6fcd088022117 P c561ec3374b94b95be0c155d80ff0bff: attempting to promote NON_VOTER 25286c485fa04c86a5d07c8f88643a47 to VOTER
I20250112 15:07:43.003311 22003 consensus_queue.cc:237] T bfaeae584ec04772b2d6fcd088022117 P c561ec3374b94b95be0c155d80ff0bff [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 28, Committed index: 28, Last appended: 3.28, Last appended by leader: 27, Current term: 3, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "2537bbbd1a9947ae9facd87567e3ccb9" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 33173 } } peers { permanent_uuid: "c561ec3374b94b95be0c155d80ff0bff" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 33627 } } peers { permanent_uuid: "3710b74ed2044ce5995533df32e67351" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 38589 } attrs { replace: true } } peers { permanent_uuid: "25286c485fa04c86a5d07c8f88643a47" member_type: VOTER last_known_addr { host: "127.17.60.198" port: 37089 } attrs { promote: false } }
I20250112 15:07:43.003716 21069 raft_consensus.cc:1270] T bfaeae584ec04772b2d6fcd088022117 P 2537bbbd1a9947ae9facd87567e3ccb9 [term 3 FOLLOWER]: Refusing update from remote peer c561ec3374b94b95be0c155d80ff0bff: Log matching property violated. Preceding OpId in replica: term: 3 index: 28. Preceding OpId from leader: term: 3 index: 29. (index mismatch)
I20250112 15:07:43.003845 21624 raft_consensus.cc:1270] T bfaeae584ec04772b2d6fcd088022117 P 25286c485fa04c86a5d07c8f88643a47 [term 3 LEARNER]: Refusing update from remote peer c561ec3374b94b95be0c155d80ff0bff: Log matching property violated. Preceding OpId in replica: term: 3 index: 28. Preceding OpId from leader: term: 3 index: 29. (index mismatch)
I20250112 15:07:43.003856 21222 raft_consensus.cc:1270] T bfaeae584ec04772b2d6fcd088022117 P 3710b74ed2044ce5995533df32e67351 [term 3 FOLLOWER]: Refusing update from remote peer c561ec3374b94b95be0c155d80ff0bff: Log matching property violated. Preceding OpId in replica: term: 3 index: 28. Preceding OpId from leader: term: 3 index: 29. (index mismatch)
I20250112 15:07:43.004020 21935 consensus_queue.cc:1035] T bfaeae584ec04772b2d6fcd088022117 P c561ec3374b94b95be0c155d80ff0bff [LEADER]: Connected to new peer: Peer: permanent_uuid: "25286c485fa04c86a5d07c8f88643a47" member_type: VOTER last_known_addr { host: "127.17.60.198" port: 37089 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.000s
I20250112 15:07:43.004123 21935 consensus_queue.cc:1035] T bfaeae584ec04772b2d6fcd088022117 P c561ec3374b94b95be0c155d80ff0bff [LEADER]: Connected to new peer: Peer: permanent_uuid: "2537bbbd1a9947ae9facd87567e3ccb9" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 33173 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.000s
I20250112 15:07:43.004177 22015 consensus_queue.cc:1035] T bfaeae584ec04772b2d6fcd088022117 P c561ec3374b94b95be0c155d80ff0bff [LEADER]: Connected to new peer: Peer: permanent_uuid: "3710b74ed2044ce5995533df32e67351" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 38589 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.000s
I20250112 15:07:43.004879 22010 raft_consensus.cc:2949] T bfaeae584ec04772b2d6fcd088022117 P c561ec3374b94b95be0c155d80ff0bff [term 3 LEADER]: Committing config change with OpId 3.29: config changed from index 27 to 29, 25286c485fa04c86a5d07c8f88643a47 (127.17.60.198) changed from NON_VOTER to VOTER. New config: { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "2537bbbd1a9947ae9facd87567e3ccb9" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 33173 } } peers { permanent_uuid: "c561ec3374b94b95be0c155d80ff0bff" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 33627 } } peers { permanent_uuid: "3710b74ed2044ce5995533df32e67351" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 38589 } attrs { replace: true } } peers { permanent_uuid: "25286c485fa04c86a5d07c8f88643a47" member_type: VOTER last_known_addr { host: "127.17.60.198" port: 37089 } attrs { promote: false } } }
I20250112 15:07:43.005080 21222 raft_consensus.cc:2949] T bfaeae584ec04772b2d6fcd088022117 P 3710b74ed2044ce5995533df32e67351 [term 3 FOLLOWER]: Committing config change with OpId 3.29: config changed from index 27 to 29, 25286c485fa04c86a5d07c8f88643a47 (127.17.60.198) changed from NON_VOTER to VOTER. New config: { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "2537bbbd1a9947ae9facd87567e3ccb9" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 33173 } } peers { permanent_uuid: "c561ec3374b94b95be0c155d80ff0bff" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 33627 } } peers { permanent_uuid: "3710b74ed2044ce5995533df32e67351" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 38589 } attrs { replace: true } } peers { permanent_uuid: "25286c485fa04c86a5d07c8f88643a47" member_type: VOTER last_known_addr { host: "127.17.60.198" port: 37089 } attrs { promote: false } } }
I20250112 15:07:43.005872 21069 raft_consensus.cc:2949] T bfaeae584ec04772b2d6fcd088022117 P 2537bbbd1a9947ae9facd87567e3ccb9 [term 3 FOLLOWER]: Committing config change with OpId 3.29: config changed from index 27 to 29, 25286c485fa04c86a5d07c8f88643a47 (127.17.60.198) changed from NON_VOTER to VOTER. New config: { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "2537bbbd1a9947ae9facd87567e3ccb9" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 33173 } } peers { permanent_uuid: "c561ec3374b94b95be0c155d80ff0bff" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 33627 } } peers { permanent_uuid: "3710b74ed2044ce5995533df32e67351" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 38589 } attrs { replace: true } } peers { permanent_uuid: "25286c485fa04c86a5d07c8f88643a47" member_type: VOTER last_known_addr { host: "127.17.60.198" port: 37089 } attrs { promote: false } } }
I20250112 15:07:43.006040 20965 catalog_manager.cc:5526] T bfaeae584ec04772b2d6fcd088022117 P 3710b74ed2044ce5995533df32e67351 reported cstate change: config changed from index 27 to 29, 25286c485fa04c86a5d07c8f88643a47 (127.17.60.198) changed from NON_VOTER to VOTER. New cstate: current_term: 3 leader_uuid: "c561ec3374b94b95be0c155d80ff0bff" committed_config { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "2537bbbd1a9947ae9facd87567e3ccb9" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 33173 } } peers { permanent_uuid: "c561ec3374b94b95be0c155d80ff0bff" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 33627 } } peers { permanent_uuid: "3710b74ed2044ce5995533df32e67351" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 38589 } attrs { replace: true } } peers { permanent_uuid: "25286c485fa04c86a5d07c8f88643a47" member_type: VOTER last_known_addr { host: "127.17.60.198" port: 37089 } attrs { promote: false } } }
I20250112 15:07:43.007064 21624 raft_consensus.cc:2949] T bfaeae584ec04772b2d6fcd088022117 P 25286c485fa04c86a5d07c8f88643a47 [term 3 FOLLOWER]: Committing config change with OpId 3.29: config changed from index 27 to 29, 25286c485fa04c86a5d07c8f88643a47 (127.17.60.198) changed from NON_VOTER to VOTER. New config: { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "2537bbbd1a9947ae9facd87567e3ccb9" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 33173 } } peers { permanent_uuid: "c561ec3374b94b95be0c155d80ff0bff" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 33627 } } peers { permanent_uuid: "3710b74ed2044ce5995533df32e67351" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 38589 } attrs { replace: true } } peers { permanent_uuid: "25286c485fa04c86a5d07c8f88643a47" member_type: VOTER last_known_addr { host: "127.17.60.198" port: 37089 } attrs { promote: false } } }
I20250112 15:07:43.007061 21147 consensus_queue.cc:237] T bfaeae584ec04772b2d6fcd088022117 P c561ec3374b94b95be0c155d80ff0bff [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 29, Committed index: 29, Last appended: 3.29, Last appended by leader: 27, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "2537bbbd1a9947ae9facd87567e3ccb9" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 33173 } } peers { permanent_uuid: "c561ec3374b94b95be0c155d80ff0bff" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 33627 } } peers { permanent_uuid: "25286c485fa04c86a5d07c8f88643a47" member_type: VOTER last_known_addr { host: "127.17.60.198" port: 37089 } attrs { promote: false } }
I20250112 15:07:43.008102 21624 raft_consensus.cc:1270] T bfaeae584ec04772b2d6fcd088022117 P 25286c485fa04c86a5d07c8f88643a47 [term 3 FOLLOWER]: Refusing update from remote peer c561ec3374b94b95be0c155d80ff0bff: Log matching property violated. Preceding OpId in replica: term: 3 index: 29. Preceding OpId from leader: term: 3 index: 30. (index mismatch)
I20250112 15:07:43.008102 21069 raft_consensus.cc:1270] T bfaeae584ec04772b2d6fcd088022117 P 2537bbbd1a9947ae9facd87567e3ccb9 [term 3 FOLLOWER]: Refusing update from remote peer c561ec3374b94b95be0c155d80ff0bff: Log matching property violated. Preceding OpId in replica: term: 3 index: 29. Preceding OpId from leader: term: 3 index: 30. (index mismatch)
I20250112 15:07:43.008273 22003 consensus_queue.cc:1035] T bfaeae584ec04772b2d6fcd088022117 P c561ec3374b94b95be0c155d80ff0bff [LEADER]: Connected to new peer: Peer: permanent_uuid: "25286c485fa04c86a5d07c8f88643a47" member_type: VOTER last_known_addr { host: "127.17.60.198" port: 37089 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 30, Last known committed idx: 29, Time since last communication: 0.000s
I20250112 15:07:43.008358 22015 consensus_queue.cc:1035] T bfaeae584ec04772b2d6fcd088022117 P c561ec3374b94b95be0c155d80ff0bff [LEADER]: Connected to new peer: Peer: permanent_uuid: "2537bbbd1a9947ae9facd87567e3ccb9" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 33173 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 30, Last known committed idx: 29, Time since last communication: 0.000s
I20250112 15:07:43.008868 22015 raft_consensus.cc:2949] T bfaeae584ec04772b2d6fcd088022117 P c561ec3374b94b95be0c155d80ff0bff [term 3 LEADER]: Committing config change with OpId 3.30: config changed from index 29 to 30, VOTER 3710b74ed2044ce5995533df32e67351 (127.17.60.195) evicted. New config: { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "2537bbbd1a9947ae9facd87567e3ccb9" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 33173 } } peers { permanent_uuid: "c561ec3374b94b95be0c155d80ff0bff" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 33627 } } peers { permanent_uuid: "25286c485fa04c86a5d07c8f88643a47" member_type: VOTER last_known_addr { host: "127.17.60.198" port: 37089 } attrs { promote: false } } }
I20250112 15:07:43.009563 20958 catalog_manager.cc:5039] ChangeConfig:REMOVE_PEER RPC for tablet bfaeae584ec04772b2d6fcd088022117 with cas_config_opid_index 29: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250112 15:07:43.009718 20970 catalog_manager.cc:5526] T bfaeae584ec04772b2d6fcd088022117 P c561ec3374b94b95be0c155d80ff0bff reported cstate change: config changed from index 29 to 30, VOTER 3710b74ed2044ce5995533df32e67351 (127.17.60.195) evicted. New cstate: current_term: 3 leader_uuid: "c561ec3374b94b95be0c155d80ff0bff" committed_config { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "2537bbbd1a9947ae9facd87567e3ccb9" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 33173 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c561ec3374b94b95be0c155d80ff0bff" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 33627 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "25286c485fa04c86a5d07c8f88643a47" member_type: VOTER last_known_addr { host: "127.17.60.198" port: 37089 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20250112 15:07:43.009756 21069 raft_consensus.cc:2949] T bfaeae584ec04772b2d6fcd088022117 P 2537bbbd1a9947ae9facd87567e3ccb9 [term 3 FOLLOWER]: Committing config change with OpId 3.30: config changed from index 29 to 30, VOTER 3710b74ed2044ce5995533df32e67351 (127.17.60.195) evicted. New config: { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "2537bbbd1a9947ae9facd87567e3ccb9" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 33173 } } peers { permanent_uuid: "c561ec3374b94b95be0c155d80ff0bff" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 33627 } } peers { permanent_uuid: "25286c485fa04c86a5d07c8f88643a47" member_type: VOTER last_known_addr { host: "127.17.60.198" port: 37089 } attrs { promote: false } } }
I20250112 15:07:43.009989 21624 raft_consensus.cc:2949] T bfaeae584ec04772b2d6fcd088022117 P 25286c485fa04c86a5d07c8f88643a47 [term 3 FOLLOWER]: Committing config change with OpId 3.30: config changed from index 29 to 30, VOTER 3710b74ed2044ce5995533df32e67351 (127.17.60.195) evicted. New config: { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "2537bbbd1a9947ae9facd87567e3ccb9" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 33173 } } peers { permanent_uuid: "c561ec3374b94b95be0c155d80ff0bff" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 33627 } } peers { permanent_uuid: "25286c485fa04c86a5d07c8f88643a47" member_type: VOTER last_known_addr { host: "127.17.60.198" port: 37089 } attrs { promote: false } } }
I20250112 15:07:43.010758 21212 tablet_service.cc:1514] Processing DeleteTablet for tablet bfaeae584ec04772b2d6fcd088022117 with delete_type TABLET_DATA_TOMBSTONED (TS 3710b74ed2044ce5995533df32e67351 not found in new config with opid_index 30) from {username='slave'} at 127.0.0.1:33192
I20250112 15:07:43.010879 22018 tablet_replica.cc:331] T bfaeae584ec04772b2d6fcd088022117 P 3710b74ed2044ce5995533df32e67351: stopping tablet replica
I20250112 15:07:43.010967 22018 raft_consensus.cc:2238] T bfaeae584ec04772b2d6fcd088022117 P 3710b74ed2044ce5995533df32e67351 [term 3 FOLLOWER]: Raft consensus shutting down.
I20250112 15:07:43.011046 22018 raft_consensus.cc:2267] T bfaeae584ec04772b2d6fcd088022117 P 3710b74ed2044ce5995533df32e67351 [term 3 FOLLOWER]: Raft consensus is shut down!
I20250112 15:07:43.011324 22018 ts_tablet_manager.cc:1905] T bfaeae584ec04772b2d6fcd088022117 P 3710b74ed2044ce5995533df32e67351: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250112 15:07:43.012346 22018 ts_tablet_manager.cc:1918] T bfaeae584ec04772b2d6fcd088022117 P 3710b74ed2044ce5995533df32e67351: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 3.29
I20250112 15:07:43.012389 22018 log.cc:1198] T bfaeae584ec04772b2d6fcd088022117 P 3710b74ed2044ce5995533df32e67351: Deleting WAL directory at /tmp/dist-test-task95dLSD/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1736694351070047-17651-0/minicluster-data/ts-2-root/wals/bfaeae584ec04772b2d6fcd088022117
I20250112 15:07:43.012707 20957 catalog_manager.cc:4872] TS 3710b74ed2044ce5995533df32e67351 (127.17.60.195:38589): tablet bfaeae584ec04772b2d6fcd088022117 (table test-workload [id=cdaf6d3a65a54ea08e8e2c0ecdfc9de7]) successfully deleted
I20250112 15:07:43.084792 21222 raft_consensus.cc:1270] T 9601bcda24ed42e7b93ab9740f1e271e P 3710b74ed2044ce5995533df32e67351 [term 3 FOLLOWER]: Refusing update from remote peer 2537bbbd1a9947ae9facd87567e3ccb9: Log matching property violated. Preceding OpId in replica: term: 2 index: 29. Preceding OpId from leader: term: 3 index: 30. (index mismatch)
I20250112 15:07:43.085130 22009 consensus_queue.cc:1035] T 9601bcda24ed42e7b93ab9740f1e271e P 2537bbbd1a9947ae9facd87567e3ccb9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3710b74ed2044ce5995533df32e67351" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 38589 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 30, Last known committed idx: 29, Time since last communication: 0.000s
I20250112 15:07:43.085994 22021 raft_consensus.cc:988] T 9601bcda24ed42e7b93ab9740f1e271e P 2537bbbd1a9947ae9facd87567e3ccb9: : Instructing follower 3710b74ed2044ce5995533df32e67351 to start an election
I20250112 15:07:43.086063 22021 raft_consensus.cc:1076] T 9601bcda24ed42e7b93ab9740f1e271e P 2537bbbd1a9947ae9facd87567e3ccb9 [term 3 LEADER]: Signalling peer 3710b74ed2044ce5995533df32e67351 to start an election
I20250112 15:07:43.086359 21474 raft_consensus.cc:1270] T 9601bcda24ed42e7b93ab9740f1e271e P a2a5bba4ac1145d5bab94a69c9f4df40 [term 3 FOLLOWER]: Refusing update from remote peer 2537bbbd1a9947ae9facd87567e3ccb9: Log matching property violated. Preceding OpId in replica: term: 2 index: 29. Preceding OpId from leader: term: 3 index: 30. (index mismatch)
I20250112 15:07:43.086650 21222 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "9601bcda24ed42e7b93ab9740f1e271e"
dest_uuid: "3710b74ed2044ce5995533df32e67351"
from {username='slave'} at 127.0.0.1:33194
I20250112 15:07:43.086721 21222 raft_consensus.cc:491] T 9601bcda24ed42e7b93ab9740f1e271e P 3710b74ed2044ce5995533df32e67351 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20250112 15:07:43.086732 22009 consensus_queue.cc:1035] T 9601bcda24ed42e7b93ab9740f1e271e P 2537bbbd1a9947ae9facd87567e3ccb9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2a5bba4ac1145d5bab94a69c9f4df40" member_type: VOTER last_known_addr { host: "127.17.60.196" port: 41807 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 30, Last known committed idx: 29, Time since last communication: 0.000s
I20250112 15:07:43.086755 21222 raft_consensus.cc:3054] T 9601bcda24ed42e7b93ab9740f1e271e P 3710b74ed2044ce5995533df32e67351 [term 3 FOLLOWER]: Advancing to term 4
I20250112 15:07:43.087028 22009 raft_consensus.cc:1059] T 9601bcda24ed42e7b93ab9740f1e271e P 2537bbbd1a9947ae9facd87567e3ccb9: attempting to promote NON_VOTER 25286c485fa04c86a5d07c8f88643a47 to VOTER
W20250112 15:07:43.087075 22009 raft_consensus.cc:1062] T 9601bcda24ed42e7b93ab9740f1e271e P 2537bbbd1a9947ae9facd87567e3ccb9: Unable to promote non-voter 25286c485fa04c86a5d07c8f88643a47: Service unavailable: leader transfer in progress
I20250112 15:07:43.087661 21222 raft_consensus.cc:513] T 9601bcda24ed42e7b93ab9740f1e271e P 3710b74ed2044ce5995533df32e67351 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "2537bbbd1a9947ae9facd87567e3ccb9" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 33173 } attrs { replace: true } } peers { permanent_uuid: "3710b74ed2044ce5995533df32e67351" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 38589 } } peers { permanent_uuid: "a2a5bba4ac1145d5bab94a69c9f4df40" member_type: VOTER last_known_addr { host: "127.17.60.196" port: 41807 } attrs { promote: false } } peers { permanent_uuid: "25286c485fa04c86a5d07c8f88643a47" member_type: NON_VOTER last_known_addr { host: "127.17.60.198" port: 37089 } attrs { promote: true } }
I20250112 15:07:43.087785 21222 leader_election.cc:290] T 9601bcda24ed42e7b93ab9740f1e271e P 3710b74ed2044ce5995533df32e67351 [CANDIDATE]: Term 4 election: Requested vote from peers 2537bbbd1a9947ae9facd87567e3ccb9 (127.17.60.193:33173), a2a5bba4ac1145d5bab94a69c9f4df40 (127.17.60.196:41807)
I20250112 15:07:43.087911 21069 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9601bcda24ed42e7b93ab9740f1e271e" candidate_uuid: "3710b74ed2044ce5995533df32e67351" candidate_term: 4 candidate_status { last_received { term: 3 index: 30 } } ignore_live_leader: true dest_uuid: "2537bbbd1a9947ae9facd87567e3ccb9"
I20250112 15:07:43.087934 21473 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9601bcda24ed42e7b93ab9740f1e271e" candidate_uuid: "3710b74ed2044ce5995533df32e67351" candidate_term: 4 candidate_status { last_received { term: 3 index: 30 } } ignore_live_leader: true dest_uuid: "a2a5bba4ac1145d5bab94a69c9f4df40"
I20250112 15:07:43.087980 21069 raft_consensus.cc:3049] T 9601bcda24ed42e7b93ab9740f1e271e P 2537bbbd1a9947ae9facd87567e3ccb9 [term 3 LEADER]: Stepping down as leader of term 3
I20250112 15:07:43.087991 21473 raft_consensus.cc:3054] T 9601bcda24ed42e7b93ab9740f1e271e P a2a5bba4ac1145d5bab94a69c9f4df40 [term 3 FOLLOWER]: Advancing to term 4
I20250112 15:07:43.088008 21069 raft_consensus.cc:738] T 9601bcda24ed42e7b93ab9740f1e271e P 2537bbbd1a9947ae9facd87567e3ccb9 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 2537bbbd1a9947ae9facd87567e3ccb9, State: Running, Role: LEADER
I20250112 15:07:43.088059 21069 consensus_queue.cc:260] T 9601bcda24ed42e7b93ab9740f1e271e P 2537bbbd1a9947ae9facd87567e3ccb9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 30, Committed index: 30, Last appended: 3.30, Last appended by leader: 30, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "2537bbbd1a9947ae9facd87567e3ccb9" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 33173 } attrs { replace: true } } peers { permanent_uuid: "3710b74ed2044ce5995533df32e67351" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 38589 } } peers { permanent_uuid: "a2a5bba4ac1145d5bab94a69c9f4df40" member_type: VOTER last_known_addr { host: "127.17.60.196" port: 41807 } attrs { promote: false } } peers { permanent_uuid: "25286c485fa04c86a5d07c8f88643a47" member_type: NON_VOTER last_known_addr { host: "127.17.60.198" port: 37089 } attrs { promote: true } }
I20250112 15:07:43.088145 21069 raft_consensus.cc:3054] T 9601bcda24ed42e7b93ab9740f1e271e P 2537bbbd1a9947ae9facd87567e3ccb9 [term 3 FOLLOWER]: Advancing to term 4
I20250112 15:07:43.088766 21473 raft_consensus.cc:2463] T 9601bcda24ed42e7b93ab9740f1e271e P a2a5bba4ac1145d5bab94a69c9f4df40 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3710b74ed2044ce5995533df32e67351 in term 4.
I20250112 15:07:43.088766 21069 raft_consensus.cc:2463] T 9601bcda24ed42e7b93ab9740f1e271e P 2537bbbd1a9947ae9facd87567e3ccb9 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3710b74ed2044ce5995533df32e67351 in term 4.
I20250112 15:07:43.088825 21474 raft_consensus.cc:1235] T 9601bcda24ed42e7b93ab9740f1e271e P a2a5bba4ac1145d5bab94a69c9f4df40 [term 4 FOLLOWER]: Rejecting Update request from peer 2537bbbd1a9947ae9facd87567e3ccb9 for earlier term 3. Current term is 4. Ops: [3.30-3.30]
I20250112 15:07:43.088949 21189 leader_election.cc:304] T 9601bcda24ed42e7b93ab9740f1e271e P 3710b74ed2044ce5995533df32e67351 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2537bbbd1a9947ae9facd87567e3ccb9, 3710b74ed2044ce5995533df32e67351; no voters:
I20250112 15:07:43.089040 22001 raft_consensus.cc:2798] T 9601bcda24ed42e7b93ab9740f1e271e P 3710b74ed2044ce5995533df32e67351 [term 4 FOLLOWER]: Leader election won for term 4
I20250112 15:07:43.089094 22001 raft_consensus.cc:695] T 9601bcda24ed42e7b93ab9740f1e271e P 3710b74ed2044ce5995533df32e67351 [term 4 LEADER]: Becoming Leader. State: Replica: 3710b74ed2044ce5995533df32e67351, State: Running, Role: LEADER
I20250112 15:07:43.089146 22001 consensus_queue.cc:237] T 9601bcda24ed42e7b93ab9740f1e271e P 3710b74ed2044ce5995533df32e67351 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 30, Committed index: 30, Last appended: 3.30, Last appended by leader: 30, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "2537bbbd1a9947ae9facd87567e3ccb9" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 33173 } attrs { replace: true } } peers { permanent_uuid: "3710b74ed2044ce5995533df32e67351" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 38589 } } peers { permanent_uuid: "a2a5bba4ac1145d5bab94a69c9f4df40" member_type: VOTER last_known_addr { host: "127.17.60.196" port: 41807 } attrs { promote: false } } peers { permanent_uuid: "25286c485fa04c86a5d07c8f88643a47" member_type: NON_VOTER last_known_addr { host: "127.17.60.198" port: 37089 } attrs { promote: true } }
I20250112 15:07:43.089495 20970 catalog_manager.cc:5526] T 9601bcda24ed42e7b93ab9740f1e271e P 3710b74ed2044ce5995533df32e67351 reported cstate change: term changed from 3 to 4, leader changed from 2537bbbd1a9947ae9facd87567e3ccb9 (127.17.60.193) to 3710b74ed2044ce5995533df32e67351 (127.17.60.195). New cstate: current_term: 4 leader_uuid: "3710b74ed2044ce5995533df32e67351" committed_config { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "2537bbbd1a9947ae9facd87567e3ccb9" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 33173 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3710b74ed2044ce5995533df32e67351" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 38589 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a2a5bba4ac1145d5bab94a69c9f4df40" member_type: VOTER last_known_addr { host: "127.17.60.196" port: 41807 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "25286c485fa04c86a5d07c8f88643a47" member_type: NON_VOTER last_known_addr { host: "127.17.60.198" port: 37089 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20250112 15:07:43.514549 21474 raft_consensus.cc:1270] T 9601bcda24ed42e7b93ab9740f1e271e P a2a5bba4ac1145d5bab94a69c9f4df40 [term 4 FOLLOWER]: Refusing update from remote peer 3710b74ed2044ce5995533df32e67351: Log matching property violated. Preceding OpId in replica: term: 2 index: 29. Preceding OpId from leader: term: 4 index: 31. (index mismatch)
I20250112 15:07:43.514789 22001 consensus_queue.cc:1035] T 9601bcda24ed42e7b93ab9740f1e271e P 3710b74ed2044ce5995533df32e67351 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2a5bba4ac1145d5bab94a69c9f4df40" member_type: VOTER last_known_addr { host: "127.17.60.196" port: 41807 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 31, Last known committed idx: 29, Time since last communication: 0.000s
I20250112 15:07:43.515717 21624 raft_consensus.cc:3054] T 9601bcda24ed42e7b93ab9740f1e271e P 25286c485fa04c86a5d07c8f88643a47 [term 3 LEARNER]: Advancing to term 4
I20250112 15:07:43.516460 21624 raft_consensus.cc:1270] T 9601bcda24ed42e7b93ab9740f1e271e P 25286c485fa04c86a5d07c8f88643a47 [term 4 LEARNER]: Refusing update from remote peer 3710b74ed2044ce5995533df32e67351: Log matching property violated. Preceding OpId in replica: term: 3 index: 30. Preceding OpId from leader: term: 4 index: 31. (index mismatch)
I20250112 15:07:43.516659 21069 raft_consensus.cc:1270] T 9601bcda24ed42e7b93ab9740f1e271e P 2537bbbd1a9947ae9facd87567e3ccb9 [term 4 FOLLOWER]: Refusing update from remote peer 3710b74ed2044ce5995533df32e67351: Log matching property violated. Preceding OpId in replica: term: 3 index: 30. Preceding OpId from leader: term: 4 index: 31. (index mismatch)
I20250112 15:07:43.516664 22053 consensus_queue.cc:1035] T 9601bcda24ed42e7b93ab9740f1e271e P 3710b74ed2044ce5995533df32e67351 [LEADER]: Connected to new peer: Peer: permanent_uuid: "25286c485fa04c86a5d07c8f88643a47" member_type: NON_VOTER last_known_addr { host: "127.17.60.198" port: 37089 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 31, Last known committed idx: 30, Time since last communication: 0.000s
I20250112 15:07:43.516837 22042 consensus_queue.cc:1035] T 9601bcda24ed42e7b93ab9740f1e271e P 3710b74ed2044ce5995533df32e67351 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2537bbbd1a9947ae9facd87567e3ccb9" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 33173 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 31, Last known committed idx: 30, Time since last communication: 0.000s
I20250112 15:07:43.517349 22001 raft_consensus.cc:1059] T 9601bcda24ed42e7b93ab9740f1e271e P 3710b74ed2044ce5995533df32e67351: attempting to promote NON_VOTER 25286c485fa04c86a5d07c8f88643a47 to VOTER
I20250112 15:07:43.517503 22001 consensus_queue.cc:237] T 9601bcda24ed42e7b93ab9740f1e271e P 3710b74ed2044ce5995533df32e67351 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 31, Committed index: 31, Last appended: 4.31, Last appended by leader: 30, Current term: 4, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "2537bbbd1a9947ae9facd87567e3ccb9" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 33173 } attrs { replace: true } } peers { permanent_uuid: "3710b74ed2044ce5995533df32e67351" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 38589 } } peers { permanent_uuid: "a2a5bba4ac1145d5bab94a69c9f4df40" member_type: VOTER last_known_addr { host: "127.17.60.196" port: 41807 } attrs { promote: false } } peers { permanent_uuid: "25286c485fa04c86a5d07c8f88643a47" member_type: VOTER last_known_addr { host: "127.17.60.198" port: 37089 } attrs { promote: false } }
I20250112 15:07:43.517948 21069 raft_consensus.cc:1270] T 9601bcda24ed42e7b93ab9740f1e271e P 2537bbbd1a9947ae9facd87567e3ccb9 [term 4 FOLLOWER]: Refusing update from remote peer 3710b74ed2044ce5995533df32e67351: Log matching property violated. Preceding OpId in replica: term: 4 index: 31. Preceding OpId from leader: term: 4 index: 32. (index mismatch)
I20250112 15:07:43.517957 21474 raft_consensus.cc:1270] T 9601bcda24ed42e7b93ab9740f1e271e P a2a5bba4ac1145d5bab94a69c9f4df40 [term 4 FOLLOWER]: Refusing update from remote peer 3710b74ed2044ce5995533df32e67351: Log matching property violated. Preceding OpId in replica: term: 4 index: 31. Preceding OpId from leader: term: 4 index: 32. (index mismatch)
I20250112 15:07:43.518013 21624 raft_consensus.cc:1270] T 9601bcda24ed42e7b93ab9740f1e271e P 25286c485fa04c86a5d07c8f88643a47 [term 4 LEARNER]: Refusing update from remote peer 3710b74ed2044ce5995533df32e67351: Log matching property violated. Preceding OpId in replica: term: 4 index: 31. Preceding OpId from leader: term: 4 index: 32. (index mismatch)
I20250112 15:07:43.518118 22054 consensus_queue.cc:1035] T 9601bcda24ed42e7b93ab9740f1e271e P 3710b74ed2044ce5995533df32e67351 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2a5bba4ac1145d5bab94a69c9f4df40" member_type: VOTER last_known_addr { host: "127.17.60.196" port: 41807 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 32, Last known committed idx: 31, Time since last communication: 0.000s
I20250112 15:07:43.518246 22054 consensus_queue.cc:1035] T 9601bcda24ed42e7b93ab9740f1e271e P 3710b74ed2044ce5995533df32e67351 [LEADER]: Connected to new peer: Peer: permanent_uuid: "25286c485fa04c86a5d07c8f88643a47" member_type: VOTER last_known_addr { host: "127.17.60.198" port: 37089 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 32, Last known committed idx: 31, Time since last communication: 0.000s
I20250112 15:07:43.518443 22042 consensus_queue.cc:1035] T 9601bcda24ed42e7b93ab9740f1e271e P 3710b74ed2044ce5995533df32e67351 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2537bbbd1a9947ae9facd87567e3ccb9" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 33173 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 32, Last known committed idx: 31, Time since last communication: 0.000s
I20250112 15:07:43.518997 22053 raft_consensus.cc:2949] T 9601bcda24ed42e7b93ab9740f1e271e P 3710b74ed2044ce5995533df32e67351 [term 4 LEADER]: Committing config change with OpId 4.32: config changed from index 29 to 32, 25286c485fa04c86a5d07c8f88643a47 (127.17.60.198) changed from NON_VOTER to VOTER. New config: { opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "2537bbbd1a9947ae9facd87567e3ccb9" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 33173 } attrs { replace: true } } peers { permanent_uuid: "3710b74ed2044ce5995533df32e67351" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 38589 } } peers { permanent_uuid: "a2a5bba4ac1145d5bab94a69c9f4df40" member_type: VOTER last_known_addr { host: "127.17.60.196" port: 41807 } attrs { promote: false } } peers { permanent_uuid: "25286c485fa04c86a5d07c8f88643a47" member_type: VOTER last_known_addr { host: "127.17.60.198" port: 37089 } attrs { promote: false } } }
I20250112 15:07:43.519097 21624 raft_consensus.cc:2949] T 9601bcda24ed42e7b93ab9740f1e271e P 25286c485fa04c86a5d07c8f88643a47 [term 4 FOLLOWER]: Committing config change with OpId 4.32: config changed from index 29 to 32, 25286c485fa04c86a5d07c8f88643a47 (127.17.60.198) changed from NON_VOTER to VOTER. New config: { opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "2537bbbd1a9947ae9facd87567e3ccb9" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 33173 } attrs { replace: true } } peers { permanent_uuid: "3710b74ed2044ce5995533df32e67351" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 38589 } } peers { permanent_uuid: "a2a5bba4ac1145d5bab94a69c9f4df40" member_type: VOTER last_known_addr { host: "127.17.60.196" port: 41807 } attrs { promote: false } } peers { permanent_uuid: "25286c485fa04c86a5d07c8f88643a47" member_type: VOTER last_known_addr { host: "127.17.60.198" port: 37089 } attrs { promote: false } } }
I20250112 15:07:43.519111 21069 raft_consensus.cc:2949] T 9601bcda24ed42e7b93ab9740f1e271e P 2537bbbd1a9947ae9facd87567e3ccb9 [term 4 FOLLOWER]: Committing config change with OpId 4.32: config changed from index 29 to 32, 25286c485fa04c86a5d07c8f88643a47 (127.17.60.198) changed from NON_VOTER to VOTER. New config: { opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "2537bbbd1a9947ae9facd87567e3ccb9" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 33173 } attrs { replace: true } } peers { permanent_uuid: "3710b74ed2044ce5995533df32e67351" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 38589 } } peers { permanent_uuid: "a2a5bba4ac1145d5bab94a69c9f4df40" member_type: VOTER last_known_addr { host: "127.17.60.196" port: 41807 } attrs { promote: false } } peers { permanent_uuid: "25286c485fa04c86a5d07c8f88643a47" member_type: VOTER last_known_addr { host: "127.17.60.198" port: 37089 } attrs { promote: false } } }
I20250112 15:07:43.520143 20965 catalog_manager.cc:5526] T 9601bcda24ed42e7b93ab9740f1e271e P 2537bbbd1a9947ae9facd87567e3ccb9 reported cstate change: config changed from index 29 to 32, 25286c485fa04c86a5d07c8f88643a47 (127.17.60.198) changed from NON_VOTER to VOTER. New cstate: current_term: 4 leader_uuid: "3710b74ed2044ce5995533df32e67351" committed_config { opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "2537bbbd1a9947ae9facd87567e3ccb9" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 33173 } attrs { replace: true } } peers { permanent_uuid: "3710b74ed2044ce5995533df32e67351" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 38589 } } peers { permanent_uuid: "a2a5bba4ac1145d5bab94a69c9f4df40" member_type: VOTER last_known_addr { host: "127.17.60.196" port: 41807 } attrs { promote: false } } peers { permanent_uuid: "25286c485fa04c86a5d07c8f88643a47" member_type: VOTER last_known_addr { host: "127.17.60.198" port: 37089 } attrs { promote: false } } }
I20250112 15:07:43.520521 21474 raft_consensus.cc:2949] T 9601bcda24ed42e7b93ab9740f1e271e P a2a5bba4ac1145d5bab94a69c9f4df40 [term 4 FOLLOWER]: Committing config change with OpId 4.32: config changed from index 29 to 32, 25286c485fa04c86a5d07c8f88643a47 (127.17.60.198) changed from NON_VOTER to VOTER. New config: { opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "2537bbbd1a9947ae9facd87567e3ccb9" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 33173 } attrs { replace: true } } peers { permanent_uuid: "3710b74ed2044ce5995533df32e67351" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 38589 } } peers { permanent_uuid: "a2a5bba4ac1145d5bab94a69c9f4df40" member_type: VOTER last_known_addr { host: "127.17.60.196" port: 41807 } attrs { promote: false } } peers { permanent_uuid: "25286c485fa04c86a5d07c8f88643a47" member_type: VOTER last_known_addr { host: "127.17.60.198" port: 37089 } attrs { promote: false } } }
I20250112 15:07:43.521620 21222 consensus_queue.cc:237] T 9601bcda24ed42e7b93ab9740f1e271e P 3710b74ed2044ce5995533df32e67351 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 32, Committed index: 32, Last appended: 4.32, Last appended by leader: 30, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 33 OBSOLETE_local: false peers { permanent_uuid: "3710b74ed2044ce5995533df32e67351" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 38589 } } peers { permanent_uuid: "a2a5bba4ac1145d5bab94a69c9f4df40" member_type: VOTER last_known_addr { host: "127.17.60.196" port: 41807 } attrs { promote: false } } peers { permanent_uuid: "25286c485fa04c86a5d07c8f88643a47" member_type: VOTER last_known_addr { host: "127.17.60.198" port: 37089 } attrs { promote: false } }
I20250112 15:07:43.521930 21624 raft_consensus.cc:1270] T 9601bcda24ed42e7b93ab9740f1e271e P 25286c485fa04c86a5d07c8f88643a47 [term 4 FOLLOWER]: Refusing update from remote peer 3710b74ed2044ce5995533df32e67351: Log matching property violated. Preceding OpId in replica: term: 4 index: 32. Preceding OpId from leader: term: 4 index: 33. (index mismatch)
I20250112 15:07:43.521939 21474 raft_consensus.cc:1270] T 9601bcda24ed42e7b93ab9740f1e271e P a2a5bba4ac1145d5bab94a69c9f4df40 [term 4 FOLLOWER]: Refusing update from remote peer 3710b74ed2044ce5995533df32e67351: Log matching property violated. Preceding OpId in replica: term: 4 index: 32. Preceding OpId from leader: term: 4 index: 33. (index mismatch)
I20250112 15:07:43.522091 22054 consensus_queue.cc:1035] T 9601bcda24ed42e7b93ab9740f1e271e P 3710b74ed2044ce5995533df32e67351 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2a5bba4ac1145d5bab94a69c9f4df40" member_type: VOTER last_known_addr { host: "127.17.60.196" port: 41807 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 33, Last known committed idx: 32, Time since last communication: 0.000s
I20250112 15:07:43.522156 22001 consensus_queue.cc:1035] T 9601bcda24ed42e7b93ab9740f1e271e P 3710b74ed2044ce5995533df32e67351 [LEADER]: Connected to new peer: Peer: permanent_uuid: "25286c485fa04c86a5d07c8f88643a47" member_type: VOTER last_known_addr { host: "127.17.60.198" port: 37089 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 33, Last known committed idx: 32, Time since last communication: 0.000s
I20250112 15:07:43.522655 22042 raft_consensus.cc:2949] T 9601bcda24ed42e7b93ab9740f1e271e P 3710b74ed2044ce5995533df32e67351 [term 4 LEADER]: Committing config change with OpId 4.33: config changed from index 32 to 33, VOTER 2537bbbd1a9947ae9facd87567e3ccb9 (127.17.60.193) evicted. New config: { opid_index: 33 OBSOLETE_local: false peers { permanent_uuid: "3710b74ed2044ce5995533df32e67351" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 38589 } } peers { permanent_uuid: "a2a5bba4ac1145d5bab94a69c9f4df40" member_type: VOTER last_known_addr { host: "127.17.60.196" port: 41807 } attrs { promote: false } } peers { permanent_uuid: "25286c485fa04c86a5d07c8f88643a47" member_type: VOTER last_known_addr { host: "127.17.60.198" port: 37089 } attrs { promote: false } } }
I20250112 15:07:43.522751 21474 raft_consensus.cc:2949] T 9601bcda24ed42e7b93ab9740f1e271e P a2a5bba4ac1145d5bab94a69c9f4df40 [term 4 FOLLOWER]: Committing config change with OpId 4.33: config changed from index 32 to 33, VOTER 2537bbbd1a9947ae9facd87567e3ccb9 (127.17.60.193) evicted. New config: { opid_index: 33 OBSOLETE_local: false peers { permanent_uuid: "3710b74ed2044ce5995533df32e67351" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 38589 } } peers { permanent_uuid: "a2a5bba4ac1145d5bab94a69c9f4df40" member_type: VOTER last_known_addr { host: "127.17.60.196" port: 41807 } attrs { promote: false } } peers { permanent_uuid: "25286c485fa04c86a5d07c8f88643a47" member_type: VOTER last_known_addr { host: "127.17.60.198" port: 37089 } attrs { promote: false } } }
I20250112 15:07:43.522769 21624 raft_consensus.cc:2949] T 9601bcda24ed42e7b93ab9740f1e271e P 25286c485fa04c86a5d07c8f88643a47 [term 4 FOLLOWER]: Committing config change with OpId 4.33: config changed from index 32 to 33, VOTER 2537bbbd1a9947ae9facd87567e3ccb9 (127.17.60.193) evicted. New config: { opid_index: 33 OBSOLETE_local: false peers { permanent_uuid: "3710b74ed2044ce5995533df32e67351" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 38589 } } peers { permanent_uuid: "a2a5bba4ac1145d5bab94a69c9f4df40" member_type: VOTER last_known_addr { host: "127.17.60.196" port: 41807 } attrs { promote: false } } peers { permanent_uuid: "25286c485fa04c86a5d07c8f88643a47" member_type: VOTER last_known_addr { host: "127.17.60.198" port: 37089 } attrs { promote: false } } }
I20250112 15:07:43.523550 20957 catalog_manager.cc:5039] ChangeConfig:REMOVE_PEER RPC for tablet 9601bcda24ed42e7b93ab9740f1e271e with cas_config_opid_index 32: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250112 15:07:43.523665 20970 catalog_manager.cc:5526] T 9601bcda24ed42e7b93ab9740f1e271e P 25286c485fa04c86a5d07c8f88643a47 reported cstate change: config changed from index 32 to 33, VOTER 2537bbbd1a9947ae9facd87567e3ccb9 (127.17.60.193) evicted. New cstate: current_term: 4 leader_uuid: "3710b74ed2044ce5995533df32e67351" committed_config { opid_index: 33 OBSOLETE_local: false peers { permanent_uuid: "3710b74ed2044ce5995533df32e67351" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 38589 } } peers { permanent_uuid: "a2a5bba4ac1145d5bab94a69c9f4df40" member_type: VOTER last_known_addr { host: "127.17.60.196" port: 41807 } attrs { promote: false } } peers { permanent_uuid: "25286c485fa04c86a5d07c8f88643a47" member_type: VOTER last_known_addr { host: "127.17.60.198" port: 37089 } attrs { promote: false } } }
I20250112 15:07:43.524358 21065 tablet_service.cc:1514] Processing DeleteTablet for tablet 9601bcda24ed42e7b93ab9740f1e271e with delete_type TABLET_DATA_TOMBSTONED (TS 2537bbbd1a9947ae9facd87567e3ccb9 not found in new config with opid_index 33) from {username='slave'} at 127.0.0.1:37188
I20250112 15:07:43.524550 22056 tablet_replica.cc:331] T 9601bcda24ed42e7b93ab9740f1e271e P 2537bbbd1a9947ae9facd87567e3ccb9: stopping tablet replica
I20250112 15:07:43.524638 22056 raft_consensus.cc:2238] T 9601bcda24ed42e7b93ab9740f1e271e P 2537bbbd1a9947ae9facd87567e3ccb9 [term 4 FOLLOWER]: Raft consensus shutting down.
I20250112 15:07:43.524696 22056 raft_consensus.cc:2267] T 9601bcda24ed42e7b93ab9740f1e271e P 2537bbbd1a9947ae9facd87567e3ccb9 [term 4 FOLLOWER]: Raft consensus is shut down!
I20250112 15:07:43.524917 22056 ts_tablet_manager.cc:1905] T 9601bcda24ed42e7b93ab9740f1e271e P 2537bbbd1a9947ae9facd87567e3ccb9: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250112 15:07:43.526124 22056 ts_tablet_manager.cc:1918] T 9601bcda24ed42e7b93ab9740f1e271e P 2537bbbd1a9947ae9facd87567e3ccb9: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 4.32
I20250112 15:07:43.526173 22056 log.cc:1198] T 9601bcda24ed42e7b93ab9740f1e271e P 2537bbbd1a9947ae9facd87567e3ccb9: Deleting WAL directory at /tmp/dist-test-task95dLSD/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1736694351070047-17651-0/minicluster-data/ts-0-root/wals/9601bcda24ed42e7b93ab9740f1e271e
I20250112 15:07:43.526458 20956 catalog_manager.cc:4872] TS 2537bbbd1a9947ae9facd87567e3ccb9 (127.17.60.193:33173): tablet 9601bcda24ed42e7b93ab9740f1e271e (table test-workload [id=cdaf6d3a65a54ea08e8e2c0ecdfc9de7]) successfully deleted
I20250112 15:07:44.010603 21222 consensus_queue.cc:237] T 8c7dbaf285a04da9b4eec92a29ab7d6c P 3710b74ed2044ce5995533df32e67351 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 31, Committed index: 31, Last appended: 4.31, Last appended by leader: 28, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "c561ec3374b94b95be0c155d80ff0bff" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 33627 } attrs { replace: true } } peers { permanent_uuid: "3710b74ed2044ce5995533df32e67351" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 38589 } } peers { permanent_uuid: "a2a5bba4ac1145d5bab94a69c9f4df40" member_type: VOTER last_known_addr { host: "127.17.60.196" port: 41807 } attrs { promote: false } } peers { permanent_uuid: "b5e9178a7a83402e8c4a6943ab78589d" member_type: NON_VOTER last_known_addr { host: "127.17.60.197" port: 36573 } attrs { promote: true } }
I20250112 15:07:44.011211 21474 raft_consensus.cc:1270] T 8c7dbaf285a04da9b4eec92a29ab7d6c P a2a5bba4ac1145d5bab94a69c9f4df40 [term 4 FOLLOWER]: Refusing update from remote peer 3710b74ed2044ce5995533df32e67351: Log matching property violated. Preceding OpId in replica: term: 4 index: 31. Preceding OpId from leader: term: 4 index: 32. (index mismatch)
I20250112 15:07:44.011241 21149 raft_consensus.cc:1270] T 8c7dbaf285a04da9b4eec92a29ab7d6c P c561ec3374b94b95be0c155d80ff0bff [term 4 FOLLOWER]: Refusing update from remote peer 3710b74ed2044ce5995533df32e67351: Log matching property violated. Preceding OpId in replica: term: 4 index: 31. Preceding OpId from leader: term: 4 index: 32. (index mismatch)
W20250112 15:07:44.011348 21189 consensus_peers.cc:487] T 8c7dbaf285a04da9b4eec92a29ab7d6c P 3710b74ed2044ce5995533df32e67351 -> Peer b5e9178a7a83402e8c4a6943ab78589d (127.17.60.197:36573): Couldn't send request to peer b5e9178a7a83402e8c4a6943ab78589d. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 8c7dbaf285a04da9b4eec92a29ab7d6c. This is attempt 1: this message will repeat every 5th retry.
I20250112 15:07:44.011412 22054 consensus_queue.cc:1035] T 8c7dbaf285a04da9b4eec92a29ab7d6c P 3710b74ed2044ce5995533df32e67351 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c561ec3374b94b95be0c155d80ff0bff" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 33627 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 32, Last known committed idx: 31, Time since last communication: 0.000s
I20250112 15:07:44.011615 22053 consensus_queue.cc:1035] T 8c7dbaf285a04da9b4eec92a29ab7d6c P 3710b74ed2044ce5995533df32e67351 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2a5bba4ac1145d5bab94a69c9f4df40" member_type: VOTER last_known_addr { host: "127.17.60.196" port: 41807 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 32, Last known committed idx: 31, Time since last communication: 0.000s
I20250112 15:07:44.012144 22054 raft_consensus.cc:2949] T 8c7dbaf285a04da9b4eec92a29ab7d6c P 3710b74ed2044ce5995533df32e67351 [term 4 LEADER]: Committing config change with OpId 4.32: config changed from index 31 to 32, NON_VOTER b5e9178a7a83402e8c4a6943ab78589d (127.17.60.197) added. New config: { opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "c561ec3374b94b95be0c155d80ff0bff" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 33627 } attrs { replace: true } } peers { permanent_uuid: "3710b74ed2044ce5995533df32e67351" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 38589 } } peers { permanent_uuid: "a2a5bba4ac1145d5bab94a69c9f4df40" member_type: VOTER last_known_addr { host: "127.17.60.196" port: 41807 } attrs { promote: false } } peers { permanent_uuid: "b5e9178a7a83402e8c4a6943ab78589d" member_type: NON_VOTER last_known_addr { host: "127.17.60.197" port: 36573 } attrs { promote: true } } }
I20250112 15:07:44.012252 21149 raft_consensus.cc:2949] T 8c7dbaf285a04da9b4eec92a29ab7d6c P c561ec3374b94b95be0c155d80ff0bff [term 4 FOLLOWER]: Committing config change with OpId 4.32: config changed from index 31 to 32, NON_VOTER b5e9178a7a83402e8c4a6943ab78589d (127.17.60.197) added. New config: { opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "c561ec3374b94b95be0c155d80ff0bff" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 33627 } attrs { replace: true } } peers { permanent_uuid: "3710b74ed2044ce5995533df32e67351" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 38589 } } peers { permanent_uuid: "a2a5bba4ac1145d5bab94a69c9f4df40" member_type: VOTER last_known_addr { host: "127.17.60.196" port: 41807 } attrs { promote: false } } peers { permanent_uuid: "b5e9178a7a83402e8c4a6943ab78589d" member_type: NON_VOTER last_known_addr { host: "127.17.60.197" port: 36573 } attrs { promote: true } } }
I20250112 15:07:44.012509 21474 raft_consensus.cc:2949] T 8c7dbaf285a04da9b4eec92a29ab7d6c P a2a5bba4ac1145d5bab94a69c9f4df40 [term 4 FOLLOWER]: Committing config change with OpId 4.32: config changed from index 31 to 32, NON_VOTER b5e9178a7a83402e8c4a6943ab78589d (127.17.60.197) added. New config: { opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "c561ec3374b94b95be0c155d80ff0bff" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 33627 } attrs { replace: true } } peers { permanent_uuid: "3710b74ed2044ce5995533df32e67351" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 38589 } } peers { permanent_uuid: "a2a5bba4ac1145d5bab94a69c9f4df40" member_type: VOTER last_known_addr { host: "127.17.60.196" port: 41807 } attrs { promote: false } } peers { permanent_uuid: "b5e9178a7a83402e8c4a6943ab78589d" member_type: NON_VOTER last_known_addr { host: "127.17.60.197" port: 36573 } attrs { promote: true } } }
I20250112 15:07:44.013422 20964 catalog_manager.cc:5526] T 8c7dbaf285a04da9b4eec92a29ab7d6c P c561ec3374b94b95be0c155d80ff0bff reported cstate change: config changed from index 31 to 32, NON_VOTER b5e9178a7a83402e8c4a6943ab78589d (127.17.60.197) added. New cstate: current_term: 4 leader_uuid: "3710b74ed2044ce5995533df32e67351" committed_config { opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "c561ec3374b94b95be0c155d80ff0bff" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 33627 } attrs { replace: true } } peers { permanent_uuid: "3710b74ed2044ce5995533df32e67351" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 38589 } } peers { permanent_uuid: "a2a5bba4ac1145d5bab94a69c9f4df40" member_type: VOTER last_known_addr { host: "127.17.60.196" port: 41807 } attrs { promote: false } } peers { permanent_uuid: "b5e9178a7a83402e8c4a6943ab78589d" member_type: NON_VOTER last_known_addr { host: "127.17.60.197" port: 36573 } attrs { promote: true } } }
I20250112 15:07:44.013916 21069 consensus_queue.cc:237] T ef90b704208140a3983ffe6105939319 P 2537bbbd1a9947ae9facd87567e3ccb9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 28, Committed index: 28, Last appended: 4.28, Last appended by leader: 27, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "2537bbbd1a9947ae9facd87567e3ccb9" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 33173 } } peers { permanent_uuid: "c561ec3374b94b95be0c155d80ff0bff" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 33627 } attrs { replace: true } } peers { permanent_uuid: "3710b74ed2044ce5995533df32e67351" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 38589 } } peers { permanent_uuid: "25286c485fa04c86a5d07c8f88643a47" member_type: NON_VOTER last_known_addr { host: "127.17.60.198" port: 37089 } attrs { promote: true } }
I20250112 15:07:44.014371 21149 raft_consensus.cc:1270] T ef90b704208140a3983ffe6105939319 P c561ec3374b94b95be0c155d80ff0bff [term 4 FOLLOWER]: Refusing update from remote peer 2537bbbd1a9947ae9facd87567e3ccb9: Log matching property violated. Preceding OpId in replica: term: 4 index: 28. Preceding OpId from leader: term: 4 index: 29. (index mismatch)
I20250112 15:07:44.014452 21222 raft_consensus.cc:1270] T ef90b704208140a3983ffe6105939319 P 3710b74ed2044ce5995533df32e67351 [term 4 FOLLOWER]: Refusing update from remote peer 2537bbbd1a9947ae9facd87567e3ccb9: Log matching property violated. Preceding OpId in replica: term: 4 index: 28. Preceding OpId from leader: term: 4 index: 29. (index mismatch)
I20250112 15:07:44.014536 22061 consensus_queue.cc:1035] T ef90b704208140a3983ffe6105939319 P 2537bbbd1a9947ae9facd87567e3ccb9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c561ec3374b94b95be0c155d80ff0bff" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 33627 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.000s
W20250112 15:07:44.014451 21038 consensus_peers.cc:487] T ef90b704208140a3983ffe6105939319 P 2537bbbd1a9947ae9facd87567e3ccb9 -> Peer 25286c485fa04c86a5d07c8f88643a47 (127.17.60.198:37089): Couldn't send request to peer 25286c485fa04c86a5d07c8f88643a47. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: ef90b704208140a3983ffe6105939319. This is attempt 1: this message will repeat every 5th retry.
I20250112 15:07:44.014679 22061 consensus_queue.cc:1035] T ef90b704208140a3983ffe6105939319 P 2537bbbd1a9947ae9facd87567e3ccb9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3710b74ed2044ce5995533df32e67351" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 38589 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.000s
I20250112 15:07:44.015347 22062 raft_consensus.cc:2949] T ef90b704208140a3983ffe6105939319 P 2537bbbd1a9947ae9facd87567e3ccb9 [term 4 LEADER]: Committing config change with OpId 4.29: config changed from index -1 to 29, NON_VOTER 25286c485fa04c86a5d07c8f88643a47 (127.17.60.198) added. New config: { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "2537bbbd1a9947ae9facd87567e3ccb9" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 33173 } } peers { permanent_uuid: "c561ec3374b94b95be0c155d80ff0bff" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 33627 } attrs { replace: true } } peers { permanent_uuid: "3710b74ed2044ce5995533df32e67351" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 38589 } } peers { permanent_uuid: "25286c485fa04c86a5d07c8f88643a47" member_type: NON_VOTER last_known_addr { host: "127.17.60.198" port: 37089 } attrs { promote: true } } }
I20250112 15:07:44.015486 21149 raft_consensus.cc:2949] T ef90b704208140a3983ffe6105939319 P c561ec3374b94b95be0c155d80ff0bff [term 4 FOLLOWER]: Committing config change with OpId 4.29: config changed from index -1 to 29, NON_VOTER 25286c485fa04c86a5d07c8f88643a47 (127.17.60.198) added. New config: { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "2537bbbd1a9947ae9facd87567e3ccb9" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 33173 } } peers { permanent_uuid: "c561ec3374b94b95be0c155d80ff0bff" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 33627 } attrs { replace: true } } peers { permanent_uuid: "3710b74ed2044ce5995533df32e67351" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 38589 } } peers { permanent_uuid: "25286c485fa04c86a5d07c8f88643a47" member_type: NON_VOTER last_known_addr { host: "127.17.60.198" port: 37089 } attrs { promote: true } } }
I20250112 15:07:44.015769 21222 raft_consensus.cc:2949] T ef90b704208140a3983ffe6105939319 P 3710b74ed2044ce5995533df32e67351 [term 4 FOLLOWER]: Committing config change with OpId 4.29: config changed from index -1 to 29, NON_VOTER 25286c485fa04c86a5d07c8f88643a47 (127.17.60.198) added. New config: { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "2537bbbd1a9947ae9facd87567e3ccb9" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 33173 } } peers { permanent_uuid: "c561ec3374b94b95be0c155d80ff0bff" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 33627 } attrs { replace: true } } peers { permanent_uuid: "3710b74ed2044ce5995533df32e67351" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 38589 } } peers { permanent_uuid: "25286c485fa04c86a5d07c8f88643a47" member_type: NON_VOTER last_known_addr { host: "127.17.60.198" port: 37089 } attrs { promote: true } } }
I20250112 15:07:44.016661 20970 catalog_manager.cc:5526] T ef90b704208140a3983ffe6105939319 P 2537bbbd1a9947ae9facd87567e3ccb9 reported cstate change: config changed from index -1 to 29, NON_VOTER 25286c485fa04c86a5d07c8f88643a47 (127.17.60.198) added. New cstate: current_term: 4 leader_uuid: "2537bbbd1a9947ae9facd87567e3ccb9" committed_config { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "2537bbbd1a9947ae9facd87567e3ccb9" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 33173 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c561ec3374b94b95be0c155d80ff0bff" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 33627 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3710b74ed2044ce5995533df32e67351" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 38589 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "25286c485fa04c86a5d07c8f88643a47" member_type: NON_VOTER last_known_addr { host: "127.17.60.198" port: 37089 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20250112 15:07:44.016749 21149 consensus_queue.cc:237] T bfaeae584ec04772b2d6fcd088022117 P c561ec3374b94b95be0c155d80ff0bff [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 30, Committed index: 30, Last appended: 3.30, Last appended by leader: 27, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "2537bbbd1a9947ae9facd87567e3ccb9" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 33173 } attrs { replace: true } } peers { permanent_uuid: "c561ec3374b94b95be0c155d80ff0bff" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 33627 } } peers { permanent_uuid: "25286c485fa04c86a5d07c8f88643a47" member_type: VOTER last_known_addr { host: "127.17.60.198" port: 37089 } attrs { promote: false } } peers { permanent_uuid: "a2a5bba4ac1145d5bab94a69c9f4df40" member_type: NON_VOTER last_known_addr { host: "127.17.60.196" port: 41807 } attrs { promote: true } }
W20250112 15:07:44.017287 21116 consensus_peers.cc:487] T bfaeae584ec04772b2d6fcd088022117 P c561ec3374b94b95be0c155d80ff0bff -> Peer a2a5bba4ac1145d5bab94a69c9f4df40 (127.17.60.196:41807): Couldn't send request to peer a2a5bba4ac1145d5bab94a69c9f4df40. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: bfaeae584ec04772b2d6fcd088022117. This is attempt 1: this message will repeat every 5th retry.
I20250112 15:07:44.017293 21069 raft_consensus.cc:1270] T bfaeae584ec04772b2d6fcd088022117 P 2537bbbd1a9947ae9facd87567e3ccb9 [term 3 FOLLOWER]: Refusing update from remote peer c561ec3374b94b95be0c155d80ff0bff: Log matching property violated. Preceding OpId in replica: term: 3 index: 30. Preceding OpId from leader: term: 3 index: 31. (index mismatch)
I20250112 15:07:44.017369 21624 raft_consensus.cc:1270] T bfaeae584ec04772b2d6fcd088022117 P 25286c485fa04c86a5d07c8f88643a47 [term 3 FOLLOWER]: Refusing update from remote peer c561ec3374b94b95be0c155d80ff0bff: Log matching property violated. Preceding OpId in replica: term: 3 index: 30. Preceding OpId from leader: term: 3 index: 31. (index mismatch)
I20250112 15:07:44.017508 22015 consensus_queue.cc:1035] T bfaeae584ec04772b2d6fcd088022117 P c561ec3374b94b95be0c155d80ff0bff [LEADER]: Connected to new peer: Peer: permanent_uuid: "2537bbbd1a9947ae9facd87567e3ccb9" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 33173 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 31, Last known committed idx: 30, Time since last communication: 0.000s
I20250112 15:07:44.017831 22066 consensus_queue.cc:1035] T bfaeae584ec04772b2d6fcd088022117 P c561ec3374b94b95be0c155d80ff0bff [LEADER]: Connected to new peer: Peer: permanent_uuid: "25286c485fa04c86a5d07c8f88643a47" member_type: VOTER last_known_addr { host: "127.17.60.198" port: 37089 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 31, Last known committed idx: 30, Time since last communication: 0.000s
I20250112 15:07:44.018239 22067 raft_consensus.cc:2949] T bfaeae584ec04772b2d6fcd088022117 P c561ec3374b94b95be0c155d80ff0bff [term 3 LEADER]: Committing config change with OpId 3.31: config changed from index 30 to 31, NON_VOTER a2a5bba4ac1145d5bab94a69c9f4df40 (127.17.60.196) added. New config: { opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "2537bbbd1a9947ae9facd87567e3ccb9" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 33173 } attrs { replace: true } } peers { permanent_uuid: "c561ec3374b94b95be0c155d80ff0bff" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 33627 } } peers { permanent_uuid: "25286c485fa04c86a5d07c8f88643a47" member_type: VOTER last_known_addr { host: "127.17.60.198" port: 37089 } attrs { promote: false } } peers { permanent_uuid: "a2a5bba4ac1145d5bab94a69c9f4df40" member_type: NON_VOTER last_known_addr { host: "127.17.60.196" port: 41807 } attrs { promote: true } } }
I20250112 15:07:44.018457 21069 raft_consensus.cc:2949] T bfaeae584ec04772b2d6fcd088022117 P 2537bbbd1a9947ae9facd87567e3ccb9 [term 3 FOLLOWER]: Committing config change with OpId 3.31: config changed from index 30 to 31, NON_VOTER a2a5bba4ac1145d5bab94a69c9f4df40 (127.17.60.196) added. New config: { opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "2537bbbd1a9947ae9facd87567e3ccb9" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 33173 } attrs { replace: true } } peers { permanent_uuid: "c561ec3374b94b95be0c155d80ff0bff" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 33627 } } peers { permanent_uuid: "25286c485fa04c86a5d07c8f88643a47" member_type: VOTER last_known_addr { host: "127.17.60.198" port: 37089 } attrs { promote: false } } peers { permanent_uuid: "a2a5bba4ac1145d5bab94a69c9f4df40" member_type: NON_VOTER last_known_addr { host: "127.17.60.196" port: 41807 } attrs { promote: true } } }
I20250112 15:07:44.019455 21624 raft_consensus.cc:2949] T bfaeae584ec04772b2d6fcd088022117 P 25286c485fa04c86a5d07c8f88643a47 [term 3 FOLLOWER]: Committing config change with OpId 3.31: config changed from index 30 to 31, NON_VOTER a2a5bba4ac1145d5bab94a69c9f4df40 (127.17.60.196) added. New config: { opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "2537bbbd1a9947ae9facd87567e3ccb9" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 33173 } attrs { replace: true } } peers { permanent_uuid: "c561ec3374b94b95be0c155d80ff0bff" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 33627 } } peers { permanent_uuid: "25286c485fa04c86a5d07c8f88643a47" member_type: VOTER last_known_addr { host: "127.17.60.198" port: 37089 } attrs { promote: false } } peers { permanent_uuid: "a2a5bba4ac1145d5bab94a69c9f4df40" member_type: NON_VOTER last_known_addr { host: "127.17.60.196" port: 41807 } attrs { promote: true } } }
I20250112 15:07:44.019476 20970 catalog_manager.cc:5526] T bfaeae584ec04772b2d6fcd088022117 P 2537bbbd1a9947ae9facd87567e3ccb9 reported cstate change: config changed from index 30 to 31, NON_VOTER a2a5bba4ac1145d5bab94a69c9f4df40 (127.17.60.196) added. New cstate: current_term: 3 leader_uuid: "c561ec3374b94b95be0c155d80ff0bff" committed_config { opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "2537bbbd1a9947ae9facd87567e3ccb9" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 33173 } attrs { replace: true } } peers { permanent_uuid: "c561ec3374b94b95be0c155d80ff0bff" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 33627 } } peers { permanent_uuid: "25286c485fa04c86a5d07c8f88643a47" member_type: VOTER last_known_addr { host: "127.17.60.198" port: 37089 } attrs { promote: false } } peers { permanent_uuid: "a2a5bba4ac1145d5bab94a69c9f4df40" member_type: NON_VOTER last_known_addr { host: "127.17.60.196" port: 41807 } attrs { promote: true } } }
I20250112 15:07:44.020454 21069 consensus_queue.cc:237] T 16f16fcd5cf347e5853189dc4da55c7f P 2537bbbd1a9947ae9facd87567e3ccb9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 29, Committed index: 29, Last appended: 2.29, Last appended by leader: 25, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "2537bbbd1a9947ae9facd87567e3ccb9" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 33173 } } peers { permanent_uuid: "3710b74ed2044ce5995533df32e67351" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 38589 } attrs { replace: true } } peers { permanent_uuid: "25286c485fa04c86a5d07c8f88643a47" member_type: VOTER last_known_addr { host: "127.17.60.198" port: 37089 } attrs { promote: false } } peers { permanent_uuid: "b5e9178a7a83402e8c4a6943ab78589d" member_type: NON_VOTER last_known_addr { host: "127.17.60.197" port: 36573 } attrs { promote: true } }
I20250112 15:07:44.020903 21222 raft_consensus.cc:1270] T 16f16fcd5cf347e5853189dc4da55c7f P 3710b74ed2044ce5995533df32e67351 [term 2 FOLLOWER]: Refusing update from remote peer 2537bbbd1a9947ae9facd87567e3ccb9: Log matching property violated. Preceding OpId in replica: term: 2 index: 29. Preceding OpId from leader: term: 2 index: 30. (index mismatch)
I20250112 15:07:44.020924 21624 raft_consensus.cc:1270] T 16f16fcd5cf347e5853189dc4da55c7f P 25286c485fa04c86a5d07c8f88643a47 [term 2 FOLLOWER]: Refusing update from remote peer 2537bbbd1a9947ae9facd87567e3ccb9: Log matching property violated. Preceding OpId in replica: term: 2 index: 29. Preceding OpId from leader: term: 2 index: 30. (index mismatch)
I20250112 15:07:44.021064 22061 consensus_queue.cc:1035] T 16f16fcd5cf347e5853189dc4da55c7f P 2537bbbd1a9947ae9facd87567e3ccb9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3710b74ed2044ce5995533df32e67351" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 38589 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 30, Last known committed idx: 29, Time since last communication: 0.000s
W20250112 15:07:44.021168 21038 consensus_peers.cc:487] T 16f16fcd5cf347e5853189dc4da55c7f P 2537bbbd1a9947ae9facd87567e3ccb9 -> Peer b5e9178a7a83402e8c4a6943ab78589d (127.17.60.197:36573): Couldn't send request to peer b5e9178a7a83402e8c4a6943ab78589d. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 16f16fcd5cf347e5853189dc4da55c7f. This is attempt 1: this message will repeat every 5th retry.
I20250112 15:07:44.021241 22061 consensus_queue.cc:1035] T 16f16fcd5cf347e5853189dc4da55c7f P 2537bbbd1a9947ae9facd87567e3ccb9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "25286c485fa04c86a5d07c8f88643a47" member_type: VOTER last_known_addr { host: "127.17.60.198" port: 37089 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 30, Last known committed idx: 29, Time since last communication: 0.000s
I20250112 15:07:44.021665 22062 raft_consensus.cc:2949] T 16f16fcd5cf347e5853189dc4da55c7f P 2537bbbd1a9947ae9facd87567e3ccb9 [term 2 LEADER]: Committing config change with OpId 2.30: config changed from index 29 to 30, NON_VOTER b5e9178a7a83402e8c4a6943ab78589d (127.17.60.197) added. New config: { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "2537bbbd1a9947ae9facd87567e3ccb9" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 33173 } } peers { permanent_uuid: "3710b74ed2044ce5995533df32e67351" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 38589 } attrs { replace: true } } peers { permanent_uuid: "25286c485fa04c86a5d07c8f88643a47" member_type: VOTER last_known_addr { host: "127.17.60.198" port: 37089 } attrs { promote: false } } peers { permanent_uuid: "b5e9178a7a83402e8c4a6943ab78589d" member_type: NON_VOTER last_known_addr { host: "127.17.60.197" port: 36573 } attrs { promote: true } } }
I20250112 15:07:44.022094 21624 raft_consensus.cc:2949] T 16f16fcd5cf347e5853189dc4da55c7f P 25286c485fa04c86a5d07c8f88643a47 [term 2 FOLLOWER]: Committing config change with OpId 2.30: config changed from index 29 to 30, NON_VOTER b5e9178a7a83402e8c4a6943ab78589d (127.17.60.197) added. New config: { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "2537bbbd1a9947ae9facd87567e3ccb9" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 33173 } } peers { permanent_uuid: "3710b74ed2044ce5995533df32e67351" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 38589 } attrs { replace: true } } peers { permanent_uuid: "25286c485fa04c86a5d07c8f88643a47" member_type: VOTER last_known_addr { host: "127.17.60.198" port: 37089 } attrs { promote: false } } peers { permanent_uuid: "b5e9178a7a83402e8c4a6943ab78589d" member_type: NON_VOTER last_known_addr { host: "127.17.60.197" port: 36573 } attrs { promote: true } } }
I20250112 15:07:44.022543 21222 raft_consensus.cc:2949] T 16f16fcd5cf347e5853189dc4da55c7f P 3710b74ed2044ce5995533df32e67351 [term 2 FOLLOWER]: Committing config change with OpId 2.30: config changed from index 29 to 30, NON_VOTER b5e9178a7a83402e8c4a6943ab78589d (127.17.60.197) added. New config: { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "2537bbbd1a9947ae9facd87567e3ccb9" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 33173 } } peers { permanent_uuid: "3710b74ed2044ce5995533df32e67351" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 38589 } attrs { replace: true } } peers { permanent_uuid: "25286c485fa04c86a5d07c8f88643a47" member_type: VOTER last_known_addr { host: "127.17.60.198" port: 37089 } attrs { promote: false } } peers { permanent_uuid: "b5e9178a7a83402e8c4a6943ab78589d" member_type: NON_VOTER last_known_addr { host: "127.17.60.197" port: 36573 } attrs { promote: true } } }
I20250112 15:07:44.022861 20965 catalog_manager.cc:5526] T 16f16fcd5cf347e5853189dc4da55c7f P 2537bbbd1a9947ae9facd87567e3ccb9 reported cstate change: config changed from index 29 to 30, NON_VOTER b5e9178a7a83402e8c4a6943ab78589d (127.17.60.197) added. New cstate: current_term: 2 leader_uuid: "2537bbbd1a9947ae9facd87567e3ccb9" committed_config { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "2537bbbd1a9947ae9facd87567e3ccb9" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 33173 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3710b74ed2044ce5995533df32e67351" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 38589 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "25286c485fa04c86a5d07c8f88643a47" member_type: VOTER last_known_addr { host: "127.17.60.198" port: 37089 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b5e9178a7a83402e8c4a6943ab78589d" member_type: NON_VOTER last_known_addr { host: "127.17.60.197" port: 36573 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20250112 15:07:44.093672 21026 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250112 15:07:44.094080 21222 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "02c33d457b6b47138dd712850819b619"
dest_uuid: "3710b74ed2044ce5995533df32e67351"
mode: GRACEFUL
new_leader_uuid: "2537bbbd1a9947ae9facd87567e3ccb9"
from {username='slave'} at 127.0.0.1:54970
I20250112 15:07:44.094211 21222 raft_consensus.cc:604] T 02c33d457b6b47138dd712850819b619 P 3710b74ed2044ce5995533df32e67351 [term 4 LEADER]: Received request to transfer leadership to TS 2537bbbd1a9947ae9facd87567e3ccb9
I20250112 15:07:44.094506 21149 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "69c23627ab854000b360030c384dc966"
dest_uuid: "c561ec3374b94b95be0c155d80ff0bff"
mode: GRACEFUL
new_leader_uuid: "a2a5bba4ac1145d5bab94a69c9f4df40"
from {username='slave'} at 127.0.0.1:48104
I20250112 15:07:44.094587 21149 raft_consensus.cc:604] T 69c23627ab854000b360030c384dc966 P c561ec3374b94b95be0c155d80ff0bff [term 3 LEADER]: Received request to transfer leadership to TS a2a5bba4ac1145d5bab94a69c9f4df40
I20250112 15:07:44.094969 21222 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "80cc505a04394c53a18c5beeded4655d"
dest_uuid: "3710b74ed2044ce5995533df32e67351"
mode: GRACEFUL
new_leader_uuid: "25286c485fa04c86a5d07c8f88643a47"
from {username='slave'} at 127.0.0.1:54970
I20250112 15:07:44.095047 21222 raft_consensus.cc:604] T 80cc505a04394c53a18c5beeded4655d P 3710b74ed2044ce5995533df32e67351 [term 2 LEADER]: Received request to transfer leadership to TS 25286c485fa04c86a5d07c8f88643a47
I20250112 15:07:44.095185 21026 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 3
I20250112 15:07:44.095245 21026 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20250112 15:07:44.250633 22001 raft_consensus.cc:988] T 80cc505a04394c53a18c5beeded4655d P 3710b74ed2044ce5995533df32e67351: : Instructing follower 25286c485fa04c86a5d07c8f88643a47 to start an election
I20250112 15:07:44.250736 22001 raft_consensus.cc:1076] T 80cc505a04394c53a18c5beeded4655d P 3710b74ed2044ce5995533df32e67351 [term 2 LEADER]: Signalling peer 25286c485fa04c86a5d07c8f88643a47 to start an election
I20250112 15:07:44.250917 21624 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "80cc505a04394c53a18c5beeded4655d"
dest_uuid: "25286c485fa04c86a5d07c8f88643a47"
from {username='slave'} at 127.0.0.1:43050
I20250112 15:07:44.250998 21624 raft_consensus.cc:491] T 80cc505a04394c53a18c5beeded4655d P 25286c485fa04c86a5d07c8f88643a47 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20250112 15:07:44.251040 21624 raft_consensus.cc:3054] T 80cc505a04394c53a18c5beeded4655d P 25286c485fa04c86a5d07c8f88643a47 [term 2 FOLLOWER]: Advancing to term 3
I20250112 15:07:44.251717 21624 raft_consensus.cc:513] T 80cc505a04394c53a18c5beeded4655d P 25286c485fa04c86a5d07c8f88643a47 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "c561ec3374b94b95be0c155d80ff0bff" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 33627 } } peers { permanent_uuid: "3710b74ed2044ce5995533df32e67351" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 38589 } } peers { permanent_uuid: "25286c485fa04c86a5d07c8f88643a47" member_type: VOTER last_known_addr { host: "127.17.60.198" port: 37089 } attrs { promote: false } }
I20250112 15:07:44.251920 21624 leader_election.cc:290] T 80cc505a04394c53a18c5beeded4655d P 25286c485fa04c86a5d07c8f88643a47 [CANDIDATE]: Term 3 election: Requested vote from peers c561ec3374b94b95be0c155d80ff0bff (127.17.60.194:33627), 3710b74ed2044ce5995533df32e67351 (127.17.60.195:38589)
I20250112 15:07:44.252020 21222 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "80cc505a04394c53a18c5beeded4655d" candidate_uuid: "25286c485fa04c86a5d07c8f88643a47" candidate_term: 3 candidate_status { last_received { term: 2 index: 29 } } ignore_live_leader: true dest_uuid: "3710b74ed2044ce5995533df32e67351"
I20250112 15:07:44.252094 21222 raft_consensus.cc:3049] T 80cc505a04394c53a18c5beeded4655d P 3710b74ed2044ce5995533df32e67351 [term 2 LEADER]: Stepping down as leader of term 2
I20250112 15:07:44.252025 21149 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "80cc505a04394c53a18c5beeded4655d" candidate_uuid: "25286c485fa04c86a5d07c8f88643a47" candidate_term: 3 candidate_status { last_received { term: 2 index: 29 } } ignore_live_leader: true dest_uuid: "c561ec3374b94b95be0c155d80ff0bff"
I20250112 15:07:44.252123 21222 raft_consensus.cc:738] T 80cc505a04394c53a18c5beeded4655d P 3710b74ed2044ce5995533df32e67351 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 3710b74ed2044ce5995533df32e67351, State: Running, Role: LEADER
I20250112 15:07:44.252140 21149 raft_consensus.cc:3054] T 80cc505a04394c53a18c5beeded4655d P c561ec3374b94b95be0c155d80ff0bff [term 2 FOLLOWER]: Advancing to term 3
I20250112 15:07:44.252171 21222 consensus_queue.cc:260] T 80cc505a04394c53a18c5beeded4655d P 3710b74ed2044ce5995533df32e67351 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 29, Committed index: 29, Last appended: 2.29, Last appended by leader: 29, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "c561ec3374b94b95be0c155d80ff0bff" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 33627 } } peers { permanent_uuid: "3710b74ed2044ce5995533df32e67351" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 38589 } } peers { permanent_uuid: "25286c485fa04c86a5d07c8f88643a47" member_type: VOTER last_known_addr { host: "127.17.60.198" port: 37089 } attrs { promote: false } }
I20250112 15:07:44.252236 21222 raft_consensus.cc:3054] T 80cc505a04394c53a18c5beeded4655d P 3710b74ed2044ce5995533df32e67351 [term 2 FOLLOWER]: Advancing to term 3
I20250112 15:07:44.252696 21149 raft_consensus.cc:2463] T 80cc505a04394c53a18c5beeded4655d P c561ec3374b94b95be0c155d80ff0bff [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 25286c485fa04c86a5d07c8f88643a47 in term 3.
I20250112 15:07:44.252796 21222 raft_consensus.cc:2463] T 80cc505a04394c53a18c5beeded4655d P 3710b74ed2044ce5995533df32e67351 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 25286c485fa04c86a5d07c8f88643a47 in term 3.
I20250112 15:07:44.252867 21589 leader_election.cc:304] T 80cc505a04394c53a18c5beeded4655d P 25286c485fa04c86a5d07c8f88643a47 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 25286c485fa04c86a5d07c8f88643a47, c561ec3374b94b95be0c155d80ff0bff; no voters:
I20250112 15:07:44.252957 21753 raft_consensus.cc:2798] T 80cc505a04394c53a18c5beeded4655d P 25286c485fa04c86a5d07c8f88643a47 [term 3 FOLLOWER]: Leader election won for term 3
I20250112 15:07:44.253053 21753 raft_consensus.cc:695] T 80cc505a04394c53a18c5beeded4655d P 25286c485fa04c86a5d07c8f88643a47 [term 3 LEADER]: Becoming Leader. State: Replica: 25286c485fa04c86a5d07c8f88643a47, State: Running, Role: LEADER
I20250112 15:07:44.253088 21753 consensus_queue.cc:237] T 80cc505a04394c53a18c5beeded4655d P 25286c485fa04c86a5d07c8f88643a47 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 29, Committed index: 29, Last appended: 2.29, Last appended by leader: 29, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "c561ec3374b94b95be0c155d80ff0bff" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 33627 } } peers { permanent_uuid: "3710b74ed2044ce5995533df32e67351" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 38589 } } peers { permanent_uuid: "25286c485fa04c86a5d07c8f88643a47" member_type: VOTER last_known_addr { host: "127.17.60.198" port: 37089 } attrs { promote: false } }
I20250112 15:07:44.253582 20965 catalog_manager.cc:5526] T 80cc505a04394c53a18c5beeded4655d P 25286c485fa04c86a5d07c8f88643a47 reported cstate change: term changed from 2 to 3, leader changed from 3710b74ed2044ce5995533df32e67351 (127.17.60.195) to 25286c485fa04c86a5d07c8f88643a47 (127.17.60.198). New cstate: current_term: 3 leader_uuid: "25286c485fa04c86a5d07c8f88643a47" committed_config { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "c561ec3374b94b95be0c155d80ff0bff" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 33627 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3710b74ed2044ce5995533df32e67351" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 38589 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "25286c485fa04c86a5d07c8f88643a47" member_type: VOTER last_known_addr { host: "127.17.60.198" port: 37089 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20250112 15:07:44.456920 22015 raft_consensus.cc:988] T 69c23627ab854000b360030c384dc966 P c561ec3374b94b95be0c155d80ff0bff: : Instructing follower a2a5bba4ac1145d5bab94a69c9f4df40 to start an election
I20250112 15:07:44.457021 22015 raft_consensus.cc:1076] T 69c23627ab854000b360030c384dc966 P c561ec3374b94b95be0c155d80ff0bff [term 3 LEADER]: Signalling peer a2a5bba4ac1145d5bab94a69c9f4df40 to start an election
I20250112 15:07:44.457165 21474 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "69c23627ab854000b360030c384dc966"
dest_uuid: "a2a5bba4ac1145d5bab94a69c9f4df40"
from {username='slave'} at 127.0.0.1:43292
I20250112 15:07:44.457230 21474 raft_consensus.cc:491] T 69c23627ab854000b360030c384dc966 P a2a5bba4ac1145d5bab94a69c9f4df40 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20250112 15:07:44.457252 21474 raft_consensus.cc:3054] T 69c23627ab854000b360030c384dc966 P a2a5bba4ac1145d5bab94a69c9f4df40 [term 3 FOLLOWER]: Advancing to term 4
I20250112 15:07:44.457955 21474 raft_consensus.cc:513] T 69c23627ab854000b360030c384dc966 P a2a5bba4ac1145d5bab94a69c9f4df40 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "c561ec3374b94b95be0c155d80ff0bff" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 33627 } } peers { permanent_uuid: "b5e9178a7a83402e8c4a6943ab78589d" member_type: VOTER last_known_addr { host: "127.17.60.197" port: 36573 } attrs { promote: false } } peers { permanent_uuid: "a2a5bba4ac1145d5bab94a69c9f4df40" member_type: VOTER last_known_addr { host: "127.17.60.196" port: 41807 } attrs { promote: false } }
I20250112 15:07:44.458113 21474 leader_election.cc:290] T 69c23627ab854000b360030c384dc966 P a2a5bba4ac1145d5bab94a69c9f4df40 [CANDIDATE]: Term 4 election: Requested vote from peers c561ec3374b94b95be0c155d80ff0bff (127.17.60.194:33627), b5e9178a7a83402e8c4a6943ab78589d (127.17.60.197:36573)
I20250112 15:07:44.458206 21149 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "69c23627ab854000b360030c384dc966" candidate_uuid: "a2a5bba4ac1145d5bab94a69c9f4df40" candidate_term: 4 candidate_status { last_received { term: 3 index: 32 } } ignore_live_leader: true dest_uuid: "c561ec3374b94b95be0c155d80ff0bff"
I20250112 15:07:44.458238 21549 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "69c23627ab854000b360030c384dc966" candidate_uuid: "a2a5bba4ac1145d5bab94a69c9f4df40" candidate_term: 4 candidate_status { last_received { term: 3 index: 32 } } ignore_live_leader: true dest_uuid: "b5e9178a7a83402e8c4a6943ab78589d"
I20250112 15:07:44.458312 21149 raft_consensus.cc:3049] T 69c23627ab854000b360030c384dc966 P c561ec3374b94b95be0c155d80ff0bff [term 3 LEADER]: Stepping down as leader of term 3
I20250112 15:07:44.458326 21549 raft_consensus.cc:3054] T 69c23627ab854000b360030c384dc966 P b5e9178a7a83402e8c4a6943ab78589d [term 3 FOLLOWER]: Advancing to term 4
I20250112 15:07:44.458981 21549 raft_consensus.cc:2463] T 69c23627ab854000b360030c384dc966 P b5e9178a7a83402e8c4a6943ab78589d [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2a5bba4ac1145d5bab94a69c9f4df40 in term 4.
I20250112 15:07:44.458356 21149 raft_consensus.cc:738] T 69c23627ab854000b360030c384dc966 P c561ec3374b94b95be0c155d80ff0bff [term 3 LEADER]: Becoming Follower/Learner. State: Replica: c561ec3374b94b95be0c155d80ff0bff, State: Running, Role: LEADER
I20250112 15:07:44.459110 21437 leader_election.cc:304] T 69c23627ab854000b360030c384dc966 P a2a5bba4ac1145d5bab94a69c9f4df40 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a2a5bba4ac1145d5bab94a69c9f4df40, b5e9178a7a83402e8c4a6943ab78589d; no voters:
I20250112 15:07:44.459097 21149 consensus_queue.cc:260] T 69c23627ab854000b360030c384dc966 P c561ec3374b94b95be0c155d80ff0bff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 32, Committed index: 32, Last appended: 3.32, Last appended by leader: 32, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "c561ec3374b94b95be0c155d80ff0bff" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 33627 } } peers { permanent_uuid: "b5e9178a7a83402e8c4a6943ab78589d" member_type: VOTER last_known_addr { host: "127.17.60.197" port: 36573 } attrs { promote: false } } peers { permanent_uuid: "a2a5bba4ac1145d5bab94a69c9f4df40" member_type: VOTER last_known_addr { host: "127.17.60.196" port: 41807 } attrs { promote: false } }
I20250112 15:07:44.459218 22041 raft_consensus.cc:2798] T 69c23627ab854000b360030c384dc966 P a2a5bba4ac1145d5bab94a69c9f4df40 [term 4 FOLLOWER]: Leader election won for term 4
I20250112 15:07:44.459244 21149 raft_consensus.cc:3054] T 69c23627ab854000b360030c384dc966 P c561ec3374b94b95be0c155d80ff0bff [term 3 FOLLOWER]: Advancing to term 4
I20250112 15:07:44.459367 22041 raft_consensus.cc:695] T 69c23627ab854000b360030c384dc966 P a2a5bba4ac1145d5bab94a69c9f4df40 [term 4 LEADER]: Becoming Leader. State: Replica: a2a5bba4ac1145d5bab94a69c9f4df40, State: Running, Role: LEADER
I20250112 15:07:44.459436 22041 consensus_queue.cc:237] T 69c23627ab854000b360030c384dc966 P a2a5bba4ac1145d5bab94a69c9f4df40 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 32, Committed index: 32, Last appended: 3.32, Last appended by leader: 32, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "c561ec3374b94b95be0c155d80ff0bff" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 33627 } } peers { permanent_uuid: "b5e9178a7a83402e8c4a6943ab78589d" member_type: VOTER last_known_addr { host: "127.17.60.197" port: 36573 } attrs { promote: false } } peers { permanent_uuid: "a2a5bba4ac1145d5bab94a69c9f4df40" member_type: VOTER last_known_addr { host: "127.17.60.196" port: 41807 } attrs { promote: false } }
I20250112 15:07:44.459857 21149 raft_consensus.cc:2463] T 69c23627ab854000b360030c384dc966 P c561ec3374b94b95be0c155d80ff0bff [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2a5bba4ac1145d5bab94a69c9f4df40 in term 4.
I20250112 15:07:44.460007 20965 catalog_manager.cc:5526] T 69c23627ab854000b360030c384dc966 P a2a5bba4ac1145d5bab94a69c9f4df40 reported cstate change: term changed from 3 to 4, leader changed from c561ec3374b94b95be0c155d80ff0bff (127.17.60.194) to a2a5bba4ac1145d5bab94a69c9f4df40 (127.17.60.196). New cstate: current_term: 4 leader_uuid: "a2a5bba4ac1145d5bab94a69c9f4df40" committed_config { opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "c561ec3374b94b95be0c155d80ff0bff" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 33627 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b5e9178a7a83402e8c4a6943ab78589d" member_type: VOTER last_known_addr { host: "127.17.60.197" port: 36573 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a2a5bba4ac1145d5bab94a69c9f4df40" member_type: VOTER last_known_addr { host: "127.17.60.196" port: 41807 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20250112 15:07:44.537494 22077 ts_tablet_manager.cc:927] T 8c7dbaf285a04da9b4eec92a29ab7d6c P b5e9178a7a83402e8c4a6943ab78589d: Initiating tablet copy from peer 3710b74ed2044ce5995533df32e67351 (127.17.60.195:38589)
I20250112 15:07:44.537712 22077 tablet_copy_client.cc:323] T 8c7dbaf285a04da9b4eec92a29ab7d6c P b5e9178a7a83402e8c4a6943ab78589d: tablet copy: Beginning tablet copy session from remote peer at address 127.17.60.195:38589
I20250112 15:07:44.540989 21236 tablet_copy_service.cc:140] P 3710b74ed2044ce5995533df32e67351: Received BeginTabletCopySession request for tablet 8c7dbaf285a04da9b4eec92a29ab7d6c from peer b5e9178a7a83402e8c4a6943ab78589d ({username='slave'} at 127.0.0.1:52204)
I20250112 15:07:44.541074 21236 tablet_copy_service.cc:161] P 3710b74ed2044ce5995533df32e67351: Beginning new tablet copy session on tablet 8c7dbaf285a04da9b4eec92a29ab7d6c from peer b5e9178a7a83402e8c4a6943ab78589d at {username='slave'} at 127.0.0.1:52204: session id = b5e9178a7a83402e8c4a6943ab78589d-8c7dbaf285a04da9b4eec92a29ab7d6c
I20250112 15:07:44.541728 21236 tablet_copy_source_session.cc:215] T 8c7dbaf285a04da9b4eec92a29ab7d6c P 3710b74ed2044ce5995533df32e67351: Tablet Copy: opened 0 blocks and 1 log segments
I20250112 15:07:44.542042 22077 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8c7dbaf285a04da9b4eec92a29ab7d6c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 15:07:44.543589 22077 tablet_copy_client.cc:806] T 8c7dbaf285a04da9b4eec92a29ab7d6c P b5e9178a7a83402e8c4a6943ab78589d: tablet copy: Starting download of 0 data blocks...
I20250112 15:07:44.543738 22077 tablet_copy_client.cc:670] T 8c7dbaf285a04da9b4eec92a29ab7d6c P b5e9178a7a83402e8c4a6943ab78589d: tablet copy: Starting download of 1 WAL segments...
I20250112 15:07:44.544198 22078 tablet_copy_client.cc:1025] T 8c7dbaf285a04da9b4eec92a29ab7d6c P b5e9178a7a83402e8c4a6943ab78589d: tablet copy: Injecting latency into file download: 2000
I20250112 15:07:44.564311 22081 ts_tablet_manager.cc:927] T 16f16fcd5cf347e5853189dc4da55c7f P b5e9178a7a83402e8c4a6943ab78589d: Initiating tablet copy from peer 2537bbbd1a9947ae9facd87567e3ccb9 (127.17.60.193:33173)
I20250112 15:07:44.564517 22081 tablet_copy_client.cc:323] T 16f16fcd5cf347e5853189dc4da55c7f P b5e9178a7a83402e8c4a6943ab78589d: tablet copy: Beginning tablet copy session from remote peer at address 127.17.60.193:33173
I20250112 15:07:44.564694 21085 tablet_copy_service.cc:140] P 2537bbbd1a9947ae9facd87567e3ccb9: Received BeginTabletCopySession request for tablet 16f16fcd5cf347e5853189dc4da55c7f from peer b5e9178a7a83402e8c4a6943ab78589d ({username='slave'} at 127.0.0.1:54448)
I20250112 15:07:44.564740 21085 tablet_copy_service.cc:161] P 2537bbbd1a9947ae9facd87567e3ccb9: Beginning new tablet copy session on tablet 16f16fcd5cf347e5853189dc4da55c7f from peer b5e9178a7a83402e8c4a6943ab78589d at {username='slave'} at 127.0.0.1:54448: session id = b5e9178a7a83402e8c4a6943ab78589d-16f16fcd5cf347e5853189dc4da55c7f
I20250112 15:07:44.565372 21085 tablet_copy_source_session.cc:215] T 16f16fcd5cf347e5853189dc4da55c7f P 2537bbbd1a9947ae9facd87567e3ccb9: Tablet Copy: opened 0 blocks and 1 log segments
I20250112 15:07:44.565690 22081 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 16f16fcd5cf347e5853189dc4da55c7f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 15:07:44.568452 22081 tablet_copy_client.cc:806] T 16f16fcd5cf347e5853189dc4da55c7f P b5e9178a7a83402e8c4a6943ab78589d: tablet copy: Starting download of 0 data blocks...
I20250112 15:07:44.568626 22081 tablet_copy_client.cc:670] T 16f16fcd5cf347e5853189dc4da55c7f P b5e9178a7a83402e8c4a6943ab78589d: tablet copy: Starting download of 1 WAL segments...
I20250112 15:07:44.569185 22082 tablet_copy_client.cc:1025] T 16f16fcd5cf347e5853189dc4da55c7f P b5e9178a7a83402e8c4a6943ab78589d: tablet copy: Injecting latency into file download: 2000
I20250112 15:07:44.619834 22083 ts_tablet_manager.cc:927] T bfaeae584ec04772b2d6fcd088022117 P a2a5bba4ac1145d5bab94a69c9f4df40: Initiating tablet copy from peer c561ec3374b94b95be0c155d80ff0bff (127.17.60.194:33627)
I20250112 15:07:44.620079 22083 tablet_copy_client.cc:323] T bfaeae584ec04772b2d6fcd088022117 P a2a5bba4ac1145d5bab94a69c9f4df40: tablet copy: Beginning tablet copy session from remote peer at address 127.17.60.194:33627
I20250112 15:07:44.620252 21161 tablet_copy_service.cc:140] P c561ec3374b94b95be0c155d80ff0bff: Received BeginTabletCopySession request for tablet bfaeae584ec04772b2d6fcd088022117 from peer a2a5bba4ac1145d5bab94a69c9f4df40 ({username='slave'} at 127.0.0.1:58522)
I20250112 15:07:44.620306 21161 tablet_copy_service.cc:161] P c561ec3374b94b95be0c155d80ff0bff: Beginning new tablet copy session on tablet bfaeae584ec04772b2d6fcd088022117 from peer a2a5bba4ac1145d5bab94a69c9f4df40 at {username='slave'} at 127.0.0.1:58522: session id = a2a5bba4ac1145d5bab94a69c9f4df40-bfaeae584ec04772b2d6fcd088022117
I20250112 15:07:44.620853 21161 tablet_copy_source_session.cc:215] T bfaeae584ec04772b2d6fcd088022117 P c561ec3374b94b95be0c155d80ff0bff: Tablet Copy: opened 0 blocks and 1 log segments
I20250112 15:07:44.621132 22083 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bfaeae584ec04772b2d6fcd088022117. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 15:07:44.622519 22083 tablet_copy_client.cc:806] T bfaeae584ec04772b2d6fcd088022117 P a2a5bba4ac1145d5bab94a69c9f4df40: tablet copy: Starting download of 0 data blocks...
I20250112 15:07:44.622665 22083 tablet_copy_client.cc:670] T bfaeae584ec04772b2d6fcd088022117 P a2a5bba4ac1145d5bab94a69c9f4df40: tablet copy: Starting download of 1 WAL segments...
I20250112 15:07:44.623122 22084 tablet_copy_client.cc:1025] T bfaeae584ec04772b2d6fcd088022117 P a2a5bba4ac1145d5bab94a69c9f4df40: tablet copy: Injecting latency into file download: 2000
I20250112 15:07:44.628165 22042 raft_consensus.cc:988] T 02c33d457b6b47138dd712850819b619 P 3710b74ed2044ce5995533df32e67351: : Instructing follower 2537bbbd1a9947ae9facd87567e3ccb9 to start an election
I20250112 15:07:44.628232 22042 raft_consensus.cc:1076] T 02c33d457b6b47138dd712850819b619 P 3710b74ed2044ce5995533df32e67351 [term 4 LEADER]: Signalling peer 2537bbbd1a9947ae9facd87567e3ccb9 to start an election
I20250112 15:07:44.628378 21068 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "02c33d457b6b47138dd712850819b619"
dest_uuid: "2537bbbd1a9947ae9facd87567e3ccb9"
from {username='slave'} at 127.0.0.1:37204
I20250112 15:07:44.628454 21068 raft_consensus.cc:491] T 02c33d457b6b47138dd712850819b619 P 2537bbbd1a9947ae9facd87567e3ccb9 [term 4 FOLLOWER]: Starting forced leader election (received explicit request)
I20250112 15:07:44.628494 21068 raft_consensus.cc:3054] T 02c33d457b6b47138dd712850819b619 P 2537bbbd1a9947ae9facd87567e3ccb9 [term 4 FOLLOWER]: Advancing to term 5
I20250112 15:07:44.629177 21068 raft_consensus.cc:513] T 02c33d457b6b47138dd712850819b619 P 2537bbbd1a9947ae9facd87567e3ccb9 [term 5 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3710b74ed2044ce5995533df32e67351" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 38589 } } peers { permanent_uuid: "2537bbbd1a9947ae9facd87567e3ccb9" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 33173 } } peers { permanent_uuid: "c561ec3374b94b95be0c155d80ff0bff" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 33627 } }
I20250112 15:07:44.629316 21068 leader_election.cc:290] T 02c33d457b6b47138dd712850819b619 P 2537bbbd1a9947ae9facd87567e3ccb9 [CANDIDATE]: Term 5 election: Requested vote from peers 3710b74ed2044ce5995533df32e67351 (127.17.60.195:38589), c561ec3374b94b95be0c155d80ff0bff (127.17.60.194:33627)
I20250112 15:07:44.629434 21222 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "02c33d457b6b47138dd712850819b619" candidate_uuid: "2537bbbd1a9947ae9facd87567e3ccb9" candidate_term: 5 candidate_status { last_received { term: 4 index: 28 } } ignore_live_leader: true dest_uuid: "3710b74ed2044ce5995533df32e67351"
I20250112 15:07:44.629451 21149 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "02c33d457b6b47138dd712850819b619" candidate_uuid: "2537bbbd1a9947ae9facd87567e3ccb9" candidate_term: 5 candidate_status { last_received { term: 4 index: 28 } } ignore_live_leader: true dest_uuid: "c561ec3374b94b95be0c155d80ff0bff"
I20250112 15:07:44.629524 21222 raft_consensus.cc:3049] T 02c33d457b6b47138dd712850819b619 P 3710b74ed2044ce5995533df32e67351 [term 4 LEADER]: Stepping down as leader of term 4
I20250112 15:07:44.629534 21149 raft_consensus.cc:3054] T 02c33d457b6b47138dd712850819b619 P c561ec3374b94b95be0c155d80ff0bff [term 4 FOLLOWER]: Advancing to term 5
I20250112 15:07:44.629562 21222 raft_consensus.cc:738] T 02c33d457b6b47138dd712850819b619 P 3710b74ed2044ce5995533df32e67351 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 3710b74ed2044ce5995533df32e67351, State: Running, Role: LEADER
I20250112 15:07:44.629616 21222 consensus_queue.cc:260] T 02c33d457b6b47138dd712850819b619 P 3710b74ed2044ce5995533df32e67351 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 28, Committed index: 28, Last appended: 4.28, Last appended by leader: 28, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3710b74ed2044ce5995533df32e67351" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 38589 } } peers { permanent_uuid: "2537bbbd1a9947ae9facd87567e3ccb9" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 33173 } } peers { permanent_uuid: "c561ec3374b94b95be0c155d80ff0bff" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 33627 } }
I20250112 15:07:44.629683 21222 raft_consensus.cc:3054] T 02c33d457b6b47138dd712850819b619 P 3710b74ed2044ce5995533df32e67351 [term 4 FOLLOWER]: Advancing to term 5
I20250112 15:07:44.630187 21149 raft_consensus.cc:2463] T 02c33d457b6b47138dd712850819b619 P c561ec3374b94b95be0c155d80ff0bff [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2537bbbd1a9947ae9facd87567e3ccb9 in term 5.
I20250112 15:07:44.630327 21222 raft_consensus.cc:2463] T 02c33d457b6b47138dd712850819b619 P 3710b74ed2044ce5995533df32e67351 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2537bbbd1a9947ae9facd87567e3ccb9 in term 5.
I20250112 15:07:44.630450 21039 leader_election.cc:304] T 02c33d457b6b47138dd712850819b619 P 2537bbbd1a9947ae9facd87567e3ccb9 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2537bbbd1a9947ae9facd87567e3ccb9, 3710b74ed2044ce5995533df32e67351; no voters:
I20250112 15:07:44.630556 21969 raft_consensus.cc:2798] T 02c33d457b6b47138dd712850819b619 P 2537bbbd1a9947ae9facd87567e3ccb9 [term 5 FOLLOWER]: Leader election won for term 5
I20250112 15:07:44.630673 21969 raft_consensus.cc:695] T 02c33d457b6b47138dd712850819b619 P 2537bbbd1a9947ae9facd87567e3ccb9 [term 5 LEADER]: Becoming Leader. State: Replica: 2537bbbd1a9947ae9facd87567e3ccb9, State: Running, Role: LEADER
I20250112 15:07:44.630717 21969 consensus_queue.cc:237] T 02c33d457b6b47138dd712850819b619 P 2537bbbd1a9947ae9facd87567e3ccb9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 28, Committed index: 28, Last appended: 4.28, Last appended by leader: 28, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3710b74ed2044ce5995533df32e67351" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 38589 } } peers { permanent_uuid: "2537bbbd1a9947ae9facd87567e3ccb9" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 33173 } } peers { permanent_uuid: "c561ec3374b94b95be0c155d80ff0bff" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 33627 } }
I20250112 15:07:44.631215 20965 catalog_manager.cc:5526] T 02c33d457b6b47138dd712850819b619 P 2537bbbd1a9947ae9facd87567e3ccb9 reported cstate change: term changed from 4 to 5, leader changed from 3710b74ed2044ce5995533df32e67351 (127.17.60.195) to 2537bbbd1a9947ae9facd87567e3ccb9 (127.17.60.193). New cstate: current_term: 5 leader_uuid: "2537bbbd1a9947ae9facd87567e3ccb9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3710b74ed2044ce5995533df32e67351" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 38589 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2537bbbd1a9947ae9facd87567e3ccb9" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 33173 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c561ec3374b94b95be0c155d80ff0bff" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 33627 } health_report { overall_health: UNKNOWN } } }
I20250112 15:07:44.636266 22087 ts_tablet_manager.cc:927] T ef90b704208140a3983ffe6105939319 P 25286c485fa04c86a5d07c8f88643a47: Initiating tablet copy from peer 2537bbbd1a9947ae9facd87567e3ccb9 (127.17.60.193:33173)
I20250112 15:07:44.636461 22087 tablet_copy_client.cc:323] T ef90b704208140a3983ffe6105939319 P 25286c485fa04c86a5d07c8f88643a47: tablet copy: Beginning tablet copy session from remote peer at address 127.17.60.193:33173
I20250112 15:07:44.636638 21085 tablet_copy_service.cc:140] P 2537bbbd1a9947ae9facd87567e3ccb9: Received BeginTabletCopySession request for tablet ef90b704208140a3983ffe6105939319 from peer 25286c485fa04c86a5d07c8f88643a47 ({username='slave'} at 127.0.0.1:54434)
I20250112 15:07:44.636690 21085 tablet_copy_service.cc:161] P 2537bbbd1a9947ae9facd87567e3ccb9: Beginning new tablet copy session on tablet ef90b704208140a3983ffe6105939319 from peer 25286c485fa04c86a5d07c8f88643a47 at {username='slave'} at 127.0.0.1:54434: session id = 25286c485fa04c86a5d07c8f88643a47-ef90b704208140a3983ffe6105939319
I20250112 15:07:44.637264 21085 tablet_copy_source_session.cc:215] T ef90b704208140a3983ffe6105939319 P 2537bbbd1a9947ae9facd87567e3ccb9: Tablet Copy: opened 0 blocks and 1 log segments
I20250112 15:07:44.637544 22087 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ef90b704208140a3983ffe6105939319. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 15:07:44.638921 22087 tablet_copy_client.cc:806] T ef90b704208140a3983ffe6105939319 P 25286c485fa04c86a5d07c8f88643a47: tablet copy: Starting download of 0 data blocks...
I20250112 15:07:44.639096 22087 tablet_copy_client.cc:670] T ef90b704208140a3983ffe6105939319 P 25286c485fa04c86a5d07c8f88643a47: tablet copy: Starting download of 1 WAL segments...
I20250112 15:07:44.639547 22088 tablet_copy_client.cc:1025] T ef90b704208140a3983ffe6105939319 P 25286c485fa04c86a5d07c8f88643a47: tablet copy: Injecting latency into file download: 2000
I20250112 15:07:44.726716 21149 raft_consensus.cc:1270] T 80cc505a04394c53a18c5beeded4655d P c561ec3374b94b95be0c155d80ff0bff [term 3 FOLLOWER]: Refusing update from remote peer 25286c485fa04c86a5d07c8f88643a47: Log matching property violated. Preceding OpId in replica: term: 2 index: 29. Preceding OpId from leader: term: 3 index: 30. (index mismatch)
I20250112 15:07:44.727115 22073 consensus_queue.cc:1035] T 80cc505a04394c53a18c5beeded4655d P 25286c485fa04c86a5d07c8f88643a47 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c561ec3374b94b95be0c155d80ff0bff" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 33627 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 30, Last known committed idx: 29, Time since last communication: 0.000s
I20250112 15:07:44.728463 21222 raft_consensus.cc:1270] T 80cc505a04394c53a18c5beeded4655d P 3710b74ed2044ce5995533df32e67351 [term 3 FOLLOWER]: Refusing update from remote peer 25286c485fa04c86a5d07c8f88643a47: Log matching property violated. Preceding OpId in replica: term: 2 index: 29. Preceding OpId from leader: term: 3 index: 30. (index mismatch)
I20250112 15:07:44.728663 21753 consensus_queue.cc:1035] T 80cc505a04394c53a18c5beeded4655d P 25286c485fa04c86a5d07c8f88643a47 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3710b74ed2044ce5995533df32e67351" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 38589 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 30, Last known committed idx: 29, Time since last communication: 0.000s
I20250112 15:07:44.928401 21149 raft_consensus.cc:1270] T 69c23627ab854000b360030c384dc966 P c561ec3374b94b95be0c155d80ff0bff [term 4 FOLLOWER]: Refusing update from remote peer a2a5bba4ac1145d5bab94a69c9f4df40: Log matching property violated. Preceding OpId in replica: term: 3 index: 32. Preceding OpId from leader: term: 4 index: 33. (index mismatch)
I20250112 15:07:44.928669 22075 consensus_queue.cc:1035] T 69c23627ab854000b360030c384dc966 P a2a5bba4ac1145d5bab94a69c9f4df40 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c561ec3374b94b95be0c155d80ff0bff" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 33627 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 33, Last known committed idx: 32, Time since last communication: 0.000s
I20250112 15:07:44.929682 21549 raft_consensus.cc:1270] T 69c23627ab854000b360030c384dc966 P b5e9178a7a83402e8c4a6943ab78589d [term 4 FOLLOWER]: Refusing update from remote peer a2a5bba4ac1145d5bab94a69c9f4df40: Log matching property violated. Preceding OpId in replica: term: 3 index: 32. Preceding OpId from leader: term: 4 index: 33. (index mismatch)
I20250112 15:07:44.929934 22041 consensus_queue.cc:1035] T 69c23627ab854000b360030c384dc966 P a2a5bba4ac1145d5bab94a69c9f4df40 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b5e9178a7a83402e8c4a6943ab78589d" member_type: VOTER last_known_addr { host: "127.17.60.197" port: 36573 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 33, Last known committed idx: 32, Time since last communication: 0.000s
I20250112 15:07:45.131052 21226 raft_consensus.cc:1270] T 02c33d457b6b47138dd712850819b619 P 3710b74ed2044ce5995533df32e67351 [term 5 FOLLOWER]: Refusing update from remote peer 2537bbbd1a9947ae9facd87567e3ccb9: Log matching property violated. Preceding OpId in replica: term: 4 index: 28. Preceding OpId from leader: term: 5 index: 29. (index mismatch)
I20250112 15:07:45.131280 22085 consensus_queue.cc:1035] T 02c33d457b6b47138dd712850819b619 P 2537bbbd1a9947ae9facd87567e3ccb9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3710b74ed2044ce5995533df32e67351" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 38589 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.000s
I20250112 15:07:45.132413 21149 raft_consensus.cc:1270] T 02c33d457b6b47138dd712850819b619 P c561ec3374b94b95be0c155d80ff0bff [term 5 FOLLOWER]: Refusing update from remote peer 2537bbbd1a9947ae9facd87567e3ccb9: Log matching property violated. Preceding OpId in replica: term: 4 index: 28. Preceding OpId from leader: term: 5 index: 29. (index mismatch)
I20250112 15:07:45.132642 22098 consensus_queue.cc:1035] T 02c33d457b6b47138dd712850819b619 P 2537bbbd1a9947ae9facd87567e3ccb9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c561ec3374b94b95be0c155d80ff0bff" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 33627 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.000s
I20250112 15:07:46.095367 21026 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250112 15:07:46.095799 21069 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "02c33d457b6b47138dd712850819b619"
dest_uuid: "2537bbbd1a9947ae9facd87567e3ccb9"
mode: GRACEFUL
new_leader_uuid: "c561ec3374b94b95be0c155d80ff0bff"
from {username='slave'} at 127.0.0.1:54414
I20250112 15:07:46.095896 21069 raft_consensus.cc:604] T 02c33d457b6b47138dd712850819b619 P 2537bbbd1a9947ae9facd87567e3ccb9 [term 5 LEADER]: Received request to transfer leadership to TS c561ec3374b94b95be0c155d80ff0bff
I20250112 15:07:46.096163 21026 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 1
I20250112 15:07:46.096246 21026 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20250112 15:07:46.355566 21969 raft_consensus.cc:988] T 02c33d457b6b47138dd712850819b619 P 2537bbbd1a9947ae9facd87567e3ccb9: : Instructing follower c561ec3374b94b95be0c155d80ff0bff to start an election
I20250112 15:07:46.355667 21969 raft_consensus.cc:1076] T 02c33d457b6b47138dd712850819b619 P 2537bbbd1a9947ae9facd87567e3ccb9 [term 5 LEADER]: Signalling peer c561ec3374b94b95be0c155d80ff0bff to start an election
I20250112 15:07:46.355840 21147 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "02c33d457b6b47138dd712850819b619"
dest_uuid: "c561ec3374b94b95be0c155d80ff0bff"
from {username='slave'} at 127.0.0.1:48082
I20250112 15:07:46.355916 21147 raft_consensus.cc:491] T 02c33d457b6b47138dd712850819b619 P c561ec3374b94b95be0c155d80ff0bff [term 5 FOLLOWER]: Starting forced leader election (received explicit request)
I20250112 15:07:46.355952 21147 raft_consensus.cc:3054] T 02c33d457b6b47138dd712850819b619 P c561ec3374b94b95be0c155d80ff0bff [term 5 FOLLOWER]: Advancing to term 6
I20250112 15:07:46.356856 21147 raft_consensus.cc:513] T 02c33d457b6b47138dd712850819b619 P c561ec3374b94b95be0c155d80ff0bff [term 6 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3710b74ed2044ce5995533df32e67351" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 38589 } } peers { permanent_uuid: "2537bbbd1a9947ae9facd87567e3ccb9" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 33173 } } peers { permanent_uuid: "c561ec3374b94b95be0c155d80ff0bff" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 33627 } }
I20250112 15:07:46.356976 21147 leader_election.cc:290] T 02c33d457b6b47138dd712850819b619 P c561ec3374b94b95be0c155d80ff0bff [CANDIDATE]: Term 6 election: Requested vote from peers 3710b74ed2044ce5995533df32e67351 (127.17.60.195:38589), 2537bbbd1a9947ae9facd87567e3ccb9 (127.17.60.193:33173)
I20250112 15:07:46.357070 21222 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "02c33d457b6b47138dd712850819b619" candidate_uuid: "c561ec3374b94b95be0c155d80ff0bff" candidate_term: 6 candidate_status { last_received { term: 5 index: 29 } } ignore_live_leader: true dest_uuid: "3710b74ed2044ce5995533df32e67351"
I20250112 15:07:46.357136 21068 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "02c33d457b6b47138dd712850819b619" candidate_uuid: "c561ec3374b94b95be0c155d80ff0bff" candidate_term: 6 candidate_status { last_received { term: 5 index: 29 } } ignore_live_leader: true dest_uuid: "2537bbbd1a9947ae9facd87567e3ccb9"
I20250112 15:07:46.357153 21222 raft_consensus.cc:3054] T 02c33d457b6b47138dd712850819b619 P 3710b74ed2044ce5995533df32e67351 [term 5 FOLLOWER]: Advancing to term 6
I20250112 15:07:46.357193 21068 raft_consensus.cc:3049] T 02c33d457b6b47138dd712850819b619 P 2537bbbd1a9947ae9facd87567e3ccb9 [term 5 LEADER]: Stepping down as leader of term 5
I20250112 15:07:46.357226 21068 raft_consensus.cc:738] T 02c33d457b6b47138dd712850819b619 P 2537bbbd1a9947ae9facd87567e3ccb9 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 2537bbbd1a9947ae9facd87567e3ccb9, State: Running, Role: LEADER
I20250112 15:07:46.357275 21068 consensus_queue.cc:260] T 02c33d457b6b47138dd712850819b619 P 2537bbbd1a9947ae9facd87567e3ccb9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 29, Committed index: 29, Last appended: 5.29, Last appended by leader: 29, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3710b74ed2044ce5995533df32e67351" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 38589 } } peers { permanent_uuid: "2537bbbd1a9947ae9facd87567e3ccb9" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 33173 } } peers { permanent_uuid: "c561ec3374b94b95be0c155d80ff0bff" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 33627 } }
I20250112 15:07:46.357334 21068 raft_consensus.cc:3054] T 02c33d457b6b47138dd712850819b619 P 2537bbbd1a9947ae9facd87567e3ccb9 [term 5 FOLLOWER]: Advancing to term 6
I20250112 15:07:46.357841 21222 raft_consensus.cc:2463] T 02c33d457b6b47138dd712850819b619 P 3710b74ed2044ce5995533df32e67351 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c561ec3374b94b95be0c155d80ff0bff in term 6.
I20250112 15:07:46.357909 21068 raft_consensus.cc:2463] T 02c33d457b6b47138dd712850819b619 P 2537bbbd1a9947ae9facd87567e3ccb9 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c561ec3374b94b95be0c155d80ff0bff in term 6.
I20250112 15:07:46.357970 21115 leader_election.cc:304] T 02c33d457b6b47138dd712850819b619 P c561ec3374b94b95be0c155d80ff0bff [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3710b74ed2044ce5995533df32e67351, c561ec3374b94b95be0c155d80ff0bff; no voters:
I20250112 15:07:46.358062 22103 raft_consensus.cc:2798] T 02c33d457b6b47138dd712850819b619 P c561ec3374b94b95be0c155d80ff0bff [term 6 FOLLOWER]: Leader election won for term 6
I20250112 15:07:46.358151 22103 raft_consensus.cc:695] T 02c33d457b6b47138dd712850819b619 P c561ec3374b94b95be0c155d80ff0bff [term 6 LEADER]: Becoming Leader. State: Replica: c561ec3374b94b95be0c155d80ff0bff, State: Running, Role: LEADER
I20250112 15:07:46.358197 22103 consensus_queue.cc:237] T 02c33d457b6b47138dd712850819b619 P c561ec3374b94b95be0c155d80ff0bff [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 29, Committed index: 29, Last appended: 5.29, Last appended by leader: 29, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3710b74ed2044ce5995533df32e67351" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 38589 } } peers { permanent_uuid: "2537bbbd1a9947ae9facd87567e3ccb9" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 33173 } } peers { permanent_uuid: "c561ec3374b94b95be0c155d80ff0bff" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 33627 } }
I20250112 15:07:46.358624 20965 catalog_manager.cc:5526] T 02c33d457b6b47138dd712850819b619 P c561ec3374b94b95be0c155d80ff0bff reported cstate change: term changed from 5 to 6, leader changed from 2537bbbd1a9947ae9facd87567e3ccb9 (127.17.60.193) to c561ec3374b94b95be0c155d80ff0bff (127.17.60.194). New cstate: current_term: 6 leader_uuid: "c561ec3374b94b95be0c155d80ff0bff" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3710b74ed2044ce5995533df32e67351" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 38589 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2537bbbd1a9947ae9facd87567e3ccb9" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 33173 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c561ec3374b94b95be0c155d80ff0bff" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 33627 } health_report { overall_health: HEALTHY } } }
I20250112 15:07:46.544420 22077 tablet_copy_client.cc:538] T 8c7dbaf285a04da9b4eec92a29ab7d6c P b5e9178a7a83402e8c4a6943ab78589d: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250112 15:07:46.545676 22077 tablet_bootstrap.cc:492] T 8c7dbaf285a04da9b4eec92a29ab7d6c P b5e9178a7a83402e8c4a6943ab78589d: Bootstrap starting.
I20250112 15:07:46.550433 22077 tablet_bootstrap.cc:492] T 8c7dbaf285a04da9b4eec92a29ab7d6c P b5e9178a7a83402e8c4a6943ab78589d: Bootstrap replayed 1/1 log segments. Stats: ops{read=32 overwritten=0 applied=32 ignored=0} inserts{seen=105 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250112 15:07:46.550668 22077 tablet_bootstrap.cc:492] T 8c7dbaf285a04da9b4eec92a29ab7d6c P b5e9178a7a83402e8c4a6943ab78589d: Bootstrap complete.
I20250112 15:07:46.550786 22077 ts_tablet_manager.cc:1397] T 8c7dbaf285a04da9b4eec92a29ab7d6c P b5e9178a7a83402e8c4a6943ab78589d: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.004s
I20250112 15:07:46.550964 22077 raft_consensus.cc:357] T 8c7dbaf285a04da9b4eec92a29ab7d6c P b5e9178a7a83402e8c4a6943ab78589d [term 4 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "c561ec3374b94b95be0c155d80ff0bff" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 33627 } attrs { replace: true } } peers { permanent_uuid: "3710b74ed2044ce5995533df32e67351" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 38589 } } peers { permanent_uuid: "a2a5bba4ac1145d5bab94a69c9f4df40" member_type: VOTER last_known_addr { host: "127.17.60.196" port: 41807 } attrs { promote: false } } peers { permanent_uuid: "b5e9178a7a83402e8c4a6943ab78589d" member_type: NON_VOTER last_known_addr { host: "127.17.60.197" port: 36573 } attrs { promote: true } }
I20250112 15:07:46.551064 22077 raft_consensus.cc:738] T 8c7dbaf285a04da9b4eec92a29ab7d6c P b5e9178a7a83402e8c4a6943ab78589d [term 4 LEARNER]: Becoming Follower/Learner. State: Replica: b5e9178a7a83402e8c4a6943ab78589d, State: Initialized, Role: LEARNER
I20250112 15:07:46.551106 22077 consensus_queue.cc:260] T 8c7dbaf285a04da9b4eec92a29ab7d6c P b5e9178a7a83402e8c4a6943ab78589d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 32, Last appended: 4.32, Last appended by leader: 32, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "c561ec3374b94b95be0c155d80ff0bff" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 33627 } attrs { replace: true } } peers { permanent_uuid: "3710b74ed2044ce5995533df32e67351" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 38589 } } peers { permanent_uuid: "a2a5bba4ac1145d5bab94a69c9f4df40" member_type: VOTER last_known_addr { host: "127.17.60.196" port: 41807 } attrs { promote: false } } peers { permanent_uuid: "b5e9178a7a83402e8c4a6943ab78589d" member_type: NON_VOTER last_known_addr { host: "127.17.60.197" port: 36573 } attrs { promote: true } }
I20250112 15:07:46.551236 22077 ts_tablet_manager.cc:1428] T 8c7dbaf285a04da9b4eec92a29ab7d6c P b5e9178a7a83402e8c4a6943ab78589d: Time spent starting tablet: real 0.000s user 0.000s sys 0.003s
I20250112 15:07:46.551414 21236 tablet_copy_service.cc:342] P 3710b74ed2044ce5995533df32e67351: Request end of tablet copy session b5e9178a7a83402e8c4a6943ab78589d-8c7dbaf285a04da9b4eec92a29ab7d6c received from {username='slave'} at 127.0.0.1:52204
I20250112 15:07:46.551472 21236 tablet_copy_service.cc:434] P 3710b74ed2044ce5995533df32e67351: ending tablet copy session b5e9178a7a83402e8c4a6943ab78589d-8c7dbaf285a04da9b4eec92a29ab7d6c on tablet 8c7dbaf285a04da9b4eec92a29ab7d6c with peer b5e9178a7a83402e8c4a6943ab78589d
I20250112 15:07:46.569388 22081 tablet_copy_client.cc:538] T 16f16fcd5cf347e5853189dc4da55c7f P b5e9178a7a83402e8c4a6943ab78589d: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250112 15:07:46.569646 21549 raft_consensus.cc:1212] T 8c7dbaf285a04da9b4eec92a29ab7d6c P b5e9178a7a83402e8c4a6943ab78589d [term 4 LEARNER]: Deduplicated request from leader. Original: 4.31->[4.32-4.32] Dedup: 4.32->[]
I20250112 15:07:46.570510 22081 tablet_bootstrap.cc:492] T 16f16fcd5cf347e5853189dc4da55c7f P b5e9178a7a83402e8c4a6943ab78589d: Bootstrap starting.
I20250112 15:07:46.574148 22081 tablet_bootstrap.cc:492] T 16f16fcd5cf347e5853189dc4da55c7f P b5e9178a7a83402e8c4a6943ab78589d: Bootstrap replayed 1/1 log segments. Stats: ops{read=30 overwritten=0 applied=30 ignored=0} inserts{seen=120 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250112 15:07:46.574347 22081 tablet_bootstrap.cc:492] T 16f16fcd5cf347e5853189dc4da55c7f P b5e9178a7a83402e8c4a6943ab78589d: Bootstrap complete.
I20250112 15:07:46.574443 22081 ts_tablet_manager.cc:1397] T 16f16fcd5cf347e5853189dc4da55c7f P b5e9178a7a83402e8c4a6943ab78589d: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20250112 15:07:46.574595 22081 raft_consensus.cc:357] T 16f16fcd5cf347e5853189dc4da55c7f P b5e9178a7a83402e8c4a6943ab78589d [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "2537bbbd1a9947ae9facd87567e3ccb9" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 33173 } } peers { permanent_uuid: "3710b74ed2044ce5995533df32e67351" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 38589 } attrs { replace: true } } peers { permanent_uuid: "25286c485fa04c86a5d07c8f88643a47" member_type: VOTER last_known_addr { host: "127.17.60.198" port: 37089 } attrs { promote: false } } peers { permanent_uuid: "b5e9178a7a83402e8c4a6943ab78589d" member_type: NON_VOTER last_known_addr { host: "127.17.60.197" port: 36573 } attrs { promote: true } }
I20250112 15:07:46.574661 22081 raft_consensus.cc:738] T 16f16fcd5cf347e5853189dc4da55c7f P b5e9178a7a83402e8c4a6943ab78589d [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: b5e9178a7a83402e8c4a6943ab78589d, State: Initialized, Role: LEARNER
I20250112 15:07:46.574692 22081 consensus_queue.cc:260] T 16f16fcd5cf347e5853189dc4da55c7f P b5e9178a7a83402e8c4a6943ab78589d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 30, Last appended: 2.30, Last appended by leader: 30, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "2537bbbd1a9947ae9facd87567e3ccb9" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 33173 } } peers { permanent_uuid: "3710b74ed2044ce5995533df32e67351" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 38589 } attrs { replace: true } } peers { permanent_uuid: "25286c485fa04c86a5d07c8f88643a47" member_type: VOTER last_known_addr { host: "127.17.60.198" port: 37089 } attrs { promote: false } } peers { permanent_uuid: "b5e9178a7a83402e8c4a6943ab78589d" member_type: NON_VOTER last_known_addr { host: "127.17.60.197" port: 36573 } attrs { promote: true } }
I20250112 15:07:46.574826 22081 ts_tablet_manager.cc:1428] T 16f16fcd5cf347e5853189dc4da55c7f P b5e9178a7a83402e8c4a6943ab78589d: Time spent starting tablet: real 0.000s user 0.002s sys 0.000s
I20250112 15:07:46.575032 21085 tablet_copy_service.cc:342] P 2537bbbd1a9947ae9facd87567e3ccb9: Request end of tablet copy session b5e9178a7a83402e8c4a6943ab78589d-16f16fcd5cf347e5853189dc4da55c7f received from {username='slave'} at 127.0.0.1:54448
I20250112 15:07:46.575083 21085 tablet_copy_service.cc:434] P 2537bbbd1a9947ae9facd87567e3ccb9: ending tablet copy session b5e9178a7a83402e8c4a6943ab78589d-16f16fcd5cf347e5853189dc4da55c7f on tablet 16f16fcd5cf347e5853189dc4da55c7f with peer b5e9178a7a83402e8c4a6943ab78589d
I20250112 15:07:46.623363 22083 tablet_copy_client.cc:538] T bfaeae584ec04772b2d6fcd088022117 P a2a5bba4ac1145d5bab94a69c9f4df40: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250112 15:07:46.624390 22083 tablet_bootstrap.cc:492] T bfaeae584ec04772b2d6fcd088022117 P a2a5bba4ac1145d5bab94a69c9f4df40: Bootstrap starting.
I20250112 15:07:46.628278 22083 tablet_bootstrap.cc:492] T bfaeae584ec04772b2d6fcd088022117 P a2a5bba4ac1145d5bab94a69c9f4df40: Bootstrap replayed 1/1 log segments. Stats: ops{read=31 overwritten=0 applied=31 ignored=0} inserts{seen=100 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250112 15:07:46.628479 22083 tablet_bootstrap.cc:492] T bfaeae584ec04772b2d6fcd088022117 P a2a5bba4ac1145d5bab94a69c9f4df40: Bootstrap complete.
I20250112 15:07:46.628621 22083 ts_tablet_manager.cc:1397] T bfaeae584ec04772b2d6fcd088022117 P a2a5bba4ac1145d5bab94a69c9f4df40: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.005s
I20250112 15:07:46.628779 22083 raft_consensus.cc:357] T bfaeae584ec04772b2d6fcd088022117 P a2a5bba4ac1145d5bab94a69c9f4df40 [term 3 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "2537bbbd1a9947ae9facd87567e3ccb9" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 33173 } attrs { replace: true } } peers { permanent_uuid: "c561ec3374b94b95be0c155d80ff0bff" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 33627 } } peers { permanent_uuid: "25286c485fa04c86a5d07c8f88643a47" member_type: VOTER last_known_addr { host: "127.17.60.198" port: 37089 } attrs { promote: false } } peers { permanent_uuid: "a2a5bba4ac1145d5bab94a69c9f4df40" member_type: NON_VOTER last_known_addr { host: "127.17.60.196" port: 41807 } attrs { promote: true } }
I20250112 15:07:46.628865 22083 raft_consensus.cc:738] T bfaeae584ec04772b2d6fcd088022117 P a2a5bba4ac1145d5bab94a69c9f4df40 [term 3 LEARNER]: Becoming Follower/Learner. State: Replica: a2a5bba4ac1145d5bab94a69c9f4df40, State: Initialized, Role: LEARNER
I20250112 15:07:46.628924 22083 consensus_queue.cc:260] T bfaeae584ec04772b2d6fcd088022117 P a2a5bba4ac1145d5bab94a69c9f4df40 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 31, Last appended: 3.31, Last appended by leader: 31, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "2537bbbd1a9947ae9facd87567e3ccb9" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 33173 } attrs { replace: true } } peers { permanent_uuid: "c561ec3374b94b95be0c155d80ff0bff" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 33627 } } peers { permanent_uuid: "25286c485fa04c86a5d07c8f88643a47" member_type: VOTER last_known_addr { host: "127.17.60.198" port: 37089 } attrs { promote: false } } peers { permanent_uuid: "a2a5bba4ac1145d5bab94a69c9f4df40" member_type: NON_VOTER last_known_addr { host: "127.17.60.196" port: 41807 } attrs { promote: true } }
I20250112 15:07:46.629031 22083 ts_tablet_manager.cc:1428] T bfaeae584ec04772b2d6fcd088022117 P a2a5bba4ac1145d5bab94a69c9f4df40: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20250112 15:07:46.629215 21161 tablet_copy_service.cc:342] P c561ec3374b94b95be0c155d80ff0bff: Request end of tablet copy session a2a5bba4ac1145d5bab94a69c9f4df40-bfaeae584ec04772b2d6fcd088022117 received from {username='slave'} at 127.0.0.1:58522
I20250112 15:07:46.629276 21161 tablet_copy_service.cc:434] P c561ec3374b94b95be0c155d80ff0bff: ending tablet copy session a2a5bba4ac1145d5bab94a69c9f4df40-bfaeae584ec04772b2d6fcd088022117 on tablet bfaeae584ec04772b2d6fcd088022117 with peer a2a5bba4ac1145d5bab94a69c9f4df40
I20250112 15:07:46.639739 22087 tablet_copy_client.cc:538] T ef90b704208140a3983ffe6105939319 P 25286c485fa04c86a5d07c8f88643a47: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250112 15:07:46.640820 22087 tablet_bootstrap.cc:492] T ef90b704208140a3983ffe6105939319 P 25286c485fa04c86a5d07c8f88643a47: Bootstrap starting.
I20250112 15:07:46.644443 22087 tablet_bootstrap.cc:492] T ef90b704208140a3983ffe6105939319 P 25286c485fa04c86a5d07c8f88643a47: Bootstrap replayed 1/1 log segments. Stats: ops{read=29 overwritten=0 applied=29 ignored=0} inserts{seen=92 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250112 15:07:46.644703 22087 tablet_bootstrap.cc:492] T ef90b704208140a3983ffe6105939319 P 25286c485fa04c86a5d07c8f88643a47: Bootstrap complete.
I20250112 15:07:46.644812 22087 ts_tablet_manager.cc:1397] T ef90b704208140a3983ffe6105939319 P 25286c485fa04c86a5d07c8f88643a47: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.004s
I20250112 15:07:46.644974 22087 raft_consensus.cc:357] T ef90b704208140a3983ffe6105939319 P 25286c485fa04c86a5d07c8f88643a47 [term 4 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "2537bbbd1a9947ae9facd87567e3ccb9" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 33173 } } peers { permanent_uuid: "c561ec3374b94b95be0c155d80ff0bff" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 33627 } attrs { replace: true } } peers { permanent_uuid: "3710b74ed2044ce5995533df32e67351" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 38589 } } peers { permanent_uuid: "25286c485fa04c86a5d07c8f88643a47" member_type: NON_VOTER last_known_addr { host: "127.17.60.198" port: 37089 } attrs { promote: true } }
I20250112 15:07:46.645061 22087 raft_consensus.cc:738] T ef90b704208140a3983ffe6105939319 P 25286c485fa04c86a5d07c8f88643a47 [term 4 LEARNER]: Becoming Follower/Learner. State: Replica: 25286c485fa04c86a5d07c8f88643a47, State: Initialized, Role: LEARNER
I20250112 15:07:46.645097 22087 consensus_queue.cc:260] T ef90b704208140a3983ffe6105939319 P 25286c485fa04c86a5d07c8f88643a47 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 29, Last appended: 4.29, Last appended by leader: 29, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "2537bbbd1a9947ae9facd87567e3ccb9" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 33173 } } peers { permanent_uuid: "c561ec3374b94b95be0c155d80ff0bff" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 33627 } attrs { replace: true } } peers { permanent_uuid: "3710b74ed2044ce5995533df32e67351" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 38589 } } peers { permanent_uuid: "25286c485fa04c86a5d07c8f88643a47" member_type: NON_VOTER last_known_addr { host: "127.17.60.198" port: 37089 } attrs { promote: true } }
I20250112 15:07:46.645272 22087 ts_tablet_manager.cc:1428] T ef90b704208140a3983ffe6105939319 P 25286c485fa04c86a5d07c8f88643a47: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20250112 15:07:46.645473 21085 tablet_copy_service.cc:342] P 2537bbbd1a9947ae9facd87567e3ccb9: Request end of tablet copy session 25286c485fa04c86a5d07c8f88643a47-ef90b704208140a3983ffe6105939319 received from {username='slave'} at 127.0.0.1:54434
I20250112 15:07:46.645525 21085 tablet_copy_service.cc:434] P 2537bbbd1a9947ae9facd87567e3ccb9: ending tablet copy session 25286c485fa04c86a5d07c8f88643a47-ef90b704208140a3983ffe6105939319 on tablet ef90b704208140a3983ffe6105939319 with peer 25286c485fa04c86a5d07c8f88643a47
I20250112 15:07:46.665524 21549 raft_consensus.cc:1212] T 16f16fcd5cf347e5853189dc4da55c7f P b5e9178a7a83402e8c4a6943ab78589d [term 2 LEARNER]: Deduplicated request from leader. Original: 2.29->[2.30-2.30] Dedup: 2.30->[]
I20250112 15:07:46.706946 21624 raft_consensus.cc:1212] T ef90b704208140a3983ffe6105939319 P 25286c485fa04c86a5d07c8f88643a47 [term 4 LEARNER]: Deduplicated request from leader. Original: 4.28->[4.29-4.29] Dedup: 4.29->[]
I20250112 15:07:46.836541 21068 raft_consensus.cc:1270] T 02c33d457b6b47138dd712850819b619 P 2537bbbd1a9947ae9facd87567e3ccb9 [term 6 FOLLOWER]: Refusing update from remote peer c561ec3374b94b95be0c155d80ff0bff: Log matching property violated. Preceding OpId in replica: term: 5 index: 29. Preceding OpId from leader: term: 6 index: 30. (index mismatch)
I20250112 15:07:46.836755 22103 consensus_queue.cc:1035] T 02c33d457b6b47138dd712850819b619 P c561ec3374b94b95be0c155d80ff0bff [LEADER]: Connected to new peer: Peer: permanent_uuid: "2537bbbd1a9947ae9facd87567e3ccb9" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 33173 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 30, Last known committed idx: 29, Time since last communication: 0.000s
I20250112 15:07:46.837884 21222 raft_consensus.cc:1270] T 02c33d457b6b47138dd712850819b619 P 3710b74ed2044ce5995533df32e67351 [term 6 FOLLOWER]: Refusing update from remote peer c561ec3374b94b95be0c155d80ff0bff: Log matching property violated. Preceding OpId in replica: term: 5 index: 29. Preceding OpId from leader: term: 6 index: 30. (index mismatch)
I20250112 15:07:46.838099 22108 consensus_queue.cc:1035] T 02c33d457b6b47138dd712850819b619 P c561ec3374b94b95be0c155d80ff0bff [LEADER]: Connected to new peer: Peer: permanent_uuid: "3710b74ed2044ce5995533df32e67351" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 38589 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 30, Last known committed idx: 29, Time since last communication: 0.000s
I20250112 15:07:46.877475 21474 raft_consensus.cc:1212] T bfaeae584ec04772b2d6fcd088022117 P a2a5bba4ac1145d5bab94a69c9f4df40 [term 3 LEARNER]: Deduplicated request from leader. Original: 3.30->[3.31-3.31] Dedup: 3.31->[]
I20250112 15:07:46.958068 22001 raft_consensus.cc:1059] T 8c7dbaf285a04da9b4eec92a29ab7d6c P 3710b74ed2044ce5995533df32e67351: attempting to promote NON_VOTER b5e9178a7a83402e8c4a6943ab78589d to VOTER
I20250112 15:07:46.958298 22001 consensus_queue.cc:237] T 8c7dbaf285a04da9b4eec92a29ab7d6c P 3710b74ed2044ce5995533df32e67351 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 32, Committed index: 32, Last appended: 4.32, Last appended by leader: 28, Current term: 4, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 33 OBSOLETE_local: false peers { permanent_uuid: "c561ec3374b94b95be0c155d80ff0bff" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 33627 } attrs { replace: true } } peers { permanent_uuid: "3710b74ed2044ce5995533df32e67351" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 38589 } } peers { permanent_uuid: "a2a5bba4ac1145d5bab94a69c9f4df40" member_type: VOTER last_known_addr { host: "127.17.60.196" port: 41807 } attrs { promote: false } } peers { permanent_uuid: "b5e9178a7a83402e8c4a6943ab78589d" member_type: VOTER last_known_addr { host: "127.17.60.197" port: 36573 } attrs { promote: false } }
I20250112 15:07:46.959698 21474 raft_consensus.cc:1270] T 8c7dbaf285a04da9b4eec92a29ab7d6c P a2a5bba4ac1145d5bab94a69c9f4df40 [term 4 FOLLOWER]: Refusing update from remote peer 3710b74ed2044ce5995533df32e67351: Log matching property violated. Preceding OpId in replica: term: 4 index: 32. Preceding OpId from leader: term: 4 index: 33. (index mismatch)
I20250112 15:07:46.959700 21147 raft_consensus.cc:1270] T 8c7dbaf285a04da9b4eec92a29ab7d6c P c561ec3374b94b95be0c155d80ff0bff [term 4 FOLLOWER]: Refusing update from remote peer 3710b74ed2044ce5995533df32e67351: Log matching property violated. Preceding OpId in replica: term: 4 index: 32. Preceding OpId from leader: term: 4 index: 33. (index mismatch)
I20250112 15:07:46.959915 21549 raft_consensus.cc:1270] T 8c7dbaf285a04da9b4eec92a29ab7d6c P b5e9178a7a83402e8c4a6943ab78589d [term 4 LEARNER]: Refusing update from remote peer 3710b74ed2044ce5995533df32e67351: Log matching property violated. Preceding OpId in replica: term: 4 index: 32. Preceding OpId from leader: term: 4 index: 33. (index mismatch)
I20250112 15:07:46.960189 22001 consensus_queue.cc:1035] T 8c7dbaf285a04da9b4eec92a29ab7d6c P 3710b74ed2044ce5995533df32e67351 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b5e9178a7a83402e8c4a6943ab78589d" member_type: VOTER last_known_addr { host: "127.17.60.197" port: 36573 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 33, Last known committed idx: 32, Time since last communication: 0.000s
I20250112 15:07:46.960310 22126 consensus_queue.cc:1035] T 8c7dbaf285a04da9b4eec92a29ab7d6c P 3710b74ed2044ce5995533df32e67351 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2a5bba4ac1145d5bab94a69c9f4df40" member_type: VOTER last_known_addr { host: "127.17.60.196" port: 41807 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 33, Last known committed idx: 32, Time since last communication: 0.000s
I20250112 15:07:46.960826 22126 consensus_queue.cc:1035] T 8c7dbaf285a04da9b4eec92a29ab7d6c P 3710b74ed2044ce5995533df32e67351 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c561ec3374b94b95be0c155d80ff0bff" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 33627 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 33, Last known committed idx: 32, Time since last communication: 0.000s
I20250112 15:07:46.962085 21147 raft_consensus.cc:2949] T 8c7dbaf285a04da9b4eec92a29ab7d6c P c561ec3374b94b95be0c155d80ff0bff [term 4 FOLLOWER]: Committing config change with OpId 4.33: config changed from index 32 to 33, b5e9178a7a83402e8c4a6943ab78589d (127.17.60.197) changed from NON_VOTER to VOTER. New config: { opid_index: 33 OBSOLETE_local: false peers { permanent_uuid: "c561ec3374b94b95be0c155d80ff0bff" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 33627 } attrs { replace: true } } peers { permanent_uuid: "3710b74ed2044ce5995533df32e67351" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 38589 } } peers { permanent_uuid: "a2a5bba4ac1145d5bab94a69c9f4df40" member_type: VOTER last_known_addr { host: "127.17.60.196" port: 41807 } attrs { promote: false } } peers { permanent_uuid: "b5e9178a7a83402e8c4a6943ab78589d" member_type: VOTER last_known_addr { host: "127.17.60.197" port: 36573 } attrs { promote: false } } }
I20250112 15:07:46.961951 22126 raft_consensus.cc:2949] T 8c7dbaf285a04da9b4eec92a29ab7d6c P 3710b74ed2044ce5995533df32e67351 [term 4 LEADER]: Committing config change with OpId 4.33: config changed from index 32 to 33, b5e9178a7a83402e8c4a6943ab78589d (127.17.60.197) changed from NON_VOTER to VOTER. New config: { opid_index: 33 OBSOLETE_local: false peers { permanent_uuid: "c561ec3374b94b95be0c155d80ff0bff" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 33627 } attrs { replace: true } } peers { permanent_uuid: "3710b74ed2044ce5995533df32e67351" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 38589 } } peers { permanent_uuid: "a2a5bba4ac1145d5bab94a69c9f4df40" member_type: VOTER last_known_addr { host: "127.17.60.196" port: 41807 } attrs { promote: false } } peers { permanent_uuid: "b5e9178a7a83402e8c4a6943ab78589d" member_type: VOTER last_known_addr { host: "127.17.60.197" port: 36573 } attrs { promote: false } } }
I20250112 15:07:46.963105 21549 raft_consensus.cc:2949] T 8c7dbaf285a04da9b4eec92a29ab7d6c P b5e9178a7a83402e8c4a6943ab78589d [term 4 FOLLOWER]: Committing config change with OpId 4.33: config changed from index 32 to 33, b5e9178a7a83402e8c4a6943ab78589d (127.17.60.197) changed from NON_VOTER to VOTER. New config: { opid_index: 33 OBSOLETE_local: false peers { permanent_uuid: "c561ec3374b94b95be0c155d80ff0bff" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 33627 } attrs { replace: true } } peers { permanent_uuid: "3710b74ed2044ce5995533df32e67351" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 38589 } } peers { permanent_uuid: "a2a5bba4ac1145d5bab94a69c9f4df40" member_type: VOTER last_known_addr { host: "127.17.60.196" port: 41807 } attrs { promote: false } } peers { permanent_uuid: "b5e9178a7a83402e8c4a6943ab78589d" member_type: VOTER last_known_addr { host: "127.17.60.197" port: 36573 } attrs { promote: false } } }
I20250112 15:07:46.963394 20970 catalog_manager.cc:5526] T 8c7dbaf285a04da9b4eec92a29ab7d6c P c561ec3374b94b95be0c155d80ff0bff reported cstate change: config changed from index 32 to 33, b5e9178a7a83402e8c4a6943ab78589d (127.17.60.197) changed from NON_VOTER to VOTER. New cstate: current_term: 4 leader_uuid: "3710b74ed2044ce5995533df32e67351" committed_config { opid_index: 33 OBSOLETE_local: false peers { permanent_uuid: "c561ec3374b94b95be0c155d80ff0bff" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 33627 } attrs { replace: true } } peers { permanent_uuid: "3710b74ed2044ce5995533df32e67351" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 38589 } } peers { permanent_uuid: "a2a5bba4ac1145d5bab94a69c9f4df40" member_type: VOTER last_known_addr { host: "127.17.60.196" port: 41807 } attrs { promote: false } } peers { permanent_uuid: "b5e9178a7a83402e8c4a6943ab78589d" member_type: VOTER last_known_addr { host: "127.17.60.197" port: 36573 } attrs { promote: false } } }
I20250112 15:07:46.964601 21474 raft_consensus.cc:2949] T 8c7dbaf285a04da9b4eec92a29ab7d6c P a2a5bba4ac1145d5bab94a69c9f4df40 [term 4 FOLLOWER]: Committing config change with OpId 4.33: config changed from index 32 to 33, b5e9178a7a83402e8c4a6943ab78589d (127.17.60.197) changed from NON_VOTER to VOTER. New config: { opid_index: 33 OBSOLETE_local: false peers { permanent_uuid: "c561ec3374b94b95be0c155d80ff0bff" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 33627 } attrs { replace: true } } peers { permanent_uuid: "3710b74ed2044ce5995533df32e67351" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 38589 } } peers { permanent_uuid: "a2a5bba4ac1145d5bab94a69c9f4df40" member_type: VOTER last_known_addr { host: "127.17.60.196" port: 41807 } attrs { promote: false } } peers { permanent_uuid: "b5e9178a7a83402e8c4a6943ab78589d" member_type: VOTER last_known_addr { host: "127.17.60.197" port: 36573 } attrs { promote: false } } }
I20250112 15:07:46.966109 21222 consensus_queue.cc:237] T 8c7dbaf285a04da9b4eec92a29ab7d6c P 3710b74ed2044ce5995533df32e67351 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 33, Committed index: 33, Last appended: 4.33, Last appended by leader: 28, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 34 OBSOLETE_local: false peers { permanent_uuid: "3710b74ed2044ce5995533df32e67351" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 38589 } } peers { permanent_uuid: "a2a5bba4ac1145d5bab94a69c9f4df40" member_type: VOTER last_known_addr { host: "127.17.60.196" port: 41807 } attrs { promote: false } } peers { permanent_uuid: "b5e9178a7a83402e8c4a6943ab78589d" member_type: VOTER last_known_addr { host: "127.17.60.197" port: 36573 } attrs { promote: false } }
I20250112 15:07:46.966543 21549 raft_consensus.cc:1270] T 8c7dbaf285a04da9b4eec92a29ab7d6c P b5e9178a7a83402e8c4a6943ab78589d [term 4 FOLLOWER]: Refusing update from remote peer 3710b74ed2044ce5995533df32e67351: Log matching property violated. Preceding OpId in replica: term: 4 index: 33. Preceding OpId from leader: term: 4 index: 34. (index mismatch)
I20250112 15:07:46.966571 21474 raft_consensus.cc:1270] T 8c7dbaf285a04da9b4eec92a29ab7d6c P a2a5bba4ac1145d5bab94a69c9f4df40 [term 4 FOLLOWER]: Refusing update from remote peer 3710b74ed2044ce5995533df32e67351: Log matching property violated. Preceding OpId in replica: term: 4 index: 33. Preceding OpId from leader: term: 4 index: 34. (index mismatch)
I20250112 15:07:46.966755 22133 consensus_queue.cc:1035] T 8c7dbaf285a04da9b4eec92a29ab7d6c P 3710b74ed2044ce5995533df32e67351 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2a5bba4ac1145d5bab94a69c9f4df40" member_type: VOTER last_known_addr { host: "127.17.60.196" port: 41807 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 34, Last known committed idx: 33, Time since last communication: 0.000s
I20250112 15:07:46.966873 22001 consensus_queue.cc:1035] T 8c7dbaf285a04da9b4eec92a29ab7d6c P 3710b74ed2044ce5995533df32e67351 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b5e9178a7a83402e8c4a6943ab78589d" member_type: VOTER last_known_addr { host: "127.17.60.197" port: 36573 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 34, Last known committed idx: 33, Time since last communication: 0.000s
I20250112 15:07:46.967429 22001 raft_consensus.cc:2949] T 8c7dbaf285a04da9b4eec92a29ab7d6c P 3710b74ed2044ce5995533df32e67351 [term 4 LEADER]: Committing config change with OpId 4.34: config changed from index 33 to 34, VOTER c561ec3374b94b95be0c155d80ff0bff (127.17.60.194) evicted. New config: { opid_index: 34 OBSOLETE_local: false peers { permanent_uuid: "3710b74ed2044ce5995533df32e67351" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 38589 } } peers { permanent_uuid: "a2a5bba4ac1145d5bab94a69c9f4df40" member_type: VOTER last_known_addr { host: "127.17.60.196" port: 41807 } attrs { promote: false } } peers { permanent_uuid: "b5e9178a7a83402e8c4a6943ab78589d" member_type: VOTER last_known_addr { host: "127.17.60.197" port: 36573 } attrs { promote: false } } }
I20250112 15:07:46.967665 21549 raft_consensus.cc:2949] T 8c7dbaf285a04da9b4eec92a29ab7d6c P b5e9178a7a83402e8c4a6943ab78589d [term 4 FOLLOWER]: Committing config change with OpId 4.34: config changed from index 33 to 34, VOTER c561ec3374b94b95be0c155d80ff0bff (127.17.60.194) evicted. New config: { opid_index: 34 OBSOLETE_local: false peers { permanent_uuid: "3710b74ed2044ce5995533df32e67351" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 38589 } } peers { permanent_uuid: "a2a5bba4ac1145d5bab94a69c9f4df40" member_type: VOTER last_known_addr { host: "127.17.60.196" port: 41807 } attrs { promote: false } } peers { permanent_uuid: "b5e9178a7a83402e8c4a6943ab78589d" member_type: VOTER last_known_addr { host: "127.17.60.197" port: 36573 } attrs { promote: false } } }
I20250112 15:07:46.968173 20957 catalog_manager.cc:5039] ChangeConfig:REMOVE_PEER RPC for tablet 8c7dbaf285a04da9b4eec92a29ab7d6c with cas_config_opid_index 33: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250112 15:07:46.968202 21474 raft_consensus.cc:2949] T 8c7dbaf285a04da9b4eec92a29ab7d6c P a2a5bba4ac1145d5bab94a69c9f4df40 [term 4 FOLLOWER]: Committing config change with OpId 4.34: config changed from index 33 to 34, VOTER c561ec3374b94b95be0c155d80ff0bff (127.17.60.194) evicted. New config: { opid_index: 34 OBSOLETE_local: false peers { permanent_uuid: "3710b74ed2044ce5995533df32e67351" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 38589 } } peers { permanent_uuid: "a2a5bba4ac1145d5bab94a69c9f4df40" member_type: VOTER last_known_addr { host: "127.17.60.196" port: 41807 } attrs { promote: false } } peers { permanent_uuid: "b5e9178a7a83402e8c4a6943ab78589d" member_type: VOTER last_known_addr { host: "127.17.60.197" port: 36573 } attrs { promote: false } } }
I20250112 15:07:46.968561 20966 catalog_manager.cc:5526] T 8c7dbaf285a04da9b4eec92a29ab7d6c P 3710b74ed2044ce5995533df32e67351 reported cstate change: config changed from index 33 to 34, VOTER c561ec3374b94b95be0c155d80ff0bff (127.17.60.194) evicted. New cstate: current_term: 4 leader_uuid: "3710b74ed2044ce5995533df32e67351" committed_config { opid_index: 34 OBSOLETE_local: false peers { permanent_uuid: "3710b74ed2044ce5995533df32e67351" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 38589 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a2a5bba4ac1145d5bab94a69c9f4df40" member_type: VOTER last_known_addr { host: "127.17.60.196" port: 41807 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b5e9178a7a83402e8c4a6943ab78589d" member_type: VOTER last_known_addr { host: "127.17.60.197" port: 36573 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20250112 15:07:46.969471 21141 tablet_service.cc:1514] Processing DeleteTablet for tablet 8c7dbaf285a04da9b4eec92a29ab7d6c with delete_type TABLET_DATA_TOMBSTONED (TS c561ec3374b94b95be0c155d80ff0bff not found in new config with opid_index 34) from {username='slave'} at 127.0.0.1:48072
I20250112 15:07:46.969683 22134 tablet_replica.cc:331] T 8c7dbaf285a04da9b4eec92a29ab7d6c P c561ec3374b94b95be0c155d80ff0bff: stopping tablet replica
I20250112 15:07:46.969758 22134 raft_consensus.cc:2238] T 8c7dbaf285a04da9b4eec92a29ab7d6c P c561ec3374b94b95be0c155d80ff0bff [term 4 FOLLOWER]: Raft consensus shutting down.
I20250112 15:07:46.969846 22134 raft_consensus.cc:2267] T 8c7dbaf285a04da9b4eec92a29ab7d6c P c561ec3374b94b95be0c155d80ff0bff [term 4 FOLLOWER]: Raft consensus is shut down!
I20250112 15:07:46.970120 22134 ts_tablet_manager.cc:1905] T 8c7dbaf285a04da9b4eec92a29ab7d6c P c561ec3374b94b95be0c155d80ff0bff: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250112 15:07:46.971235 22134 ts_tablet_manager.cc:1918] T 8c7dbaf285a04da9b4eec92a29ab7d6c P c561ec3374b94b95be0c155d80ff0bff: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 4.33
I20250112 15:07:46.971298 22134 log.cc:1198] T 8c7dbaf285a04da9b4eec92a29ab7d6c P c561ec3374b94b95be0c155d80ff0bff: Deleting WAL directory at /tmp/dist-test-task95dLSD/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1736694351070047-17651-0/minicluster-data/ts-1-root/wals/8c7dbaf285a04da9b4eec92a29ab7d6c
I20250112 15:07:46.971635 20958 catalog_manager.cc:4872] TS c561ec3374b94b95be0c155d80ff0bff (127.17.60.194:33627): tablet 8c7dbaf285a04da9b4eec92a29ab7d6c (table test-workload [id=cdaf6d3a65a54ea08e8e2c0ecdfc9de7]) successfully deleted
I20250112 15:07:47.118763 22135 raft_consensus.cc:1059] T ef90b704208140a3983ffe6105939319 P 2537bbbd1a9947ae9facd87567e3ccb9: attempting to promote NON_VOTER 25286c485fa04c86a5d07c8f88643a47 to VOTER
I20250112 15:07:47.119078 22135 consensus_queue.cc:237] T ef90b704208140a3983ffe6105939319 P 2537bbbd1a9947ae9facd87567e3ccb9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 29, Committed index: 29, Last appended: 4.29, Last appended by leader: 27, Current term: 4, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "2537bbbd1a9947ae9facd87567e3ccb9" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 33173 } } peers { permanent_uuid: "c561ec3374b94b95be0c155d80ff0bff" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 33627 } attrs { replace: true } } peers { permanent_uuid: "3710b74ed2044ce5995533df32e67351" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 38589 } } peers { permanent_uuid: "25286c485fa04c86a5d07c8f88643a47" member_type: VOTER last_known_addr { host: "127.17.60.198" port: 37089 } attrs { promote: false } }
I20250112 15:07:47.119680 21226 raft_consensus.cc:1270] T ef90b704208140a3983ffe6105939319 P 3710b74ed2044ce5995533df32e67351 [term 4 FOLLOWER]: Refusing update from remote peer 2537bbbd1a9947ae9facd87567e3ccb9: Log matching property violated. Preceding OpId in replica: term: 4 index: 29. Preceding OpId from leader: term: 4 index: 30. (index mismatch)
I20250112 15:07:47.119719 21149 raft_consensus.cc:1270] T ef90b704208140a3983ffe6105939319 P c561ec3374b94b95be0c155d80ff0bff [term 4 FOLLOWER]: Refusing update from remote peer 2537bbbd1a9947ae9facd87567e3ccb9: Log matching property violated. Preceding OpId in replica: term: 4 index: 29. Preceding OpId from leader: term: 4 index: 30. (index mismatch)
I20250112 15:07:47.119693 21624 raft_consensus.cc:1270] T ef90b704208140a3983ffe6105939319 P 25286c485fa04c86a5d07c8f88643a47 [term 4 LEARNER]: Refusing update from remote peer 2537bbbd1a9947ae9facd87567e3ccb9: Log matching property violated. Preceding OpId in replica: term: 4 index: 29. Preceding OpId from leader: term: 4 index: 30. (index mismatch)
I20250112 15:07:47.119858 22138 consensus_queue.cc:1035] T ef90b704208140a3983ffe6105939319 P 2537bbbd1a9947ae9facd87567e3ccb9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3710b74ed2044ce5995533df32e67351" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 38589 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 30, Last known committed idx: 29, Time since last communication: 0.000s
I20250112 15:07:47.120056 22138 consensus_queue.cc:1035] T ef90b704208140a3983ffe6105939319 P 2537bbbd1a9947ae9facd87567e3ccb9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c561ec3374b94b95be0c155d80ff0bff" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 33627 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 30, Last known committed idx: 29, Time since last communication: 0.000s
I20250112 15:07:47.120136 21969 consensus_queue.cc:1035] T ef90b704208140a3983ffe6105939319 P 2537bbbd1a9947ae9facd87567e3ccb9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "25286c485fa04c86a5d07c8f88643a47" member_type: VOTER last_known_addr { host: "127.17.60.198" port: 37089 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 30, Last known committed idx: 29, Time since last communication: 0.000s
I20250112 15:07:47.120862 21969 raft_consensus.cc:2949] T ef90b704208140a3983ffe6105939319 P 2537bbbd1a9947ae9facd87567e3ccb9 [term 4 LEADER]: Committing config change with OpId 4.30: config changed from index 29 to 30, 25286c485fa04c86a5d07c8f88643a47 (127.17.60.198) changed from NON_VOTER to VOTER. New config: { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "2537bbbd1a9947ae9facd87567e3ccb9" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 33173 } } peers { permanent_uuid: "c561ec3374b94b95be0c155d80ff0bff" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 33627 } attrs { replace: true } } peers { permanent_uuid: "3710b74ed2044ce5995533df32e67351" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 38589 } } peers { permanent_uuid: "25286c485fa04c86a5d07c8f88643a47" member_type: VOTER last_known_addr { host: "127.17.60.198" port: 37089 } attrs { promote: false } } }
I20250112 15:07:47.120970 21226 raft_consensus.cc:2949] T ef90b704208140a3983ffe6105939319 P 3710b74ed2044ce5995533df32e67351 [term 4 FOLLOWER]: Committing config change with OpId 4.30: config changed from index 29 to 30, 25286c485fa04c86a5d07c8f88643a47 (127.17.60.198) changed from NON_VOTER to VOTER. New config: { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "2537bbbd1a9947ae9facd87567e3ccb9" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 33173 } } peers { permanent_uuid: "c561ec3374b94b95be0c155d80ff0bff" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 33627 } attrs { replace: true } } peers { permanent_uuid: "3710b74ed2044ce5995533df32e67351" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 38589 } } peers { permanent_uuid: "25286c485fa04c86a5d07c8f88643a47" member_type: VOTER last_known_addr { host: "127.17.60.198" port: 37089 } attrs { promote: false } } }
I20250112 15:07:47.121836 21624 raft_consensus.cc:2949] T ef90b704208140a3983ffe6105939319 P 25286c485fa04c86a5d07c8f88643a47 [term 4 FOLLOWER]: Committing config change with OpId 4.30: config changed from index 29 to 30, 25286c485fa04c86a5d07c8f88643a47 (127.17.60.198) changed from NON_VOTER to VOTER. New config: { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "2537bbbd1a9947ae9facd87567e3ccb9" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 33173 } } peers { permanent_uuid: "c561ec3374b94b95be0c155d80ff0bff" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 33627 } attrs { replace: true } } peers { permanent_uuid: "3710b74ed2044ce5995533df32e67351" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 38589 } } peers { permanent_uuid: "25286c485fa04c86a5d07c8f88643a47" member_type: VOTER last_known_addr { host: "127.17.60.198" port: 37089 } attrs { promote: false } } }
I20250112 15:07:47.121928 21147 raft_consensus.cc:2949] T ef90b704208140a3983ffe6105939319 P c561ec3374b94b95be0c155d80ff0bff [term 4 FOLLOWER]: Committing config change with OpId 4.30: config changed from index 29 to 30, 25286c485fa04c86a5d07c8f88643a47 (127.17.60.198) changed from NON_VOTER to VOTER. New config: { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "2537bbbd1a9947ae9facd87567e3ccb9" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 33173 } } peers { permanent_uuid: "c561ec3374b94b95be0c155d80ff0bff" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 33627 } attrs { replace: true } } peers { permanent_uuid: "3710b74ed2044ce5995533df32e67351" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 38589 } } peers { permanent_uuid: "25286c485fa04c86a5d07c8f88643a47" member_type: VOTER last_known_addr { host: "127.17.60.198" port: 37089 } attrs { promote: false } } }
I20250112 15:07:47.122817 20964 catalog_manager.cc:5526] T ef90b704208140a3983ffe6105939319 P 25286c485fa04c86a5d07c8f88643a47 reported cstate change: config changed from index 29 to 30, 25286c485fa04c86a5d07c8f88643a47 (127.17.60.198) changed from NON_VOTER to VOTER. New cstate: current_term: 4 leader_uuid: "2537bbbd1a9947ae9facd87567e3ccb9" committed_config { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "2537bbbd1a9947ae9facd87567e3ccb9" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 33173 } } peers { permanent_uuid: "c561ec3374b94b95be0c155d80ff0bff" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 33627 } attrs { replace: true } } peers { permanent_uuid: "3710b74ed2044ce5995533df32e67351" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 38589 } } peers { permanent_uuid: "25286c485fa04c86a5d07c8f88643a47" member_type: VOTER last_known_addr { host: "127.17.60.198" port: 37089 } attrs { promote: false } } }
I20250112 15:07:47.123878 21068 consensus_queue.cc:237] T ef90b704208140a3983ffe6105939319 P 2537bbbd1a9947ae9facd87567e3ccb9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 30, Committed index: 30, Last appended: 4.30, Last appended by leader: 27, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "2537bbbd1a9947ae9facd87567e3ccb9" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 33173 } } peers { permanent_uuid: "3710b74ed2044ce5995533df32e67351" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 38589 } } peers { permanent_uuid: "25286c485fa04c86a5d07c8f88643a47" member_type: VOTER last_known_addr { host: "127.17.60.198" port: 37089 } attrs { promote: false } }
I20250112 15:07:47.124177 21624 raft_consensus.cc:1270] T ef90b704208140a3983ffe6105939319 P 25286c485fa04c86a5d07c8f88643a47 [term 4 FOLLOWER]: Refusing update from remote peer 2537bbbd1a9947ae9facd87567e3ccb9: Log matching property violated. Preceding OpId in replica: term: 4 index: 30. Preceding OpId from leader: term: 4 index: 31. (index mismatch)
I20250112 15:07:47.124181 21226 raft_consensus.cc:1270] T ef90b704208140a3983ffe6105939319 P 3710b74ed2044ce5995533df32e67351 [term 4 FOLLOWER]: Refusing update from remote peer 2537bbbd1a9947ae9facd87567e3ccb9: Log matching property violated. Preceding OpId in replica: term: 4 index: 30. Preceding OpId from leader: term: 4 index: 31. (index mismatch)
I20250112 15:07:47.124325 22135 consensus_queue.cc:1035] T ef90b704208140a3983ffe6105939319 P 2537bbbd1a9947ae9facd87567e3ccb9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "25286c485fa04c86a5d07c8f88643a47" member_type: VOTER last_known_addr { host: "127.17.60.198" port: 37089 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 31, Last known committed idx: 30, Time since last communication: 0.000s
I20250112 15:07:47.124428 22138 consensus_queue.cc:1035] T ef90b704208140a3983ffe6105939319 P 2537bbbd1a9947ae9facd87567e3ccb9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3710b74ed2044ce5995533df32e67351" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 38589 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 31, Last known committed idx: 30, Time since last communication: 0.000s
I20250112 15:07:47.125197 21226 raft_consensus.cc:2949] T ef90b704208140a3983ffe6105939319 P 3710b74ed2044ce5995533df32e67351 [term 4 FOLLOWER]: Committing config change with OpId 4.31: config changed from index 30 to 31, VOTER c561ec3374b94b95be0c155d80ff0bff (127.17.60.194) evicted. New config: { opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "2537bbbd1a9947ae9facd87567e3ccb9" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 33173 } } peers { permanent_uuid: "3710b74ed2044ce5995533df32e67351" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 38589 } } peers { permanent_uuid: "25286c485fa04c86a5d07c8f88643a47" member_type: VOTER last_known_addr { host: "127.17.60.198" port: 37089 } attrs { promote: false } } }
I20250112 15:07:47.125221 21624 raft_consensus.cc:2949] T ef90b704208140a3983ffe6105939319 P 25286c485fa04c86a5d07c8f88643a47 [term 4 FOLLOWER]: Committing config change with OpId 4.31: config changed from index 30 to 31, VOTER c561ec3374b94b95be0c155d80ff0bff (127.17.60.194) evicted. New config: { opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "2537bbbd1a9947ae9facd87567e3ccb9" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 33173 } } peers { permanent_uuid: "3710b74ed2044ce5995533df32e67351" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 38589 } } peers { permanent_uuid: "25286c485fa04c86a5d07c8f88643a47" member_type: VOTER last_known_addr { host: "127.17.60.198" port: 37089 } attrs { promote: false } } }
I20250112 15:07:47.126062 20966 catalog_manager.cc:5526] T ef90b704208140a3983ffe6105939319 P 3710b74ed2044ce5995533df32e67351 reported cstate change: config changed from index 30 to 31, VOTER c561ec3374b94b95be0c155d80ff0bff (127.17.60.194) evicted. New cstate: current_term: 4 leader_uuid: "2537bbbd1a9947ae9facd87567e3ccb9" committed_config { opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "2537bbbd1a9947ae9facd87567e3ccb9" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 33173 } } peers { permanent_uuid: "3710b74ed2044ce5995533df32e67351" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 38589 } } peers { permanent_uuid: "25286c485fa04c86a5d07c8f88643a47" member_type: VOTER last_known_addr { host: "127.17.60.198" port: 37089 } attrs { promote: false } } }
I20250112 15:07:47.126294 22137 raft_consensus.cc:2949] T ef90b704208140a3983ffe6105939319 P 2537bbbd1a9947ae9facd87567e3ccb9 [term 4 LEADER]: Committing config change with OpId 4.31: config changed from index 30 to 31, VOTER c561ec3374b94b95be0c155d80ff0bff (127.17.60.194) evicted. New config: { opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "2537bbbd1a9947ae9facd87567e3ccb9" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 33173 } } peers { permanent_uuid: "3710b74ed2044ce5995533df32e67351" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 38589 } } peers { permanent_uuid: "25286c485fa04c86a5d07c8f88643a47" member_type: VOTER last_known_addr { host: "127.17.60.198" port: 37089 } attrs { promote: false } } }
I20250112 15:07:47.126721 21141 tablet_service.cc:1514] Processing DeleteTablet for tablet ef90b704208140a3983ffe6105939319 with delete_type TABLET_DATA_TOMBSTONED (TS c561ec3374b94b95be0c155d80ff0bff not found in new config with opid_index 31) from {username='slave'} at 127.0.0.1:48072
I20250112 15:07:47.126837 22134 tablet_replica.cc:331] T ef90b704208140a3983ffe6105939319 P c561ec3374b94b95be0c155d80ff0bff: stopping tablet replica
I20250112 15:07:47.126973 22134 raft_consensus.cc:2238] T ef90b704208140a3983ffe6105939319 P c561ec3374b94b95be0c155d80ff0bff [term 4 FOLLOWER]: Raft consensus shutting down.
I20250112 15:07:47.127063 22134 raft_consensus.cc:2267] T ef90b704208140a3983ffe6105939319 P c561ec3374b94b95be0c155d80ff0bff [term 4 FOLLOWER]: Raft consensus is shut down!
I20250112 15:07:47.127434 22134 ts_tablet_manager.cc:1905] T ef90b704208140a3983ffe6105939319 P c561ec3374b94b95be0c155d80ff0bff: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250112 15:07:47.128360 20956 catalog_manager.cc:5039] ChangeConfig:REMOVE_PEER RPC for tablet ef90b704208140a3983ffe6105939319 with cas_config_opid_index 30: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250112 15:07:47.128587 22134 ts_tablet_manager.cc:1918] T ef90b704208140a3983ffe6105939319 P c561ec3374b94b95be0c155d80ff0bff: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 4.30
I20250112 15:07:47.128639 22134 log.cc:1198] T ef90b704208140a3983ffe6105939319 P c561ec3374b94b95be0c155d80ff0bff: Deleting WAL directory at /tmp/dist-test-task95dLSD/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1736694351070047-17651-0/minicluster-data/ts-1-root/wals/ef90b704208140a3983ffe6105939319
I20250112 15:07:47.128968 20958 catalog_manager.cc:4872] TS c561ec3374b94b95be0c155d80ff0bff (127.17.60.194:33627): tablet ef90b704208140a3983ffe6105939319 (table test-workload [id=cdaf6d3a65a54ea08e8e2c0ecdfc9de7]) successfully deleted
I20250112 15:07:47.183817 22138 raft_consensus.cc:1059] T 16f16fcd5cf347e5853189dc4da55c7f P 2537bbbd1a9947ae9facd87567e3ccb9: attempting to promote NON_VOTER b5e9178a7a83402e8c4a6943ab78589d to VOTER
I20250112 15:07:47.184062 22138 consensus_queue.cc:237] T 16f16fcd5cf347e5853189dc4da55c7f P 2537bbbd1a9947ae9facd87567e3ccb9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 30, Committed index: 30, Last appended: 2.30, Last appended by leader: 25, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "2537bbbd1a9947ae9facd87567e3ccb9" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 33173 } } peers { permanent_uuid: "3710b74ed2044ce5995533df32e67351" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 38589 } attrs { replace: true } } peers { permanent_uuid: "25286c485fa04c86a5d07c8f88643a47" member_type: VOTER last_known_addr { host: "127.17.60.198" port: 37089 } attrs { promote: false } } peers { permanent_uuid: "b5e9178a7a83402e8c4a6943ab78589d" member_type: VOTER last_known_addr { host: "127.17.60.197" port: 36573 } attrs { promote: false } }
I20250112 15:07:47.184617 21226 raft_consensus.cc:1270] T 16f16fcd5cf347e5853189dc4da55c7f P 3710b74ed2044ce5995533df32e67351 [term 2 FOLLOWER]: Refusing update from remote peer 2537bbbd1a9947ae9facd87567e3ccb9: Log matching property violated. Preceding OpId in replica: term: 2 index: 30. Preceding OpId from leader: term: 2 index: 31. (index mismatch)
I20250112 15:07:47.184633 21624 raft_consensus.cc:1270] T 16f16fcd5cf347e5853189dc4da55c7f P 25286c485fa04c86a5d07c8f88643a47 [term 2 FOLLOWER]: Refusing update from remote peer 2537bbbd1a9947ae9facd87567e3ccb9: Log matching property violated. Preceding OpId in replica: term: 2 index: 30. Preceding OpId from leader: term: 2 index: 31. (index mismatch)
I20250112 15:07:47.184836 21549 raft_consensus.cc:1270] T 16f16fcd5cf347e5853189dc4da55c7f P b5e9178a7a83402e8c4a6943ab78589d [term 2 LEARNER]: Refusing update from remote peer 2537bbbd1a9947ae9facd87567e3ccb9: Log matching property violated. Preceding OpId in replica: term: 2 index: 30. Preceding OpId from leader: term: 2 index: 31. (index mismatch)
I20250112 15:07:47.184815 21969 consensus_queue.cc:1035] T 16f16fcd5cf347e5853189dc4da55c7f P 2537bbbd1a9947ae9facd87567e3ccb9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "25286c485fa04c86a5d07c8f88643a47" member_type: VOTER last_known_addr { host: "127.17.60.198" port: 37089 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 31, Last known committed idx: 30, Time since last communication: 0.000s
I20250112 15:07:47.185089 22137 consensus_queue.cc:1035] T 16f16fcd5cf347e5853189dc4da55c7f P 2537bbbd1a9947ae9facd87567e3ccb9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b5e9178a7a83402e8c4a6943ab78589d" member_type: VOTER last_known_addr { host: "127.17.60.197" port: 36573 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 31, Last known committed idx: 30, Time since last communication: 0.000s
I20250112 15:07:47.185223 22137 consensus_queue.cc:1035] T 16f16fcd5cf347e5853189dc4da55c7f P 2537bbbd1a9947ae9facd87567e3ccb9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3710b74ed2044ce5995533df32e67351" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 38589 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 31, Last known committed idx: 30, Time since last communication: 0.000s
I20250112 15:07:47.185786 21969 raft_consensus.cc:2949] T 16f16fcd5cf347e5853189dc4da55c7f P 2537bbbd1a9947ae9facd87567e3ccb9 [term 2 LEADER]: Committing config change with OpId 2.31: config changed from index 30 to 31, b5e9178a7a83402e8c4a6943ab78589d (127.17.60.197) changed from NON_VOTER to VOTER. New config: { opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "2537bbbd1a9947ae9facd87567e3ccb9" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 33173 } } peers { permanent_uuid: "3710b74ed2044ce5995533df32e67351" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 38589 } attrs { replace: true } } peers { permanent_uuid: "25286c485fa04c86a5d07c8f88643a47" member_type: VOTER last_known_addr { host: "127.17.60.198" port: 37089 } attrs { promote: false } } peers { permanent_uuid: "b5e9178a7a83402e8c4a6943ab78589d" member_type: VOTER last_known_addr { host: "127.17.60.197" port: 36573 } attrs { promote: false } } }
I20250112 15:07:47.185907 21549 raft_consensus.cc:2949] T 16f16fcd5cf347e5853189dc4da55c7f P b5e9178a7a83402e8c4a6943ab78589d [term 2 FOLLOWER]: Committing config change with OpId 2.31: config changed from index 30 to 31, b5e9178a7a83402e8c4a6943ab78589d (127.17.60.197) changed from NON_VOTER to VOTER. New config: { opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "2537bbbd1a9947ae9facd87567e3ccb9" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 33173 } } peers { permanent_uuid: "3710b74ed2044ce5995533df32e67351" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 38589 } attrs { replace: true } } peers { permanent_uuid: "25286c485fa04c86a5d07c8f88643a47" member_type: VOTER last_known_addr { host: "127.17.60.198" port: 37089 } attrs { promote: false } } peers { permanent_uuid: "b5e9178a7a83402e8c4a6943ab78589d" member_type: VOTER last_known_addr { host: "127.17.60.197" port: 36573 } attrs { promote: false } } }
I20250112 15:07:47.186887 21624 raft_consensus.cc:2949] T 16f16fcd5cf347e5853189dc4da55c7f P 25286c485fa04c86a5d07c8f88643a47 [term 2 FOLLOWER]: Committing config change with OpId 2.31: config changed from index 30 to 31, b5e9178a7a83402e8c4a6943ab78589d (127.17.60.197) changed from NON_VOTER to VOTER. New config: { opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "2537bbbd1a9947ae9facd87567e3ccb9" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 33173 } } peers { permanent_uuid: "3710b74ed2044ce5995533df32e67351" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 38589 } attrs { replace: true } } peers { permanent_uuid: "25286c485fa04c86a5d07c8f88643a47" member_type: VOTER last_known_addr { host: "127.17.60.198" port: 37089 } attrs { promote: false } } peers { permanent_uuid: "b5e9178a7a83402e8c4a6943ab78589d" member_type: VOTER last_known_addr { host: "127.17.60.197" port: 36573 } attrs { promote: false } } }
I20250112 15:07:47.187117 20966 catalog_manager.cc:5526] T 16f16fcd5cf347e5853189dc4da55c7f P b5e9178a7a83402e8c4a6943ab78589d reported cstate change: config changed from index 30 to 31, b5e9178a7a83402e8c4a6943ab78589d (127.17.60.197) changed from NON_VOTER to VOTER. New cstate: current_term: 2 leader_uuid: "2537bbbd1a9947ae9facd87567e3ccb9" committed_config { opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "2537bbbd1a9947ae9facd87567e3ccb9" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 33173 } } peers { permanent_uuid: "3710b74ed2044ce5995533df32e67351" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 38589 } attrs { replace: true } } peers { permanent_uuid: "25286c485fa04c86a5d07c8f88643a47" member_type: VOTER last_known_addr { host: "127.17.60.198" port: 37089 } attrs { promote: false } } peers { permanent_uuid: "b5e9178a7a83402e8c4a6943ab78589d" member_type: VOTER last_known_addr { host: "127.17.60.197" port: 36573 } attrs { promote: false } } }
I20250112 15:07:47.187670 21226 raft_consensus.cc:2949] T 16f16fcd5cf347e5853189dc4da55c7f P 3710b74ed2044ce5995533df32e67351 [term 2 FOLLOWER]: Committing config change with OpId 2.31: config changed from index 30 to 31, b5e9178a7a83402e8c4a6943ab78589d (127.17.60.197) changed from NON_VOTER to VOTER. New config: { opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "2537bbbd1a9947ae9facd87567e3ccb9" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 33173 } } peers { permanent_uuid: "3710b74ed2044ce5995533df32e67351" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 38589 } attrs { replace: true } } peers { permanent_uuid: "25286c485fa04c86a5d07c8f88643a47" member_type: VOTER last_known_addr { host: "127.17.60.198" port: 37089 } attrs { promote: false } } peers { permanent_uuid: "b5e9178a7a83402e8c4a6943ab78589d" member_type: VOTER last_known_addr { host: "127.17.60.197" port: 36573 } attrs { promote: false } } }
I20250112 15:07:47.188652 21068 consensus_queue.cc:237] T 16f16fcd5cf347e5853189dc4da55c7f P 2537bbbd1a9947ae9facd87567e3ccb9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 31, Committed index: 31, Last appended: 2.31, Last appended by leader: 25, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "2537bbbd1a9947ae9facd87567e3ccb9" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 33173 } } peers { permanent_uuid: "25286c485fa04c86a5d07c8f88643a47" member_type: VOTER last_known_addr { host: "127.17.60.198" port: 37089 } attrs { promote: false } } peers { permanent_uuid: "b5e9178a7a83402e8c4a6943ab78589d" member_type: VOTER last_known_addr { host: "127.17.60.197" port: 36573 } attrs { promote: false } }
I20250112 15:07:47.189028 21624 raft_consensus.cc:1270] T 16f16fcd5cf347e5853189dc4da55c7f P 25286c485fa04c86a5d07c8f88643a47 [term 2 FOLLOWER]: Refusing update from remote peer 2537bbbd1a9947ae9facd87567e3ccb9: Log matching property violated. Preceding OpId in replica: term: 2 index: 31. Preceding OpId from leader: term: 2 index: 32. (index mismatch)
I20250112 15:07:47.189036 21549 raft_consensus.cc:1270] T 16f16fcd5cf347e5853189dc4da55c7f P b5e9178a7a83402e8c4a6943ab78589d [term 2 FOLLOWER]: Refusing update from remote peer 2537bbbd1a9947ae9facd87567e3ccb9: Log matching property violated. Preceding OpId in replica: term: 2 index: 31. Preceding OpId from leader: term: 2 index: 32. (index mismatch)
I20250112 15:07:47.189236 22135 consensus_queue.cc:1035] T 16f16fcd5cf347e5853189dc4da55c7f P 2537bbbd1a9947ae9facd87567e3ccb9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "25286c485fa04c86a5d07c8f88643a47" member_type: VOTER last_known_addr { host: "127.17.60.198" port: 37089 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 32, Last known committed idx: 31, Time since last communication: 0.000s
I20250112 15:07:47.189356 22138 consensus_queue.cc:1035] T 16f16fcd5cf347e5853189dc4da55c7f P 2537bbbd1a9947ae9facd87567e3ccb9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b5e9178a7a83402e8c4a6943ab78589d" member_type: VOTER last_known_addr { host: "127.17.60.197" port: 36573 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 32, Last known committed idx: 31, Time since last communication: 0.000s
I20250112 15:07:47.189810 22138 raft_consensus.cc:2949] T 16f16fcd5cf347e5853189dc4da55c7f P 2537bbbd1a9947ae9facd87567e3ccb9 [term 2 LEADER]: Committing config change with OpId 2.32: config changed from index 31 to 32, VOTER 3710b74ed2044ce5995533df32e67351 (127.17.60.195) evicted. New config: { opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "2537bbbd1a9947ae9facd87567e3ccb9" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 33173 } } peers { permanent_uuid: "25286c485fa04c86a5d07c8f88643a47" member_type: VOTER last_known_addr { host: "127.17.60.198" port: 37089 } attrs { promote: false } } peers { permanent_uuid: "b5e9178a7a83402e8c4a6943ab78589d" member_type: VOTER last_known_addr { host: "127.17.60.197" port: 36573 } attrs { promote: false } } }
I20250112 15:07:47.189882 21624 raft_consensus.cc:2949] T 16f16fcd5cf347e5853189dc4da55c7f P 25286c485fa04c86a5d07c8f88643a47 [term 2 FOLLOWER]: Committing config change with OpId 2.32: config changed from index 31 to 32, VOTER 3710b74ed2044ce5995533df32e67351 (127.17.60.195) evicted. New config: { opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "2537bbbd1a9947ae9facd87567e3ccb9" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 33173 } } peers { permanent_uuid: "25286c485fa04c86a5d07c8f88643a47" member_type: VOTER last_known_addr { host: "127.17.60.198" port: 37089 } attrs { promote: false } } peers { permanent_uuid: "b5e9178a7a83402e8c4a6943ab78589d" member_type: VOTER last_known_addr { host: "127.17.60.197" port: 36573 } attrs { promote: false } } }
I20250112 15:07:47.190008 21549 raft_consensus.cc:2949] T 16f16fcd5cf347e5853189dc4da55c7f P b5e9178a7a83402e8c4a6943ab78589d [term 2 FOLLOWER]: Committing config change with OpId 2.32: config changed from index 31 to 32, VOTER 3710b74ed2044ce5995533df32e67351 (127.17.60.195) evicted. New config: { opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "2537bbbd1a9947ae9facd87567e3ccb9" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 33173 } } peers { permanent_uuid: "25286c485fa04c86a5d07c8f88643a47" member_type: VOTER last_known_addr { host: "127.17.60.198" port: 37089 } attrs { promote: false } } peers { permanent_uuid: "b5e9178a7a83402e8c4a6943ab78589d" member_type: VOTER last_known_addr { host: "127.17.60.197" port: 36573 } attrs { promote: false } } }
I20250112 15:07:47.190718 20970 catalog_manager.cc:5526] T 16f16fcd5cf347e5853189dc4da55c7f P 25286c485fa04c86a5d07c8f88643a47 reported cstate change: config changed from index 31 to 32, VOTER 3710b74ed2044ce5995533df32e67351 (127.17.60.195) evicted. New cstate: current_term: 2 leader_uuid: "2537bbbd1a9947ae9facd87567e3ccb9" committed_config { opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "2537bbbd1a9947ae9facd87567e3ccb9" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 33173 } } peers { permanent_uuid: "25286c485fa04c86a5d07c8f88643a47" member_type: VOTER last_known_addr { host: "127.17.60.198" port: 37089 } attrs { promote: false } } peers { permanent_uuid: "b5e9178a7a83402e8c4a6943ab78589d" member_type: VOTER last_known_addr { host: "127.17.60.197" port: 36573 } attrs { promote: false } } }
I20250112 15:07:47.191128 20956 catalog_manager.cc:5039] ChangeConfig:REMOVE_PEER RPC for tablet 16f16fcd5cf347e5853189dc4da55c7f with cas_config_opid_index 31: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250112 15:07:47.191444 21212 tablet_service.cc:1514] Processing DeleteTablet for tablet 16f16fcd5cf347e5853189dc4da55c7f with delete_type TABLET_DATA_TOMBSTONED (TS 3710b74ed2044ce5995533df32e67351 not found in new config with opid_index 32) from {username='slave'} at 127.0.0.1:33192
I20250112 15:07:47.191663 22146 tablet_replica.cc:331] T 16f16fcd5cf347e5853189dc4da55c7f P 3710b74ed2044ce5995533df32e67351: stopping tablet replica
I20250112 15:07:47.191733 22146 raft_consensus.cc:2238] T 16f16fcd5cf347e5853189dc4da55c7f P 3710b74ed2044ce5995533df32e67351 [term 2 FOLLOWER]: Raft consensus shutting down.
I20250112 15:07:47.191821 22146 raft_consensus.cc:2267] T 16f16fcd5cf347e5853189dc4da55c7f P 3710b74ed2044ce5995533df32e67351 [term 2 FOLLOWER]: Raft consensus is shut down!
I20250112 15:07:47.192047 22146 ts_tablet_manager.cc:1905] T 16f16fcd5cf347e5853189dc4da55c7f P 3710b74ed2044ce5995533df32e67351: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250112 15:07:47.193166 22146 ts_tablet_manager.cc:1918] T 16f16fcd5cf347e5853189dc4da55c7f P 3710b74ed2044ce5995533df32e67351: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.31
I20250112 15:07:47.193215 22146 log.cc:1198] T 16f16fcd5cf347e5853189dc4da55c7f P 3710b74ed2044ce5995533df32e67351: Deleting WAL directory at /tmp/dist-test-task95dLSD/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1736694351070047-17651-0/minicluster-data/ts-2-root/wals/16f16fcd5cf347e5853189dc4da55c7f
I20250112 15:07:47.193517 20957 catalog_manager.cc:4872] TS 3710b74ed2044ce5995533df32e67351 (127.17.60.195:38589): tablet 16f16fcd5cf347e5853189dc4da55c7f (table test-workload [id=cdaf6d3a65a54ea08e8e2c0ecdfc9de7]) successfully deleted
I20250112 15:07:47.343214 22108 raft_consensus.cc:1059] T bfaeae584ec04772b2d6fcd088022117 P c561ec3374b94b95be0c155d80ff0bff: attempting to promote NON_VOTER a2a5bba4ac1145d5bab94a69c9f4df40 to VOTER
I20250112 15:07:47.343447 22108 consensus_queue.cc:237] T bfaeae584ec04772b2d6fcd088022117 P c561ec3374b94b95be0c155d80ff0bff [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 31, Committed index: 31, Last appended: 3.31, Last appended by leader: 27, Current term: 3, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "2537bbbd1a9947ae9facd87567e3ccb9" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 33173 } attrs { replace: true } } peers { permanent_uuid: "c561ec3374b94b95be0c155d80ff0bff" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 33627 } } peers { permanent_uuid: "25286c485fa04c86a5d07c8f88643a47" member_type: VOTER last_known_addr { host: "127.17.60.198" port: 37089 } attrs { promote: false } } peers { permanent_uuid: "a2a5bba4ac1145d5bab94a69c9f4df40" member_type: VOTER last_known_addr { host: "127.17.60.196" port: 41807 } attrs { promote: false } }
I20250112 15:07:47.343987 21474 raft_consensus.cc:1270] T bfaeae584ec04772b2d6fcd088022117 P a2a5bba4ac1145d5bab94a69c9f4df40 [term 3 LEARNER]: Refusing update from remote peer c561ec3374b94b95be0c155d80ff0bff: Log matching property violated. Preceding OpId in replica: term: 3 index: 31. Preceding OpId from leader: term: 3 index: 32. (index mismatch)
I20250112 15:07:47.343987 21069 raft_consensus.cc:1270] T bfaeae584ec04772b2d6fcd088022117 P 2537bbbd1a9947ae9facd87567e3ccb9 [term 3 FOLLOWER]: Refusing update from remote peer c561ec3374b94b95be0c155d80ff0bff: Log matching property violated. Preceding OpId in replica: term: 3 index: 31. Preceding OpId from leader: term: 3 index: 32. (index mismatch)
I20250112 15:07:47.344038 21624 raft_consensus.cc:1270] T bfaeae584ec04772b2d6fcd088022117 P 25286c485fa04c86a5d07c8f88643a47 [term 3 FOLLOWER]: Refusing update from remote peer c561ec3374b94b95be0c155d80ff0bff: Log matching property violated. Preceding OpId in replica: term: 3 index: 31. Preceding OpId from leader: term: 3 index: 32. (index mismatch)
I20250112 15:07:47.344277 22149 consensus_queue.cc:1035] T bfaeae584ec04772b2d6fcd088022117 P c561ec3374b94b95be0c155d80ff0bff [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2a5bba4ac1145d5bab94a69c9f4df40" member_type: VOTER last_known_addr { host: "127.17.60.196" port: 41807 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 32, Last known committed idx: 31, Time since last communication: 0.000s
I20250112 15:07:47.344449 22148 consensus_queue.cc:1035] T bfaeae584ec04772b2d6fcd088022117 P c561ec3374b94b95be0c155d80ff0bff [LEADER]: Connected to new peer: Peer: permanent_uuid: "25286c485fa04c86a5d07c8f88643a47" member_type: VOTER last_known_addr { host: "127.17.60.198" port: 37089 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 32, Last known committed idx: 31, Time since last communication: 0.000s
I20250112 15:07:47.344669 22108 consensus_queue.cc:1035] T bfaeae584ec04772b2d6fcd088022117 P c561ec3374b94b95be0c155d80ff0bff [LEADER]: Connected to new peer: Peer: permanent_uuid: "2537bbbd1a9947ae9facd87567e3ccb9" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 33173 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 32, Last known committed idx: 31, Time since last communication: 0.000s
I20250112 15:07:47.345329 22149 raft_consensus.cc:2949] T bfaeae584ec04772b2d6fcd088022117 P c561ec3374b94b95be0c155d80ff0bff [term 3 LEADER]: Committing config change with OpId 3.32: config changed from index 31 to 32, a2a5bba4ac1145d5bab94a69c9f4df40 (127.17.60.196) changed from NON_VOTER to VOTER. New config: { opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "2537bbbd1a9947ae9facd87567e3ccb9" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 33173 } attrs { replace: true } } peers { permanent_uuid: "c561ec3374b94b95be0c155d80ff0bff" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 33627 } } peers { permanent_uuid: "25286c485fa04c86a5d07c8f88643a47" member_type: VOTER last_known_addr { host: "127.17.60.198" port: 37089 } attrs { promote: false } } peers { permanent_uuid: "a2a5bba4ac1145d5bab94a69c9f4df40" member_type: VOTER last_known_addr { host: "127.17.60.196" port: 41807 } attrs { promote: false } } }
I20250112 15:07:47.345427 21474 raft_consensus.cc:2949] T bfaeae584ec04772b2d6fcd088022117 P a2a5bba4ac1145d5bab94a69c9f4df40 [term 3 FOLLOWER]: Committing config change with OpId 3.32: config changed from index 31 to 32, a2a5bba4ac1145d5bab94a69c9f4df40 (127.17.60.196) changed from NON_VOTER to VOTER. New config: { opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "2537bbbd1a9947ae9facd87567e3ccb9" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 33173 } attrs { replace: true } } peers { permanent_uuid: "c561ec3374b94b95be0c155d80ff0bff" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 33627 } } peers { permanent_uuid: "25286c485fa04c86a5d07c8f88643a47" member_type: VOTER last_known_addr { host: "127.17.60.198" port: 37089 } attrs { promote: false } } peers { permanent_uuid: "a2a5bba4ac1145d5bab94a69c9f4df40" member_type: VOTER last_known_addr { host: "127.17.60.196" port: 41807 } attrs { promote: false } } }
I20250112 15:07:47.346426 20966 catalog_manager.cc:5526] T bfaeae584ec04772b2d6fcd088022117 P a2a5bba4ac1145d5bab94a69c9f4df40 reported cstate change: config changed from index 31 to 32, a2a5bba4ac1145d5bab94a69c9f4df40 (127.17.60.196) changed from NON_VOTER to VOTER. New cstate: current_term: 3 leader_uuid: "c561ec3374b94b95be0c155d80ff0bff" committed_config { opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "2537bbbd1a9947ae9facd87567e3ccb9" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 33173 } attrs { replace: true } } peers { permanent_uuid: "c561ec3374b94b95be0c155d80ff0bff" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 33627 } } peers { permanent_uuid: "25286c485fa04c86a5d07c8f88643a47" member_type: VOTER last_known_addr { host: "127.17.60.198" port: 37089 } attrs { promote: false } } peers { permanent_uuid: "a2a5bba4ac1145d5bab94a69c9f4df40" member_type: VOTER last_known_addr { host: "127.17.60.196" port: 41807 } attrs { promote: false } } }
I20250112 15:07:47.346741 21069 raft_consensus.cc:2949] T bfaeae584ec04772b2d6fcd088022117 P 2537bbbd1a9947ae9facd87567e3ccb9 [term 3 FOLLOWER]: Committing config change with OpId 3.32: config changed from index 31 to 32, a2a5bba4ac1145d5bab94a69c9f4df40 (127.17.60.196) changed from NON_VOTER to VOTER. New config: { opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "2537bbbd1a9947ae9facd87567e3ccb9" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 33173 } attrs { replace: true } } peers { permanent_uuid: "c561ec3374b94b95be0c155d80ff0bff" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 33627 } } peers { permanent_uuid: "25286c485fa04c86a5d07c8f88643a47" member_type: VOTER last_known_addr { host: "127.17.60.198" port: 37089 } attrs { promote: false } } peers { permanent_uuid: "a2a5bba4ac1145d5bab94a69c9f4df40" member_type: VOTER last_known_addr { host: "127.17.60.196" port: 41807 } attrs { promote: false } } }
I20250112 15:07:47.346742 21624 raft_consensus.cc:2949] T bfaeae584ec04772b2d6fcd088022117 P 25286c485fa04c86a5d07c8f88643a47 [term 3 FOLLOWER]: Committing config change with OpId 3.32: config changed from index 31 to 32, a2a5bba4ac1145d5bab94a69c9f4df40 (127.17.60.196) changed from NON_VOTER to VOTER. New config: { opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "2537bbbd1a9947ae9facd87567e3ccb9" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 33173 } attrs { replace: true } } peers { permanent_uuid: "c561ec3374b94b95be0c155d80ff0bff" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 33627 } } peers { permanent_uuid: "25286c485fa04c86a5d07c8f88643a47" member_type: VOTER last_known_addr { host: "127.17.60.198" port: 37089 } attrs { promote: false } } peers { permanent_uuid: "a2a5bba4ac1145d5bab94a69c9f4df40" member_type: VOTER last_known_addr { host: "127.17.60.196" port: 41807 } attrs { promote: false } } }
I20250112 15:07:47.348265 21147 consensus_queue.cc:237] T bfaeae584ec04772b2d6fcd088022117 P c561ec3374b94b95be0c155d80ff0bff [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 32, Committed index: 32, Last appended: 3.32, Last appended by leader: 27, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 33 OBSOLETE_local: false peers { permanent_uuid: "c561ec3374b94b95be0c155d80ff0bff" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 33627 } } peers { permanent_uuid: "25286c485fa04c86a5d07c8f88643a47" member_type: VOTER last_known_addr { host: "127.17.60.198" port: 37089 } attrs { promote: false } } peers { permanent_uuid: "a2a5bba4ac1145d5bab94a69c9f4df40" member_type: VOTER last_known_addr { host: "127.17.60.196" port: 41807 } attrs { promote: false } }
I20250112 15:07:47.348604 21474 raft_consensus.cc:1270] T bfaeae584ec04772b2d6fcd088022117 P a2a5bba4ac1145d5bab94a69c9f4df40 [term 3 FOLLOWER]: Refusing update from remote peer c561ec3374b94b95be0c155d80ff0bff: Log matching property violated. Preceding OpId in replica: term: 3 index: 32. Preceding OpId from leader: term: 3 index: 33. (index mismatch)
I20250112 15:07:47.348645 21624 raft_consensus.cc:1270] T bfaeae584ec04772b2d6fcd088022117 P 25286c485fa04c86a5d07c8f88643a47 [term 3 FOLLOWER]: Refusing update from remote peer c561ec3374b94b95be0c155d80ff0bff: Log matching property violated. Preceding OpId in replica: term: 3 index: 32. Preceding OpId from leader: term: 3 index: 33. (index mismatch)
I20250112 15:07:47.348791 22148 consensus_queue.cc:1035] T bfaeae584ec04772b2d6fcd088022117 P c561ec3374b94b95be0c155d80ff0bff [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2a5bba4ac1145d5bab94a69c9f4df40" member_type: VOTER last_known_addr { host: "127.17.60.196" port: 41807 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 33, Last known committed idx: 32, Time since last communication: 0.000s
I20250112 15:07:47.348870 22108 consensus_queue.cc:1035] T bfaeae584ec04772b2d6fcd088022117 P c561ec3374b94b95be0c155d80ff0bff [LEADER]: Connected to new peer: Peer: permanent_uuid: "25286c485fa04c86a5d07c8f88643a47" member_type: VOTER last_known_addr { host: "127.17.60.198" port: 37089 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 33, Last known committed idx: 32, Time since last communication: 0.000s
I20250112 15:07:47.349386 22108 raft_consensus.cc:2949] T bfaeae584ec04772b2d6fcd088022117 P c561ec3374b94b95be0c155d80ff0bff [term 3 LEADER]: Committing config change with OpId 3.33: config changed from index 32 to 33, VOTER 2537bbbd1a9947ae9facd87567e3ccb9 (127.17.60.193) evicted. New config: { opid_index: 33 OBSOLETE_local: false peers { permanent_uuid: "c561ec3374b94b95be0c155d80ff0bff" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 33627 } } peers { permanent_uuid: "25286c485fa04c86a5d07c8f88643a47" member_type: VOTER last_known_addr { host: "127.17.60.198" port: 37089 } attrs { promote: false } } peers { permanent_uuid: "a2a5bba4ac1145d5bab94a69c9f4df40" member_type: VOTER last_known_addr { host: "127.17.60.196" port: 41807 } attrs { promote: false } } }
I20250112 15:07:47.350196 20958 catalog_manager.cc:5039] ChangeConfig:REMOVE_PEER RPC for tablet bfaeae584ec04772b2d6fcd088022117 with cas_config_opid_index 32: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250112 15:07:47.350353 21624 raft_consensus.cc:2949] T bfaeae584ec04772b2d6fcd088022117 P 25286c485fa04c86a5d07c8f88643a47 [term 3 FOLLOWER]: Committing config change with OpId 3.33: config changed from index 32 to 33, VOTER 2537bbbd1a9947ae9facd87567e3ccb9 (127.17.60.193) evicted. New config: { opid_index: 33 OBSOLETE_local: false peers { permanent_uuid: "c561ec3374b94b95be0c155d80ff0bff" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 33627 } } peers { permanent_uuid: "25286c485fa04c86a5d07c8f88643a47" member_type: VOTER last_known_addr { host: "127.17.60.198" port: 37089 } attrs { promote: false } } peers { permanent_uuid: "a2a5bba4ac1145d5bab94a69c9f4df40" member_type: VOTER last_known_addr { host: "127.17.60.196" port: 41807 } attrs { promote: false } } }
I20250112 15:07:47.350409 20966 catalog_manager.cc:5526] T bfaeae584ec04772b2d6fcd088022117 P c561ec3374b94b95be0c155d80ff0bff reported cstate change: config changed from index 32 to 33, VOTER 2537bbbd1a9947ae9facd87567e3ccb9 (127.17.60.193) evicted. New cstate: current_term: 3 leader_uuid: "c561ec3374b94b95be0c155d80ff0bff" committed_config { opid_index: 33 OBSOLETE_local: false peers { permanent_uuid: "c561ec3374b94b95be0c155d80ff0bff" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 33627 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "25286c485fa04c86a5d07c8f88643a47" member_type: VOTER last_known_addr { host: "127.17.60.198" port: 37089 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a2a5bba4ac1145d5bab94a69c9f4df40" member_type: VOTER last_known_addr { host: "127.17.60.196" port: 41807 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20250112 15:07:47.350353 21474 raft_consensus.cc:2949] T bfaeae584ec04772b2d6fcd088022117 P a2a5bba4ac1145d5bab94a69c9f4df40 [term 3 FOLLOWER]: Committing config change with OpId 3.33: config changed from index 32 to 33, VOTER 2537bbbd1a9947ae9facd87567e3ccb9 (127.17.60.193) evicted. New config: { opid_index: 33 OBSOLETE_local: false peers { permanent_uuid: "c561ec3374b94b95be0c155d80ff0bff" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 33627 } } peers { permanent_uuid: "25286c485fa04c86a5d07c8f88643a47" member_type: VOTER last_known_addr { host: "127.17.60.198" port: 37089 } attrs { promote: false } } peers { permanent_uuid: "a2a5bba4ac1145d5bab94a69c9f4df40" member_type: VOTER last_known_addr { host: "127.17.60.196" port: 41807 } attrs { promote: false } } }
I20250112 15:07:47.351846 21065 tablet_service.cc:1514] Processing DeleteTablet for tablet bfaeae584ec04772b2d6fcd088022117 with delete_type TABLET_DATA_TOMBSTONED (TS 2537bbbd1a9947ae9facd87567e3ccb9 not found in new config with opid_index 33) from {username='slave'} at 127.0.0.1:37188
I20250112 15:07:47.352074 22153 tablet_replica.cc:331] T bfaeae584ec04772b2d6fcd088022117 P 2537bbbd1a9947ae9facd87567e3ccb9: stopping tablet replica
I20250112 15:07:47.352134 22153 raft_consensus.cc:2238] T bfaeae584ec04772b2d6fcd088022117 P 2537bbbd1a9947ae9facd87567e3ccb9 [term 3 FOLLOWER]: Raft consensus shutting down.
I20250112 15:07:47.352239 22153 raft_consensus.cc:2267] T bfaeae584ec04772b2d6fcd088022117 P 2537bbbd1a9947ae9facd87567e3ccb9 [term 3 FOLLOWER]: Raft consensus is shut down!
I20250112 15:07:47.352476 22153 ts_tablet_manager.cc:1905] T bfaeae584ec04772b2d6fcd088022117 P 2537bbbd1a9947ae9facd87567e3ccb9: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250112 15:07:47.353615 22153 ts_tablet_manager.cc:1918] T bfaeae584ec04772b2d6fcd088022117 P 2537bbbd1a9947ae9facd87567e3ccb9: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 3.32
I20250112 15:07:47.353663 22153 log.cc:1198] T bfaeae584ec04772b2d6fcd088022117 P 2537bbbd1a9947ae9facd87567e3ccb9: Deleting WAL directory at /tmp/dist-test-task95dLSD/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1736694351070047-17651-0/minicluster-data/ts-0-root/wals/bfaeae584ec04772b2d6fcd088022117
I20250112 15:07:47.353993 20956 catalog_manager.cc:4872] TS 2537bbbd1a9947ae9facd87567e3ccb9 (127.17.60.193:33173): tablet bfaeae584ec04772b2d6fcd088022117 (table test-workload [id=cdaf6d3a65a54ea08e8e2c0ecdfc9de7]) successfully deleted
I20250112 15:07:48.096465 21026 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250112 15:07:48.097115 21149 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "bfaeae584ec04772b2d6fcd088022117"
dest_uuid: "c561ec3374b94b95be0c155d80ff0bff"
mode: GRACEFUL
new_leader_uuid: "25286c485fa04c86a5d07c8f88643a47"
from {username='slave'} at 127.0.0.1:48104
I20250112 15:07:48.097205 21149 raft_consensus.cc:604] T bfaeae584ec04772b2d6fcd088022117 P c561ec3374b94b95be0c155d80ff0bff [term 3 LEADER]: Received request to transfer leadership to TS 25286c485fa04c86a5d07c8f88643a47
I20250112 15:07:48.097407 21026 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 1
I20250112 15:07:48.097489 21026 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20250112 15:07:48.342015 22103 raft_consensus.cc:988] T bfaeae584ec04772b2d6fcd088022117 P c561ec3374b94b95be0c155d80ff0bff: : Instructing follower 25286c485fa04c86a5d07c8f88643a47 to start an election
I20250112 15:07:48.342104 22103 raft_consensus.cc:1076] T bfaeae584ec04772b2d6fcd088022117 P c561ec3374b94b95be0c155d80ff0bff [term 3 LEADER]: Signalling peer 25286c485fa04c86a5d07c8f88643a47 to start an election
I20250112 15:07:48.342280 21624 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "bfaeae584ec04772b2d6fcd088022117"
dest_uuid: "25286c485fa04c86a5d07c8f88643a47"
from {username='slave'} at 127.0.0.1:35738
I20250112 15:07:48.342365 21624 raft_consensus.cc:491] T bfaeae584ec04772b2d6fcd088022117 P 25286c485fa04c86a5d07c8f88643a47 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20250112 15:07:48.342388 21624 raft_consensus.cc:3054] T bfaeae584ec04772b2d6fcd088022117 P 25286c485fa04c86a5d07c8f88643a47 [term 3 FOLLOWER]: Advancing to term 4
I20250112 15:07:48.343041 21624 raft_consensus.cc:513] T bfaeae584ec04772b2d6fcd088022117 P 25286c485fa04c86a5d07c8f88643a47 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: 33 OBSOLETE_local: false peers { permanent_uuid: "c561ec3374b94b95be0c155d80ff0bff" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 33627 } } peers { permanent_uuid: "25286c485fa04c86a5d07c8f88643a47" member_type: VOTER last_known_addr { host: "127.17.60.198" port: 37089 } attrs { promote: false } } peers { permanent_uuid: "a2a5bba4ac1145d5bab94a69c9f4df40" member_type: VOTER last_known_addr { host: "127.17.60.196" port: 41807 } attrs { promote: false } }
I20250112 15:07:48.343204 21624 leader_election.cc:290] T bfaeae584ec04772b2d6fcd088022117 P 25286c485fa04c86a5d07c8f88643a47 [CANDIDATE]: Term 4 election: Requested vote from peers c561ec3374b94b95be0c155d80ff0bff (127.17.60.194:33627), a2a5bba4ac1145d5bab94a69c9f4df40 (127.17.60.196:41807)
I20250112 15:07:48.343408 21149 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bfaeae584ec04772b2d6fcd088022117" candidate_uuid: "25286c485fa04c86a5d07c8f88643a47" candidate_term: 4 candidate_status { last_received { term: 3 index: 33 } } ignore_live_leader: true dest_uuid: "c561ec3374b94b95be0c155d80ff0bff"
I20250112 15:07:48.343495 21149 raft_consensus.cc:3049] T bfaeae584ec04772b2d6fcd088022117 P c561ec3374b94b95be0c155d80ff0bff [term 3 LEADER]: Stepping down as leader of term 3
I20250112 15:07:48.343519 21149 raft_consensus.cc:738] T bfaeae584ec04772b2d6fcd088022117 P c561ec3374b94b95be0c155d80ff0bff [term 3 LEADER]: Becoming Follower/Learner. State: Replica: c561ec3374b94b95be0c155d80ff0bff, State: Running, Role: LEADER
I20250112 15:07:48.343582 21149 consensus_queue.cc:260] T bfaeae584ec04772b2d6fcd088022117 P c561ec3374b94b95be0c155d80ff0bff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 33, Committed index: 33, Last appended: 3.33, Last appended by leader: 33, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 33 OBSOLETE_local: false peers { permanent_uuid: "c561ec3374b94b95be0c155d80ff0bff" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 33627 } } peers { permanent_uuid: "25286c485fa04c86a5d07c8f88643a47" member_type: VOTER last_known_addr { host: "127.17.60.198" port: 37089 } attrs { promote: false } } peers { permanent_uuid: "a2a5bba4ac1145d5bab94a69c9f4df40" member_type: VOTER last_known_addr { host: "127.17.60.196" port: 41807 } attrs { promote: false } }
I20250112 15:07:48.343678 21149 raft_consensus.cc:3054] T bfaeae584ec04772b2d6fcd088022117 P c561ec3374b94b95be0c155d80ff0bff [term 3 FOLLOWER]: Advancing to term 4
I20250112 15:07:48.344424 21149 raft_consensus.cc:2463] T bfaeae584ec04772b2d6fcd088022117 P c561ec3374b94b95be0c155d80ff0bff [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 25286c485fa04c86a5d07c8f88643a47 in term 4.
I20250112 15:07:48.344627 21589 leader_election.cc:304] T bfaeae584ec04772b2d6fcd088022117 P 25286c485fa04c86a5d07c8f88643a47 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 25286c485fa04c86a5d07c8f88643a47, c561ec3374b94b95be0c155d80ff0bff; no voters:
I20250112 15:07:48.344739 22099 raft_consensus.cc:2798] T bfaeae584ec04772b2d6fcd088022117 P 25286c485fa04c86a5d07c8f88643a47 [term 4 FOLLOWER]: Leader election won for term 4
I20250112 15:07:48.344851 22099 raft_consensus.cc:695] T bfaeae584ec04772b2d6fcd088022117 P 25286c485fa04c86a5d07c8f88643a47 [term 4 LEADER]: Becoming Leader. State: Replica: 25286c485fa04c86a5d07c8f88643a47, State: Running, Role: LEADER
I20250112 15:07:48.344925 22099 consensus_queue.cc:237] T bfaeae584ec04772b2d6fcd088022117 P 25286c485fa04c86a5d07c8f88643a47 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 33, Committed index: 33, Last appended: 3.33, Last appended by leader: 33, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 33 OBSOLETE_local: false peers { permanent_uuid: "c561ec3374b94b95be0c155d80ff0bff" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 33627 } } peers { permanent_uuid: "25286c485fa04c86a5d07c8f88643a47" member_type: VOTER last_known_addr { host: "127.17.60.198" port: 37089 } attrs { promote: false } } peers { permanent_uuid: "a2a5bba4ac1145d5bab94a69c9f4df40" member_type: VOTER last_known_addr { host: "127.17.60.196" port: 41807 } attrs { promote: false } }
I20250112 15:07:48.345424 20966 catalog_manager.cc:5526] T bfaeae584ec04772b2d6fcd088022117 P 25286c485fa04c86a5d07c8f88643a47 reported cstate change: term changed from 3 to 4, leader changed from c561ec3374b94b95be0c155d80ff0bff (127.17.60.194) to 25286c485fa04c86a5d07c8f88643a47 (127.17.60.198). New cstate: current_term: 4 leader_uuid: "25286c485fa04c86a5d07c8f88643a47" committed_config { opid_index: 33 OBSOLETE_local: false peers { permanent_uuid: "c561ec3374b94b95be0c155d80ff0bff" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 33627 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "25286c485fa04c86a5d07c8f88643a47" member_type: VOTER last_known_addr { host: "127.17.60.198" port: 37089 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a2a5bba4ac1145d5bab94a69c9f4df40" member_type: VOTER last_known_addr { host: "127.17.60.196" port: 41807 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20250112 15:07:48.346297 21474 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bfaeae584ec04772b2d6fcd088022117" candidate_uuid: "25286c485fa04c86a5d07c8f88643a47" candidate_term: 4 candidate_status { last_received { term: 3 index: 33 } } ignore_live_leader: true dest_uuid: "a2a5bba4ac1145d5bab94a69c9f4df40"
I20250112 15:07:48.346364 21474 raft_consensus.cc:3054] T bfaeae584ec04772b2d6fcd088022117 P a2a5bba4ac1145d5bab94a69c9f4df40 [term 3 FOLLOWER]: Advancing to term 4
I20250112 15:07:48.346918 21474 raft_consensus.cc:2463] T bfaeae584ec04772b2d6fcd088022117 P a2a5bba4ac1145d5bab94a69c9f4df40 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 25286c485fa04c86a5d07c8f88643a47 in term 4.
I20250112 15:07:48.846812 21474 raft_consensus.cc:1270] T bfaeae584ec04772b2d6fcd088022117 P a2a5bba4ac1145d5bab94a69c9f4df40 [term 4 FOLLOWER]: Refusing update from remote peer 25286c485fa04c86a5d07c8f88643a47: Log matching property violated. Preceding OpId in replica: term: 3 index: 33. Preceding OpId from leader: term: 4 index: 34. (index mismatch)
I20250112 15:07:48.847071 22099 consensus_queue.cc:1035] T bfaeae584ec04772b2d6fcd088022117 P 25286c485fa04c86a5d07c8f88643a47 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2a5bba4ac1145d5bab94a69c9f4df40" member_type: VOTER last_known_addr { host: "127.17.60.196" port: 41807 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 34, Last known committed idx: 33, Time since last communication: 0.000s
I20250112 15:07:48.848222 21149 raft_consensus.cc:1270] T bfaeae584ec04772b2d6fcd088022117 P c561ec3374b94b95be0c155d80ff0bff [term 4 FOLLOWER]: Refusing update from remote peer 25286c485fa04c86a5d07c8f88643a47: Log matching property violated. Preceding OpId in replica: term: 3 index: 33. Preceding OpId from leader: term: 4 index: 34. (index mismatch)
I20250112 15:07:48.848380 22163 consensus_queue.cc:1035] T bfaeae584ec04772b2d6fcd088022117 P 25286c485fa04c86a5d07c8f88643a47 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c561ec3374b94b95be0c155d80ff0bff" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 33627 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 34, Last known committed idx: 33, Time since last communication: 0.000s
I20250112 15:07:50.097786 21026 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250112 15:07:50.100751 21624 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "1eb2b7e6edd34a7b83f9b918d3cdf2ed"
dest_uuid: "25286c485fa04c86a5d07c8f88643a47"
mode: GRACEFUL
new_leader_uuid: "2537bbbd1a9947ae9facd87567e3ccb9"
from {username='slave'} at 127.0.0.1:56310
I20250112 15:07:50.100831 21624 raft_consensus.cc:604] T 1eb2b7e6edd34a7b83f9b918d3cdf2ed P 25286c485fa04c86a5d07c8f88643a47 [term 2 LEADER]: Received request to transfer leadership to TS 2537bbbd1a9947ae9facd87567e3ccb9
I20250112 15:07:50.101025 21026 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 1
I20250112 15:07:50.101075 21026 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20250112 15:07:50.420171 22099 raft_consensus.cc:988] T 1eb2b7e6edd34a7b83f9b918d3cdf2ed P 25286c485fa04c86a5d07c8f88643a47: : Instructing follower 2537bbbd1a9947ae9facd87567e3ccb9 to start an election
I20250112 15:07:50.420259 22099 raft_consensus.cc:1076] T 1eb2b7e6edd34a7b83f9b918d3cdf2ed P 25286c485fa04c86a5d07c8f88643a47 [term 2 LEADER]: Signalling peer 2537bbbd1a9947ae9facd87567e3ccb9 to start an election
I20250112 15:07:50.420461 21069 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "1eb2b7e6edd34a7b83f9b918d3cdf2ed"
dest_uuid: "2537bbbd1a9947ae9facd87567e3ccb9"
from {username='slave'} at 127.0.0.1:54434
I20250112 15:07:50.420550 21069 raft_consensus.cc:491] T 1eb2b7e6edd34a7b83f9b918d3cdf2ed P 2537bbbd1a9947ae9facd87567e3ccb9 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20250112 15:07:50.420601 21069 raft_consensus.cc:3054] T 1eb2b7e6edd34a7b83f9b918d3cdf2ed P 2537bbbd1a9947ae9facd87567e3ccb9 [term 2 FOLLOWER]: Advancing to term 3
I20250112 15:07:50.421342 21069 raft_consensus.cc:513] T 1eb2b7e6edd34a7b83f9b918d3cdf2ed P 2537bbbd1a9947ae9facd87567e3ccb9 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "2537bbbd1a9947ae9facd87567e3ccb9" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 33173 } } peers { permanent_uuid: "c561ec3374b94b95be0c155d80ff0bff" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 33627 } } peers { permanent_uuid: "25286c485fa04c86a5d07c8f88643a47" member_type: VOTER last_known_addr { host: "127.17.60.198" port: 37089 } attrs { promote: false } }
I20250112 15:07:50.421496 21069 leader_election.cc:290] T 1eb2b7e6edd34a7b83f9b918d3cdf2ed P 2537bbbd1a9947ae9facd87567e3ccb9 [CANDIDATE]: Term 3 election: Requested vote from peers c561ec3374b94b95be0c155d80ff0bff (127.17.60.194:33627), 25286c485fa04c86a5d07c8f88643a47 (127.17.60.198:37089)
I20250112 15:07:50.421656 21624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1eb2b7e6edd34a7b83f9b918d3cdf2ed" candidate_uuid: "2537bbbd1a9947ae9facd87567e3ccb9" candidate_term: 3 candidate_status { last_received { term: 2 index: 29 } } ignore_live_leader: true dest_uuid: "25286c485fa04c86a5d07c8f88643a47"
I20250112 15:07:50.421655 21149 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1eb2b7e6edd34a7b83f9b918d3cdf2ed" candidate_uuid: "2537bbbd1a9947ae9facd87567e3ccb9" candidate_term: 3 candidate_status { last_received { term: 2 index: 29 } } ignore_live_leader: true dest_uuid: "c561ec3374b94b95be0c155d80ff0bff"
I20250112 15:07:50.421764 21624 raft_consensus.cc:3049] T 1eb2b7e6edd34a7b83f9b918d3cdf2ed P 25286c485fa04c86a5d07c8f88643a47 [term 2 LEADER]: Stepping down as leader of term 2
I20250112 15:07:50.421779 21149 raft_consensus.cc:3054] T 1eb2b7e6edd34a7b83f9b918d3cdf2ed P c561ec3374b94b95be0c155d80ff0bff [term 2 FOLLOWER]: Advancing to term 3
I20250112 15:07:50.421794 21624 raft_consensus.cc:738] T 1eb2b7e6edd34a7b83f9b918d3cdf2ed P 25286c485fa04c86a5d07c8f88643a47 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 25286c485fa04c86a5d07c8f88643a47, State: Running, Role: LEADER
I20250112 15:07:50.421847 21624 consensus_queue.cc:260] T 1eb2b7e6edd34a7b83f9b918d3cdf2ed P 25286c485fa04c86a5d07c8f88643a47 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 29, Committed index: 29, Last appended: 2.29, Last appended by leader: 29, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "2537bbbd1a9947ae9facd87567e3ccb9" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 33173 } } peers { permanent_uuid: "c561ec3374b94b95be0c155d80ff0bff" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 33627 } } peers { permanent_uuid: "25286c485fa04c86a5d07c8f88643a47" member_type: VOTER last_known_addr { host: "127.17.60.198" port: 37089 } attrs { promote: false } }
I20250112 15:07:50.421938 21624 raft_consensus.cc:3054] T 1eb2b7e6edd34a7b83f9b918d3cdf2ed P 25286c485fa04c86a5d07c8f88643a47 [term 2 FOLLOWER]: Advancing to term 3
I20250112 15:07:50.422537 21149 raft_consensus.cc:2463] T 1eb2b7e6edd34a7b83f9b918d3cdf2ed P c561ec3374b94b95be0c155d80ff0bff [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2537bbbd1a9947ae9facd87567e3ccb9 in term 3.
I20250112 15:07:50.422608 21624 raft_consensus.cc:2463] T 1eb2b7e6edd34a7b83f9b918d3cdf2ed P 25286c485fa04c86a5d07c8f88643a47 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2537bbbd1a9947ae9facd87567e3ccb9 in term 3.
I20250112 15:07:50.422778 21038 leader_election.cc:304] T 1eb2b7e6edd34a7b83f9b918d3cdf2ed P 2537bbbd1a9947ae9facd87567e3ccb9 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 25286c485fa04c86a5d07c8f88643a47, 2537bbbd1a9947ae9facd87567e3ccb9; no voters:
I20250112 15:07:50.423012 22154 raft_consensus.cc:2798] T 1eb2b7e6edd34a7b83f9b918d3cdf2ed P 2537bbbd1a9947ae9facd87567e3ccb9 [term 3 FOLLOWER]: Leader election won for term 3
I20250112 15:07:50.423066 22154 raft_consensus.cc:695] T 1eb2b7e6edd34a7b83f9b918d3cdf2ed P 2537bbbd1a9947ae9facd87567e3ccb9 [term 3 LEADER]: Becoming Leader. State: Replica: 2537bbbd1a9947ae9facd87567e3ccb9, State: Running, Role: LEADER
I20250112 15:07:50.423113 22154 consensus_queue.cc:237] T 1eb2b7e6edd34a7b83f9b918d3cdf2ed P 2537bbbd1a9947ae9facd87567e3ccb9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 29, Committed index: 29, Last appended: 2.29, Last appended by leader: 29, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "2537bbbd1a9947ae9facd87567e3ccb9" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 33173 } } peers { permanent_uuid: "c561ec3374b94b95be0c155d80ff0bff" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 33627 } } peers { permanent_uuid: "25286c485fa04c86a5d07c8f88643a47" member_type: VOTER last_known_addr { host: "127.17.60.198" port: 37089 } attrs { promote: false } }
I20250112 15:07:50.423702 20966 catalog_manager.cc:5526] T 1eb2b7e6edd34a7b83f9b918d3cdf2ed P 2537bbbd1a9947ae9facd87567e3ccb9 reported cstate change: term changed from 2 to 3, leader changed from 25286c485fa04c86a5d07c8f88643a47 (127.17.60.198) to 2537bbbd1a9947ae9facd87567e3ccb9 (127.17.60.193). New cstate: current_term: 3 leader_uuid: "2537bbbd1a9947ae9facd87567e3ccb9" committed_config { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "2537bbbd1a9947ae9facd87567e3ccb9" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 33173 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c561ec3374b94b95be0c155d80ff0bff" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 33627 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "25286c485fa04c86a5d07c8f88643a47" member_type: VOTER last_known_addr { host: "127.17.60.198" port: 37089 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20250112 15:07:50.827858 21149 raft_consensus.cc:1270] T 1eb2b7e6edd34a7b83f9b918d3cdf2ed P c561ec3374b94b95be0c155d80ff0bff [term 3 FOLLOWER]: Refusing update from remote peer 2537bbbd1a9947ae9facd87567e3ccb9: Log matching property violated. Preceding OpId in replica: term: 2 index: 29. Preceding OpId from leader: term: 3 index: 30. (index mismatch)
I20250112 15:07:50.828161 22173 consensus_queue.cc:1035] T 1eb2b7e6edd34a7b83f9b918d3cdf2ed P 2537bbbd1a9947ae9facd87567e3ccb9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c561ec3374b94b95be0c155d80ff0bff" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 33627 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 30, Last known committed idx: 29, Time since last communication: 0.000s
I20250112 15:07:50.829275 21624 raft_consensus.cc:1270] T 1eb2b7e6edd34a7b83f9b918d3cdf2ed P 25286c485fa04c86a5d07c8f88643a47 [term 3 FOLLOWER]: Refusing update from remote peer 2537bbbd1a9947ae9facd87567e3ccb9: Log matching property violated. Preceding OpId in replica: term: 2 index: 29. Preceding OpId from leader: term: 3 index: 30. (index mismatch)
I20250112 15:07:50.829568 22173 consensus_queue.cc:1035] T 1eb2b7e6edd34a7b83f9b918d3cdf2ed P 2537bbbd1a9947ae9facd87567e3ccb9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "25286c485fa04c86a5d07c8f88643a47" member_type: VOTER last_known_addr { host: "127.17.60.198" port: 37089 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 30, Last known committed idx: 29, Time since last communication: 0.000s
I20250112 15:07:52.101365 21026 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250112 15:07:52.102043 21069 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "ef90b704208140a3983ffe6105939319"
dest_uuid: "2537bbbd1a9947ae9facd87567e3ccb9"
mode: GRACEFUL
new_leader_uuid: "3710b74ed2044ce5995533df32e67351"
from {username='slave'} at 127.0.0.1:54414
I20250112 15:07:52.102136 21069 raft_consensus.cc:604] T ef90b704208140a3983ffe6105939319 P 2537bbbd1a9947ae9facd87567e3ccb9 [term 4 LEADER]: Received request to transfer leadership to TS 3710b74ed2044ce5995533df32e67351
I20250112 15:07:52.102375 21026 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 1
I20250112 15:07:52.102439 21026 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20250112 15:07:52.330248 22154 raft_consensus.cc:988] T ef90b704208140a3983ffe6105939319 P 2537bbbd1a9947ae9facd87567e3ccb9: : Instructing follower 3710b74ed2044ce5995533df32e67351 to start an election
I20250112 15:07:52.330332 22154 raft_consensus.cc:1076] T ef90b704208140a3983ffe6105939319 P 2537bbbd1a9947ae9facd87567e3ccb9 [term 4 LEADER]: Signalling peer 3710b74ed2044ce5995533df32e67351 to start an election
I20250112 15:07:52.330530 21226 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "ef90b704208140a3983ffe6105939319"
dest_uuid: "3710b74ed2044ce5995533df32e67351"
from {username='slave'} at 127.0.0.1:33194
I20250112 15:07:52.330596 21226 raft_consensus.cc:491] T ef90b704208140a3983ffe6105939319 P 3710b74ed2044ce5995533df32e67351 [term 4 FOLLOWER]: Starting forced leader election (received explicit request)
I20250112 15:07:52.330618 21226 raft_consensus.cc:3054] T ef90b704208140a3983ffe6105939319 P 3710b74ed2044ce5995533df32e67351 [term 4 FOLLOWER]: Advancing to term 5
I20250112 15:07:52.331272 21226 raft_consensus.cc:513] T ef90b704208140a3983ffe6105939319 P 3710b74ed2044ce5995533df32e67351 [term 5 FOLLOWER]: Starting forced leader election with config: opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "2537bbbd1a9947ae9facd87567e3ccb9" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 33173 } } peers { permanent_uuid: "3710b74ed2044ce5995533df32e67351" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 38589 } } peers { permanent_uuid: "25286c485fa04c86a5d07c8f88643a47" member_type: VOTER last_known_addr { host: "127.17.60.198" port: 37089 } attrs { promote: false } }
I20250112 15:07:52.331400 21226 leader_election.cc:290] T ef90b704208140a3983ffe6105939319 P 3710b74ed2044ce5995533df32e67351 [CANDIDATE]: Term 5 election: Requested vote from peers 2537bbbd1a9947ae9facd87567e3ccb9 (127.17.60.193:33173), 25286c485fa04c86a5d07c8f88643a47 (127.17.60.198:37089)
I20250112 15:07:52.331568 21069 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef90b704208140a3983ffe6105939319" candidate_uuid: "3710b74ed2044ce5995533df32e67351" candidate_term: 5 candidate_status { last_received { term: 4 index: 31 } } ignore_live_leader: true dest_uuid: "2537bbbd1a9947ae9facd87567e3ccb9"
I20250112 15:07:52.331580 21624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef90b704208140a3983ffe6105939319" candidate_uuid: "3710b74ed2044ce5995533df32e67351" candidate_term: 5 candidate_status { last_received { term: 4 index: 31 } } ignore_live_leader: true dest_uuid: "25286c485fa04c86a5d07c8f88643a47"
I20250112 15:07:52.331684 21069 raft_consensus.cc:3049] T ef90b704208140a3983ffe6105939319 P 2537bbbd1a9947ae9facd87567e3ccb9 [term 4 LEADER]: Stepping down as leader of term 4
I20250112 15:07:52.331689 21624 raft_consensus.cc:3054] T ef90b704208140a3983ffe6105939319 P 25286c485fa04c86a5d07c8f88643a47 [term 4 FOLLOWER]: Advancing to term 5
I20250112 15:07:52.331725 21069 raft_consensus.cc:738] T ef90b704208140a3983ffe6105939319 P 2537bbbd1a9947ae9facd87567e3ccb9 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 2537bbbd1a9947ae9facd87567e3ccb9, State: Running, Role: LEADER
I20250112 15:07:52.331786 21069 consensus_queue.cc:260] T ef90b704208140a3983ffe6105939319 P 2537bbbd1a9947ae9facd87567e3ccb9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 31, Committed index: 31, Last appended: 4.31, Last appended by leader: 31, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "2537bbbd1a9947ae9facd87567e3ccb9" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 33173 } } peers { permanent_uuid: "3710b74ed2044ce5995533df32e67351" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 38589 } } peers { permanent_uuid: "25286c485fa04c86a5d07c8f88643a47" member_type: VOTER last_known_addr { host: "127.17.60.198" port: 37089 } attrs { promote: false } }
I20250112 15:07:52.331871 21069 raft_consensus.cc:3054] T ef90b704208140a3983ffe6105939319 P 2537bbbd1a9947ae9facd87567e3ccb9 [term 4 FOLLOWER]: Advancing to term 5
I20250112 15:07:52.332473 21069 raft_consensus.cc:2463] T ef90b704208140a3983ffe6105939319 P 2537bbbd1a9947ae9facd87567e3ccb9 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3710b74ed2044ce5995533df32e67351 in term 5.
I20250112 15:07:52.332473 21624 raft_consensus.cc:2463] T ef90b704208140a3983ffe6105939319 P 25286c485fa04c86a5d07c8f88643a47 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3710b74ed2044ce5995533df32e67351 in term 5.
I20250112 15:07:52.332700 21189 leader_election.cc:304] T ef90b704208140a3983ffe6105939319 P 3710b74ed2044ce5995533df32e67351 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2537bbbd1a9947ae9facd87567e3ccb9, 3710b74ed2044ce5995533df32e67351; no voters:
I20250112 15:07:52.332809 22123 raft_consensus.cc:2798] T ef90b704208140a3983ffe6105939319 P 3710b74ed2044ce5995533df32e67351 [term 5 FOLLOWER]: Leader election won for term 5
I20250112 15:07:52.332922 22123 raft_consensus.cc:695] T ef90b704208140a3983ffe6105939319 P 3710b74ed2044ce5995533df32e67351 [term 5 LEADER]: Becoming Leader. State: Replica: 3710b74ed2044ce5995533df32e67351, State: Running, Role: LEADER
I20250112 15:07:52.332973 22123 consensus_queue.cc:237] T ef90b704208140a3983ffe6105939319 P 3710b74ed2044ce5995533df32e67351 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 31, Committed index: 31, Last appended: 4.31, Last appended by leader: 31, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "2537bbbd1a9947ae9facd87567e3ccb9" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 33173 } } peers { permanent_uuid: "3710b74ed2044ce5995533df32e67351" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 38589 } } peers { permanent_uuid: "25286c485fa04c86a5d07c8f88643a47" member_type: VOTER last_known_addr { host: "127.17.60.198" port: 37089 } attrs { promote: false } }
I20250112 15:07:52.333454 20966 catalog_manager.cc:5526] T ef90b704208140a3983ffe6105939319 P 3710b74ed2044ce5995533df32e67351 reported cstate change: term changed from 4 to 5, leader changed from 2537bbbd1a9947ae9facd87567e3ccb9 (127.17.60.193) to 3710b74ed2044ce5995533df32e67351 (127.17.60.195). New cstate: current_term: 5 leader_uuid: "3710b74ed2044ce5995533df32e67351" committed_config { opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "2537bbbd1a9947ae9facd87567e3ccb9" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 33173 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3710b74ed2044ce5995533df32e67351" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 38589 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "25286c485fa04c86a5d07c8f88643a47" member_type: VOTER last_known_addr { host: "127.17.60.198" port: 37089 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20250112 15:07:52.801810 21069 raft_consensus.cc:1270] T ef90b704208140a3983ffe6105939319 P 2537bbbd1a9947ae9facd87567e3ccb9 [term 5 FOLLOWER]: Refusing update from remote peer 3710b74ed2044ce5995533df32e67351: Log matching property violated. Preceding OpId in replica: term: 4 index: 31. Preceding OpId from leader: term: 5 index: 32. (index mismatch)
I20250112 15:07:52.802047 22123 consensus_queue.cc:1035] T ef90b704208140a3983ffe6105939319 P 3710b74ed2044ce5995533df32e67351 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2537bbbd1a9947ae9facd87567e3ccb9" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 33173 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 32, Last known committed idx: 31, Time since last communication: 0.000s
I20250112 15:07:52.803164 21624 raft_consensus.cc:1270] T ef90b704208140a3983ffe6105939319 P 25286c485fa04c86a5d07c8f88643a47 [term 5 FOLLOWER]: Refusing update from remote peer 3710b74ed2044ce5995533df32e67351: Log matching property violated. Preceding OpId in replica: term: 4 index: 31. Preceding OpId from leader: term: 5 index: 32. (index mismatch)
I20250112 15:07:52.803356 22123 consensus_queue.cc:1035] T ef90b704208140a3983ffe6105939319 P 3710b74ed2044ce5995533df32e67351 [LEADER]: Connected to new peer: Peer: permanent_uuid: "25286c485fa04c86a5d07c8f88643a47" member_type: VOTER last_known_addr { host: "127.17.60.198" port: 37089 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 32, Last known committed idx: 31, Time since last communication: 0.000s
I20250112 15:07:54.102633 21026 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250112 15:07:54.103152 21226 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "9601bcda24ed42e7b93ab9740f1e271e"
dest_uuid: "3710b74ed2044ce5995533df32e67351"
mode: GRACEFUL
new_leader_uuid: "a2a5bba4ac1145d5bab94a69c9f4df40"
from {username='slave'} at 127.0.0.1:54970
I20250112 15:07:54.103242 21226 raft_consensus.cc:604] T 9601bcda24ed42e7b93ab9740f1e271e P 3710b74ed2044ce5995533df32e67351 [term 4 LEADER]: Received request to transfer leadership to TS a2a5bba4ac1145d5bab94a69c9f4df40
I20250112 15:07:54.103451 21026 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 1
I20250112 15:07:54.103514 21026 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20250112 15:07:54.487718 22197 raft_consensus.cc:988] T 9601bcda24ed42e7b93ab9740f1e271e P 3710b74ed2044ce5995533df32e67351: : Instructing follower a2a5bba4ac1145d5bab94a69c9f4df40 to start an election
I20250112 15:07:54.487798 22197 raft_consensus.cc:1076] T 9601bcda24ed42e7b93ab9740f1e271e P 3710b74ed2044ce5995533df32e67351 [term 4 LEADER]: Signalling peer a2a5bba4ac1145d5bab94a69c9f4df40 to start an election
I20250112 15:07:54.488049 21474 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "9601bcda24ed42e7b93ab9740f1e271e"
dest_uuid: "a2a5bba4ac1145d5bab94a69c9f4df40"
from {username='slave'} at 127.0.0.1:43316
I20250112 15:07:54.488137 21474 raft_consensus.cc:491] T 9601bcda24ed42e7b93ab9740f1e271e P a2a5bba4ac1145d5bab94a69c9f4df40 [term 4 FOLLOWER]: Starting forced leader election (received explicit request)
I20250112 15:07:54.488168 21474 raft_consensus.cc:3054] T 9601bcda24ed42e7b93ab9740f1e271e P a2a5bba4ac1145d5bab94a69c9f4df40 [term 4 FOLLOWER]: Advancing to term 5
I20250112 15:07:54.488867 21474 raft_consensus.cc:513] T 9601bcda24ed42e7b93ab9740f1e271e P a2a5bba4ac1145d5bab94a69c9f4df40 [term 5 FOLLOWER]: Starting forced leader election with config: opid_index: 33 OBSOLETE_local: false peers { permanent_uuid: "3710b74ed2044ce5995533df32e67351" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 38589 } } peers { permanent_uuid: "a2a5bba4ac1145d5bab94a69c9f4df40" member_type: VOTER last_known_addr { host: "127.17.60.196" port: 41807 } attrs { promote: false } } peers { permanent_uuid: "25286c485fa04c86a5d07c8f88643a47" member_type: VOTER last_known_addr { host: "127.17.60.198" port: 37089 } attrs { promote: false } }
I20250112 15:07:54.489054 21474 leader_election.cc:290] T 9601bcda24ed42e7b93ab9740f1e271e P a2a5bba4ac1145d5bab94a69c9f4df40 [CANDIDATE]: Term 5 election: Requested vote from peers 3710b74ed2044ce5995533df32e67351 (127.17.60.195:38589), 25286c485fa04c86a5d07c8f88643a47 (127.17.60.198:37089)
I20250112 15:07:54.489257 21226 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9601bcda24ed42e7b93ab9740f1e271e" candidate_uuid: "a2a5bba4ac1145d5bab94a69c9f4df40" candidate_term: 5 candidate_status { last_received { term: 4 index: 33 } } ignore_live_leader: true dest_uuid: "3710b74ed2044ce5995533df32e67351"
I20250112 15:07:54.489334 21226 raft_consensus.cc:3049] T 9601bcda24ed42e7b93ab9740f1e271e P 3710b74ed2044ce5995533df32e67351 [term 4 LEADER]: Stepping down as leader of term 4
I20250112 15:07:54.489363 21226 raft_consensus.cc:738] T 9601bcda24ed42e7b93ab9740f1e271e P 3710b74ed2044ce5995533df32e67351 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 3710b74ed2044ce5995533df32e67351, State: Running, Role: LEADER
I20250112 15:07:54.489423 21226 consensus_queue.cc:260] T 9601bcda24ed42e7b93ab9740f1e271e P 3710b74ed2044ce5995533df32e67351 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 33, Committed index: 33, Last appended: 4.33, Last appended by leader: 33, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 33 OBSOLETE_local: false peers { permanent_uuid: "3710b74ed2044ce5995533df32e67351" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 38589 } } peers { permanent_uuid: "a2a5bba4ac1145d5bab94a69c9f4df40" member_type: VOTER last_known_addr { host: "127.17.60.196" port: 41807 } attrs { promote: false } } peers { permanent_uuid: "25286c485fa04c86a5d07c8f88643a47" member_type: VOTER last_known_addr { host: "127.17.60.198" port: 37089 } attrs { promote: false } }
I20250112 15:07:54.489514 21226 raft_consensus.cc:3054] T 9601bcda24ed42e7b93ab9740f1e271e P 3710b74ed2044ce5995533df32e67351 [term 4 FOLLOWER]: Advancing to term 5
I20250112 15:07:54.490180 21226 raft_consensus.cc:2463] T 9601bcda24ed42e7b93ab9740f1e271e P 3710b74ed2044ce5995533df32e67351 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2a5bba4ac1145d5bab94a69c9f4df40 in term 5.
I20250112 15:07:54.490368 21438 leader_election.cc:304] T 9601bcda24ed42e7b93ab9740f1e271e P a2a5bba4ac1145d5bab94a69c9f4df40 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3710b74ed2044ce5995533df32e67351, a2a5bba4ac1145d5bab94a69c9f4df40; no voters:
I20250112 15:07:54.490451 22104 raft_consensus.cc:2798] T 9601bcda24ed42e7b93ab9740f1e271e P a2a5bba4ac1145d5bab94a69c9f4df40 [term 5 FOLLOWER]: Leader election won for term 5
I20250112 15:07:54.490541 22104 raft_consensus.cc:695] T 9601bcda24ed42e7b93ab9740f1e271e P a2a5bba4ac1145d5bab94a69c9f4df40 [term 5 LEADER]: Becoming Leader. State: Replica: a2a5bba4ac1145d5bab94a69c9f4df40, State: Running, Role: LEADER
I20250112 15:07:54.490602 22104 consensus_queue.cc:237] T 9601bcda24ed42e7b93ab9740f1e271e P a2a5bba4ac1145d5bab94a69c9f4df40 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 33, Committed index: 33, Last appended: 4.33, Last appended by leader: 33, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 33 OBSOLETE_local: false peers { permanent_uuid: "3710b74ed2044ce5995533df32e67351" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 38589 } } peers { permanent_uuid: "a2a5bba4ac1145d5bab94a69c9f4df40" member_type: VOTER last_known_addr { host: "127.17.60.196" port: 41807 } attrs { promote: false } } peers { permanent_uuid: "25286c485fa04c86a5d07c8f88643a47" member_type: VOTER last_known_addr { host: "127.17.60.198" port: 37089 } attrs { promote: false } }
I20250112 15:07:54.491137 20966 catalog_manager.cc:5526] T 9601bcda24ed42e7b93ab9740f1e271e P a2a5bba4ac1145d5bab94a69c9f4df40 reported cstate change: term changed from 4 to 5, leader changed from 3710b74ed2044ce5995533df32e67351 (127.17.60.195) to a2a5bba4ac1145d5bab94a69c9f4df40 (127.17.60.196). New cstate: current_term: 5 leader_uuid: "a2a5bba4ac1145d5bab94a69c9f4df40" committed_config { opid_index: 33 OBSOLETE_local: false peers { permanent_uuid: "3710b74ed2044ce5995533df32e67351" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 38589 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a2a5bba4ac1145d5bab94a69c9f4df40" member_type: VOTER last_known_addr { host: "127.17.60.196" port: 41807 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "25286c485fa04c86a5d07c8f88643a47" member_type: VOTER last_known_addr { host: "127.17.60.198" port: 37089 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20250112 15:07:54.491993 21624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9601bcda24ed42e7b93ab9740f1e271e" candidate_uuid: "a2a5bba4ac1145d5bab94a69c9f4df40" candidate_term: 5 candidate_status { last_received { term: 4 index: 33 } } ignore_live_leader: true dest_uuid: "25286c485fa04c86a5d07c8f88643a47"
I20250112 15:07:54.492064 21624 raft_consensus.cc:3054] T 9601bcda24ed42e7b93ab9740f1e271e P 25286c485fa04c86a5d07c8f88643a47 [term 4 FOLLOWER]: Advancing to term 5
I20250112 15:07:54.492609 21624 raft_consensus.cc:2463] T 9601bcda24ed42e7b93ab9740f1e271e P 25286c485fa04c86a5d07c8f88643a47 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2a5bba4ac1145d5bab94a69c9f4df40 in term 5.
I20250112 15:07:54.907231 21226 raft_consensus.cc:1270] T 9601bcda24ed42e7b93ab9740f1e271e P 3710b74ed2044ce5995533df32e67351 [term 5 FOLLOWER]: Refusing update from remote peer a2a5bba4ac1145d5bab94a69c9f4df40: Log matching property violated. Preceding OpId in replica: term: 4 index: 33. Preceding OpId from leader: term: 5 index: 34. (index mismatch)
I20250112 15:07:54.907516 22104 consensus_queue.cc:1035] T 9601bcda24ed42e7b93ab9740f1e271e P a2a5bba4ac1145d5bab94a69c9f4df40 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3710b74ed2044ce5995533df32e67351" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 38589 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 34, Last known committed idx: 33, Time since last communication: 0.000s
I20250112 15:07:54.908794 21624 raft_consensus.cc:1270] T 9601bcda24ed42e7b93ab9740f1e271e P 25286c485fa04c86a5d07c8f88643a47 [term 5 FOLLOWER]: Refusing update from remote peer a2a5bba4ac1145d5bab94a69c9f4df40: Log matching property violated. Preceding OpId in replica: term: 4 index: 33. Preceding OpId from leader: term: 5 index: 34. (index mismatch)
I20250112 15:07:54.909109 22104 consensus_queue.cc:1035] T 9601bcda24ed42e7b93ab9740f1e271e P a2a5bba4ac1145d5bab94a69c9f4df40 [LEADER]: Connected to new peer: Peer: permanent_uuid: "25286c485fa04c86a5d07c8f88643a47" member_type: VOTER last_known_addr { host: "127.17.60.198" port: 37089 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 34, Last known committed idx: 33, Time since last communication: 0.000s
I20250112 15:07:56.103682 21026 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250112 15:07:56.112617 21474 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "9601bcda24ed42e7b93ab9740f1e271e"
dest_uuid: "a2a5bba4ac1145d5bab94a69c9f4df40"
mode: GRACEFUL
new_leader_uuid: "3710b74ed2044ce5995533df32e67351"
from {username='slave'} at 127.0.0.1:57292
I20250112 15:07:56.112715 21474 raft_consensus.cc:604] T 9601bcda24ed42e7b93ab9740f1e271e P a2a5bba4ac1145d5bab94a69c9f4df40 [term 5 LEADER]: Received request to transfer leadership to TS 3710b74ed2044ce5995533df32e67351
I20250112 15:07:56.112871 21026 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 1
I20250112 15:07:56.112929 21026 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20250112 15:07:56.368623 22212 raft_consensus.cc:988] T 9601bcda24ed42e7b93ab9740f1e271e P a2a5bba4ac1145d5bab94a69c9f4df40: : Instructing follower 3710b74ed2044ce5995533df32e67351 to start an election
I20250112 15:07:56.368703 22212 raft_consensus.cc:1076] T 9601bcda24ed42e7b93ab9740f1e271e P a2a5bba4ac1145d5bab94a69c9f4df40 [term 5 LEADER]: Signalling peer 3710b74ed2044ce5995533df32e67351 to start an election
I20250112 15:07:56.368867 21226 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "9601bcda24ed42e7b93ab9740f1e271e"
dest_uuid: "3710b74ed2044ce5995533df32e67351"
from {username='slave'} at 127.0.0.1:54984
I20250112 15:07:56.368928 21226 raft_consensus.cc:491] T 9601bcda24ed42e7b93ab9740f1e271e P 3710b74ed2044ce5995533df32e67351 [term 5 FOLLOWER]: Starting forced leader election (received explicit request)
I20250112 15:07:56.368947 21226 raft_consensus.cc:3054] T 9601bcda24ed42e7b93ab9740f1e271e P 3710b74ed2044ce5995533df32e67351 [term 5 FOLLOWER]: Advancing to term 6
I20250112 15:07:56.369575 21226 raft_consensus.cc:513] T 9601bcda24ed42e7b93ab9740f1e271e P 3710b74ed2044ce5995533df32e67351 [term 6 FOLLOWER]: Starting forced leader election with config: opid_index: 33 OBSOLETE_local: false peers { permanent_uuid: "3710b74ed2044ce5995533df32e67351" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 38589 } } peers { permanent_uuid: "a2a5bba4ac1145d5bab94a69c9f4df40" member_type: VOTER last_known_addr { host: "127.17.60.196" port: 41807 } attrs { promote: false } } peers { permanent_uuid: "25286c485fa04c86a5d07c8f88643a47" member_type: VOTER last_known_addr { host: "127.17.60.198" port: 37089 } attrs { promote: false } }
I20250112 15:07:56.369747 21226 leader_election.cc:290] T 9601bcda24ed42e7b93ab9740f1e271e P 3710b74ed2044ce5995533df32e67351 [CANDIDATE]: Term 6 election: Requested vote from peers a2a5bba4ac1145d5bab94a69c9f4df40 (127.17.60.196:41807), 25286c485fa04c86a5d07c8f88643a47 (127.17.60.198:37089)
I20250112 15:07:56.369870 21474 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9601bcda24ed42e7b93ab9740f1e271e" candidate_uuid: "3710b74ed2044ce5995533df32e67351" candidate_term: 6 candidate_status { last_received { term: 5 index: 34 } } ignore_live_leader: true dest_uuid: "a2a5bba4ac1145d5bab94a69c9f4df40"
I20250112 15:07:56.369894 21624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9601bcda24ed42e7b93ab9740f1e271e" candidate_uuid: "3710b74ed2044ce5995533df32e67351" candidate_term: 6 candidate_status { last_received { term: 5 index: 34 } } ignore_live_leader: true dest_uuid: "25286c485fa04c86a5d07c8f88643a47"
I20250112 15:07:56.369933 21474 raft_consensus.cc:3049] T 9601bcda24ed42e7b93ab9740f1e271e P a2a5bba4ac1145d5bab94a69c9f4df40 [term 5 LEADER]: Stepping down as leader of term 5
I20250112 15:07:56.369946 21624 raft_consensus.cc:3054] T 9601bcda24ed42e7b93ab9740f1e271e P 25286c485fa04c86a5d07c8f88643a47 [term 5 FOLLOWER]: Advancing to term 6
I20250112 15:07:56.369956 21474 raft_consensus.cc:738] T 9601bcda24ed42e7b93ab9740f1e271e P a2a5bba4ac1145d5bab94a69c9f4df40 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: a2a5bba4ac1145d5bab94a69c9f4df40, State: Running, Role: LEADER
I20250112 15:07:56.370028 21474 consensus_queue.cc:260] T 9601bcda24ed42e7b93ab9740f1e271e P a2a5bba4ac1145d5bab94a69c9f4df40 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 34, Committed index: 34, Last appended: 5.34, Last appended by leader: 34, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 33 OBSOLETE_local: false peers { permanent_uuid: "3710b74ed2044ce5995533df32e67351" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 38589 } } peers { permanent_uuid: "a2a5bba4ac1145d5bab94a69c9f4df40" member_type: VOTER last_known_addr { host: "127.17.60.196" port: 41807 } attrs { promote: false } } peers { permanent_uuid: "25286c485fa04c86a5d07c8f88643a47" member_type: VOTER last_known_addr { host: "127.17.60.198" port: 37089 } attrs { promote: false } }
I20250112 15:07:56.370118 21474 raft_consensus.cc:3054] T 9601bcda24ed42e7b93ab9740f1e271e P a2a5bba4ac1145d5bab94a69c9f4df40 [term 5 FOLLOWER]: Advancing to term 6
I20250112 15:07:56.370683 21624 raft_consensus.cc:2463] T 9601bcda24ed42e7b93ab9740f1e271e P 25286c485fa04c86a5d07c8f88643a47 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3710b74ed2044ce5995533df32e67351 in term 6.
I20250112 15:07:56.370707 21474 raft_consensus.cc:2463] T 9601bcda24ed42e7b93ab9740f1e271e P a2a5bba4ac1145d5bab94a69c9f4df40 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3710b74ed2044ce5995533df32e67351 in term 6.
I20250112 15:07:56.370884 21189 leader_election.cc:304] T 9601bcda24ed42e7b93ab9740f1e271e P 3710b74ed2044ce5995533df32e67351 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 25286c485fa04c86a5d07c8f88643a47, 3710b74ed2044ce5995533df32e67351; no voters:
I20250112 15:07:56.370997 22198 raft_consensus.cc:2798] T 9601bcda24ed42e7b93ab9740f1e271e P 3710b74ed2044ce5995533df32e67351 [term 6 FOLLOWER]: Leader election won for term 6
I20250112 15:07:56.371119 22198 raft_consensus.cc:695] T 9601bcda24ed42e7b93ab9740f1e271e P 3710b74ed2044ce5995533df32e67351 [term 6 LEADER]: Becoming Leader. State: Replica: 3710b74ed2044ce5995533df32e67351, State: Running, Role: LEADER
I20250112 15:07:56.371306 22198 consensus_queue.cc:237] T 9601bcda24ed42e7b93ab9740f1e271e P 3710b74ed2044ce5995533df32e67351 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 34, Committed index: 34, Last appended: 5.34, Last appended by leader: 34, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 33 OBSOLETE_local: false peers { permanent_uuid: "3710b74ed2044ce5995533df32e67351" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 38589 } } peers { permanent_uuid: "a2a5bba4ac1145d5bab94a69c9f4df40" member_type: VOTER last_known_addr { host: "127.17.60.196" port: 41807 } attrs { promote: false } } peers { permanent_uuid: "25286c485fa04c86a5d07c8f88643a47" member_type: VOTER last_known_addr { host: "127.17.60.198" port: 37089 } attrs { promote: false } }
I20250112 15:07:56.371867 20966 catalog_manager.cc:5526] T 9601bcda24ed42e7b93ab9740f1e271e P 3710b74ed2044ce5995533df32e67351 reported cstate change: term changed from 5 to 6, leader changed from a2a5bba4ac1145d5bab94a69c9f4df40 (127.17.60.196) to 3710b74ed2044ce5995533df32e67351 (127.17.60.195). New cstate: current_term: 6 leader_uuid: "3710b74ed2044ce5995533df32e67351" committed_config { opid_index: 33 OBSOLETE_local: false peers { permanent_uuid: "3710b74ed2044ce5995533df32e67351" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 38589 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a2a5bba4ac1145d5bab94a69c9f4df40" member_type: VOTER last_known_addr { host: "127.17.60.196" port: 41807 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "25286c485fa04c86a5d07c8f88643a47" member_type: VOTER last_known_addr { host: "127.17.60.198" port: 37089 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20250112 15:07:56.829869 21474 raft_consensus.cc:1270] T 9601bcda24ed42e7b93ab9740f1e271e P a2a5bba4ac1145d5bab94a69c9f4df40 [term 6 FOLLOWER]: Refusing update from remote peer 3710b74ed2044ce5995533df32e67351: Log matching property violated. Preceding OpId in replica: term: 5 index: 34. Preceding OpId from leader: term: 6 index: 35. (index mismatch)
I20250112 15:07:56.830165 22198 consensus_queue.cc:1035] T 9601bcda24ed42e7b93ab9740f1e271e P 3710b74ed2044ce5995533df32e67351 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2a5bba4ac1145d5bab94a69c9f4df40" member_type: VOTER last_known_addr { host: "127.17.60.196" port: 41807 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 35, Last known committed idx: 34, Time since last communication: 0.000s
I20250112 15:07:56.831326 21624 raft_consensus.cc:1270] T 9601bcda24ed42e7b93ab9740f1e271e P 25286c485fa04c86a5d07c8f88643a47 [term 6 FOLLOWER]: Refusing update from remote peer 3710b74ed2044ce5995533df32e67351: Log matching property violated. Preceding OpId in replica: term: 5 index: 34. Preceding OpId from leader: term: 6 index: 35. (index mismatch)
I20250112 15:07:56.831517 22216 consensus_queue.cc:1035] T 9601bcda24ed42e7b93ab9740f1e271e P 3710b74ed2044ce5995533df32e67351 [LEADER]: Connected to new peer: Peer: permanent_uuid: "25286c485fa04c86a5d07c8f88643a47" member_type: VOTER last_known_addr { host: "127.17.60.198" port: 37089 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 35, Last known committed idx: 34, Time since last communication: 0.000s
I20250112 15:07:58.113092 21026 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250112 15:07:58.113607 21226 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "ef90b704208140a3983ffe6105939319"
dest_uuid: "3710b74ed2044ce5995533df32e67351"
mode: GRACEFUL
new_leader_uuid: "2537bbbd1a9947ae9facd87567e3ccb9"
from {username='slave'} at 127.0.0.1:54970
I20250112 15:07:58.113698 21226 raft_consensus.cc:604] T ef90b704208140a3983ffe6105939319 P 3710b74ed2044ce5995533df32e67351 [term 5 LEADER]: Received request to transfer leadership to TS 2537bbbd1a9947ae9facd87567e3ccb9
I20250112 15:07:58.113891 21026 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 1
I20250112 15:07:58.113957 21026 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20250112 15:07:58.344619 22228 raft_consensus.cc:988] T ef90b704208140a3983ffe6105939319 P 3710b74ed2044ce5995533df32e67351: : Instructing follower 2537bbbd1a9947ae9facd87567e3ccb9 to start an election
I20250112 15:07:58.344698 22228 raft_consensus.cc:1076] T ef90b704208140a3983ffe6105939319 P 3710b74ed2044ce5995533df32e67351 [term 5 LEADER]: Signalling peer 2537bbbd1a9947ae9facd87567e3ccb9 to start an election
I20250112 15:07:58.344867 21069 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "ef90b704208140a3983ffe6105939319"
dest_uuid: "2537bbbd1a9947ae9facd87567e3ccb9"
from {username='slave'} at 127.0.0.1:37204
I20250112 15:07:58.344959 21069 raft_consensus.cc:491] T ef90b704208140a3983ffe6105939319 P 2537bbbd1a9947ae9facd87567e3ccb9 [term 5 FOLLOWER]: Starting forced leader election (received explicit request)
I20250112 15:07:58.344981 21069 raft_consensus.cc:3054] T ef90b704208140a3983ffe6105939319 P 2537bbbd1a9947ae9facd87567e3ccb9 [term 5 FOLLOWER]: Advancing to term 6
I20250112 15:07:58.345643 21069 raft_consensus.cc:513] T ef90b704208140a3983ffe6105939319 P 2537bbbd1a9947ae9facd87567e3ccb9 [term 6 FOLLOWER]: Starting forced leader election with config: opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "2537bbbd1a9947ae9facd87567e3ccb9" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 33173 } } peers { permanent_uuid: "3710b74ed2044ce5995533df32e67351" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 38589 } } peers { permanent_uuid: "25286c485fa04c86a5d07c8f88643a47" member_type: VOTER last_known_addr { host: "127.17.60.198" port: 37089 } attrs { promote: false } }
I20250112 15:07:58.345817 21069 leader_election.cc:290] T ef90b704208140a3983ffe6105939319 P 2537bbbd1a9947ae9facd87567e3ccb9 [CANDIDATE]: Term 6 election: Requested vote from peers 3710b74ed2044ce5995533df32e67351 (127.17.60.195:38589), 25286c485fa04c86a5d07c8f88643a47 (127.17.60.198:37089)
I20250112 15:07:58.346002 21624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef90b704208140a3983ffe6105939319" candidate_uuid: "2537bbbd1a9947ae9facd87567e3ccb9" candidate_term: 6 candidate_status { last_received { term: 5 index: 32 } } ignore_live_leader: true dest_uuid: "25286c485fa04c86a5d07c8f88643a47"
I20250112 15:07:58.346002 21226 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef90b704208140a3983ffe6105939319" candidate_uuid: "2537bbbd1a9947ae9facd87567e3ccb9" candidate_term: 6 candidate_status { last_received { term: 5 index: 32 } } ignore_live_leader: true dest_uuid: "3710b74ed2044ce5995533df32e67351"
I20250112 15:07:58.346107 21624 raft_consensus.cc:3054] T ef90b704208140a3983ffe6105939319 P 25286c485fa04c86a5d07c8f88643a47 [term 5 FOLLOWER]: Advancing to term 6
I20250112 15:07:58.346108 21226 raft_consensus.cc:3049] T ef90b704208140a3983ffe6105939319 P 3710b74ed2044ce5995533df32e67351 [term 5 LEADER]: Stepping down as leader of term 5
I20250112 15:07:58.346243 21226 raft_consensus.cc:738] T ef90b704208140a3983ffe6105939319 P 3710b74ed2044ce5995533df32e67351 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 3710b74ed2044ce5995533df32e67351, State: Running, Role: LEADER
I20250112 15:07:58.346292 21226 consensus_queue.cc:260] T ef90b704208140a3983ffe6105939319 P 3710b74ed2044ce5995533df32e67351 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 32, Committed index: 32, Last appended: 5.32, Last appended by leader: 32, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "2537bbbd1a9947ae9facd87567e3ccb9" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 33173 } } peers { permanent_uuid: "3710b74ed2044ce5995533df32e67351" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 38589 } } peers { permanent_uuid: "25286c485fa04c86a5d07c8f88643a47" member_type: VOTER last_known_addr { host: "127.17.60.198" port: 37089 } attrs { promote: false } }
I20250112 15:07:58.346382 21226 raft_consensus.cc:3054] T ef90b704208140a3983ffe6105939319 P 3710b74ed2044ce5995533df32e67351 [term 5 FOLLOWER]: Advancing to term 6
I20250112 15:07:58.346917 21624 raft_consensus.cc:2463] T ef90b704208140a3983ffe6105939319 P 25286c485fa04c86a5d07c8f88643a47 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2537bbbd1a9947ae9facd87567e3ccb9 in term 6.
I20250112 15:07:58.347033 21226 raft_consensus.cc:2463] T ef90b704208140a3983ffe6105939319 P 3710b74ed2044ce5995533df32e67351 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2537bbbd1a9947ae9facd87567e3ccb9 in term 6.
I20250112 15:07:58.347132 21038 leader_election.cc:304] T ef90b704208140a3983ffe6105939319 P 2537bbbd1a9947ae9facd87567e3ccb9 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 25286c485fa04c86a5d07c8f88643a47, 2537bbbd1a9947ae9facd87567e3ccb9; no voters:
I20250112 15:07:58.347239 22154 raft_consensus.cc:2798] T ef90b704208140a3983ffe6105939319 P 2537bbbd1a9947ae9facd87567e3ccb9 [term 6 FOLLOWER]: Leader election won for term 6
I20250112 15:07:58.347332 22154 raft_consensus.cc:695] T ef90b704208140a3983ffe6105939319 P 2537bbbd1a9947ae9facd87567e3ccb9 [term 6 LEADER]: Becoming Leader. State: Replica: 2537bbbd1a9947ae9facd87567e3ccb9, State: Running, Role: LEADER
I20250112 15:07:58.347397 22154 consensus_queue.cc:237] T ef90b704208140a3983ffe6105939319 P 2537bbbd1a9947ae9facd87567e3ccb9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 32, Committed index: 32, Last appended: 5.32, Last appended by leader: 32, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "2537bbbd1a9947ae9facd87567e3ccb9" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 33173 } } peers { permanent_uuid: "3710b74ed2044ce5995533df32e67351" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 38589 } } peers { permanent_uuid: "25286c485fa04c86a5d07c8f88643a47" member_type: VOTER last_known_addr { host: "127.17.60.198" port: 37089 } attrs { promote: false } }
I20250112 15:07:58.347883 20970 catalog_manager.cc:5526] T ef90b704208140a3983ffe6105939319 P 2537bbbd1a9947ae9facd87567e3ccb9 reported cstate change: term changed from 5 to 6, leader changed from 3710b74ed2044ce5995533df32e67351 (127.17.60.195) to 2537bbbd1a9947ae9facd87567e3ccb9 (127.17.60.193). New cstate: current_term: 6 leader_uuid: "2537bbbd1a9947ae9facd87567e3ccb9" committed_config { opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "2537bbbd1a9947ae9facd87567e3ccb9" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 33173 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3710b74ed2044ce5995533df32e67351" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 38589 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "25286c485fa04c86a5d07c8f88643a47" member_type: VOTER last_known_addr { host: "127.17.60.198" port: 37089 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20250112 15:07:58.734958 21226 raft_consensus.cc:1270] T ef90b704208140a3983ffe6105939319 P 3710b74ed2044ce5995533df32e67351 [term 6 FOLLOWER]: Refusing update from remote peer 2537bbbd1a9947ae9facd87567e3ccb9: Log matching property violated. Preceding OpId in replica: term: 5 index: 32. Preceding OpId from leader: term: 6 index: 33. (index mismatch)
I20250112 15:07:58.735234 22154 consensus_queue.cc:1035] T ef90b704208140a3983ffe6105939319 P 2537bbbd1a9947ae9facd87567e3ccb9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3710b74ed2044ce5995533df32e67351" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 38589 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 33, Last known committed idx: 32, Time since last communication: 0.000s
I20250112 15:07:58.736510 21624 raft_consensus.cc:1270] T ef90b704208140a3983ffe6105939319 P 25286c485fa04c86a5d07c8f88643a47 [term 6 FOLLOWER]: Refusing update from remote peer 2537bbbd1a9947ae9facd87567e3ccb9: Log matching property violated. Preceding OpId in replica: term: 5 index: 32. Preceding OpId from leader: term: 6 index: 33. (index mismatch)
I20250112 15:07:58.736759 22239 consensus_queue.cc:1035] T ef90b704208140a3983ffe6105939319 P 2537bbbd1a9947ae9facd87567e3ccb9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "25286c485fa04c86a5d07c8f88643a47" member_type: VOTER last_known_addr { host: "127.17.60.198" port: 37089 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 33, Last known committed idx: 32, Time since last communication: 0.000s
/home/jenkins-slave/workspace/build_and_test_flaky/src/kudu/master/auto_rebalancer-test.cc:196: Failure
Expected equality of these values:
0
NumMovesScheduled(leader_idx, BalanceThreadType::LEADER_REBALANCE)
Which is: 1
/home/jenkins-slave/workspace/build_and_test_flaky/src/kudu/util/test_util.cc:401: Failure
Failed
Timed out waiting for assertion to pass.
/home/jenkins-slave/workspace/build_and_test_flaky/src/kudu/master/auto_rebalancer-test.cc:575: Failure
Expected: CheckNoLeaderMovesScheduled() doesn't generate new fatal failures in the current thread.
Actual: it does.
I20250112 15:07:58.841625 17651 tablet_server.cc:178] TabletServer@127.17.60.193:0 shutting down...
I20250112 15:07:58.846439 17651 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250112 15:07:58.846621 17651 tablet_replica.cc:331] T 1eb2b7e6edd34a7b83f9b918d3cdf2ed P 2537bbbd1a9947ae9facd87567e3ccb9: stopping tablet replica
I20250112 15:07:58.846704 17651 raft_consensus.cc:2238] T 1eb2b7e6edd34a7b83f9b918d3cdf2ed P 2537bbbd1a9947ae9facd87567e3ccb9 [term 3 LEADER]: Raft consensus shutting down.
I20250112 15:07:58.846781 17651 raft_consensus.cc:2267] T 1eb2b7e6edd34a7b83f9b918d3cdf2ed P 2537bbbd1a9947ae9facd87567e3ccb9 [term 3 FOLLOWER]: Raft consensus is shut down!
I20250112 15:07:58.846971 17651 tablet_replica.cc:331] T 16f16fcd5cf347e5853189dc4da55c7f P 2537bbbd1a9947ae9facd87567e3ccb9: stopping tablet replica
I20250112 15:07:58.847008 17651 raft_consensus.cc:2238] T 16f16fcd5cf347e5853189dc4da55c7f P 2537bbbd1a9947ae9facd87567e3ccb9 [term 2 LEADER]: Raft consensus shutting down.
I20250112 15:07:58.847054 17651 raft_consensus.cc:2267] T 16f16fcd5cf347e5853189dc4da55c7f P 2537bbbd1a9947ae9facd87567e3ccb9 [term 2 FOLLOWER]: Raft consensus is shut down!
I20250112 15:07:58.847194 17651 tablet_replica.cc:331] T 9d4bbba799324d7fbbac082520c65686 P 2537bbbd1a9947ae9facd87567e3ccb9: stopping tablet replica
I20250112 15:07:58.847234 17651 raft_consensus.cc:2238] T 9d4bbba799324d7fbbac082520c65686 P 2537bbbd1a9947ae9facd87567e3ccb9 [term 3 FOLLOWER]: Raft consensus shutting down.
I20250112 15:07:58.847280 17651 raft_consensus.cc:2267] T 9d4bbba799324d7fbbac082520c65686 P 2537bbbd1a9947ae9facd87567e3ccb9 [term 3 FOLLOWER]: Raft consensus is shut down!
I20250112 15:07:58.847402 17651 tablet_replica.cc:331] T 02c33d457b6b47138dd712850819b619 P 2537bbbd1a9947ae9facd87567e3ccb9: stopping tablet replica
I20250112 15:07:58.847433 17651 raft_consensus.cc:2238] T 02c33d457b6b47138dd712850819b619 P 2537bbbd1a9947ae9facd87567e3ccb9 [term 6 FOLLOWER]: Raft consensus shutting down.
I20250112 15:07:58.847453 17651 raft_consensus.cc:2267] T 02c33d457b6b47138dd712850819b619 P 2537bbbd1a9947ae9facd87567e3ccb9 [term 6 FOLLOWER]: Raft consensus is shut down!
I20250112 15:07:58.847628 17651 tablet_replica.cc:331] T 57e47b07fdcf434e96ca5dd0354e93b2 P 2537bbbd1a9947ae9facd87567e3ccb9: stopping tablet replica
I20250112 15:07:58.847674 17651 raft_consensus.cc:2238] T 57e47b07fdcf434e96ca5dd0354e93b2 P 2537bbbd1a9947ae9facd87567e3ccb9 [term 3 FOLLOWER]: Raft consensus shutting down.
I20250112 15:07:58.847716 17651 raft_consensus.cc:2267] T 57e47b07fdcf434e96ca5dd0354e93b2 P 2537bbbd1a9947ae9facd87567e3ccb9 [term 3 FOLLOWER]: Raft consensus is shut down!
I20250112 15:07:58.847831 17651 tablet_replica.cc:331] T ef90b704208140a3983ffe6105939319 P 2537bbbd1a9947ae9facd87567e3ccb9: stopping tablet replica
I20250112 15:07:58.847858 17651 raft_consensus.cc:2238] T ef90b704208140a3983ffe6105939319 P 2537bbbd1a9947ae9facd87567e3ccb9 [term 6 LEADER]: Raft consensus shutting down.
I20250112 15:07:58.847899 17651 raft_consensus.cc:2267] T ef90b704208140a3983ffe6105939319 P 2537bbbd1a9947ae9facd87567e3ccb9 [term 6 FOLLOWER]: Raft consensus is shut down!
I20250112 15:07:58.850806 17651 tablet_server.cc:195] TabletServer@127.17.60.193:0 shutdown complete.
I20250112 15:07:58.852217 17651 tablet_server.cc:178] TabletServer@127.17.60.194:0 shutting down...
I20250112 15:07:58.857746 17651 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250112 15:07:58.857895 17651 tablet_replica.cc:331] T 80cc505a04394c53a18c5beeded4655d P c561ec3374b94b95be0c155d80ff0bff: stopping tablet replica
I20250112 15:07:58.857997 17651 raft_consensus.cc:2238] T 80cc505a04394c53a18c5beeded4655d P c561ec3374b94b95be0c155d80ff0bff [term 3 FOLLOWER]: Raft consensus shutting down.
I20250112 15:07:58.858052 17651 raft_consensus.cc:2267] T 80cc505a04394c53a18c5beeded4655d P c561ec3374b94b95be0c155d80ff0bff [term 3 FOLLOWER]: Raft consensus is shut down!
I20250112 15:07:58.858240 17651 tablet_replica.cc:331] T 9d4bbba799324d7fbbac082520c65686 P c561ec3374b94b95be0c155d80ff0bff: stopping tablet replica
I20250112 15:07:58.858287 17651 raft_consensus.cc:2238] T 9d4bbba799324d7fbbac082520c65686 P c561ec3374b94b95be0c155d80ff0bff [term 3 LEADER]: Raft consensus shutting down.
I20250112 15:07:58.858350 17651 raft_consensus.cc:2267] T 9d4bbba799324d7fbbac082520c65686 P c561ec3374b94b95be0c155d80ff0bff [term 3 FOLLOWER]: Raft consensus is shut down!
I20250112 15:07:58.858486 17651 tablet_replica.cc:331] T 02c33d457b6b47138dd712850819b619 P c561ec3374b94b95be0c155d80ff0bff: stopping tablet replica
I20250112 15:07:58.858529 17651 raft_consensus.cc:2238] T 02c33d457b6b47138dd712850819b619 P c561ec3374b94b95be0c155d80ff0bff [term 6 LEADER]: Raft consensus shutting down.
I20250112 15:07:58.858577 17651 raft_consensus.cc:2267] T 02c33d457b6b47138dd712850819b619 P c561ec3374b94b95be0c155d80ff0bff [term 6 FOLLOWER]: Raft consensus is shut down!
I20250112 15:07:58.858723 17651 tablet_replica.cc:331] T 69c23627ab854000b360030c384dc966 P c561ec3374b94b95be0c155d80ff0bff: stopping tablet replica
I20250112 15:07:58.858768 17651 raft_consensus.cc:2238] T 69c23627ab854000b360030c384dc966 P c561ec3374b94b95be0c155d80ff0bff [term 4 FOLLOWER]: Raft consensus shutting down.
I20250112 15:07:58.858806 17651 raft_consensus.cc:2267] T 69c23627ab854000b360030c384dc966 P c561ec3374b94b95be0c155d80ff0bff [term 4 FOLLOWER]: Raft consensus is shut down!
I20250112 15:07:58.858963 17651 tablet_replica.cc:331] T bfaeae584ec04772b2d6fcd088022117 P c561ec3374b94b95be0c155d80ff0bff: stopping tablet replica
I20250112 15:07:58.859009 17651 raft_consensus.cc:2238] T bfaeae584ec04772b2d6fcd088022117 P c561ec3374b94b95be0c155d80ff0bff [term 4 FOLLOWER]: Raft consensus shutting down.
I20250112 15:07:58.859050 17651 raft_consensus.cc:2267] T bfaeae584ec04772b2d6fcd088022117 P c561ec3374b94b95be0c155d80ff0bff [term 4 FOLLOWER]: Raft consensus is shut down!
I20250112 15:07:58.859192 17651 tablet_replica.cc:331] T 1eb2b7e6edd34a7b83f9b918d3cdf2ed P c561ec3374b94b95be0c155d80ff0bff: stopping tablet replica
I20250112 15:07:58.859238 17651 raft_consensus.cc:2238] T 1eb2b7e6edd34a7b83f9b918d3cdf2ed P c561ec3374b94b95be0c155d80ff0bff [term 3 FOLLOWER]: Raft consensus shutting down.
I20250112 15:07:58.859278 17651 raft_consensus.cc:2267] T 1eb2b7e6edd34a7b83f9b918d3cdf2ed P c561ec3374b94b95be0c155d80ff0bff [term 3 FOLLOWER]: Raft consensus is shut down!
I20250112 15:07:58.871866 17651 tablet_server.cc:195] TabletServer@127.17.60.194:0 shutdown complete.
I20250112 15:07:58.874253 17651 tablet_server.cc:178] TabletServer@127.17.60.195:0 shutting down...
I20250112 15:07:58.879343 17651 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250112 15:07:58.879552 17651 tablet_replica.cc:331] T 5bcec60d3b974a9bb24aedfc738e1594 P 3710b74ed2044ce5995533df32e67351: stopping tablet replica
I20250112 15:07:58.879645 17651 raft_consensus.cc:2238] T 5bcec60d3b974a9bb24aedfc738e1594 P 3710b74ed2044ce5995533df32e67351 [term 2 FOLLOWER]: Raft consensus shutting down.
I20250112 15:07:58.879766 17651 raft_consensus.cc:2267] T 5bcec60d3b974a9bb24aedfc738e1594 P 3710b74ed2044ce5995533df32e67351 [term 2 FOLLOWER]: Raft consensus is shut down!
I20250112 15:07:58.879964 17651 tablet_replica.cc:331] T 9601bcda24ed42e7b93ab9740f1e271e P 3710b74ed2044ce5995533df32e67351: stopping tablet replica
I20250112 15:07:58.880016 17651 raft_consensus.cc:2238] T 9601bcda24ed42e7b93ab9740f1e271e P 3710b74ed2044ce5995533df32e67351 [term 6 LEADER]: Raft consensus shutting down.
I20250112 15:07:58.880088 17651 raft_consensus.cc:2267] T 9601bcda24ed42e7b93ab9740f1e271e P 3710b74ed2044ce5995533df32e67351 [term 6 FOLLOWER]: Raft consensus is shut down!
I20250112 15:07:58.880234 17651 tablet_replica.cc:331] T 80cc505a04394c53a18c5beeded4655d P 3710b74ed2044ce5995533df32e67351: stopping tablet replica
I20250112 15:07:58.880281 17651 raft_consensus.cc:2238] T 80cc505a04394c53a18c5beeded4655d P 3710b74ed2044ce5995533df32e67351 [term 3 FOLLOWER]: Raft consensus shutting down.
I20250112 15:07:58.880319 17651 raft_consensus.cc:2267] T 80cc505a04394c53a18c5beeded4655d P 3710b74ed2044ce5995533df32e67351 [term 3 FOLLOWER]: Raft consensus is shut down!
I20250112 15:07:58.880446 17651 tablet_replica.cc:331] T 02c33d457b6b47138dd712850819b619 P 3710b74ed2044ce5995533df32e67351: stopping tablet replica
I20250112 15:07:58.880491 17651 raft_consensus.cc:2238] T 02c33d457b6b47138dd712850819b619 P 3710b74ed2044ce5995533df32e67351 [term 6 FOLLOWER]: Raft consensus shutting down.
I20250112 15:07:58.880532 17651 raft_consensus.cc:2267] T 02c33d457b6b47138dd712850819b619 P 3710b74ed2044ce5995533df32e67351 [term 6 FOLLOWER]: Raft consensus is shut down!
I20250112 15:07:58.880697 17651 tablet_replica.cc:331] T ef90b704208140a3983ffe6105939319 P 3710b74ed2044ce5995533df32e67351: stopping tablet replica
I20250112 15:07:58.880741 17651 raft_consensus.cc:2238] T ef90b704208140a3983ffe6105939319 P 3710b74ed2044ce5995533df32e67351 [term 6 FOLLOWER]: Raft consensus shutting down.
I20250112 15:07:58.880779 17651 raft_consensus.cc:2267] T ef90b704208140a3983ffe6105939319 P 3710b74ed2044ce5995533df32e67351 [term 6 FOLLOWER]: Raft consensus is shut down!
I20250112 15:07:58.880990 17651 tablet_replica.cc:331] T 8c7dbaf285a04da9b4eec92a29ab7d6c P 3710b74ed2044ce5995533df32e67351: stopping tablet replica
I20250112 15:07:58.881047 17651 raft_consensus.cc:2238] T 8c7dbaf285a04da9b4eec92a29ab7d6c P 3710b74ed2044ce5995533df32e67351 [term 4 LEADER]: Raft consensus shutting down.
I20250112 15:07:58.881110 17651 raft_consensus.cc:2267] T 8c7dbaf285a04da9b4eec92a29ab7d6c P 3710b74ed2044ce5995533df32e67351 [term 4 FOLLOWER]: Raft consensus is shut down!
I20250112 15:07:58.883795 17651 tablet_server.cc:195] TabletServer@127.17.60.195:0 shutdown complete.
I20250112 15:07:58.885843 17651 tablet_server.cc:178] TabletServer@127.17.60.196:0 shutting down...
I20250112 15:07:58.889916 17651 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250112 15:07:58.890089 17651 tablet_replica.cc:331] T 57e47b07fdcf434e96ca5dd0354e93b2 P a2a5bba4ac1145d5bab94a69c9f4df40: stopping tablet replica
I20250112 15:07:58.890173 17651 raft_consensus.cc:2238] T 57e47b07fdcf434e96ca5dd0354e93b2 P a2a5bba4ac1145d5bab94a69c9f4df40 [term 3 FOLLOWER]: Raft consensus shutting down.
I20250112 15:07:58.890242 17651 raft_consensus.cc:2267] T 57e47b07fdcf434e96ca5dd0354e93b2 P a2a5bba4ac1145d5bab94a69c9f4df40 [term 3 FOLLOWER]: Raft consensus is shut down!
I20250112 15:07:58.890429 17651 tablet_replica.cc:331] T 8c7dbaf285a04da9b4eec92a29ab7d6c P a2a5bba4ac1145d5bab94a69c9f4df40: stopping tablet replica
I20250112 15:07:58.890477 17651 raft_consensus.cc:2238] T 8c7dbaf285a04da9b4eec92a29ab7d6c P a2a5bba4ac1145d5bab94a69c9f4df40 [term 4 FOLLOWER]: Raft consensus shutting down.
I20250112 15:07:58.890551 17651 raft_consensus.cc:2267] T 8c7dbaf285a04da9b4eec92a29ab7d6c P a2a5bba4ac1145d5bab94a69c9f4df40 [term 4 FOLLOWER]: Raft consensus is shut down!
I20250112 15:07:58.890681 17651 tablet_replica.cc:331] T 69c23627ab854000b360030c384dc966 P a2a5bba4ac1145d5bab94a69c9f4df40: stopping tablet replica
I20250112 15:07:58.890725 17651 raft_consensus.cc:2238] T 69c23627ab854000b360030c384dc966 P a2a5bba4ac1145d5bab94a69c9f4df40 [term 4 LEADER]: Raft consensus shutting down.
I20250112 15:07:58.890797 17651 raft_consensus.cc:2267] T 69c23627ab854000b360030c384dc966 P a2a5bba4ac1145d5bab94a69c9f4df40 [term 4 FOLLOWER]: Raft consensus is shut down!
I20250112 15:07:58.890929 17651 tablet_replica.cc:331] T bfaeae584ec04772b2d6fcd088022117 P a2a5bba4ac1145d5bab94a69c9f4df40: stopping tablet replica
I20250112 15:07:58.890985 17651 raft_consensus.cc:2238] T bfaeae584ec04772b2d6fcd088022117 P a2a5bba4ac1145d5bab94a69c9f4df40 [term 4 FOLLOWER]: Raft consensus shutting down.
I20250112 15:07:58.891037 17651 raft_consensus.cc:2267] T bfaeae584ec04772b2d6fcd088022117 P a2a5bba4ac1145d5bab94a69c9f4df40 [term 4 FOLLOWER]: Raft consensus is shut down!
I20250112 15:07:58.891160 17651 tablet_replica.cc:331] T 9601bcda24ed42e7b93ab9740f1e271e P a2a5bba4ac1145d5bab94a69c9f4df40: stopping tablet replica
I20250112 15:07:58.891216 17651 raft_consensus.cc:2238] T 9601bcda24ed42e7b93ab9740f1e271e P a2a5bba4ac1145d5bab94a69c9f4df40 [term 6 FOLLOWER]: Raft consensus shutting down.
I20250112 15:07:58.891250 17651 raft_consensus.cc:2267] T 9601bcda24ed42e7b93ab9740f1e271e P a2a5bba4ac1145d5bab94a69c9f4df40 [term 6 FOLLOWER]: Raft consensus is shut down!
I20250112 15:07:58.891364 17651 tablet_replica.cc:331] T 5bcec60d3b974a9bb24aedfc738e1594 P a2a5bba4ac1145d5bab94a69c9f4df40: stopping tablet replica
I20250112 15:07:58.891405 17651 raft_consensus.cc:2238] T 5bcec60d3b974a9bb24aedfc738e1594 P a2a5bba4ac1145d5bab94a69c9f4df40 [term 2 LEADER]: Raft consensus shutting down.
I20250112 15:07:58.891464 17651 raft_consensus.cc:2267] T 5bcec60d3b974a9bb24aedfc738e1594 P a2a5bba4ac1145d5bab94a69c9f4df40 [term 2 FOLLOWER]: Raft consensus is shut down!
I20250112 15:07:58.893436 17651 tablet_server.cc:195] TabletServer@127.17.60.196:0 shutdown complete.
I20250112 15:07:58.894577 17651 tablet_server.cc:178] TabletServer@127.17.60.197:0 shutting down...
I20250112 15:07:58.898963 17651 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250112 15:07:58.899155 17651 tablet_replica.cc:331] T 16f16fcd5cf347e5853189dc4da55c7f P b5e9178a7a83402e8c4a6943ab78589d: stopping tablet replica
I20250112 15:07:58.899217 17651 raft_consensus.cc:2238] T 16f16fcd5cf347e5853189dc4da55c7f P b5e9178a7a83402e8c4a6943ab78589d [term 2 FOLLOWER]: Raft consensus shutting down.
I20250112 15:07:58.899281 17651 raft_consensus.cc:2267] T 16f16fcd5cf347e5853189dc4da55c7f P b5e9178a7a83402e8c4a6943ab78589d [term 2 FOLLOWER]: Raft consensus is shut down!
I20250112 15:07:58.899433 17651 tablet_replica.cc:331] T 8c7dbaf285a04da9b4eec92a29ab7d6c P b5e9178a7a83402e8c4a6943ab78589d: stopping tablet replica
I20250112 15:07:58.899468 17651 raft_consensus.cc:2238] T 8c7dbaf285a04da9b4eec92a29ab7d6c P b5e9178a7a83402e8c4a6943ab78589d [term 4 FOLLOWER]: Raft consensus shutting down.
I20250112 15:07:58.899503 17651 raft_consensus.cc:2267] T 8c7dbaf285a04da9b4eec92a29ab7d6c P b5e9178a7a83402e8c4a6943ab78589d [term 4 FOLLOWER]: Raft consensus is shut down!
I20250112 15:07:58.899626 17651 tablet_replica.cc:331] T 9d4bbba799324d7fbbac082520c65686 P b5e9178a7a83402e8c4a6943ab78589d: stopping tablet replica
I20250112 15:07:58.899658 17651 raft_consensus.cc:2238] T 9d4bbba799324d7fbbac082520c65686 P b5e9178a7a83402e8c4a6943ab78589d [term 3 FOLLOWER]: Raft consensus shutting down.
I20250112 15:07:58.899691 17651 raft_consensus.cc:2267] T 9d4bbba799324d7fbbac082520c65686 P b5e9178a7a83402e8c4a6943ab78589d [term 3 FOLLOWER]: Raft consensus is shut down!
I20250112 15:07:58.899801 17651 tablet_replica.cc:331] T 5bcec60d3b974a9bb24aedfc738e1594 P b5e9178a7a83402e8c4a6943ab78589d: stopping tablet replica
I20250112 15:07:58.899835 17651 raft_consensus.cc:2238] T 5bcec60d3b974a9bb24aedfc738e1594 P b5e9178a7a83402e8c4a6943ab78589d [term 2 FOLLOWER]: Raft consensus shutting down.
I20250112 15:07:58.899868 17651 raft_consensus.cc:2267] T 5bcec60d3b974a9bb24aedfc738e1594 P b5e9178a7a83402e8c4a6943ab78589d [term 2 FOLLOWER]: Raft consensus is shut down!
I20250112 15:07:58.899984 17651 tablet_replica.cc:331] T 69c23627ab854000b360030c384dc966 P b5e9178a7a83402e8c4a6943ab78589d: stopping tablet replica
I20250112 15:07:58.900017 17651 raft_consensus.cc:2238] T 69c23627ab854000b360030c384dc966 P b5e9178a7a83402e8c4a6943ab78589d [term 4 FOLLOWER]: Raft consensus shutting down.
I20250112 15:07:58.900076 17651 raft_consensus.cc:2267] T 69c23627ab854000b360030c384dc966 P b5e9178a7a83402e8c4a6943ab78589d [term 4 FOLLOWER]: Raft consensus is shut down!
I20250112 15:07:58.900199 17651 tablet_replica.cc:331] T 57e47b07fdcf434e96ca5dd0354e93b2 P b5e9178a7a83402e8c4a6943ab78589d: stopping tablet replica
I20250112 15:07:58.900228 17651 raft_consensus.cc:2238] T 57e47b07fdcf434e96ca5dd0354e93b2 P b5e9178a7a83402e8c4a6943ab78589d [term 3 LEADER]: Raft consensus shutting down.
I20250112 15:07:58.900266 17651 raft_consensus.cc:2267] T 57e47b07fdcf434e96ca5dd0354e93b2 P b5e9178a7a83402e8c4a6943ab78589d [term 3 FOLLOWER]: Raft consensus is shut down!
I20250112 15:07:58.902693 17651 tablet_server.cc:195] TabletServer@127.17.60.197:0 shutdown complete.
I20250112 15:07:58.903947 17651 tablet_server.cc:178] TabletServer@127.17.60.198:0 shutting down...
I20250112 15:07:58.908077 17651 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250112 15:07:58.908205 17651 tablet_replica.cc:331] T ef90b704208140a3983ffe6105939319 P 25286c485fa04c86a5d07c8f88643a47: stopping tablet replica
I20250112 15:07:58.908268 17651 raft_consensus.cc:2238] T ef90b704208140a3983ffe6105939319 P 25286c485fa04c86a5d07c8f88643a47 [term 6 FOLLOWER]: Raft consensus shutting down.
I20250112 15:07:58.908337 17651 raft_consensus.cc:2267] T ef90b704208140a3983ffe6105939319 P 25286c485fa04c86a5d07c8f88643a47 [term 6 FOLLOWER]: Raft consensus is shut down!
I20250112 15:07:58.908571 17651 tablet_replica.cc:331] T 16f16fcd5cf347e5853189dc4da55c7f P 25286c485fa04c86a5d07c8f88643a47: stopping tablet replica
I20250112 15:07:58.908638 17651 raft_consensus.cc:2238] T 16f16fcd5cf347e5853189dc4da55c7f P 25286c485fa04c86a5d07c8f88643a47 [term 2 FOLLOWER]: Raft consensus shutting down.
I20250112 15:07:58.908705 17651 raft_consensus.cc:2267] T 16f16fcd5cf347e5853189dc4da55c7f P 25286c485fa04c86a5d07c8f88643a47 [term 2 FOLLOWER]: Raft consensus is shut down!
I20250112 15:07:58.908819 17651 tablet_replica.cc:331] T 9601bcda24ed42e7b93ab9740f1e271e P 25286c485fa04c86a5d07c8f88643a47: stopping tablet replica
I20250112 15:07:58.908854 17651 raft_consensus.cc:2238] T 9601bcda24ed42e7b93ab9740f1e271e P 25286c485fa04c86a5d07c8f88643a47 [term 6 FOLLOWER]: Raft consensus shutting down.
I20250112 15:07:58.908896 17651 raft_consensus.cc:2267] T 9601bcda24ed42e7b93ab9740f1e271e P 25286c485fa04c86a5d07c8f88643a47 [term 6 FOLLOWER]: Raft consensus is shut down!
I20250112 15:07:58.909018 17651 tablet_replica.cc:331] T bfaeae584ec04772b2d6fcd088022117 P 25286c485fa04c86a5d07c8f88643a47: stopping tablet replica
I20250112 15:07:58.909049 17651 raft_consensus.cc:2238] T bfaeae584ec04772b2d6fcd088022117 P 25286c485fa04c86a5d07c8f88643a47 [term 4 LEADER]: Raft consensus shutting down.
I20250112 15:07:58.909096 17651 raft_consensus.cc:2267] T bfaeae584ec04772b2d6fcd088022117 P 25286c485fa04c86a5d07c8f88643a47 [term 4 FOLLOWER]: Raft consensus is shut down!
I20250112 15:07:58.909211 17651 tablet_replica.cc:331] T 1eb2b7e6edd34a7b83f9b918d3cdf2ed P 25286c485fa04c86a5d07c8f88643a47: stopping tablet replica
I20250112 15:07:58.909245 17651 raft_consensus.cc:2238] T 1eb2b7e6edd34a7b83f9b918d3cdf2ed P 25286c485fa04c86a5d07c8f88643a47 [term 3 FOLLOWER]: Raft consensus shutting down.
I20250112 15:07:58.909282 17651 raft_consensus.cc:2267] T 1eb2b7e6edd34a7b83f9b918d3cdf2ed P 25286c485fa04c86a5d07c8f88643a47 [term 3 FOLLOWER]: Raft consensus is shut down!
I20250112 15:07:58.909390 17651 tablet_replica.cc:331] T 80cc505a04394c53a18c5beeded4655d P 25286c485fa04c86a5d07c8f88643a47: stopping tablet replica
I20250112 15:07:58.909420 17651 raft_consensus.cc:2238] T 80cc505a04394c53a18c5beeded4655d P 25286c485fa04c86a5d07c8f88643a47 [term 3 LEADER]: Raft consensus shutting down.
I20250112 15:07:58.909471 17651 raft_consensus.cc:2267] T 80cc505a04394c53a18c5beeded4655d P 25286c485fa04c86a5d07c8f88643a47 [term 3 FOLLOWER]: Raft consensus is shut down!
I20250112 15:07:58.918234 17651 tablet_server.cc:195] TabletServer@127.17.60.198:0 shutdown complete.
I20250112 15:07:58.919595 17651 master.cc:537] Master@127.17.60.254:42121 shutting down...
I20250112 15:07:58.922117 17651 raft_consensus.cc:2238] T 00000000000000000000000000000000 P 99889f1e9af44c26ae27b2aa189394be [term 1 LEADER]: Raft consensus shutting down.
I20250112 15:07:58.922180 17651 raft_consensus.cc:2267] T 00000000000000000000000000000000 P 99889f1e9af44c26ae27b2aa189394be [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 15:07:58.922215 17651 tablet_replica.cc:331] T 00000000000000000000000000000000 P 99889f1e9af44c26ae27b2aa189394be: stopping tablet replica
I20250112 15:07:58.934136 17651 master.cc:559] Master@127.17.60.254:42121 shutdown complete.
I20250112 15:07:58.936017 17651 test_util.cc:181] -----------------------------------------------
I20250112 15:07:58.936060 17651 test_util.cc:182] Had failures, leaving test files at /tmp/dist-test-task95dLSD/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1736694351070047-17651-0
[ FAILED ] AutoRebalancerTest.TestMaxMovesPerServer (34912 ms)
[ RUN ] AutoRebalancerTest.AutoRebalancingUnstableCluster
I20250112 15:07:58.936520 17651 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.17.60.254:38743
I20250112 15:07:58.936694 17651 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250112 15:07:58.938326 22243 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250112 15:07:58.939360 17651 server_base.cc:1034] running on GCE node
W20250112 15:07:58.940001 22246 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250112 15:07:58.940696 22244 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250112 15:07:58.940846 17651 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250112 15:07:58.940879 17651 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250112 15:07:58.940891 17651 hybrid_clock.cc:648] HybridClock initialized: now 1736694478940892 us; error 0 us; skew 500 ppm
I20250112 15:07:58.941540 17651 webserver.cc:458] Webserver started at http://127.17.60.254:33651/ using document root <none> and password file <none>
I20250112 15:07:58.941607 17651 fs_manager.cc:362] Metadata directory not provided
I20250112 15:07:58.941641 17651 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250112 15:07:58.941677 17651 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250112 15:07:58.941951 17651 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task95dLSD/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingUnstableCluster.1736694351070047-17651-0/minicluster-data/master-0-root/instance:
uuid: "80d1abdf9414405ebe25c3b7fd85b15d"
format_stamp: "Formatted at 2025-01-12 15:07:58 on dist-test-slave-b4nz"
I20250112 15:07:58.942797 17651 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20250112 15:07:58.943207 22251 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250112 15:07:58.943323 17651 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20250112 15:07:58.943362 17651 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task95dLSD/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingUnstableCluster.1736694351070047-17651-0/minicluster-data/master-0-root
uuid: "80d1abdf9414405ebe25c3b7fd85b15d"
format_stamp: "Formatted at 2025-01-12 15:07:58 on dist-test-slave-b4nz"
I20250112 15:07:58.943405 17651 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task95dLSD/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingUnstableCluster.1736694351070047-17651-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-task95dLSD/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingUnstableCluster.1736694351070047-17651-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-task95dLSD/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingUnstableCluster.1736694351070047-17651-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250112 15:07:58.951303 17651 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250112 15:07:58.951470 17651 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250112 15:07:58.953837 17651 rpc_server.cc:307] RPC server started. Bound to: 127.17.60.254:38743
I20250112 15:07:58.954422 22303 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.17.60.254:38743 every 8 connection(s)
I20250112 15:07:58.955401 22304 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 15:07:58.963397 22304 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 80d1abdf9414405ebe25c3b7fd85b15d: Bootstrap starting.
I20250112 15:07:58.963738 22304 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 80d1abdf9414405ebe25c3b7fd85b15d: Neither blocks nor log segments found. Creating new log.
I20250112 15:07:58.964274 22304 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 80d1abdf9414405ebe25c3b7fd85b15d: No bootstrap required, opened a new log
I20250112 15:07:58.964398 22304 raft_consensus.cc:357] T 00000000000000000000000000000000 P 80d1abdf9414405ebe25c3b7fd85b15d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "80d1abdf9414405ebe25c3b7fd85b15d" member_type: VOTER }
I20250112 15:07:58.964449 22304 raft_consensus.cc:383] T 00000000000000000000000000000000 P 80d1abdf9414405ebe25c3b7fd85b15d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 15:07:58.964469 22304 raft_consensus.cc:738] T 00000000000000000000000000000000 P 80d1abdf9414405ebe25c3b7fd85b15d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 80d1abdf9414405ebe25c3b7fd85b15d, State: Initialized, Role: FOLLOWER
I20250112 15:07:58.964517 22304 consensus_queue.cc:260] T 00000000000000000000000000000000 P 80d1abdf9414405ebe25c3b7fd85b15d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "80d1abdf9414405ebe25c3b7fd85b15d" member_type: VOTER }
I20250112 15:07:58.964577 22304 raft_consensus.cc:397] T 00000000000000000000000000000000 P 80d1abdf9414405ebe25c3b7fd85b15d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250112 15:07:58.964607 22304 raft_consensus.cc:491] T 00000000000000000000000000000000 P 80d1abdf9414405ebe25c3b7fd85b15d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250112 15:07:58.964637 22304 raft_consensus.cc:3054] T 00000000000000000000000000000000 P 80d1abdf9414405ebe25c3b7fd85b15d [term 0 FOLLOWER]: Advancing to term 1
I20250112 15:07:58.965144 22304 raft_consensus.cc:513] T 00000000000000000000000000000000 P 80d1abdf9414405ebe25c3b7fd85b15d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "80d1abdf9414405ebe25c3b7fd85b15d" member_type: VOTER }
I20250112 15:07:58.965199 22304 leader_election.cc:304] T 00000000000000000000000000000000 P 80d1abdf9414405ebe25c3b7fd85b15d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 80d1abdf9414405ebe25c3b7fd85b15d; no voters:
I20250112 15:07:58.965310 22304 leader_election.cc:290] T 00000000000000000000000000000000 P 80d1abdf9414405ebe25c3b7fd85b15d [CANDIDATE]: Term 1 election: Requested vote from peers
I20250112 15:07:58.965368 22307 raft_consensus.cc:2798] T 00000000000000000000000000000000 P 80d1abdf9414405ebe25c3b7fd85b15d [term 1 FOLLOWER]: Leader election won for term 1
I20250112 15:07:58.965442 22304 sys_catalog.cc:564] T 00000000000000000000000000000000 P 80d1abdf9414405ebe25c3b7fd85b15d [sys.catalog]: configured and running, proceeding with master startup.
I20250112 15:07:58.965508 22307 raft_consensus.cc:695] T 00000000000000000000000000000000 P 80d1abdf9414405ebe25c3b7fd85b15d [term 1 LEADER]: Becoming Leader. State: Replica: 80d1abdf9414405ebe25c3b7fd85b15d, State: Running, Role: LEADER
I20250112 15:07:58.965559 22307 consensus_queue.cc:237] T 00000000000000000000000000000000 P 80d1abdf9414405ebe25c3b7fd85b15d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "80d1abdf9414405ebe25c3b7fd85b15d" member_type: VOTER }
I20250112 15:07:58.965762 22308 sys_catalog.cc:455] T 00000000000000000000000000000000 P 80d1abdf9414405ebe25c3b7fd85b15d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "80d1abdf9414405ebe25c3b7fd85b15d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "80d1abdf9414405ebe25c3b7fd85b15d" member_type: VOTER } }
I20250112 15:07:58.965793 22309 sys_catalog.cc:455] T 00000000000000000000000000000000 P 80d1abdf9414405ebe25c3b7fd85b15d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 80d1abdf9414405ebe25c3b7fd85b15d. Latest consensus state: current_term: 1 leader_uuid: "80d1abdf9414405ebe25c3b7fd85b15d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "80d1abdf9414405ebe25c3b7fd85b15d" member_type: VOTER } }
I20250112 15:07:58.965837 22308 sys_catalog.cc:458] T 00000000000000000000000000000000 P 80d1abdf9414405ebe25c3b7fd85b15d [sys.catalog]: This master's current role is: LEADER
I20250112 15:07:58.965859 22309 sys_catalog.cc:458] T 00000000000000000000000000000000 P 80d1abdf9414405ebe25c3b7fd85b15d [sys.catalog]: This master's current role is: LEADER
I20250112 15:07:58.966095 22313 catalog_manager.cc:1476] Loading table and tablet metadata into memory...
I20250112 15:07:58.966229 22313 catalog_manager.cc:1485] Initializing Kudu cluster ID...
I20250112 15:07:58.966715 17651 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20250112 15:07:58.966751 22313 catalog_manager.cc:1348] Generated new cluster ID: 912d9555b8704c7eb3e2125c847a5973
I20250112 15:07:58.966800 22313 catalog_manager.cc:1496] Initializing Kudu internal certificate authority...
I20250112 15:07:58.974033 22313 catalog_manager.cc:1371] Generated new certificate authority record
I20250112 15:07:58.974432 22313 catalog_manager.cc:1505] Loading token signing keys...
I20250112 15:07:58.978363 22313 catalog_manager.cc:5899] T 00000000000000000000000000000000 P 80d1abdf9414405ebe25c3b7fd85b15d: Generated new TSK 0
I20250112 15:07:58.978447 22313 catalog_manager.cc:1515] Initializing in-progress tserver states...
I20250112 15:07:58.982367 17651 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250112 15:07:58.984145 22325 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250112 15:07:58.984544 22326 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250112 15:07:58.985186 22328 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250112 15:07:58.985875 17651 server_base.cc:1034] running on GCE node
I20250112 15:07:58.985986 17651 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250112 15:07:58.986016 17651 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250112 15:07:58.986028 17651 hybrid_clock.cc:648] HybridClock initialized: now 1736694478986029 us; error 0 us; skew 500 ppm
I20250112 15:07:58.986694 17651 webserver.cc:458] Webserver started at http://127.17.60.193:40013/ using document root <none> and password file <none>
I20250112 15:07:58.986778 17651 fs_manager.cc:362] Metadata directory not provided
I20250112 15:07:58.986824 17651 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250112 15:07:58.986874 17651 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250112 15:07:58.987175 17651 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task95dLSD/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingUnstableCluster.1736694351070047-17651-0/minicluster-data/ts-0-root/instance:
uuid: "a93555b72ffe4d3aa2f57067b8927f89"
format_stamp: "Formatted at 2025-01-12 15:07:58 on dist-test-slave-b4nz"
I20250112 15:07:58.988018 17651 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20250112 15:07:58.988399 22333 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250112 15:07:58.988520 17651 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20250112 15:07:58.988572 17651 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task95dLSD/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingUnstableCluster.1736694351070047-17651-0/minicluster-data/ts-0-root
uuid: "a93555b72ffe4d3aa2f57067b8927f89"
format_stamp: "Formatted at 2025-01-12 15:07:58 on dist-test-slave-b4nz"
I20250112 15:07:58.988617 17651 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task95dLSD/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingUnstableCluster.1736694351070047-17651-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-task95dLSD/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingUnstableCluster.1736694351070047-17651-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-task95dLSD/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingUnstableCluster.1736694351070047-17651-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250112 15:07:58.998005 17651 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250112 15:07:58.998227 17651 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250112 15:07:58.998359 17651 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250112 15:07:58.998538 17651 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250112 15:07:58.998564 17651 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250112 15:07:58.998595 17651 ts_tablet_manager.cc:610] Registered 0 tablets
I20250112 15:07:58.998620 17651 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250112 15:07:59.001276 17651 rpc_server.cc:307] RPC server started. Bound to: 127.17.60.193:36981
I20250112 15:07:59.001318 22396 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.17.60.193:36981 every 8 connection(s)
I20250112 15:07:59.001639 17651 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250112 15:07:59.003340 22402 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250112 15:07:59.003748 22401 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250112 15:07:59.004308 22404 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250112 15:07:59.004645 22397 heartbeater.cc:346] Connected to a master server at 127.17.60.254:38743
I20250112 15:07:59.004700 22397 heartbeater.cc:463] Registering TS with master...
I20250112 15:07:59.004796 22397 heartbeater.cc:510] Master 127.17.60.254:38743 requested a full tablet report, sending...
I20250112 15:07:59.004979 17651 server_base.cc:1034] running on GCE node
I20250112 15:07:59.005115 22268 ts_manager.cc:194] Registered new tserver with Master: a93555b72ffe4d3aa2f57067b8927f89 (127.17.60.193:36981)
I20250112 15:07:59.005124 17651 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250112 15:07:59.005173 17651 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250112 15:07:59.005189 17651 hybrid_clock.cc:648] HybridClock initialized: now 1736694479005189 us; error 0 us; skew 500 ppm
I20250112 15:07:59.005810 17651 webserver.cc:458] Webserver started at http://127.17.60.194:37307/ using document root <none> and password file <none>
I20250112 15:07:59.005874 22268 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:59772
I20250112 15:07:59.005878 17651 fs_manager.cc:362] Metadata directory not provided
I20250112 15:07:59.005955 17651 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250112 15:07:59.006001 17651 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250112 15:07:59.006285 17651 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task95dLSD/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingUnstableCluster.1736694351070047-17651-0/minicluster-data/ts-1-root/instance:
uuid: "d2e717f485ad442c8bfccf44cf80aa3e"
format_stamp: "Formatted at 2025-01-12 15:07:59 on dist-test-slave-b4nz"
I20250112 15:07:59.007129 17651 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20250112 15:07:59.007550 22409 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250112 15:07:59.007678 17651 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20250112 15:07:59.007730 17651 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task95dLSD/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingUnstableCluster.1736694351070047-17651-0/minicluster-data/ts-1-root
uuid: "d2e717f485ad442c8bfccf44cf80aa3e"
format_stamp: "Formatted at 2025-01-12 15:07:59 on dist-test-slave-b4nz"
I20250112 15:07:59.007773 17651 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task95dLSD/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingUnstableCluster.1736694351070047-17651-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-task95dLSD/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingUnstableCluster.1736694351070047-17651-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-task95dLSD/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingUnstableCluster.1736694351070047-17651-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250112 15:07:59.051808 17651 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250112 15:07:59.052055 17651 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250112 15:07:59.052207 17651 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250112 15:07:59.052402 17651 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250112 15:07:59.052428 17651 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250112 15:07:59.052459 17651 ts_tablet_manager.cc:610] Registered 0 tablets
I20250112 15:07:59.052488 17651 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250112 15:07:59.055099 17651 rpc_server.cc:307] RPC server started. Bound to: 127.17.60.194:40357
I20250112 15:07:59.055174 22472 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.17.60.194:40357 every 8 connection(s)
I20250112 15:07:59.055516 17651 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250112 15:07:59.056953 22476 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250112 15:07:59.057715 22477 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250112 15:07:59.058403 22479 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250112 15:07:59.058511 22473 heartbeater.cc:346] Connected to a master server at 127.17.60.254:38743
I20250112 15:07:59.058560 22473 heartbeater.cc:463] Registering TS with master...
I20250112 15:07:59.058640 22473 heartbeater.cc:510] Master 127.17.60.254:38743 requested a full tablet report, sending...
I20250112 15:07:59.058832 22268 ts_manager.cc:194] Registered new tserver with Master: d2e717f485ad442c8bfccf44cf80aa3e (127.17.60.194:40357)
I20250112 15:07:59.059176 22268 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:59774
I20250112 15:07:59.059486 17651 server_base.cc:1034] running on GCE node
I20250112 15:07:59.059602 17651 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250112 15:07:59.059630 17651 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250112 15:07:59.059643 17651 hybrid_clock.cc:648] HybridClock initialized: now 1736694479059644 us; error 0 us; skew 500 ppm
I20250112 15:07:59.060340 17651 webserver.cc:458] Webserver started at http://127.17.60.195:44099/ using document root <none> and password file <none>
I20250112 15:07:59.060408 17651 fs_manager.cc:362] Metadata directory not provided
I20250112 15:07:59.060444 17651 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250112 15:07:59.060478 17651 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250112 15:07:59.060752 17651 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task95dLSD/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingUnstableCluster.1736694351070047-17651-0/minicluster-data/ts-2-root/instance:
uuid: "74f47306030d436f90298b8a17297d29"
format_stamp: "Formatted at 2025-01-12 15:07:59 on dist-test-slave-b4nz"
I20250112 15:07:59.061542 17651 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20250112 15:07:59.061949 22484 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250112 15:07:59.062079 17651 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20250112 15:07:59.062122 17651 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task95dLSD/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingUnstableCluster.1736694351070047-17651-0/minicluster-data/ts-2-root
uuid: "74f47306030d436f90298b8a17297d29"
format_stamp: "Formatted at 2025-01-12 15:07:59 on dist-test-slave-b4nz"
I20250112 15:07:59.062168 17651 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task95dLSD/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingUnstableCluster.1736694351070047-17651-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-task95dLSD/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingUnstableCluster.1736694351070047-17651-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-task95dLSD/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingUnstableCluster.1736694351070047-17651-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250112 15:07:59.093349 17651 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250112 15:07:59.093576 17651 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250112 15:07:59.093734 17651 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250112 15:07:59.093941 17651 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250112 15:07:59.093966 17651 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250112 15:07:59.093997 17651 ts_tablet_manager.cc:610] Registered 0 tablets
I20250112 15:07:59.094022 17651 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250112 15:07:59.096830 17651 rpc_server.cc:307] RPC server started. Bound to: 127.17.60.195:34175
I20250112 15:07:59.096909 22547 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.17.60.195:34175 every 8 connection(s)
I20250112 15:07:59.099759 22548 heartbeater.cc:346] Connected to a master server at 127.17.60.254:38743
I20250112 15:07:59.099813 22548 heartbeater.cc:463] Registering TS with master...
I20250112 15:07:59.099907 22548 heartbeater.cc:510] Master 127.17.60.254:38743 requested a full tablet report, sending...
I20250112 15:07:59.100131 22268 ts_manager.cc:194] Registered new tserver with Master: 74f47306030d436f90298b8a17297d29 (127.17.60.195:34175)
I20250112 15:07:59.100335 17651 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.003314279s
I20250112 15:07:59.100518 22268 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:59780
I20250112 15:08:00.006495 22397 heartbeater.cc:502] Master 127.17.60.254:38743 was elected leader, sending a full tablet report...
I20250112 15:08:00.059648 22473 heartbeater.cc:502] Master 127.17.60.254:38743 was elected leader, sending a full tablet report...
I20250112 15:08:00.101058 22548 heartbeater.cc:502] Master 127.17.60.254:38743 was elected leader, sending a full tablet report...
I20250112 15:08:00.125840 17651 test_util.cc:274] Using random seed: -1315815539
I20250112 15:08:00.130659 22268 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50510:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "\004\001\000\252\252\252*\004\001\000TUUU""\004\001\000\252\252\252*\004\001\000TUUU"
indirect_data: """"
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 15:08:00.130800 22268 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-workload in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 15:08:00.135597 22512 tablet_service.cc:1467] Processing CreateTablet for tablet 50c711f4e65148faa8062f3ae6d87990 (DEFAULT_TABLE table=test-workload [id=cf7476fdd897444a8f48782f82fbc535]), partition=RANGE (key) PARTITION VALUES < 715827882
I20250112 15:08:00.135699 22361 tablet_service.cc:1467] Processing CreateTablet for tablet 50c711f4e65148faa8062f3ae6d87990 (DEFAULT_TABLE table=test-workload [id=cf7476fdd897444a8f48782f82fbc535]), partition=RANGE (key) PARTITION VALUES < 715827882
I20250112 15:08:00.135732 22512 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 50c711f4e65148faa8062f3ae6d87990. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 15:08:00.135782 22361 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 50c711f4e65148faa8062f3ae6d87990. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 15:08:00.135733 22359 tablet_service.cc:1467] Processing CreateTablet for tablet 68bb4bb68a93481cb8a3406f5f7d0b91 (DEFAULT_TABLE table=test-workload [id=cf7476fdd897444a8f48782f82fbc535]), partition=RANGE (key) PARTITION 1431655764 <= VALUES
I20250112 15:08:00.135955 22359 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 68bb4bb68a93481cb8a3406f5f7d0b91. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 15:08:00.136126 22437 tablet_service.cc:1467] Processing CreateTablet for tablet 50c711f4e65148faa8062f3ae6d87990 (DEFAULT_TABLE table=test-workload [id=cf7476fdd897444a8f48782f82fbc535]), partition=RANGE (key) PARTITION VALUES < 715827882
I20250112 15:08:00.137215 22568 tablet_bootstrap.cc:492] T 68bb4bb68a93481cb8a3406f5f7d0b91 P a93555b72ffe4d3aa2f57067b8927f89: Bootstrap starting.
I20250112 15:08:00.137271 22435 tablet_service.cc:1467] Processing CreateTablet for tablet 68bb4bb68a93481cb8a3406f5f7d0b91 (DEFAULT_TABLE table=test-workload [id=cf7476fdd897444a8f48782f82fbc535]), partition=RANGE (key) PARTITION 1431655764 <= VALUES
I20250112 15:08:00.137282 22569 tablet_bootstrap.cc:492] T 50c711f4e65148faa8062f3ae6d87990 P 74f47306030d436f90298b8a17297d29: Bootstrap starting.
I20250112 15:08:00.137352 22510 tablet_service.cc:1467] Processing CreateTablet for tablet 68bb4bb68a93481cb8a3406f5f7d0b91 (DEFAULT_TABLE table=test-workload [id=cf7476fdd897444a8f48782f82fbc535]), partition=RANGE (key) PARTITION 1431655764 <= VALUES
I20250112 15:08:00.137370 22435 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 68bb4bb68a93481cb8a3406f5f7d0b91. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 15:08:00.137434 22510 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 68bb4bb68a93481cb8a3406f5f7d0b91. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 15:08:00.137626 22568 tablet_bootstrap.cc:654] T 68bb4bb68a93481cb8a3406f5f7d0b91 P a93555b72ffe4d3aa2f57067b8927f89: Neither blocks nor log segments found. Creating new log.
I20250112 15:08:00.138015 22569 tablet_bootstrap.cc:654] T 50c711f4e65148faa8062f3ae6d87990 P 74f47306030d436f90298b8a17297d29: Neither blocks nor log segments found. Creating new log.
I20250112 15:08:00.138568 22572 tablet_bootstrap.cc:492] T 68bb4bb68a93481cb8a3406f5f7d0b91 P d2e717f485ad442c8bfccf44cf80aa3e: Bootstrap starting.
I20250112 15:08:00.138640 22569 tablet_bootstrap.cc:492] T 50c711f4e65148faa8062f3ae6d87990 P 74f47306030d436f90298b8a17297d29: No bootstrap required, opened a new log
I20250112 15:08:00.138687 22569 ts_tablet_manager.cc:1397] T 50c711f4e65148faa8062f3ae6d87990 P 74f47306030d436f90298b8a17297d29: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20250112 15:08:00.137229 22436 tablet_service.cc:1467] Processing CreateTablet for tablet 8b7046b625f84ac3b2fe6d67946fe6bf (DEFAULT_TABLE table=test-workload [id=cf7476fdd897444a8f48782f82fbc535]), partition=RANGE (key) PARTITION 715827882 <= VALUES < 1431655764
I20250112 15:08:00.138849 22569 raft_consensus.cc:357] T 50c711f4e65148faa8062f3ae6d87990 P 74f47306030d436f90298b8a17297d29 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74f47306030d436f90298b8a17297d29" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 34175 } } peers { permanent_uuid: "a93555b72ffe4d3aa2f57067b8927f89" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 36981 } } peers { permanent_uuid: "d2e717f485ad442c8bfccf44cf80aa3e" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 40357 } }
I20250112 15:08:00.138921 22569 raft_consensus.cc:383] T 50c711f4e65148faa8062f3ae6d87990 P 74f47306030d436f90298b8a17297d29 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 15:08:00.138955 22569 raft_consensus.cc:738] T 50c711f4e65148faa8062f3ae6d87990 P 74f47306030d436f90298b8a17297d29 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 74f47306030d436f90298b8a17297d29, State: Initialized, Role: FOLLOWER
I20250112 15:08:00.135733 22360 tablet_service.cc:1467] Processing CreateTablet for tablet 8b7046b625f84ac3b2fe6d67946fe6bf (DEFAULT_TABLE table=test-workload [id=cf7476fdd897444a8f48782f82fbc535]), partition=RANGE (key) PARTITION 715827882 <= VALUES < 1431655764
I20250112 15:08:00.139052 22572 tablet_bootstrap.cc:654] T 68bb4bb68a93481cb8a3406f5f7d0b91 P d2e717f485ad442c8bfccf44cf80aa3e: Neither blocks nor log segments found. Creating new log.
I20250112 15:08:00.139065 22569 consensus_queue.cc:260] T 50c711f4e65148faa8062f3ae6d87990 P 74f47306030d436f90298b8a17297d29 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74f47306030d436f90298b8a17297d29" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 34175 } } peers { permanent_uuid: "a93555b72ffe4d3aa2f57067b8927f89" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 36981 } } peers { permanent_uuid: "d2e717f485ad442c8bfccf44cf80aa3e" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 40357 } }
I20250112 15:08:00.137308 22511 tablet_service.cc:1467] Processing CreateTablet for tablet 8b7046b625f84ac3b2fe6d67946fe6bf (DEFAULT_TABLE table=test-workload [id=cf7476fdd897444a8f48782f82fbc535]), partition=RANGE (key) PARTITION 715827882 <= VALUES < 1431655764
I20250112 15:08:00.138895 22436 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8b7046b625f84ac3b2fe6d67946fe6bf. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 15:08:00.139209 22511 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8b7046b625f84ac3b2fe6d67946fe6bf. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 15:08:00.139273 22569 ts_tablet_manager.cc:1428] T 50c711f4e65148faa8062f3ae6d87990 P 74f47306030d436f90298b8a17297d29: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20250112 15:08:00.139356 22569 tablet_bootstrap.cc:492] T 68bb4bb68a93481cb8a3406f5f7d0b91 P 74f47306030d436f90298b8a17297d29: Bootstrap starting.
I20250112 15:08:00.139418 22437 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 50c711f4e65148faa8062f3ae6d87990. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 15:08:00.139722 22569 tablet_bootstrap.cc:654] T 68bb4bb68a93481cb8a3406f5f7d0b91 P 74f47306030d436f90298b8a17297d29: Neither blocks nor log segments found. Creating new log.
I20250112 15:08:00.140429 22569 tablet_bootstrap.cc:492] T 68bb4bb68a93481cb8a3406f5f7d0b91 P 74f47306030d436f90298b8a17297d29: No bootstrap required, opened a new log
I20250112 15:08:00.140470 22569 ts_tablet_manager.cc:1397] T 68bb4bb68a93481cb8a3406f5f7d0b91 P 74f47306030d436f90298b8a17297d29: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20250112 15:08:00.140632 22569 raft_consensus.cc:357] T 68bb4bb68a93481cb8a3406f5f7d0b91 P 74f47306030d436f90298b8a17297d29 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a93555b72ffe4d3aa2f57067b8927f89" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 36981 } } peers { permanent_uuid: "d2e717f485ad442c8bfccf44cf80aa3e" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 40357 } } peers { permanent_uuid: "74f47306030d436f90298b8a17297d29" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 34175 } }
I20250112 15:08:00.140695 22569 raft_consensus.cc:383] T 68bb4bb68a93481cb8a3406f5f7d0b91 P 74f47306030d436f90298b8a17297d29 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 15:08:00.140722 22569 raft_consensus.cc:738] T 68bb4bb68a93481cb8a3406f5f7d0b91 P 74f47306030d436f90298b8a17297d29 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 74f47306030d436f90298b8a17297d29, State: Initialized, Role: FOLLOWER
I20250112 15:08:00.140767 22569 consensus_queue.cc:260] T 68bb4bb68a93481cb8a3406f5f7d0b91 P 74f47306030d436f90298b8a17297d29 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a93555b72ffe4d3aa2f57067b8927f89" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 36981 } } peers { permanent_uuid: "d2e717f485ad442c8bfccf44cf80aa3e" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 40357 } } peers { permanent_uuid: "74f47306030d436f90298b8a17297d29" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 34175 } }
I20250112 15:08:00.140832 22573 raft_consensus.cc:491] T 68bb4bb68a93481cb8a3406f5f7d0b91 P 74f47306030d436f90298b8a17297d29 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 15:08:00.140897 22573 raft_consensus.cc:513] T 68bb4bb68a93481cb8a3406f5f7d0b91 P 74f47306030d436f90298b8a17297d29 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a93555b72ffe4d3aa2f57067b8927f89" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 36981 } } peers { permanent_uuid: "d2e717f485ad442c8bfccf44cf80aa3e" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 40357 } } peers { permanent_uuid: "74f47306030d436f90298b8a17297d29" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 34175 } }
I20250112 15:08:00.140933 22569 ts_tablet_manager.cc:1428] T 68bb4bb68a93481cb8a3406f5f7d0b91 P 74f47306030d436f90298b8a17297d29: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20250112 15:08:00.139107 22360 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8b7046b625f84ac3b2fe6d67946fe6bf. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 15:08:00.141079 22573 leader_election.cc:290] T 68bb4bb68a93481cb8a3406f5f7d0b91 P 74f47306030d436f90298b8a17297d29 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a93555b72ffe4d3aa2f57067b8927f89 (127.17.60.193:36981), d2e717f485ad442c8bfccf44cf80aa3e (127.17.60.194:40357)
I20250112 15:08:00.141047 22569 tablet_bootstrap.cc:492] T 8b7046b625f84ac3b2fe6d67946fe6bf P 74f47306030d436f90298b8a17297d29: Bootstrap starting.
I20250112 15:08:00.141805 22569 tablet_bootstrap.cc:654] T 8b7046b625f84ac3b2fe6d67946fe6bf P 74f47306030d436f90298b8a17297d29: Neither blocks nor log segments found. Creating new log.
I20250112 15:08:00.142289 22572 tablet_bootstrap.cc:492] T 68bb4bb68a93481cb8a3406f5f7d0b91 P d2e717f485ad442c8bfccf44cf80aa3e: No bootstrap required, opened a new log
I20250112 15:08:00.142345 22572 ts_tablet_manager.cc:1397] T 68bb4bb68a93481cb8a3406f5f7d0b91 P d2e717f485ad442c8bfccf44cf80aa3e: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20250112 15:08:00.142366 22569 tablet_bootstrap.cc:492] T 8b7046b625f84ac3b2fe6d67946fe6bf P 74f47306030d436f90298b8a17297d29: No bootstrap required, opened a new log
I20250112 15:08:00.142405 22569 ts_tablet_manager.cc:1397] T 8b7046b625f84ac3b2fe6d67946fe6bf P 74f47306030d436f90298b8a17297d29: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20250112 15:08:00.142477 22572 raft_consensus.cc:357] T 68bb4bb68a93481cb8a3406f5f7d0b91 P d2e717f485ad442c8bfccf44cf80aa3e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a93555b72ffe4d3aa2f57067b8927f89" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 36981 } } peers { permanent_uuid: "d2e717f485ad442c8bfccf44cf80aa3e" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 40357 } } peers { permanent_uuid: "74f47306030d436f90298b8a17297d29" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 34175 } }
I20250112 15:08:00.142524 22572 raft_consensus.cc:383] T 68bb4bb68a93481cb8a3406f5f7d0b91 P d2e717f485ad442c8bfccf44cf80aa3e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 15:08:00.142546 22572 raft_consensus.cc:738] T 68bb4bb68a93481cb8a3406f5f7d0b91 P d2e717f485ad442c8bfccf44cf80aa3e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d2e717f485ad442c8bfccf44cf80aa3e, State: Initialized, Role: FOLLOWER
I20250112 15:08:00.142537 22569 raft_consensus.cc:357] T 8b7046b625f84ac3b2fe6d67946fe6bf P 74f47306030d436f90298b8a17297d29 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a93555b72ffe4d3aa2f57067b8927f89" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 36981 } } peers { permanent_uuid: "d2e717f485ad442c8bfccf44cf80aa3e" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 40357 } } peers { permanent_uuid: "74f47306030d436f90298b8a17297d29" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 34175 } }
I20250112 15:08:00.142596 22569 raft_consensus.cc:383] T 8b7046b625f84ac3b2fe6d67946fe6bf P 74f47306030d436f90298b8a17297d29 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 15:08:00.142612 22572 consensus_queue.cc:260] T 68bb4bb68a93481cb8a3406f5f7d0b91 P d2e717f485ad442c8bfccf44cf80aa3e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a93555b72ffe4d3aa2f57067b8927f89" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 36981 } } peers { permanent_uuid: "d2e717f485ad442c8bfccf44cf80aa3e" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 40357 } } peers { permanent_uuid: "74f47306030d436f90298b8a17297d29" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 34175 } }
I20250112 15:08:00.142771 22572 ts_tablet_manager.cc:1428] T 68bb4bb68a93481cb8a3406f5f7d0b91 P d2e717f485ad442c8bfccf44cf80aa3e: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20250112 15:08:00.142832 22572 tablet_bootstrap.cc:492] T 50c711f4e65148faa8062f3ae6d87990 P d2e717f485ad442c8bfccf44cf80aa3e: Bootstrap starting.
I20250112 15:08:00.143205 22572 tablet_bootstrap.cc:654] T 50c711f4e65148faa8062f3ae6d87990 P d2e717f485ad442c8bfccf44cf80aa3e: Neither blocks nor log segments found. Creating new log.
I20250112 15:08:00.144059 22572 tablet_bootstrap.cc:492] T 50c711f4e65148faa8062f3ae6d87990 P d2e717f485ad442c8bfccf44cf80aa3e: No bootstrap required, opened a new log
I20250112 15:08:00.144107 22572 ts_tablet_manager.cc:1397] T 50c711f4e65148faa8062f3ae6d87990 P d2e717f485ad442c8bfccf44cf80aa3e: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20250112 15:08:00.144241 22572 raft_consensus.cc:357] T 50c711f4e65148faa8062f3ae6d87990 P d2e717f485ad442c8bfccf44cf80aa3e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74f47306030d436f90298b8a17297d29" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 34175 } } peers { permanent_uuid: "a93555b72ffe4d3aa2f57067b8927f89" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 36981 } } peers { permanent_uuid: "d2e717f485ad442c8bfccf44cf80aa3e" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 40357 } }
I20250112 15:08:00.144294 22572 raft_consensus.cc:383] T 50c711f4e65148faa8062f3ae6d87990 P d2e717f485ad442c8bfccf44cf80aa3e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 15:08:00.144312 22572 raft_consensus.cc:738] T 50c711f4e65148faa8062f3ae6d87990 P d2e717f485ad442c8bfccf44cf80aa3e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d2e717f485ad442c8bfccf44cf80aa3e, State: Initialized, Role: FOLLOWER
I20250112 15:08:00.144354 22572 consensus_queue.cc:260] T 50c711f4e65148faa8062f3ae6d87990 P d2e717f485ad442c8bfccf44cf80aa3e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74f47306030d436f90298b8a17297d29" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 34175 } } peers { permanent_uuid: "a93555b72ffe4d3aa2f57067b8927f89" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 36981 } } peers { permanent_uuid: "d2e717f485ad442c8bfccf44cf80aa3e" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 40357 } }
I20250112 15:08:00.144436 22572 ts_tablet_manager.cc:1428] T 50c711f4e65148faa8062f3ae6d87990 P d2e717f485ad442c8bfccf44cf80aa3e: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20250112 15:08:00.144503 22572 tablet_bootstrap.cc:492] T 8b7046b625f84ac3b2fe6d67946fe6bf P d2e717f485ad442c8bfccf44cf80aa3e: Bootstrap starting.
I20250112 15:08:00.142618 22569 raft_consensus.cc:738] T 8b7046b625f84ac3b2fe6d67946fe6bf P 74f47306030d436f90298b8a17297d29 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 74f47306030d436f90298b8a17297d29, State: Initialized, Role: FOLLOWER
I20250112 15:08:00.144833 22569 consensus_queue.cc:260] T 8b7046b625f84ac3b2fe6d67946fe6bf P 74f47306030d436f90298b8a17297d29 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a93555b72ffe4d3aa2f57067b8927f89" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 36981 } } peers { permanent_uuid: "d2e717f485ad442c8bfccf44cf80aa3e" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 40357 } } peers { permanent_uuid: "74f47306030d436f90298b8a17297d29" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 34175 } }
I20250112 15:08:00.145001 22569 ts_tablet_manager.cc:1428] T 8b7046b625f84ac3b2fe6d67946fe6bf P 74f47306030d436f90298b8a17297d29: Time spent starting tablet: real 0.003s user 0.000s sys 0.001s
I20250112 15:08:00.144901 22572 tablet_bootstrap.cc:654] T 8b7046b625f84ac3b2fe6d67946fe6bf P d2e717f485ad442c8bfccf44cf80aa3e: Neither blocks nor log segments found. Creating new log.
I20250112 15:08:00.145495 22568 tablet_bootstrap.cc:492] T 68bb4bb68a93481cb8a3406f5f7d0b91 P a93555b72ffe4d3aa2f57067b8927f89: No bootstrap required, opened a new log
I20250112 15:08:00.145538 22568 ts_tablet_manager.cc:1397] T 68bb4bb68a93481cb8a3406f5f7d0b91 P a93555b72ffe4d3aa2f57067b8927f89: Time spent bootstrapping tablet: real 0.008s user 0.001s sys 0.000s
I20250112 15:08:00.145565 22371 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "68bb4bb68a93481cb8a3406f5f7d0b91" candidate_uuid: "74f47306030d436f90298b8a17297d29" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a93555b72ffe4d3aa2f57067b8927f89" is_pre_election: true
I20250112 15:08:00.145658 22568 raft_consensus.cc:357] T 68bb4bb68a93481cb8a3406f5f7d0b91 P a93555b72ffe4d3aa2f57067b8927f89 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a93555b72ffe4d3aa2f57067b8927f89" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 36981 } } peers { permanent_uuid: "d2e717f485ad442c8bfccf44cf80aa3e" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 40357 } } peers { permanent_uuid: "74f47306030d436f90298b8a17297d29" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 34175 } }
I20250112 15:08:00.145704 22568 raft_consensus.cc:383] T 68bb4bb68a93481cb8a3406f5f7d0b91 P a93555b72ffe4d3aa2f57067b8927f89 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 15:08:00.145723 22568 raft_consensus.cc:738] T 68bb4bb68a93481cb8a3406f5f7d0b91 P a93555b72ffe4d3aa2f57067b8927f89 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a93555b72ffe4d3aa2f57067b8927f89, State: Initialized, Role: FOLLOWER
W20250112 15:08:00.145740 22488 leader_election.cc:343] T 68bb4bb68a93481cb8a3406f5f7d0b91 P 74f47306030d436f90298b8a17297d29 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a93555b72ffe4d3aa2f57067b8927f89 (127.17.60.193:36981): Illegal state: must be running to vote when last-logged opid is not known
I20250112 15:08:00.145766 22568 consensus_queue.cc:260] T 68bb4bb68a93481cb8a3406f5f7d0b91 P a93555b72ffe4d3aa2f57067b8927f89 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a93555b72ffe4d3aa2f57067b8927f89" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 36981 } } peers { permanent_uuid: "d2e717f485ad442c8bfccf44cf80aa3e" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 40357 } } peers { permanent_uuid: "74f47306030d436f90298b8a17297d29" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 34175 } }
I20250112 15:08:00.145907 22568 ts_tablet_manager.cc:1428] T 68bb4bb68a93481cb8a3406f5f7d0b91 P a93555b72ffe4d3aa2f57067b8927f89: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20250112 15:08:00.145958 22568 tablet_bootstrap.cc:492] T 50c711f4e65148faa8062f3ae6d87990 P a93555b72ffe4d3aa2f57067b8927f89: Bootstrap starting.
I20250112 15:08:00.146286 22447 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "68bb4bb68a93481cb8a3406f5f7d0b91" candidate_uuid: "74f47306030d436f90298b8a17297d29" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2e717f485ad442c8bfccf44cf80aa3e" is_pre_election: true
I20250112 15:08:00.146353 22447 raft_consensus.cc:2463] T 68bb4bb68a93481cb8a3406f5f7d0b91 P d2e717f485ad442c8bfccf44cf80aa3e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 74f47306030d436f90298b8a17297d29 in term 0.
I20250112 15:08:00.146538 22485 leader_election.cc:304] T 68bb4bb68a93481cb8a3406f5f7d0b91 P 74f47306030d436f90298b8a17297d29 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 74f47306030d436f90298b8a17297d29, d2e717f485ad442c8bfccf44cf80aa3e; no voters: a93555b72ffe4d3aa2f57067b8927f89
I20250112 15:08:00.146338 22568 tablet_bootstrap.cc:654] T 50c711f4e65148faa8062f3ae6d87990 P a93555b72ffe4d3aa2f57067b8927f89: Neither blocks nor log segments found. Creating new log.
I20250112 15:08:00.146677 22576 raft_consensus.cc:2798] T 68bb4bb68a93481cb8a3406f5f7d0b91 P 74f47306030d436f90298b8a17297d29 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250112 15:08:00.146718 22576 raft_consensus.cc:491] T 68bb4bb68a93481cb8a3406f5f7d0b91 P 74f47306030d436f90298b8a17297d29 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250112 15:08:00.146733 22576 raft_consensus.cc:3054] T 68bb4bb68a93481cb8a3406f5f7d0b91 P 74f47306030d436f90298b8a17297d29 [term 0 FOLLOWER]: Advancing to term 1
I20250112 15:08:00.146340 22572 tablet_bootstrap.cc:492] T 8b7046b625f84ac3b2fe6d67946fe6bf P d2e717f485ad442c8bfccf44cf80aa3e: No bootstrap required, opened a new log
I20250112 15:08:00.146940 22572 ts_tablet_manager.cc:1397] T 8b7046b625f84ac3b2fe6d67946fe6bf P d2e717f485ad442c8bfccf44cf80aa3e: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20250112 15:08:00.147080 22572 raft_consensus.cc:357] T 8b7046b625f84ac3b2fe6d67946fe6bf P d2e717f485ad442c8bfccf44cf80aa3e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a93555b72ffe4d3aa2f57067b8927f89" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 36981 } } peers { permanent_uuid: "d2e717f485ad442c8bfccf44cf80aa3e" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 40357 } } peers { permanent_uuid: "74f47306030d436f90298b8a17297d29" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 34175 } }
I20250112 15:08:00.147141 22572 raft_consensus.cc:383] T 8b7046b625f84ac3b2fe6d67946fe6bf P d2e717f485ad442c8bfccf44cf80aa3e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 15:08:00.147162 22572 raft_consensus.cc:738] T 8b7046b625f84ac3b2fe6d67946fe6bf P d2e717f485ad442c8bfccf44cf80aa3e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d2e717f485ad442c8bfccf44cf80aa3e, State: Initialized, Role: FOLLOWER
I20250112 15:08:00.147172 22568 tablet_bootstrap.cc:492] T 50c711f4e65148faa8062f3ae6d87990 P a93555b72ffe4d3aa2f57067b8927f89: No bootstrap required, opened a new log
I20250112 15:08:00.147207 22568 ts_tablet_manager.cc:1397] T 50c711f4e65148faa8062f3ae6d87990 P a93555b72ffe4d3aa2f57067b8927f89: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20250112 15:08:00.147222 22572 consensus_queue.cc:260] T 8b7046b625f84ac3b2fe6d67946fe6bf P d2e717f485ad442c8bfccf44cf80aa3e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a93555b72ffe4d3aa2f57067b8927f89" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 36981 } } peers { permanent_uuid: "d2e717f485ad442c8bfccf44cf80aa3e" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 40357 } } peers { permanent_uuid: "74f47306030d436f90298b8a17297d29" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 34175 } }
I20250112 15:08:00.147312 22572 ts_tablet_manager.cc:1428] T 8b7046b625f84ac3b2fe6d67946fe6bf P d2e717f485ad442c8bfccf44cf80aa3e: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20250112 15:08:00.147336 22568 raft_consensus.cc:357] T 50c711f4e65148faa8062f3ae6d87990 P a93555b72ffe4d3aa2f57067b8927f89 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74f47306030d436f90298b8a17297d29" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 34175 } } peers { permanent_uuid: "a93555b72ffe4d3aa2f57067b8927f89" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 36981 } } peers { permanent_uuid: "d2e717f485ad442c8bfccf44cf80aa3e" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 40357 } }
I20250112 15:08:00.147351 22576 raft_consensus.cc:513] T 68bb4bb68a93481cb8a3406f5f7d0b91 P 74f47306030d436f90298b8a17297d29 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a93555b72ffe4d3aa2f57067b8927f89" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 36981 } } peers { permanent_uuid: "d2e717f485ad442c8bfccf44cf80aa3e" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 40357 } } peers { permanent_uuid: "74f47306030d436f90298b8a17297d29" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 34175 } }
I20250112 15:08:00.147389 22568 raft_consensus.cc:383] T 50c711f4e65148faa8062f3ae6d87990 P a93555b72ffe4d3aa2f57067b8927f89 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 15:08:00.147439 22568 raft_consensus.cc:738] T 50c711f4e65148faa8062f3ae6d87990 P a93555b72ffe4d3aa2f57067b8927f89 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a93555b72ffe4d3aa2f57067b8927f89, State: Initialized, Role: FOLLOWER
I20250112 15:08:00.147492 22576 leader_election.cc:290] T 68bb4bb68a93481cb8a3406f5f7d0b91 P 74f47306030d436f90298b8a17297d29 [CANDIDATE]: Term 1 election: Requested vote from peers a93555b72ffe4d3aa2f57067b8927f89 (127.17.60.193:36981), d2e717f485ad442c8bfccf44cf80aa3e (127.17.60.194:40357)
I20250112 15:08:00.147480 22568 consensus_queue.cc:260] T 50c711f4e65148faa8062f3ae6d87990 P a93555b72ffe4d3aa2f57067b8927f89 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74f47306030d436f90298b8a17297d29" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 34175 } } peers { permanent_uuid: "a93555b72ffe4d3aa2f57067b8927f89" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 36981 } } peers { permanent_uuid: "d2e717f485ad442c8bfccf44cf80aa3e" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 40357 } }
I20250112 15:08:00.147578 22568 ts_tablet_manager.cc:1428] T 50c711f4e65148faa8062f3ae6d87990 P a93555b72ffe4d3aa2f57067b8927f89: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20250112 15:08:00.147642 22568 tablet_bootstrap.cc:492] T 8b7046b625f84ac3b2fe6d67946fe6bf P a93555b72ffe4d3aa2f57067b8927f89: Bootstrap starting.
I20250112 15:08:00.147621 22447 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "68bb4bb68a93481cb8a3406f5f7d0b91" candidate_uuid: "74f47306030d436f90298b8a17297d29" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2e717f485ad442c8bfccf44cf80aa3e"
I20250112 15:08:00.147676 22447 raft_consensus.cc:3054] T 68bb4bb68a93481cb8a3406f5f7d0b91 P d2e717f485ad442c8bfccf44cf80aa3e [term 0 FOLLOWER]: Advancing to term 1
I20250112 15:08:00.147698 22371 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "68bb4bb68a93481cb8a3406f5f7d0b91" candidate_uuid: "74f47306030d436f90298b8a17297d29" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a93555b72ffe4d3aa2f57067b8927f89"
I20250112 15:08:00.147755 22371 raft_consensus.cc:3054] T 68bb4bb68a93481cb8a3406f5f7d0b91 P a93555b72ffe4d3aa2f57067b8927f89 [term 0 FOLLOWER]: Advancing to term 1
I20250112 15:08:00.148027 22568 tablet_bootstrap.cc:654] T 8b7046b625f84ac3b2fe6d67946fe6bf P a93555b72ffe4d3aa2f57067b8927f89: Neither blocks nor log segments found. Creating new log.
I20250112 15:08:00.148329 22447 raft_consensus.cc:2463] T 68bb4bb68a93481cb8a3406f5f7d0b91 P d2e717f485ad442c8bfccf44cf80aa3e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 74f47306030d436f90298b8a17297d29 in term 1.
I20250112 15:08:00.148329 22371 raft_consensus.cc:2463] T 68bb4bb68a93481cb8a3406f5f7d0b91 P a93555b72ffe4d3aa2f57067b8927f89 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 74f47306030d436f90298b8a17297d29 in term 1.
I20250112 15:08:00.148408 22576 raft_consensus.cc:491] T 8b7046b625f84ac3b2fe6d67946fe6bf P 74f47306030d436f90298b8a17297d29 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 15:08:00.148442 22576 raft_consensus.cc:513] T 8b7046b625f84ac3b2fe6d67946fe6bf P 74f47306030d436f90298b8a17297d29 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a93555b72ffe4d3aa2f57067b8927f89" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 36981 } } peers { permanent_uuid: "d2e717f485ad442c8bfccf44cf80aa3e" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 40357 } } peers { permanent_uuid: "74f47306030d436f90298b8a17297d29" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 34175 } }
I20250112 15:08:00.148530 22485 leader_election.cc:304] T 68bb4bb68a93481cb8a3406f5f7d0b91 P 74f47306030d436f90298b8a17297d29 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 74f47306030d436f90298b8a17297d29, d2e717f485ad442c8bfccf44cf80aa3e; no voters:
I20250112 15:08:00.148692 22573 raft_consensus.cc:2798] T 68bb4bb68a93481cb8a3406f5f7d0b91 P 74f47306030d436f90298b8a17297d29 [term 1 FOLLOWER]: Leader election won for term 1
I20250112 15:08:00.148766 22371 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8b7046b625f84ac3b2fe6d67946fe6bf" candidate_uuid: "74f47306030d436f90298b8a17297d29" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a93555b72ffe4d3aa2f57067b8927f89" is_pre_election: true
I20250112 15:08:00.148809 22573 raft_consensus.cc:695] T 68bb4bb68a93481cb8a3406f5f7d0b91 P 74f47306030d436f90298b8a17297d29 [term 1 LEADER]: Becoming Leader. State: Replica: 74f47306030d436f90298b8a17297d29, State: Running, Role: LEADER
I20250112 15:08:00.148850 22573 consensus_queue.cc:237] T 68bb4bb68a93481cb8a3406f5f7d0b91 P 74f47306030d436f90298b8a17297d29 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a93555b72ffe4d3aa2f57067b8927f89" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 36981 } } peers { permanent_uuid: "d2e717f485ad442c8bfccf44cf80aa3e" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 40357 } } peers { permanent_uuid: "74f47306030d436f90298b8a17297d29" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 34175 } }
W20250112 15:08:00.148983 22488 leader_election.cc:343] T 8b7046b625f84ac3b2fe6d67946fe6bf P 74f47306030d436f90298b8a17297d29 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a93555b72ffe4d3aa2f57067b8927f89 (127.17.60.193:36981): Illegal state: must be running to vote when last-logged opid is not known
I20250112 15:08:00.149134 22447 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8b7046b625f84ac3b2fe6d67946fe6bf" candidate_uuid: "74f47306030d436f90298b8a17297d29" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2e717f485ad442c8bfccf44cf80aa3e" is_pre_election: true
I20250112 15:08:00.149200 22447 raft_consensus.cc:2463] T 8b7046b625f84ac3b2fe6d67946fe6bf P d2e717f485ad442c8bfccf44cf80aa3e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 74f47306030d436f90298b8a17297d29 in term 0.
I20250112 15:08:00.149386 22267 catalog_manager.cc:5526] T 68bb4bb68a93481cb8a3406f5f7d0b91 P 74f47306030d436f90298b8a17297d29 reported cstate change: term changed from 0 to 1, leader changed from <none> to 74f47306030d436f90298b8a17297d29 (127.17.60.195). New cstate: current_term: 1 leader_uuid: "74f47306030d436f90298b8a17297d29" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a93555b72ffe4d3aa2f57067b8927f89" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 36981 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d2e717f485ad442c8bfccf44cf80aa3e" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 40357 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "74f47306030d436f90298b8a17297d29" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 34175 } health_report { overall_health: HEALTHY } } }
I20250112 15:08:00.148543 22576 leader_election.cc:290] T 8b7046b625f84ac3b2fe6d67946fe6bf P 74f47306030d436f90298b8a17297d29 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a93555b72ffe4d3aa2f57067b8927f89 (127.17.60.193:36981), d2e717f485ad442c8bfccf44cf80aa3e (127.17.60.194:40357)
I20250112 15:08:00.149420 22568 tablet_bootstrap.cc:492] T 8b7046b625f84ac3b2fe6d67946fe6bf P a93555b72ffe4d3aa2f57067b8927f89: No bootstrap required, opened a new log
I20250112 15:08:00.149497 22568 ts_tablet_manager.cc:1397] T 8b7046b625f84ac3b2fe6d67946fe6bf P a93555b72ffe4d3aa2f57067b8927f89: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20250112 15:08:00.149593 22485 leader_election.cc:304] T 8b7046b625f84ac3b2fe6d67946fe6bf P 74f47306030d436f90298b8a17297d29 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 74f47306030d436f90298b8a17297d29, d2e717f485ad442c8bfccf44cf80aa3e; no voters: a93555b72ffe4d3aa2f57067b8927f89
I20250112 15:08:00.149633 22568 raft_consensus.cc:357] T 8b7046b625f84ac3b2fe6d67946fe6bf P a93555b72ffe4d3aa2f57067b8927f89 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a93555b72ffe4d3aa2f57067b8927f89" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 36981 } } peers { permanent_uuid: "d2e717f485ad442c8bfccf44cf80aa3e" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 40357 } } peers { permanent_uuid: "74f47306030d436f90298b8a17297d29" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 34175 } }
I20250112 15:08:00.149686 22568 raft_consensus.cc:383] T 8b7046b625f84ac3b2fe6d67946fe6bf P a93555b72ffe4d3aa2f57067b8927f89 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 15:08:00.149708 22568 raft_consensus.cc:738] T 8b7046b625f84ac3b2fe6d67946fe6bf P a93555b72ffe4d3aa2f57067b8927f89 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a93555b72ffe4d3aa2f57067b8927f89, State: Initialized, Role: FOLLOWER
I20250112 15:08:00.149747 22568 consensus_queue.cc:260] T 8b7046b625f84ac3b2fe6d67946fe6bf P a93555b72ffe4d3aa2f57067b8927f89 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a93555b72ffe4d3aa2f57067b8927f89" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 36981 } } peers { permanent_uuid: "d2e717f485ad442c8bfccf44cf80aa3e" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 40357 } } peers { permanent_uuid: "74f47306030d436f90298b8a17297d29" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 34175 } }
I20250112 15:08:00.149834 22568 ts_tablet_manager.cc:1428] T 8b7046b625f84ac3b2fe6d67946fe6bf P a93555b72ffe4d3aa2f57067b8927f89: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20250112 15:08:00.149837 22576 raft_consensus.cc:2798] T 8b7046b625f84ac3b2fe6d67946fe6bf P 74f47306030d436f90298b8a17297d29 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250112 15:08:00.149899 22576 raft_consensus.cc:491] T 8b7046b625f84ac3b2fe6d67946fe6bf P 74f47306030d436f90298b8a17297d29 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250112 15:08:00.149919 22576 raft_consensus.cc:3054] T 8b7046b625f84ac3b2fe6d67946fe6bf P 74f47306030d436f90298b8a17297d29 [term 0 FOLLOWER]: Advancing to term 1
I20250112 15:08:00.150473 22576 raft_consensus.cc:513] T 8b7046b625f84ac3b2fe6d67946fe6bf P 74f47306030d436f90298b8a17297d29 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a93555b72ffe4d3aa2f57067b8927f89" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 36981 } } peers { permanent_uuid: "d2e717f485ad442c8bfccf44cf80aa3e" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 40357 } } peers { permanent_uuid: "74f47306030d436f90298b8a17297d29" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 34175 } }
I20250112 15:08:00.150630 22576 leader_election.cc:290] T 8b7046b625f84ac3b2fe6d67946fe6bf P 74f47306030d436f90298b8a17297d29 [CANDIDATE]: Term 1 election: Requested vote from peers a93555b72ffe4d3aa2f57067b8927f89 (127.17.60.193:36981), d2e717f485ad442c8bfccf44cf80aa3e (127.17.60.194:40357)
I20250112 15:08:00.150661 22371 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8b7046b625f84ac3b2fe6d67946fe6bf" candidate_uuid: "74f47306030d436f90298b8a17297d29" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a93555b72ffe4d3aa2f57067b8927f89"
I20250112 15:08:00.150686 22447 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8b7046b625f84ac3b2fe6d67946fe6bf" candidate_uuid: "74f47306030d436f90298b8a17297d29" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2e717f485ad442c8bfccf44cf80aa3e"
I20250112 15:08:00.150719 22371 raft_consensus.cc:3054] T 8b7046b625f84ac3b2fe6d67946fe6bf P a93555b72ffe4d3aa2f57067b8927f89 [term 0 FOLLOWER]: Advancing to term 1
I20250112 15:08:00.150722 22447 raft_consensus.cc:3054] T 8b7046b625f84ac3b2fe6d67946fe6bf P d2e717f485ad442c8bfccf44cf80aa3e [term 0 FOLLOWER]: Advancing to term 1
I20250112 15:08:00.151233 22371 raft_consensus.cc:2463] T 8b7046b625f84ac3b2fe6d67946fe6bf P a93555b72ffe4d3aa2f57067b8927f89 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 74f47306030d436f90298b8a17297d29 in term 1.
I20250112 15:08:00.151239 22447 raft_consensus.cc:2463] T 8b7046b625f84ac3b2fe6d67946fe6bf P d2e717f485ad442c8bfccf44cf80aa3e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 74f47306030d436f90298b8a17297d29 in term 1.
I20250112 15:08:00.151377 22488 leader_election.cc:304] T 8b7046b625f84ac3b2fe6d67946fe6bf P 74f47306030d436f90298b8a17297d29 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 74f47306030d436f90298b8a17297d29, a93555b72ffe4d3aa2f57067b8927f89; no voters:
I20250112 15:08:00.151451 22576 raft_consensus.cc:2798] T 8b7046b625f84ac3b2fe6d67946fe6bf P 74f47306030d436f90298b8a17297d29 [term 1 FOLLOWER]: Leader election won for term 1
I20250112 15:08:00.151504 22576 raft_consensus.cc:695] T 8b7046b625f84ac3b2fe6d67946fe6bf P 74f47306030d436f90298b8a17297d29 [term 1 LEADER]: Becoming Leader. State: Replica: 74f47306030d436f90298b8a17297d29, State: Running, Role: LEADER
I20250112 15:08:00.151546 22576 consensus_queue.cc:237] T 8b7046b625f84ac3b2fe6d67946fe6bf P 74f47306030d436f90298b8a17297d29 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a93555b72ffe4d3aa2f57067b8927f89" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 36981 } } peers { permanent_uuid: "d2e717f485ad442c8bfccf44cf80aa3e" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 40357 } } peers { permanent_uuid: "74f47306030d436f90298b8a17297d29" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 34175 } }
I20250112 15:08:00.151981 22268 catalog_manager.cc:5526] T 8b7046b625f84ac3b2fe6d67946fe6bf P 74f47306030d436f90298b8a17297d29 reported cstate change: term changed from 0 to 1, leader changed from <none> to 74f47306030d436f90298b8a17297d29 (127.17.60.195). New cstate: current_term: 1 leader_uuid: "74f47306030d436f90298b8a17297d29" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a93555b72ffe4d3aa2f57067b8927f89" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 36981 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d2e717f485ad442c8bfccf44cf80aa3e" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 40357 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "74f47306030d436f90298b8a17297d29" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 34175 } health_report { overall_health: HEALTHY } } }
I20250112 15:08:00.183920 22583 raft_consensus.cc:491] T 50c711f4e65148faa8062f3ae6d87990 P a93555b72ffe4d3aa2f57067b8927f89 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 15:08:00.184037 22583 raft_consensus.cc:513] T 50c711f4e65148faa8062f3ae6d87990 P a93555b72ffe4d3aa2f57067b8927f89 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74f47306030d436f90298b8a17297d29" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 34175 } } peers { permanent_uuid: "a93555b72ffe4d3aa2f57067b8927f89" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 36981 } } peers { permanent_uuid: "d2e717f485ad442c8bfccf44cf80aa3e" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 40357 } }
I20250112 15:08:00.184224 22583 leader_election.cc:290] T 50c711f4e65148faa8062f3ae6d87990 P a93555b72ffe4d3aa2f57067b8927f89 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 74f47306030d436f90298b8a17297d29 (127.17.60.195:34175), d2e717f485ad442c8bfccf44cf80aa3e (127.17.60.194:40357)
I20250112 15:08:00.187682 22447 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "50c711f4e65148faa8062f3ae6d87990" candidate_uuid: "a93555b72ffe4d3aa2f57067b8927f89" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2e717f485ad442c8bfccf44cf80aa3e" is_pre_election: true
I20250112 15:08:00.187700 22522 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "50c711f4e65148faa8062f3ae6d87990" candidate_uuid: "a93555b72ffe4d3aa2f57067b8927f89" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "74f47306030d436f90298b8a17297d29" is_pre_election: true
I20250112 15:08:00.187780 22447 raft_consensus.cc:2463] T 50c711f4e65148faa8062f3ae6d87990 P d2e717f485ad442c8bfccf44cf80aa3e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a93555b72ffe4d3aa2f57067b8927f89 in term 0.
I20250112 15:08:00.187815 22522 raft_consensus.cc:2463] T 50c711f4e65148faa8062f3ae6d87990 P 74f47306030d436f90298b8a17297d29 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a93555b72ffe4d3aa2f57067b8927f89 in term 0.
I20250112 15:08:00.187961 22334 leader_election.cc:304] T 50c711f4e65148faa8062f3ae6d87990 P a93555b72ffe4d3aa2f57067b8927f89 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a93555b72ffe4d3aa2f57067b8927f89, d2e717f485ad442c8bfccf44cf80aa3e; no voters:
I20250112 15:08:00.188055 22583 raft_consensus.cc:2798] T 50c711f4e65148faa8062f3ae6d87990 P a93555b72ffe4d3aa2f57067b8927f89 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250112 15:08:00.188104 22583 raft_consensus.cc:491] T 50c711f4e65148faa8062f3ae6d87990 P a93555b72ffe4d3aa2f57067b8927f89 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250112 15:08:00.188119 22583 raft_consensus.cc:3054] T 50c711f4e65148faa8062f3ae6d87990 P a93555b72ffe4d3aa2f57067b8927f89 [term 0 FOLLOWER]: Advancing to term 1
I20250112 15:08:00.188810 22583 raft_consensus.cc:513] T 50c711f4e65148faa8062f3ae6d87990 P a93555b72ffe4d3aa2f57067b8927f89 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74f47306030d436f90298b8a17297d29" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 34175 } } peers { permanent_uuid: "a93555b72ffe4d3aa2f57067b8927f89" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 36981 } } peers { permanent_uuid: "d2e717f485ad442c8bfccf44cf80aa3e" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 40357 } }
I20250112 15:08:00.188941 22583 leader_election.cc:290] T 50c711f4e65148faa8062f3ae6d87990 P a93555b72ffe4d3aa2f57067b8927f89 [CANDIDATE]: Term 1 election: Requested vote from peers 74f47306030d436f90298b8a17297d29 (127.17.60.195:34175), d2e717f485ad442c8bfccf44cf80aa3e (127.17.60.194:40357)
I20250112 15:08:00.189091 22522 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "50c711f4e65148faa8062f3ae6d87990" candidate_uuid: "a93555b72ffe4d3aa2f57067b8927f89" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "74f47306030d436f90298b8a17297d29"
I20250112 15:08:00.189123 22447 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "50c711f4e65148faa8062f3ae6d87990" candidate_uuid: "a93555b72ffe4d3aa2f57067b8927f89" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2e717f485ad442c8bfccf44cf80aa3e"
I20250112 15:08:00.189172 22522 raft_consensus.cc:3054] T 50c711f4e65148faa8062f3ae6d87990 P 74f47306030d436f90298b8a17297d29 [term 0 FOLLOWER]: Advancing to term 1
I20250112 15:08:00.189186 22447 raft_consensus.cc:3054] T 50c711f4e65148faa8062f3ae6d87990 P d2e717f485ad442c8bfccf44cf80aa3e [term 0 FOLLOWER]: Advancing to term 1
I20250112 15:08:00.189716 22522 raft_consensus.cc:2463] T 50c711f4e65148faa8062f3ae6d87990 P 74f47306030d436f90298b8a17297d29 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a93555b72ffe4d3aa2f57067b8927f89 in term 1.
I20250112 15:08:00.189754 22447 raft_consensus.cc:2463] T 50c711f4e65148faa8062f3ae6d87990 P d2e717f485ad442c8bfccf44cf80aa3e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a93555b72ffe4d3aa2f57067b8927f89 in term 1.
I20250112 15:08:00.189888 22336 leader_election.cc:304] T 50c711f4e65148faa8062f3ae6d87990 P a93555b72ffe4d3aa2f57067b8927f89 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 74f47306030d436f90298b8a17297d29, a93555b72ffe4d3aa2f57067b8927f89; no voters:
I20250112 15:08:00.189986 22583 raft_consensus.cc:2798] T 50c711f4e65148faa8062f3ae6d87990 P a93555b72ffe4d3aa2f57067b8927f89 [term 1 FOLLOWER]: Leader election won for term 1
I20250112 15:08:00.190111 22583 raft_consensus.cc:695] T 50c711f4e65148faa8062f3ae6d87990 P a93555b72ffe4d3aa2f57067b8927f89 [term 1 LEADER]: Becoming Leader. State: Replica: a93555b72ffe4d3aa2f57067b8927f89, State: Running, Role: LEADER
I20250112 15:08:00.190167 22583 consensus_queue.cc:237] T 50c711f4e65148faa8062f3ae6d87990 P a93555b72ffe4d3aa2f57067b8927f89 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74f47306030d436f90298b8a17297d29" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 34175 } } peers { permanent_uuid: "a93555b72ffe4d3aa2f57067b8927f89" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 36981 } } peers { permanent_uuid: "d2e717f485ad442c8bfccf44cf80aa3e" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 40357 } }
I20250112 15:08:00.190647 22268 catalog_manager.cc:5526] T 50c711f4e65148faa8062f3ae6d87990 P a93555b72ffe4d3aa2f57067b8927f89 reported cstate change: term changed from 0 to 1, leader changed from <none> to a93555b72ffe4d3aa2f57067b8927f89 (127.17.60.193). New cstate: current_term: 1 leader_uuid: "a93555b72ffe4d3aa2f57067b8927f89" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74f47306030d436f90298b8a17297d29" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 34175 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a93555b72ffe4d3aa2f57067b8927f89" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 36981 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d2e717f485ad442c8bfccf44cf80aa3e" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 40357 } health_report { overall_health: UNKNOWN } } }
I20250112 15:08:00.205423 17651 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250112 15:08:00.207044 22593 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250112 15:08:00.207854 22594 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250112 15:08:00.208688 22596 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250112 15:08:00.209564 17651 server_base.cc:1034] running on GCE node
I20250112 15:08:00.209728 17651 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250112 15:08:00.209757 17651 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250112 15:08:00.209780 17651 hybrid_clock.cc:648] HybridClock initialized: now 1736694480209780 us; error 0 us; skew 500 ppm
I20250112 15:08:00.210502 17651 webserver.cc:458] Webserver started at http://127.17.60.196:38687/ using document root <none> and password file <none>
I20250112 15:08:00.210584 17651 fs_manager.cc:362] Metadata directory not provided
I20250112 15:08:00.210633 17651 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250112 15:08:00.210680 17651 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250112 15:08:00.210984 17651 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task95dLSD/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingUnstableCluster.1736694351070047-17651-0/minicluster-data/ts-3-root/instance:
uuid: "72c5e551bf0d441f9a702da4848d7b1c"
format_stamp: "Formatted at 2025-01-12 15:08:00 on dist-test-slave-b4nz"
I20250112 15:08:00.211824 17651 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20250112 15:08:00.212221 22601 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250112 15:08:00.212383 17651 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20250112 15:08:00.212424 17651 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task95dLSD/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingUnstableCluster.1736694351070047-17651-0/minicluster-data/ts-3-root
uuid: "72c5e551bf0d441f9a702da4848d7b1c"
format_stamp: "Formatted at 2025-01-12 15:08:00 on dist-test-slave-b4nz"
I20250112 15:08:00.212487 17651 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task95dLSD/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingUnstableCluster.1736694351070047-17651-0/minicluster-data/ts-3-root
metadata directory: /tmp/dist-test-task95dLSD/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingUnstableCluster.1736694351070047-17651-0/minicluster-data/ts-3-root
1 data directories: /tmp/dist-test-task95dLSD/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingUnstableCluster.1736694351070047-17651-0/minicluster-data/ts-3-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250112 15:08:00.225493 22576 consensus_queue.cc:1035] T 68bb4bb68a93481cb8a3406f5f7d0b91 P 74f47306030d436f90298b8a17297d29 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a93555b72ffe4d3aa2f57067b8927f89" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 36981 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20250112 15:08:00.226362 22573 fault_injection.cc:43] FAULT INJECTION ENABLED!
W20250112 15:08:00.226413 22573 fault_injection.cc:44] THIS SERVER MAY CRASH!
I20250112 15:08:00.227448 17651 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250112 15:08:00.227675 17651 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250112 15:08:00.227838 17651 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250112 15:08:00.228006 17651 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250112 15:08:00.228034 17651 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250112 15:08:00.228065 17651 ts_tablet_manager.cc:610] Registered 0 tablets
I20250112 15:08:00.228089 17651 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250112 15:08:00.231143 17651 rpc_server.cc:307] RPC server started. Bound to: 127.17.60.196:35345
I20250112 15:08:00.231213 22666 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.17.60.196:35345 every 8 connection(s)
I20250112 15:08:00.232369 22586 consensus_queue.cc:1035] T 68bb4bb68a93481cb8a3406f5f7d0b91 P 74f47306030d436f90298b8a17297d29 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d2e717f485ad442c8bfccf44cf80aa3e" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 40357 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250112 15:08:00.234308 22667 heartbeater.cc:346] Connected to a master server at 127.17.60.254:38743
I20250112 15:08:00.234359 22667 heartbeater.cc:463] Registering TS with master...
I20250112 15:08:00.234454 22667 heartbeater.cc:510] Master 127.17.60.254:38743 requested a full tablet report, sending...
I20250112 15:08:00.234647 22268 ts_manager.cc:194] Registered new tserver with Master: 72c5e551bf0d441f9a702da4848d7b1c (127.17.60.196:35345)
I20250112 15:08:00.234922 22268 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:50526
I20250112 15:08:00.246958 22586 consensus_queue.cc:1035] T 8b7046b625f84ac3b2fe6d67946fe6bf P 74f47306030d436f90298b8a17297d29 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a93555b72ffe4d3aa2f57067b8927f89" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 36981 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20250112 15:08:00.252192 22398 tablet.cc:2367] T 50c711f4e65148faa8062f3ae6d87990 P a93555b72ffe4d3aa2f57067b8927f89: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250112 15:08:00.275076 22674 consensus_queue.cc:1035] T 8b7046b625f84ac3b2fe6d67946fe6bf P 74f47306030d436f90298b8a17297d29 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d2e717f485ad442c8bfccf44cf80aa3e" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 40357 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250112 15:08:00.298202 22583 consensus_queue.cc:1035] T 50c711f4e65148faa8062f3ae6d87990 P a93555b72ffe4d3aa2f57067b8927f89 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d2e717f485ad442c8bfccf44cf80aa3e" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 40357 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250112 15:08:00.304195 22583 consensus_queue.cc:1035] T 50c711f4e65148faa8062f3ae6d87990 P a93555b72ffe4d3aa2f57067b8927f89 [LEADER]: Connected to new peer: Peer: permanent_uuid: "74f47306030d436f90298b8a17297d29" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 34175 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250112 15:08:00.966748 22322 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250112 15:08:00.969874 22371 consensus_queue.cc:237] T 50c711f4e65148faa8062f3ae6d87990 P a93555b72ffe4d3aa2f57067b8927f89 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "74f47306030d436f90298b8a17297d29" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 34175 } attrs { replace: true } } peers { permanent_uuid: "a93555b72ffe4d3aa2f57067b8927f89" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 36981 } } peers { permanent_uuid: "d2e717f485ad442c8bfccf44cf80aa3e" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 40357 } } peers { permanent_uuid: "72c5e551bf0d441f9a702da4848d7b1c" member_type: NON_VOTER last_known_addr { host: "127.17.60.196" port: 35345 } attrs { promote: true } }
I20250112 15:08:00.969875 22522 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "68bb4bb68a93481cb8a3406f5f7d0b91"
dest_uuid: "74f47306030d436f90298b8a17297d29"
mode: GRACEFUL
new_leader_uuid: "d2e717f485ad442c8bfccf44cf80aa3e"
from {username='slave'} at 127.0.0.1:58114
I20250112 15:08:00.970000 22522 raft_consensus.cc:604] T 68bb4bb68a93481cb8a3406f5f7d0b91 P 74f47306030d436f90298b8a17297d29 [term 1 LEADER]: Received request to transfer leadership to TS d2e717f485ad442c8bfccf44cf80aa3e
I20250112 15:08:00.970163 22322 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 1
I20250112 15:08:00.970206 22322 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20250112 15:08:00.970463 22447 raft_consensus.cc:1270] T 50c711f4e65148faa8062f3ae6d87990 P d2e717f485ad442c8bfccf44cf80aa3e [term 1 FOLLOWER]: Refusing update from remote peer a93555b72ffe4d3aa2f57067b8927f89: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250112 15:08:00.970659 22522 raft_consensus.cc:1270] T 50c711f4e65148faa8062f3ae6d87990 P 74f47306030d436f90298b8a17297d29 [term 1 FOLLOWER]: Refusing update from remote peer a93555b72ffe4d3aa2f57067b8927f89: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250112 15:08:00.970660 22678 consensus_queue.cc:1035] T 50c711f4e65148faa8062f3ae6d87990 P a93555b72ffe4d3aa2f57067b8927f89 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d2e717f485ad442c8bfccf44cf80aa3e" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 40357 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250112 15:08:00.970877 22591 consensus_queue.cc:1035] T 50c711f4e65148faa8062f3ae6d87990 P a93555b72ffe4d3aa2f57067b8927f89 [LEADER]: Connected to new peer: Peer: permanent_uuid: "74f47306030d436f90298b8a17297d29" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 34175 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250112 15:08:00.971640 22522 raft_consensus.cc:2949] T 50c711f4e65148faa8062f3ae6d87990 P 74f47306030d436f90298b8a17297d29 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 72c5e551bf0d441f9a702da4848d7b1c (127.17.60.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "74f47306030d436f90298b8a17297d29" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 34175 } attrs { replace: true } } peers { permanent_uuid: "a93555b72ffe4d3aa2f57067b8927f89" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 36981 } } peers { permanent_uuid: "d2e717f485ad442c8bfccf44cf80aa3e" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 40357 } } peers { permanent_uuid: "72c5e551bf0d441f9a702da4848d7b1c" member_type: NON_VOTER last_known_addr { host: "127.17.60.196" port: 35345 } attrs { promote: true } } }
I20250112 15:08:00.971644 22447 raft_consensus.cc:2949] T 50c711f4e65148faa8062f3ae6d87990 P d2e717f485ad442c8bfccf44cf80aa3e [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 72c5e551bf0d441f9a702da4848d7b1c (127.17.60.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "74f47306030d436f90298b8a17297d29" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 34175 } attrs { replace: true } } peers { permanent_uuid: "a93555b72ffe4d3aa2f57067b8927f89" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 36981 } } peers { permanent_uuid: "d2e717f485ad442c8bfccf44cf80aa3e" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 40357 } } peers { permanent_uuid: "72c5e551bf0d441f9a702da4848d7b1c" member_type: NON_VOTER last_known_addr { host: "127.17.60.196" port: 35345 } attrs { promote: true } } }
I20250112 15:08:00.972798 22268 catalog_manager.cc:5526] T 50c711f4e65148faa8062f3ae6d87990 P 74f47306030d436f90298b8a17297d29 reported cstate change: config changed from index -1 to 2, NON_VOTER 72c5e551bf0d441f9a702da4848d7b1c (127.17.60.196) added. New cstate: current_term: 1 leader_uuid: "a93555b72ffe4d3aa2f57067b8927f89" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "74f47306030d436f90298b8a17297d29" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 34175 } attrs { replace: true } } peers { permanent_uuid: "a93555b72ffe4d3aa2f57067b8927f89" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 36981 } } peers { permanent_uuid: "d2e717f485ad442c8bfccf44cf80aa3e" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 40357 } } peers { permanent_uuid: "72c5e551bf0d441f9a702da4848d7b1c" member_type: NON_VOTER last_known_addr { host: "127.17.60.196" port: 35345 } attrs { promote: true } } }
W20250112 15:08:00.973766 22337 consensus_peers.cc:487] T 50c711f4e65148faa8062f3ae6d87990 P a93555b72ffe4d3aa2f57067b8927f89 -> Peer 72c5e551bf0d441f9a702da4848d7b1c (127.17.60.196:35345): Couldn't send request to peer 72c5e551bf0d441f9a702da4848d7b1c. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 50c711f4e65148faa8062f3ae6d87990. This is attempt 1: this message will repeat every 5th retry.
I20250112 15:08:01.025744 22678 raft_consensus.cc:2949] T 50c711f4e65148faa8062f3ae6d87990 P a93555b72ffe4d3aa2f57067b8927f89 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 72c5e551bf0d441f9a702da4848d7b1c (127.17.60.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "74f47306030d436f90298b8a17297d29" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 34175 } attrs { replace: true } } peers { permanent_uuid: "a93555b72ffe4d3aa2f57067b8927f89" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 36981 } } peers { permanent_uuid: "d2e717f485ad442c8bfccf44cf80aa3e" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 40357 } } peers { permanent_uuid: "72c5e551bf0d441f9a702da4848d7b1c" member_type: NON_VOTER last_known_addr { host: "127.17.60.196" port: 35345 } attrs { promote: true } } }
I20250112 15:08:01.029793 22522 consensus_queue.cc:237] T 8b7046b625f84ac3b2fe6d67946fe6bf P 74f47306030d436f90298b8a17297d29 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a93555b72ffe4d3aa2f57067b8927f89" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 36981 } attrs { replace: true } } peers { permanent_uuid: "d2e717f485ad442c8bfccf44cf80aa3e" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 40357 } } peers { permanent_uuid: "74f47306030d436f90298b8a17297d29" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 34175 } } peers { permanent_uuid: "72c5e551bf0d441f9a702da4848d7b1c" member_type: NON_VOTER last_known_addr { host: "127.17.60.196" port: 35345 } attrs { promote: true } }
I20250112 15:08:01.030342 22447 raft_consensus.cc:1270] T 8b7046b625f84ac3b2fe6d67946fe6bf P d2e717f485ad442c8bfccf44cf80aa3e [term 1 FOLLOWER]: Refusing update from remote peer 74f47306030d436f90298b8a17297d29: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250112 15:08:01.030452 22371 raft_consensus.cc:1270] T 8b7046b625f84ac3b2fe6d67946fe6bf P a93555b72ffe4d3aa2f57067b8927f89 [term 1 FOLLOWER]: Refusing update from remote peer 74f47306030d436f90298b8a17297d29: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250112 15:08:01.030563 22690 consensus_queue.cc:1035] T 8b7046b625f84ac3b2fe6d67946fe6bf P 74f47306030d436f90298b8a17297d29 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d2e717f485ad442c8bfccf44cf80aa3e" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 40357 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250112 15:08:01.030753 22576 consensus_queue.cc:1035] T 8b7046b625f84ac3b2fe6d67946fe6bf P 74f47306030d436f90298b8a17297d29 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a93555b72ffe4d3aa2f57067b8927f89" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 36981 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250112 15:08:01.031633 22447 raft_consensus.cc:2949] T 8b7046b625f84ac3b2fe6d67946fe6bf P d2e717f485ad442c8bfccf44cf80aa3e [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 72c5e551bf0d441f9a702da4848d7b1c (127.17.60.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a93555b72ffe4d3aa2f57067b8927f89" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 36981 } attrs { replace: true } } peers { permanent_uuid: "d2e717f485ad442c8bfccf44cf80aa3e" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 40357 } } peers { permanent_uuid: "74f47306030d436f90298b8a17297d29" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 34175 } } peers { permanent_uuid: "72c5e551bf0d441f9a702da4848d7b1c" member_type: NON_VOTER last_known_addr { host: "127.17.60.196" port: 35345 } attrs { promote: true } } }
I20250112 15:08:01.031633 22371 raft_consensus.cc:2949] T 8b7046b625f84ac3b2fe6d67946fe6bf P a93555b72ffe4d3aa2f57067b8927f89 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 72c5e551bf0d441f9a702da4848d7b1c (127.17.60.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a93555b72ffe4d3aa2f57067b8927f89" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 36981 } attrs { replace: true } } peers { permanent_uuid: "d2e717f485ad442c8bfccf44cf80aa3e" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 40357 } } peers { permanent_uuid: "74f47306030d436f90298b8a17297d29" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 34175 } } peers { permanent_uuid: "72c5e551bf0d441f9a702da4848d7b1c" member_type: NON_VOTER last_known_addr { host: "127.17.60.196" port: 35345 } attrs { promote: true } } }
W20250112 15:08:01.032922 22488 consensus_peers.cc:487] T 8b7046b625f84ac3b2fe6d67946fe6bf P 74f47306030d436f90298b8a17297d29 -> Peer 72c5e551bf0d441f9a702da4848d7b1c (127.17.60.196:35345): Couldn't send request to peer 72c5e551bf0d441f9a702da4848d7b1c. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 8b7046b625f84ac3b2fe6d67946fe6bf. This is attempt 1: this message will repeat every 5th retry.
I20250112 15:08:01.054728 22266 catalog_manager.cc:5526] T 8b7046b625f84ac3b2fe6d67946fe6bf P d2e717f485ad442c8bfccf44cf80aa3e reported cstate change: config changed from index -1 to 2, NON_VOTER 72c5e551bf0d441f9a702da4848d7b1c (127.17.60.196) added. New cstate: current_term: 1 leader_uuid: "74f47306030d436f90298b8a17297d29" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a93555b72ffe4d3aa2f57067b8927f89" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 36981 } attrs { replace: true } } peers { permanent_uuid: "d2e717f485ad442c8bfccf44cf80aa3e" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 40357 } } peers { permanent_uuid: "74f47306030d436f90298b8a17297d29" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 34175 } } peers { permanent_uuid: "72c5e551bf0d441f9a702da4848d7b1c" member_type: NON_VOTER last_known_addr { host: "127.17.60.196" port: 35345 } attrs { promote: true } } }
I20250112 15:08:01.065550 22693 ts_tablet_manager.cc:927] T 50c711f4e65148faa8062f3ae6d87990 P 72c5e551bf0d441f9a702da4848d7b1c: Initiating tablet copy from peer a93555b72ffe4d3aa2f57067b8927f89 (127.17.60.193:36981)
I20250112 15:08:01.065760 22693 tablet_copy_client.cc:323] T 50c711f4e65148faa8062f3ae6d87990 P 72c5e551bf0d441f9a702da4848d7b1c: tablet copy: Beginning tablet copy session from remote peer at address 127.17.60.193:36981
I20250112 15:08:01.068306 22381 tablet_copy_service.cc:140] P a93555b72ffe4d3aa2f57067b8927f89: Received BeginTabletCopySession request for tablet 50c711f4e65148faa8062f3ae6d87990 from peer 72c5e551bf0d441f9a702da4848d7b1c ({username='slave'} at 127.0.0.1:35788)
I20250112 15:08:01.068372 22381 tablet_copy_service.cc:161] P a93555b72ffe4d3aa2f57067b8927f89: Beginning new tablet copy session on tablet 50c711f4e65148faa8062f3ae6d87990 from peer 72c5e551bf0d441f9a702da4848d7b1c at {username='slave'} at 127.0.0.1:35788: session id = 72c5e551bf0d441f9a702da4848d7b1c-50c711f4e65148faa8062f3ae6d87990
I20250112 15:08:01.068799 22381 tablet_copy_source_session.cc:215] T 50c711f4e65148faa8062f3ae6d87990 P a93555b72ffe4d3aa2f57067b8927f89: Tablet Copy: opened 0 blocks and 1 log segments
I20250112 15:08:01.069150 22693 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 50c711f4e65148faa8062f3ae6d87990. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 15:08:01.070269 22693 tablet_copy_client.cc:806] T 50c711f4e65148faa8062f3ae6d87990 P 72c5e551bf0d441f9a702da4848d7b1c: tablet copy: Starting download of 0 data blocks...
I20250112 15:08:01.070374 22693 tablet_copy_client.cc:670] T 50c711f4e65148faa8062f3ae6d87990 P 72c5e551bf0d441f9a702da4848d7b1c: tablet copy: Starting download of 1 WAL segments...
I20250112 15:08:01.070864 22693 tablet_copy_client.cc:538] T 50c711f4e65148faa8062f3ae6d87990 P 72c5e551bf0d441f9a702da4848d7b1c: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250112 15:08:01.071359 22693 tablet_bootstrap.cc:492] T 50c711f4e65148faa8062f3ae6d87990 P 72c5e551bf0d441f9a702da4848d7b1c: Bootstrap starting.
I20250112 15:08:01.072559 22693 tablet_bootstrap.cc:492] T 50c711f4e65148faa8062f3ae6d87990 P 72c5e551bf0d441f9a702da4848d7b1c: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250112 15:08:01.072727 22693 tablet_bootstrap.cc:492] T 50c711f4e65148faa8062f3ae6d87990 P 72c5e551bf0d441f9a702da4848d7b1c: Bootstrap complete.
I20250112 15:08:01.072806 22693 ts_tablet_manager.cc:1397] T 50c711f4e65148faa8062f3ae6d87990 P 72c5e551bf0d441f9a702da4848d7b1c: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20250112 15:08:01.072957 22693 raft_consensus.cc:357] T 50c711f4e65148faa8062f3ae6d87990 P 72c5e551bf0d441f9a702da4848d7b1c [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "74f47306030d436f90298b8a17297d29" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 34175 } attrs { replace: true } } peers { permanent_uuid: "a93555b72ffe4d3aa2f57067b8927f89" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 36981 } } peers { permanent_uuid: "d2e717f485ad442c8bfccf44cf80aa3e" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 40357 } } peers { permanent_uuid: "72c5e551bf0d441f9a702da4848d7b1c" member_type: NON_VOTER last_known_addr { host: "127.17.60.196" port: 35345 } attrs { promote: true } }
I20250112 15:08:01.073017 22693 raft_consensus.cc:738] T 50c711f4e65148faa8062f3ae6d87990 P 72c5e551bf0d441f9a702da4848d7b1c [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 72c5e551bf0d441f9a702da4848d7b1c, State: Initialized, Role: LEARNER
I20250112 15:08:01.073050 22693 consensus_queue.cc:260] T 50c711f4e65148faa8062f3ae6d87990 P 72c5e551bf0d441f9a702da4848d7b1c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "74f47306030d436f90298b8a17297d29" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 34175 } attrs { replace: true } } peers { permanent_uuid: "a93555b72ffe4d3aa2f57067b8927f89" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 36981 } } peers { permanent_uuid: "d2e717f485ad442c8bfccf44cf80aa3e" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 40357 } } peers { permanent_uuid: "72c5e551bf0d441f9a702da4848d7b1c" member_type: NON_VOTER last_known_addr { host: "127.17.60.196" port: 35345 } attrs { promote: true } }
I20250112 15:08:01.073197 22693 ts_tablet_manager.cc:1428] T 50c711f4e65148faa8062f3ae6d87990 P 72c5e551bf0d441f9a702da4848d7b1c: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20250112 15:08:01.073246 22667 heartbeater.cc:502] Master 127.17.60.254:38743 was elected leader, sending a full tablet report...
I20250112 15:08:01.073382 22381 tablet_copy_service.cc:342] P a93555b72ffe4d3aa2f57067b8927f89: Request end of tablet copy session 72c5e551bf0d441f9a702da4848d7b1c-50c711f4e65148faa8062f3ae6d87990 received from {username='slave'} at 127.0.0.1:35788
I20250112 15:08:01.073448 22381 tablet_copy_service.cc:434] P a93555b72ffe4d3aa2f57067b8927f89: ending tablet copy session 72c5e551bf0d441f9a702da4848d7b1c-50c711f4e65148faa8062f3ae6d87990 on tablet 50c711f4e65148faa8062f3ae6d87990 with peer 72c5e551bf0d441f9a702da4848d7b1c
I20250112 15:08:01.111619 22690 raft_consensus.cc:2949] T 8b7046b625f84ac3b2fe6d67946fe6bf P 74f47306030d436f90298b8a17297d29 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 72c5e551bf0d441f9a702da4848d7b1c (127.17.60.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a93555b72ffe4d3aa2f57067b8927f89" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 36981 } attrs { replace: true } } peers { permanent_uuid: "d2e717f485ad442c8bfccf44cf80aa3e" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 40357 } } peers { permanent_uuid: "74f47306030d436f90298b8a17297d29" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 34175 } } peers { permanent_uuid: "72c5e551bf0d441f9a702da4848d7b1c" member_type: NON_VOTER last_known_addr { host: "127.17.60.196" port: 35345 } attrs { promote: true } } }
I20250112 15:08:01.130771 22693 ts_tablet_manager.cc:927] T 8b7046b625f84ac3b2fe6d67946fe6bf P 72c5e551bf0d441f9a702da4848d7b1c: Initiating tablet copy from peer 74f47306030d436f90298b8a17297d29 (127.17.60.195:34175)
I20250112 15:08:01.131006 22693 tablet_copy_client.cc:323] T 8b7046b625f84ac3b2fe6d67946fe6bf P 72c5e551bf0d441f9a702da4848d7b1c: tablet copy: Beginning tablet copy session from remote peer at address 127.17.60.195:34175
I20250112 15:08:01.132289 22674 raft_consensus.cc:988] T 68bb4bb68a93481cb8a3406f5f7d0b91 P 74f47306030d436f90298b8a17297d29: : Instructing follower d2e717f485ad442c8bfccf44cf80aa3e to start an election
I20250112 15:08:01.132364 22674 raft_consensus.cc:1076] T 68bb4bb68a93481cb8a3406f5f7d0b91 P 74f47306030d436f90298b8a17297d29 [term 1 LEADER]: Signalling peer d2e717f485ad442c8bfccf44cf80aa3e to start an election
I20250112 15:08:01.132519 22447 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "68bb4bb68a93481cb8a3406f5f7d0b91"
dest_uuid: "d2e717f485ad442c8bfccf44cf80aa3e"
from {username='slave'} at 127.0.0.1:58140
I20250112 15:08:01.132622 22447 raft_consensus.cc:491] T 68bb4bb68a93481cb8a3406f5f7d0b91 P d2e717f485ad442c8bfccf44cf80aa3e [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20250112 15:08:01.132653 22447 raft_consensus.cc:3054] T 68bb4bb68a93481cb8a3406f5f7d0b91 P d2e717f485ad442c8bfccf44cf80aa3e [term 1 FOLLOWER]: Advancing to term 2
I20250112 15:08:01.133437 22447 raft_consensus.cc:513] T 68bb4bb68a93481cb8a3406f5f7d0b91 P d2e717f485ad442c8bfccf44cf80aa3e [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a93555b72ffe4d3aa2f57067b8927f89" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 36981 } } peers { permanent_uuid: "d2e717f485ad442c8bfccf44cf80aa3e" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 40357 } } peers { permanent_uuid: "74f47306030d436f90298b8a17297d29" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 34175 } }
I20250112 15:08:01.133635 22447 leader_election.cc:290] T 68bb4bb68a93481cb8a3406f5f7d0b91 P d2e717f485ad442c8bfccf44cf80aa3e [CANDIDATE]: Term 2 election: Requested vote from peers a93555b72ffe4d3aa2f57067b8927f89 (127.17.60.193:36981), 74f47306030d436f90298b8a17297d29 (127.17.60.195:34175)
I20250112 15:08:01.133987 22532 tablet_copy_service.cc:140] P 74f47306030d436f90298b8a17297d29: Received BeginTabletCopySession request for tablet 8b7046b625f84ac3b2fe6d67946fe6bf from peer 72c5e551bf0d441f9a702da4848d7b1c ({username='slave'} at 127.0.0.1:58136)
I20250112 15:08:01.134047 22532 tablet_copy_service.cc:161] P 74f47306030d436f90298b8a17297d29: Beginning new tablet copy session on tablet 8b7046b625f84ac3b2fe6d67946fe6bf from peer 72c5e551bf0d441f9a702da4848d7b1c at {username='slave'} at 127.0.0.1:58136: session id = 72c5e551bf0d441f9a702da4848d7b1c-8b7046b625f84ac3b2fe6d67946fe6bf
I20250112 15:08:01.134683 22532 tablet_copy_source_session.cc:215] T 8b7046b625f84ac3b2fe6d67946fe6bf P 74f47306030d436f90298b8a17297d29: Tablet Copy: opened 0 blocks and 1 log segments
I20250112 15:08:01.135299 22693 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8b7046b625f84ac3b2fe6d67946fe6bf. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 15:08:01.136721 22693 tablet_copy_client.cc:806] T 8b7046b625f84ac3b2fe6d67946fe6bf P 72c5e551bf0d441f9a702da4848d7b1c: tablet copy: Starting download of 0 data blocks...
I20250112 15:08:01.136871 22371 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "68bb4bb68a93481cb8a3406f5f7d0b91" candidate_uuid: "d2e717f485ad442c8bfccf44cf80aa3e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "a93555b72ffe4d3aa2f57067b8927f89"
I20250112 15:08:01.136973 22371 raft_consensus.cc:3054] T 68bb4bb68a93481cb8a3406f5f7d0b91 P a93555b72ffe4d3aa2f57067b8927f89 [term 1 FOLLOWER]: Advancing to term 2
I20250112 15:08:01.137573 22371 raft_consensus.cc:2463] T 68bb4bb68a93481cb8a3406f5f7d0b91 P a93555b72ffe4d3aa2f57067b8927f89 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d2e717f485ad442c8bfccf44cf80aa3e in term 2.
I20250112 15:08:01.136873 22693 tablet_copy_client.cc:670] T 8b7046b625f84ac3b2fe6d67946fe6bf P 72c5e551bf0d441f9a702da4848d7b1c: tablet copy: Starting download of 1 WAL segments...
I20250112 15:08:01.137889 22413 leader_election.cc:304] T 68bb4bb68a93481cb8a3406f5f7d0b91 P d2e717f485ad442c8bfccf44cf80aa3e [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a93555b72ffe4d3aa2f57067b8927f89, d2e717f485ad442c8bfccf44cf80aa3e; no voters:
I20250112 15:08:01.137987 22522 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "68bb4bb68a93481cb8a3406f5f7d0b91" candidate_uuid: "d2e717f485ad442c8bfccf44cf80aa3e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "74f47306030d436f90298b8a17297d29"
I20250112 15:08:01.138007 22686 raft_consensus.cc:2798] T 68bb4bb68a93481cb8a3406f5f7d0b91 P d2e717f485ad442c8bfccf44cf80aa3e [term 2 FOLLOWER]: Leader election won for term 2
I20250112 15:08:01.138039 22522 raft_consensus.cc:3049] T 68bb4bb68a93481cb8a3406f5f7d0b91 P 74f47306030d436f90298b8a17297d29 [term 1 LEADER]: Stepping down as leader of term 1
I20250112 15:08:01.138064 22522 raft_consensus.cc:738] T 68bb4bb68a93481cb8a3406f5f7d0b91 P 74f47306030d436f90298b8a17297d29 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 74f47306030d436f90298b8a17297d29, State: Running, Role: LEADER
I20250112 15:08:01.138106 22522 consensus_queue.cc:260] T 68bb4bb68a93481cb8a3406f5f7d0b91 P 74f47306030d436f90298b8a17297d29 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a93555b72ffe4d3aa2f57067b8927f89" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 36981 } } peers { permanent_uuid: "d2e717f485ad442c8bfccf44cf80aa3e" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 40357 } } peers { permanent_uuid: "74f47306030d436f90298b8a17297d29" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 34175 } }
I20250112 15:08:01.138170 22686 raft_consensus.cc:695] T 68bb4bb68a93481cb8a3406f5f7d0b91 P d2e717f485ad442c8bfccf44cf80aa3e [term 2 LEADER]: Becoming Leader. State: Replica: d2e717f485ad442c8bfccf44cf80aa3e, State: Running, Role: LEADER
I20250112 15:08:01.138217 22522 raft_consensus.cc:3054] T 68bb4bb68a93481cb8a3406f5f7d0b91 P 74f47306030d436f90298b8a17297d29 [term 1 FOLLOWER]: Advancing to term 2
I20250112 15:08:01.138231 22686 consensus_queue.cc:237] T 68bb4bb68a93481cb8a3406f5f7d0b91 P d2e717f485ad442c8bfccf44cf80aa3e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a93555b72ffe4d3aa2f57067b8927f89" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 36981 } } peers { permanent_uuid: "d2e717f485ad442c8bfccf44cf80aa3e" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 40357 } } peers { permanent_uuid: "74f47306030d436f90298b8a17297d29" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 34175 } }
I20250112 15:08:01.138501 22447 raft_consensus.cc:1235] T 68bb4bb68a93481cb8a3406f5f7d0b91 P d2e717f485ad442c8bfccf44cf80aa3e [term 2 LEADER]: Rejecting Update request from peer 74f47306030d436f90298b8a17297d29 for earlier term 1. Current term is 2. Ops: []
I20250112 15:08:01.138935 22522 raft_consensus.cc:2463] T 68bb4bb68a93481cb8a3406f5f7d0b91 P 74f47306030d436f90298b8a17297d29 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d2e717f485ad442c8bfccf44cf80aa3e in term 2.
I20250112 15:08:01.139087 22267 catalog_manager.cc:5526] T 68bb4bb68a93481cb8a3406f5f7d0b91 P d2e717f485ad442c8bfccf44cf80aa3e reported cstate change: term changed from 1 to 2, leader changed from 74f47306030d436f90298b8a17297d29 (127.17.60.195) to d2e717f485ad442c8bfccf44cf80aa3e (127.17.60.194). New cstate: current_term: 2 leader_uuid: "d2e717f485ad442c8bfccf44cf80aa3e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a93555b72ffe4d3aa2f57067b8927f89" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 36981 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d2e717f485ad442c8bfccf44cf80aa3e" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 40357 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "74f47306030d436f90298b8a17297d29" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 34175 } health_report { overall_health: UNKNOWN } } }
I20250112 15:08:01.139518 22693 tablet_copy_client.cc:538] T 8b7046b625f84ac3b2fe6d67946fe6bf P 72c5e551bf0d441f9a702da4848d7b1c: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250112 15:08:01.140247 22693 tablet_bootstrap.cc:492] T 8b7046b625f84ac3b2fe6d67946fe6bf P 72c5e551bf0d441f9a702da4848d7b1c: Bootstrap starting.
I20250112 15:08:01.141738 22641 raft_consensus.cc:1212] T 50c711f4e65148faa8062f3ae6d87990 P 72c5e551bf0d441f9a702da4848d7b1c [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20250112 15:08:01.142123 22693 tablet_bootstrap.cc:492] T 8b7046b625f84ac3b2fe6d67946fe6bf P 72c5e551bf0d441f9a702da4848d7b1c: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250112 15:08:01.142319 22693 tablet_bootstrap.cc:492] T 8b7046b625f84ac3b2fe6d67946fe6bf P 72c5e551bf0d441f9a702da4848d7b1c: Bootstrap complete.
I20250112 15:08:01.142427 22693 ts_tablet_manager.cc:1397] T 8b7046b625f84ac3b2fe6d67946fe6bf P 72c5e551bf0d441f9a702da4848d7b1c: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.001s
I20250112 15:08:01.142597 22693 raft_consensus.cc:357] T 8b7046b625f84ac3b2fe6d67946fe6bf P 72c5e551bf0d441f9a702da4848d7b1c [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a93555b72ffe4d3aa2f57067b8927f89" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 36981 } attrs { replace: true } } peers { permanent_uuid: "d2e717f485ad442c8bfccf44cf80aa3e" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 40357 } } peers { permanent_uuid: "74f47306030d436f90298b8a17297d29" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 34175 } } peers { permanent_uuid: "72c5e551bf0d441f9a702da4848d7b1c" member_type: NON_VOTER last_known_addr { host: "127.17.60.196" port: 35345 } attrs { promote: true } }
I20250112 15:08:01.142661 22693 raft_consensus.cc:738] T 8b7046b625f84ac3b2fe6d67946fe6bf P 72c5e551bf0d441f9a702da4848d7b1c [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 72c5e551bf0d441f9a702da4848d7b1c, State: Initialized, Role: LEARNER
I20250112 15:08:01.142693 22693 consensus_queue.cc:260] T 8b7046b625f84ac3b2fe6d67946fe6bf P 72c5e551bf0d441f9a702da4848d7b1c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a93555b72ffe4d3aa2f57067b8927f89" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 36981 } attrs { replace: true } } peers { permanent_uuid: "d2e717f485ad442c8bfccf44cf80aa3e" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 40357 } } peers { permanent_uuid: "74f47306030d436f90298b8a17297d29" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 34175 } } peers { permanent_uuid: "72c5e551bf0d441f9a702da4848d7b1c" member_type: NON_VOTER last_known_addr { host: "127.17.60.196" port: 35345 } attrs { promote: true } }
I20250112 15:08:01.142839 22693 ts_tablet_manager.cc:1428] T 8b7046b625f84ac3b2fe6d67946fe6bf P 72c5e551bf0d441f9a702da4848d7b1c: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20250112 15:08:01.143020 22532 tablet_copy_service.cc:342] P 74f47306030d436f90298b8a17297d29: Request end of tablet copy session 72c5e551bf0d441f9a702da4848d7b1c-8b7046b625f84ac3b2fe6d67946fe6bf received from {username='slave'} at 127.0.0.1:58136
I20250112 15:08:01.143071 22532 tablet_copy_service.cc:434] P 74f47306030d436f90298b8a17297d29: ending tablet copy session 72c5e551bf0d441f9a702da4848d7b1c-8b7046b625f84ac3b2fe6d67946fe6bf on tablet 8b7046b625f84ac3b2fe6d67946fe6bf with peer 72c5e551bf0d441f9a702da4848d7b1c
I20250112 15:08:01.207197 22641 raft_consensus.cc:1212] T 8b7046b625f84ac3b2fe6d67946fe6bf P 72c5e551bf0d441f9a702da4848d7b1c [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20250112 15:08:01.255939 22521 raft_consensus.cc:1270] T 68bb4bb68a93481cb8a3406f5f7d0b91 P 74f47306030d436f90298b8a17297d29 [term 2 FOLLOWER]: Refusing update from remote peer d2e717f485ad442c8bfccf44cf80aa3e: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250112 15:08:01.256167 22702 consensus_queue.cc:1035] T 68bb4bb68a93481cb8a3406f5f7d0b91 P d2e717f485ad442c8bfccf44cf80aa3e [LEADER]: Connected to new peer: Peer: permanent_uuid: "74f47306030d436f90298b8a17297d29" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 34175 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250112 15:08:01.257624 22371 raft_consensus.cc:1270] T 68bb4bb68a93481cb8a3406f5f7d0b91 P a93555b72ffe4d3aa2f57067b8927f89 [term 2 FOLLOWER]: Refusing update from remote peer d2e717f485ad442c8bfccf44cf80aa3e: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250112 15:08:01.257814 22702 consensus_queue.cc:1035] T 68bb4bb68a93481cb8a3406f5f7d0b91 P d2e717f485ad442c8bfccf44cf80aa3e [LEADER]: Connected to new peer: Peer: permanent_uuid: "a93555b72ffe4d3aa2f57067b8927f89" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 36981 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250112 15:08:01.342195 22591 raft_consensus.cc:1059] T 50c711f4e65148faa8062f3ae6d87990 P a93555b72ffe4d3aa2f57067b8927f89: attempting to promote NON_VOTER 72c5e551bf0d441f9a702da4848d7b1c to VOTER
I20250112 15:08:01.342453 22591 consensus_queue.cc:237] T 50c711f4e65148faa8062f3ae6d87990 P a93555b72ffe4d3aa2f57067b8927f89 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "74f47306030d436f90298b8a17297d29" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 34175 } attrs { replace: true } } peers { permanent_uuid: "a93555b72ffe4d3aa2f57067b8927f89" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 36981 } } peers { permanent_uuid: "d2e717f485ad442c8bfccf44cf80aa3e" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 40357 } } peers { permanent_uuid: "72c5e551bf0d441f9a702da4848d7b1c" member_type: VOTER last_known_addr { host: "127.17.60.196" port: 35345 } attrs { promote: false } }
I20250112 15:08:01.342965 22641 raft_consensus.cc:1270] T 50c711f4e65148faa8062f3ae6d87990 P 72c5e551bf0d441f9a702da4848d7b1c [term 1 LEARNER]: Refusing update from remote peer a93555b72ffe4d3aa2f57067b8927f89: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250112 15:08:01.342931 22521 raft_consensus.cc:1270] T 50c711f4e65148faa8062f3ae6d87990 P 74f47306030d436f90298b8a17297d29 [term 1 FOLLOWER]: Refusing update from remote peer a93555b72ffe4d3aa2f57067b8927f89: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250112 15:08:01.342931 22447 raft_consensus.cc:1270] T 50c711f4e65148faa8062f3ae6d87990 P d2e717f485ad442c8bfccf44cf80aa3e [term 1 FOLLOWER]: Refusing update from remote peer a93555b72ffe4d3aa2f57067b8927f89: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250112 15:08:01.343207 22689 consensus_queue.cc:1035] T 50c711f4e65148faa8062f3ae6d87990 P a93555b72ffe4d3aa2f57067b8927f89 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d2e717f485ad442c8bfccf44cf80aa3e" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 40357 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250112 15:08:01.343346 22689 consensus_queue.cc:1035] T 50c711f4e65148faa8062f3ae6d87990 P a93555b72ffe4d3aa2f57067b8927f89 [LEADER]: Connected to new peer: Peer: permanent_uuid: "74f47306030d436f90298b8a17297d29" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 34175 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250112 15:08:01.343501 22678 consensus_queue.cc:1035] T 50c711f4e65148faa8062f3ae6d87990 P a93555b72ffe4d3aa2f57067b8927f89 [LEADER]: Connected to new peer: Peer: permanent_uuid: "72c5e551bf0d441f9a702da4848d7b1c" member_type: VOTER last_known_addr { host: "127.17.60.196" port: 35345 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250112 15:08:01.344403 22641 raft_consensus.cc:2949] T 50c711f4e65148faa8062f3ae6d87990 P 72c5e551bf0d441f9a702da4848d7b1c [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 72c5e551bf0d441f9a702da4848d7b1c (127.17.60.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "74f47306030d436f90298b8a17297d29" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 34175 } attrs { replace: true } } peers { permanent_uuid: "a93555b72ffe4d3aa2f57067b8927f89" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 36981 } } peers { permanent_uuid: "d2e717f485ad442c8bfccf44cf80aa3e" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 40357 } } peers { permanent_uuid: "72c5e551bf0d441f9a702da4848d7b1c" member_type: VOTER last_known_addr { host: "127.17.60.196" port: 35345 } attrs { promote: false } } }
I20250112 15:08:01.344905 22521 raft_consensus.cc:2949] T 50c711f4e65148faa8062f3ae6d87990 P 74f47306030d436f90298b8a17297d29 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 72c5e551bf0d441f9a702da4848d7b1c (127.17.60.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "74f47306030d436f90298b8a17297d29" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 34175 } attrs { replace: true } } peers { permanent_uuid: "a93555b72ffe4d3aa2f57067b8927f89" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 36981 } } peers { permanent_uuid: "d2e717f485ad442c8bfccf44cf80aa3e" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 40357 } } peers { permanent_uuid: "72c5e551bf0d441f9a702da4848d7b1c" member_type: VOTER last_known_addr { host: "127.17.60.196" port: 35345 } attrs { promote: false } } }
I20250112 15:08:01.345711 22267 catalog_manager.cc:5526] T 50c711f4e65148faa8062f3ae6d87990 P 72c5e551bf0d441f9a702da4848d7b1c reported cstate change: config changed from index 2 to 3, 72c5e551bf0d441f9a702da4848d7b1c (127.17.60.196) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "a93555b72ffe4d3aa2f57067b8927f89" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "74f47306030d436f90298b8a17297d29" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 34175 } attrs { replace: true } } peers { permanent_uuid: "a93555b72ffe4d3aa2f57067b8927f89" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 36981 } } peers { permanent_uuid: "d2e717f485ad442c8bfccf44cf80aa3e" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 40357 } } peers { permanent_uuid: "72c5e551bf0d441f9a702da4848d7b1c" member_type: VOTER last_known_addr { host: "127.17.60.196" port: 35345 } attrs { promote: false } } }
I20250112 15:08:01.389503 22576 raft_consensus.cc:1059] T 8b7046b625f84ac3b2fe6d67946fe6bf P 74f47306030d436f90298b8a17297d29: attempting to promote NON_VOTER 72c5e551bf0d441f9a702da4848d7b1c to VOTER
I20250112 15:08:01.389739 22576 consensus_queue.cc:237] T 8b7046b625f84ac3b2fe6d67946fe6bf P 74f47306030d436f90298b8a17297d29 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a93555b72ffe4d3aa2f57067b8927f89" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 36981 } attrs { replace: true } } peers { permanent_uuid: "d2e717f485ad442c8bfccf44cf80aa3e" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 40357 } } peers { permanent_uuid: "74f47306030d436f90298b8a17297d29" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 34175 } } peers { permanent_uuid: "72c5e551bf0d441f9a702da4848d7b1c" member_type: VOTER last_known_addr { host: "127.17.60.196" port: 35345 } attrs { promote: false } }
I20250112 15:08:01.390197 22370 raft_consensus.cc:1270] T 8b7046b625f84ac3b2fe6d67946fe6bf P a93555b72ffe4d3aa2f57067b8927f89 [term 1 FOLLOWER]: Refusing update from remote peer 74f47306030d436f90298b8a17297d29: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250112 15:08:01.390204 22641 raft_consensus.cc:1270] T 8b7046b625f84ac3b2fe6d67946fe6bf P 72c5e551bf0d441f9a702da4848d7b1c [term 1 LEARNER]: Refusing update from remote peer 74f47306030d436f90298b8a17297d29: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250112 15:08:01.390256 22447 raft_consensus.cc:1270] T 8b7046b625f84ac3b2fe6d67946fe6bf P d2e717f485ad442c8bfccf44cf80aa3e [term 1 FOLLOWER]: Refusing update from remote peer 74f47306030d436f90298b8a17297d29: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250112 15:08:01.390497 22690 consensus_queue.cc:1035] T 8b7046b625f84ac3b2fe6d67946fe6bf P 74f47306030d436f90298b8a17297d29 [LEADER]: Connected to new peer: Peer: permanent_uuid: "72c5e551bf0d441f9a702da4848d7b1c" member_type: VOTER last_known_addr { host: "127.17.60.196" port: 35345 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250112 15:08:01.390722 22698 consensus_queue.cc:1035] T 8b7046b625f84ac3b2fe6d67946fe6bf P 74f47306030d436f90298b8a17297d29 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a93555b72ffe4d3aa2f57067b8927f89" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 36981 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250112 15:08:01.390925 22690 consensus_queue.cc:1035] T 8b7046b625f84ac3b2fe6d67946fe6bf P 74f47306030d436f90298b8a17297d29 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d2e717f485ad442c8bfccf44cf80aa3e" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 40357 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250112 15:08:01.391646 22641 raft_consensus.cc:2949] T 8b7046b625f84ac3b2fe6d67946fe6bf P 72c5e551bf0d441f9a702da4848d7b1c [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 72c5e551bf0d441f9a702da4848d7b1c (127.17.60.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a93555b72ffe4d3aa2f57067b8927f89" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 36981 } attrs { replace: true } } peers { permanent_uuid: "d2e717f485ad442c8bfccf44cf80aa3e" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 40357 } } peers { permanent_uuid: "74f47306030d436f90298b8a17297d29" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 34175 } } peers { permanent_uuid: "72c5e551bf0d441f9a702da4848d7b1c" member_type: VOTER last_known_addr { host: "127.17.60.196" port: 35345 } attrs { promote: false } } }
I20250112 15:08:01.391629 22447 raft_consensus.cc:2949] T 8b7046b625f84ac3b2fe6d67946fe6bf P d2e717f485ad442c8bfccf44cf80aa3e [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 72c5e551bf0d441f9a702da4848d7b1c (127.17.60.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a93555b72ffe4d3aa2f57067b8927f89" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 36981 } attrs { replace: true } } peers { permanent_uuid: "d2e717f485ad442c8bfccf44cf80aa3e" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 40357 } } peers { permanent_uuid: "74f47306030d436f90298b8a17297d29" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 34175 } } peers { permanent_uuid: "72c5e551bf0d441f9a702da4848d7b1c" member_type: VOTER last_known_addr { host: "127.17.60.196" port: 35345 } attrs { promote: false } } }
I20250112 15:08:01.392802 22266 catalog_manager.cc:5526] T 8b7046b625f84ac3b2fe6d67946fe6bf P 72c5e551bf0d441f9a702da4848d7b1c reported cstate change: config changed from index 2 to 3, 72c5e551bf0d441f9a702da4848d7b1c (127.17.60.196) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "74f47306030d436f90298b8a17297d29" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a93555b72ffe4d3aa2f57067b8927f89" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 36981 } attrs { replace: true } } peers { permanent_uuid: "d2e717f485ad442c8bfccf44cf80aa3e" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 40357 } } peers { permanent_uuid: "74f47306030d436f90298b8a17297d29" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 34175 } } peers { permanent_uuid: "72c5e551bf0d441f9a702da4848d7b1c" member_type: VOTER last_known_addr { host: "127.17.60.196" port: 35345 } attrs { promote: false } } }
I20250112 15:08:01.402347 22689 raft_consensus.cc:2949] T 50c711f4e65148faa8062f3ae6d87990 P a93555b72ffe4d3aa2f57067b8927f89 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 72c5e551bf0d441f9a702da4848d7b1c (127.17.60.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "74f47306030d436f90298b8a17297d29" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 34175 } attrs { replace: true } } peers { permanent_uuid: "a93555b72ffe4d3aa2f57067b8927f89" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 36981 } } peers { permanent_uuid: "d2e717f485ad442c8bfccf44cf80aa3e" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 40357 } } peers { permanent_uuid: "72c5e551bf0d441f9a702da4848d7b1c" member_type: VOTER last_known_addr { host: "127.17.60.196" port: 35345 } attrs { promote: false } } }
I20250112 15:08:01.403426 22447 raft_consensus.cc:2949] T 50c711f4e65148faa8062f3ae6d87990 P d2e717f485ad442c8bfccf44cf80aa3e [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 72c5e551bf0d441f9a702da4848d7b1c (127.17.60.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "74f47306030d436f90298b8a17297d29" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 34175 } attrs { replace: true } } peers { permanent_uuid: "a93555b72ffe4d3aa2f57067b8927f89" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 36981 } } peers { permanent_uuid: "d2e717f485ad442c8bfccf44cf80aa3e" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 40357 } } peers { permanent_uuid: "72c5e551bf0d441f9a702da4848d7b1c" member_type: VOTER last_known_addr { host: "127.17.60.196" port: 35345 } attrs { promote: false } } }
I20250112 15:08:01.403802 22371 consensus_queue.cc:237] T 50c711f4e65148faa8062f3ae6d87990 P a93555b72ffe4d3aa2f57067b8927f89 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a93555b72ffe4d3aa2f57067b8927f89" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 36981 } } peers { permanent_uuid: "d2e717f485ad442c8bfccf44cf80aa3e" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 40357 } } peers { permanent_uuid: "72c5e551bf0d441f9a702da4848d7b1c" member_type: VOTER last_known_addr { host: "127.17.60.196" port: 35345 } attrs { promote: false } }
I20250112 15:08:01.404129 22641 raft_consensus.cc:1270] T 50c711f4e65148faa8062f3ae6d87990 P 72c5e551bf0d441f9a702da4848d7b1c [term 1 FOLLOWER]: Refusing update from remote peer a93555b72ffe4d3aa2f57067b8927f89: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20250112 15:08:01.404307 22712 consensus_queue.cc:1035] T 50c711f4e65148faa8062f3ae6d87990 P a93555b72ffe4d3aa2f57067b8927f89 [LEADER]: Connected to new peer: Peer: permanent_uuid: "72c5e551bf0d441f9a702da4848d7b1c" member_type: VOTER last_known_addr { host: "127.17.60.196" port: 35345 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250112 15:08:01.404543 22447 raft_consensus.cc:1270] T 50c711f4e65148faa8062f3ae6d87990 P d2e717f485ad442c8bfccf44cf80aa3e [term 1 FOLLOWER]: Refusing update from remote peer a93555b72ffe4d3aa2f57067b8927f89: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20250112 15:08:01.404918 22591 consensus_queue.cc:1035] T 50c711f4e65148faa8062f3ae6d87990 P a93555b72ffe4d3aa2f57067b8927f89 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d2e717f485ad442c8bfccf44cf80aa3e" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 40357 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250112 15:08:01.405208 22640 raft_consensus.cc:2949] T 50c711f4e65148faa8062f3ae6d87990 P 72c5e551bf0d441f9a702da4848d7b1c [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 74f47306030d436f90298b8a17297d29 (127.17.60.195) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a93555b72ffe4d3aa2f57067b8927f89" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 36981 } } peers { permanent_uuid: "d2e717f485ad442c8bfccf44cf80aa3e" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 40357 } } peers { permanent_uuid: "72c5e551bf0d441f9a702da4848d7b1c" member_type: VOTER last_known_addr { host: "127.17.60.196" port: 35345 } attrs { promote: false } } }
I20250112 15:08:01.405292 22447 raft_consensus.cc:2949] T 50c711f4e65148faa8062f3ae6d87990 P d2e717f485ad442c8bfccf44cf80aa3e [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 74f47306030d436f90298b8a17297d29 (127.17.60.195) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a93555b72ffe4d3aa2f57067b8927f89" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 36981 } } peers { permanent_uuid: "d2e717f485ad442c8bfccf44cf80aa3e" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 40357 } } peers { permanent_uuid: "72c5e551bf0d441f9a702da4848d7b1c" member_type: VOTER last_known_addr { host: "127.17.60.196" port: 35345 } attrs { promote: false } } }
I20250112 15:08:01.459614 22674 raft_consensus.cc:2949] T 8b7046b625f84ac3b2fe6d67946fe6bf P 74f47306030d436f90298b8a17297d29 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 72c5e551bf0d441f9a702da4848d7b1c (127.17.60.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a93555b72ffe4d3aa2f57067b8927f89" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 36981 } attrs { replace: true } } peers { permanent_uuid: "d2e717f485ad442c8bfccf44cf80aa3e" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 40357 } } peers { permanent_uuid: "74f47306030d436f90298b8a17297d29" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 34175 } } peers { permanent_uuid: "72c5e551bf0d441f9a702da4848d7b1c" member_type: VOTER last_known_addr { host: "127.17.60.196" port: 35345 } attrs { promote: false } } }
I20250112 15:08:01.460870 22370 raft_consensus.cc:2949] T 8b7046b625f84ac3b2fe6d67946fe6bf P a93555b72ffe4d3aa2f57067b8927f89 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 72c5e551bf0d441f9a702da4848d7b1c (127.17.60.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a93555b72ffe4d3aa2f57067b8927f89" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 36981 } attrs { replace: true } } peers { permanent_uuid: "d2e717f485ad442c8bfccf44cf80aa3e" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 40357 } } peers { permanent_uuid: "74f47306030d436f90298b8a17297d29" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 34175 } } peers { permanent_uuid: "72c5e551bf0d441f9a702da4848d7b1c" member_type: VOTER last_known_addr { host: "127.17.60.196" port: 35345 } attrs { promote: false } } }
I20250112 15:08:01.462811 22268 catalog_manager.cc:5526] T 50c711f4e65148faa8062f3ae6d87990 P 72c5e551bf0d441f9a702da4848d7b1c reported cstate change: config changed from index 3 to 4, VOTER 74f47306030d436f90298b8a17297d29 (127.17.60.195) evicted. New cstate: current_term: 1 leader_uuid: "a93555b72ffe4d3aa2f57067b8927f89" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a93555b72ffe4d3aa2f57067b8927f89" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 36981 } } peers { permanent_uuid: "d2e717f485ad442c8bfccf44cf80aa3e" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 40357 } } peers { permanent_uuid: "72c5e551bf0d441f9a702da4848d7b1c" member_type: VOTER last_known_addr { host: "127.17.60.196" port: 35345 } attrs { promote: false } } }
I20250112 15:08:01.462891 22521 consensus_queue.cc:237] T 8b7046b625f84ac3b2fe6d67946fe6bf P 74f47306030d436f90298b8a17297d29 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d2e717f485ad442c8bfccf44cf80aa3e" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 40357 } } peers { permanent_uuid: "74f47306030d436f90298b8a17297d29" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 34175 } } peers { permanent_uuid: "72c5e551bf0d441f9a702da4848d7b1c" member_type: VOTER last_known_addr { host: "127.17.60.196" port: 35345 } attrs { promote: false } }
I20250112 15:08:01.463249 22447 raft_consensus.cc:1270] T 8b7046b625f84ac3b2fe6d67946fe6bf P d2e717f485ad442c8bfccf44cf80aa3e [term 1 FOLLOWER]: Refusing update from remote peer 74f47306030d436f90298b8a17297d29: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20250112 15:08:01.463434 22640 raft_consensus.cc:1270] T 8b7046b625f84ac3b2fe6d67946fe6bf P 72c5e551bf0d441f9a702da4848d7b1c [term 1 FOLLOWER]: Refusing update from remote peer 74f47306030d436f90298b8a17297d29: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20250112 15:08:01.463565 22698 consensus_queue.cc:1035] T 8b7046b625f84ac3b2fe6d67946fe6bf P 74f47306030d436f90298b8a17297d29 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d2e717f485ad442c8bfccf44cf80aa3e" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 40357 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250112 15:08:01.463701 22713 consensus_queue.cc:1035] T 8b7046b625f84ac3b2fe6d67946fe6bf P 74f47306030d436f90298b8a17297d29 [LEADER]: Connected to new peer: Peer: permanent_uuid: "72c5e551bf0d441f9a702da4848d7b1c" member_type: VOTER last_known_addr { host: "127.17.60.196" port: 35345 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250112 15:08:01.464365 22447 raft_consensus.cc:2949] T 8b7046b625f84ac3b2fe6d67946fe6bf P d2e717f485ad442c8bfccf44cf80aa3e [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER a93555b72ffe4d3aa2f57067b8927f89 (127.17.60.193) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d2e717f485ad442c8bfccf44cf80aa3e" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 40357 } } peers { permanent_uuid: "74f47306030d436f90298b8a17297d29" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 34175 } } peers { permanent_uuid: "72c5e551bf0d441f9a702da4848d7b1c" member_type: VOTER last_known_addr { host: "127.17.60.196" port: 35345 } attrs { promote: false } } }
I20250112 15:08:01.464376 22640 raft_consensus.cc:2949] T 8b7046b625f84ac3b2fe6d67946fe6bf P 72c5e551bf0d441f9a702da4848d7b1c [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER a93555b72ffe4d3aa2f57067b8927f89 (127.17.60.193) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d2e717f485ad442c8bfccf44cf80aa3e" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 40357 } } peers { permanent_uuid: "74f47306030d436f90298b8a17297d29" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 34175 } } peers { permanent_uuid: "72c5e551bf0d441f9a702da4848d7b1c" member_type: VOTER last_known_addr { host: "127.17.60.196" port: 35345 } attrs { promote: false } } }
I20250112 15:08:01.539788 22689 raft_consensus.cc:2949] T 50c711f4e65148faa8062f3ae6d87990 P a93555b72ffe4d3aa2f57067b8927f89 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 74f47306030d436f90298b8a17297d29 (127.17.60.195) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a93555b72ffe4d3aa2f57067b8927f89" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 36981 } } peers { permanent_uuid: "d2e717f485ad442c8bfccf44cf80aa3e" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 40357 } } peers { permanent_uuid: "72c5e551bf0d441f9a702da4848d7b1c" member_type: VOTER last_known_addr { host: "127.17.60.196" port: 35345 } attrs { promote: false } } }
I20250112 15:08:01.540923 22255 catalog_manager.cc:5039] ChangeConfig:REMOVE_PEER RPC for tablet 50c711f4e65148faa8062f3ae6d87990 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250112 15:08:01.553929 22511 tablet_service.cc:1514] Processing DeleteTablet for tablet 50c711f4e65148faa8062f3ae6d87990 with delete_type TABLET_DATA_TOMBSTONED (TS 74f47306030d436f90298b8a17297d29 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:58098
I20250112 15:08:01.554029 22265 catalog_manager.cc:5526] T 8b7046b625f84ac3b2fe6d67946fe6bf P d2e717f485ad442c8bfccf44cf80aa3e reported cstate change: config changed from index 3 to 4, VOTER a93555b72ffe4d3aa2f57067b8927f89 (127.17.60.193) evicted. New cstate: current_term: 1 leader_uuid: "74f47306030d436f90298b8a17297d29" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d2e717f485ad442c8bfccf44cf80aa3e" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 40357 } } peers { permanent_uuid: "74f47306030d436f90298b8a17297d29" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 34175 } } peers { permanent_uuid: "72c5e551bf0d441f9a702da4848d7b1c" member_type: VOTER last_known_addr { host: "127.17.60.196" port: 35345 } attrs { promote: false } } }
I20250112 15:08:01.554167 22714 tablet_replica.cc:331] T 50c711f4e65148faa8062f3ae6d87990 P 74f47306030d436f90298b8a17297d29: stopping tablet replica
I20250112 15:08:01.554235 22714 raft_consensus.cc:2238] T 50c711f4e65148faa8062f3ae6d87990 P 74f47306030d436f90298b8a17297d29 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250112 15:08:01.554280 22714 raft_consensus.cc:2267] T 50c711f4e65148faa8062f3ae6d87990 P 74f47306030d436f90298b8a17297d29 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 15:08:01.554622 22714 ts_tablet_manager.cc:1905] T 50c711f4e65148faa8062f3ae6d87990 P 74f47306030d436f90298b8a17297d29: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250112 15:08:01.556227 22714 ts_tablet_manager.cc:1918] T 50c711f4e65148faa8062f3ae6d87990 P 74f47306030d436f90298b8a17297d29: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20250112 15:08:01.556278 22714 log.cc:1198] T 50c711f4e65148faa8062f3ae6d87990 P 74f47306030d436f90298b8a17297d29: Deleting WAL directory at /tmp/dist-test-task95dLSD/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingUnstableCluster.1736694351070047-17651-0/minicluster-data/ts-2-root/wals/50c711f4e65148faa8062f3ae6d87990
I20250112 15:08:01.556650 22254 catalog_manager.cc:4872] TS 74f47306030d436f90298b8a17297d29 (127.17.60.195:34175): tablet 50c711f4e65148faa8062f3ae6d87990 (table test-workload [id=cf7476fdd897444a8f48782f82fbc535]) successfully deleted
I20250112 15:08:01.587042 22360 tablet_service.cc:1514] Processing DeleteTablet for tablet 8b7046b625f84ac3b2fe6d67946fe6bf with delete_type TABLET_DATA_TOMBSTONED (TS a93555b72ffe4d3aa2f57067b8927f89 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:35760
I20250112 15:08:01.587334 22715 tablet_replica.cc:331] T 8b7046b625f84ac3b2fe6d67946fe6bf P a93555b72ffe4d3aa2f57067b8927f89: stopping tablet replica
I20250112 15:08:01.587417 22715 raft_consensus.cc:2238] T 8b7046b625f84ac3b2fe6d67946fe6bf P a93555b72ffe4d3aa2f57067b8927f89 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250112 15:08:01.587532 22715 raft_consensus.cc:2267] T 8b7046b625f84ac3b2fe6d67946fe6bf P a93555b72ffe4d3aa2f57067b8927f89 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 15:08:01.587802 22715 ts_tablet_manager.cc:1905] T 8b7046b625f84ac3b2fe6d67946fe6bf P a93555b72ffe4d3aa2f57067b8927f89: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250112 15:08:01.589442 22715 ts_tablet_manager.cc:1918] T 8b7046b625f84ac3b2fe6d67946fe6bf P a93555b72ffe4d3aa2f57067b8927f89: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20250112 15:08:01.589509 22715 log.cc:1198] T 8b7046b625f84ac3b2fe6d67946fe6bf P a93555b72ffe4d3aa2f57067b8927f89: Deleting WAL directory at /tmp/dist-test-task95dLSD/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingUnstableCluster.1736694351070047-17651-0/minicluster-data/ts-0-root/wals/8b7046b625f84ac3b2fe6d67946fe6bf
I20250112 15:08:01.589849 22255 catalog_manager.cc:4872] TS a93555b72ffe4d3aa2f57067b8927f89 (127.17.60.193:36981): tablet 8b7046b625f84ac3b2fe6d67946fe6bf (table test-workload [id=cf7476fdd897444a8f48782f82fbc535]) successfully deleted
I20250112 15:08:01.623212 22674 raft_consensus.cc:2949] T 8b7046b625f84ac3b2fe6d67946fe6bf P 74f47306030d436f90298b8a17297d29 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER a93555b72ffe4d3aa2f57067b8927f89 (127.17.60.193) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d2e717f485ad442c8bfccf44cf80aa3e" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 40357 } } peers { permanent_uuid: "74f47306030d436f90298b8a17297d29" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 34175 } } peers { permanent_uuid: "72c5e551bf0d441f9a702da4848d7b1c" member_type: VOTER last_known_addr { host: "127.17.60.196" port: 35345 } attrs { promote: false } } }
I20250112 15:08:01.624183 22254 catalog_manager.cc:5039] ChangeConfig:REMOVE_PEER RPC for tablet 8b7046b625f84ac3b2fe6d67946fe6bf with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250112 15:08:02.970366 22322 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250112 15:08:02.970531 22322 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20250112 15:08:02.970553 22322 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20250112 15:08:04.970719 22322 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250112 15:08:04.970871 22322 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20250112 15:08:04.970894 22322 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20250112 15:08:06.971045 22322 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250112 15:08:06.971197 22322 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20250112 15:08:06.971231 22322 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20250112 15:08:08.971386 22322 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250112 15:08:08.971520 22322 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20250112 15:08:08.971537 22322 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20250112 15:08:09.281931 17651 tablet_server.cc:178] TabletServer@127.17.60.193:0 shutting down...
I20250112 15:08:09.284720 17651 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250112 15:08:09.284852 17651 tablet_replica.cc:331] T 68bb4bb68a93481cb8a3406f5f7d0b91 P a93555b72ffe4d3aa2f57067b8927f89: stopping tablet replica
I20250112 15:08:09.284906 17651 raft_consensus.cc:2238] T 68bb4bb68a93481cb8a3406f5f7d0b91 P a93555b72ffe4d3aa2f57067b8927f89 [term 2 FOLLOWER]: Raft consensus shutting down.
I20250112 15:08:09.284956 17651 raft_consensus.cc:2267] T 68bb4bb68a93481cb8a3406f5f7d0b91 P a93555b72ffe4d3aa2f57067b8927f89 [term 2 FOLLOWER]: Raft consensus is shut down!
I20250112 15:08:09.285135 17651 tablet_replica.cc:331] T 50c711f4e65148faa8062f3ae6d87990 P a93555b72ffe4d3aa2f57067b8927f89: stopping tablet replica
I20250112 15:08:09.285171 17651 raft_consensus.cc:2238] T 50c711f4e65148faa8062f3ae6d87990 P a93555b72ffe4d3aa2f57067b8927f89 [term 1 LEADER]: Raft consensus shutting down.
I20250112 15:08:09.285277 17651 raft_consensus.cc:2267] T 50c711f4e65148faa8062f3ae6d87990 P a93555b72ffe4d3aa2f57067b8927f89 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 15:08:09.297302 17651 tablet_server.cc:195] TabletServer@127.17.60.193:0 shutdown complete.
I20250112 15:08:09.298446 17651 tablet_server.cc:178] TabletServer@127.17.60.194:0 shutting down...
I20250112 15:08:09.301352 17651 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250112 15:08:09.301514 17651 tablet_replica.cc:331] T 8b7046b625f84ac3b2fe6d67946fe6bf P d2e717f485ad442c8bfccf44cf80aa3e: stopping tablet replica
I20250112 15:08:09.301587 17651 raft_consensus.cc:2238] T 8b7046b625f84ac3b2fe6d67946fe6bf P d2e717f485ad442c8bfccf44cf80aa3e [term 1 FOLLOWER]: Raft consensus shutting down.
I20250112 15:08:09.301651 17651 raft_consensus.cc:2267] T 8b7046b625f84ac3b2fe6d67946fe6bf P d2e717f485ad442c8bfccf44cf80aa3e [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 15:08:09.301805 17651 tablet_replica.cc:331] T 68bb4bb68a93481cb8a3406f5f7d0b91 P d2e717f485ad442c8bfccf44cf80aa3e: stopping tablet replica
I20250112 15:08:09.301841 17651 raft_consensus.cc:2238] T 68bb4bb68a93481cb8a3406f5f7d0b91 P d2e717f485ad442c8bfccf44cf80aa3e [term 2 LEADER]: Raft consensus shutting down.
I20250112 15:08:09.301893 17651 raft_consensus.cc:2267] T 68bb4bb68a93481cb8a3406f5f7d0b91 P d2e717f485ad442c8bfccf44cf80aa3e [term 2 FOLLOWER]: Raft consensus is shut down!
I20250112 15:08:09.302024 17651 tablet_replica.cc:331] T 50c711f4e65148faa8062f3ae6d87990 P d2e717f485ad442c8bfccf44cf80aa3e: stopping tablet replica
I20250112 15:08:09.302057 17651 raft_consensus.cc:2238] T 50c711f4e65148faa8062f3ae6d87990 P d2e717f485ad442c8bfccf44cf80aa3e [term 1 FOLLOWER]: Raft consensus shutting down.
I20250112 15:08:09.302091 17651 raft_consensus.cc:2267] T 50c711f4e65148faa8062f3ae6d87990 P d2e717f485ad442c8bfccf44cf80aa3e [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 15:08:09.313905 17651 tablet_server.cc:195] TabletServer@127.17.60.194:0 shutdown complete.
I20250112 15:08:09.314889 17651 tablet_server.cc:178] TabletServer@127.17.60.195:0 shutting down...
I20250112 15:08:09.317402 17651 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250112 15:08:09.317504 17651 tablet_replica.cc:331] T 8b7046b625f84ac3b2fe6d67946fe6bf P 74f47306030d436f90298b8a17297d29: stopping tablet replica
I20250112 15:08:09.317587 17651 raft_consensus.cc:2238] T 8b7046b625f84ac3b2fe6d67946fe6bf P 74f47306030d436f90298b8a17297d29 [term 1 LEADER]: Raft consensus shutting down.
I20250112 15:08:09.317663 17651 raft_consensus.cc:2267] T 8b7046b625f84ac3b2fe6d67946fe6bf P 74f47306030d436f90298b8a17297d29 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 15:08:09.317838 17651 tablet_replica.cc:331] T 68bb4bb68a93481cb8a3406f5f7d0b91 P 74f47306030d436f90298b8a17297d29: stopping tablet replica
I20250112 15:08:09.317867 17651 raft_consensus.cc:2238] T 68bb4bb68a93481cb8a3406f5f7d0b91 P 74f47306030d436f90298b8a17297d29 [term 2 FOLLOWER]: Raft consensus shutting down.
I20250112 15:08:09.317888 17651 raft_consensus.cc:2267] T 68bb4bb68a93481cb8a3406f5f7d0b91 P 74f47306030d436f90298b8a17297d29 [term 2 FOLLOWER]: Raft consensus is shut down!
I20250112 15:08:09.329398 17651 tablet_server.cc:195] TabletServer@127.17.60.195:0 shutdown complete.
I20250112 15:08:09.330439 17651 tablet_server.cc:178] TabletServer@127.17.60.196:0 shutting down...
I20250112 15:08:09.332604 17651 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250112 15:08:09.332731 17651 tablet_replica.cc:331] T 8b7046b625f84ac3b2fe6d67946fe6bf P 72c5e551bf0d441f9a702da4848d7b1c: stopping tablet replica
I20250112 15:08:09.332801 17651 raft_consensus.cc:2238] T 8b7046b625f84ac3b2fe6d67946fe6bf P 72c5e551bf0d441f9a702da4848d7b1c [term 1 FOLLOWER]: Raft consensus shutting down.
I20250112 15:08:09.332863 17651 raft_consensus.cc:2267] T 8b7046b625f84ac3b2fe6d67946fe6bf P 72c5e551bf0d441f9a702da4848d7b1c [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 15:08:09.333026 17651 tablet_replica.cc:331] T 50c711f4e65148faa8062f3ae6d87990 P 72c5e551bf0d441f9a702da4848d7b1c: stopping tablet replica
I20250112 15:08:09.333057 17651 raft_consensus.cc:2238] T 50c711f4e65148faa8062f3ae6d87990 P 72c5e551bf0d441f9a702da4848d7b1c [term 1 FOLLOWER]: Raft consensus shutting down.
I20250112 15:08:09.333097 17651 raft_consensus.cc:2267] T 50c711f4e65148faa8062f3ae6d87990 P 72c5e551bf0d441f9a702da4848d7b1c [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 15:08:09.344467 17651 tablet_server.cc:195] TabletServer@127.17.60.196:0 shutdown complete.
I20250112 15:08:09.345444 17651 master.cc:537] Master@127.17.60.254:38743 shutting down...
I20250112 15:08:09.347787 17651 raft_consensus.cc:2238] T 00000000000000000000000000000000 P 80d1abdf9414405ebe25c3b7fd85b15d [term 1 LEADER]: Raft consensus shutting down.
I20250112 15:08:09.347857 17651 raft_consensus.cc:2267] T 00000000000000000000000000000000 P 80d1abdf9414405ebe25c3b7fd85b15d [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 15:08:09.347877 17651 tablet_replica.cc:331] T 00000000000000000000000000000000 P 80d1abdf9414405ebe25c3b7fd85b15d: stopping tablet replica
I20250112 15:08:09.359556 17651 master.cc:559] Master@127.17.60.254:38743 shutdown complete.
[ OK ] AutoRebalancerTest.AutoRebalancingUnstableCluster (10428 ms)
[ RUN ] AutoRebalancerTest.NoReplicaMovesIfCannotMeetReplicationFactor
I20250112 15:08:09.365108 17651 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.17.60.254:36431
I20250112 15:08:09.365269 17651 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250112 15:08:09.367184 22718 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250112 15:08:09.367470 22720 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250112 15:08:09.368167 22717 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250112 15:08:09.369303 17651 server_base.cc:1034] running on GCE node
I20250112 15:08:09.369402 17651 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250112 15:08:09.369439 17651 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250112 15:08:09.369452 17651 hybrid_clock.cc:648] HybridClock initialized: now 1736694489369452 us; error 0 us; skew 500 ppm
I20250112 15:08:09.370039 17651 webserver.cc:458] Webserver started at http://127.17.60.254:34829/ using document root <none> and password file <none>
I20250112 15:08:09.370106 17651 fs_manager.cc:362] Metadata directory not provided
I20250112 15:08:09.370158 17651 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250112 15:08:09.370204 17651 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250112 15:08:09.370451 17651 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task95dLSD/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotMeetReplicationFactor.1736694351070047-17651-0/minicluster-data/master-0-root/instance:
uuid: "60e8868d100f4d4492c4eddcee10628a"
format_stamp: "Formatted at 2025-01-12 15:08:09 on dist-test-slave-b4nz"
I20250112 15:08:09.371256 17651 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20250112 15:08:09.371678 22725 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250112 15:08:09.371800 17651 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20250112 15:08:09.371842 17651 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task95dLSD/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotMeetReplicationFactor.1736694351070047-17651-0/minicluster-data/master-0-root
uuid: "60e8868d100f4d4492c4eddcee10628a"
format_stamp: "Formatted at 2025-01-12 15:08:09 on dist-test-slave-b4nz"
I20250112 15:08:09.371887 17651 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task95dLSD/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotMeetReplicationFactor.1736694351070047-17651-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-task95dLSD/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotMeetReplicationFactor.1736694351070047-17651-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-task95dLSD/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotMeetReplicationFactor.1736694351070047-17651-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250112 15:08:09.376143 17651 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250112 15:08:09.376308 17651 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250112 15:08:09.378688 17651 rpc_server.cc:307] RPC server started. Bound to: 127.17.60.254:36431
I20250112 15:08:09.381922 22777 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.17.60.254:36431 every 8 connection(s)
I20250112 15:08:09.382054 22778 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 15:08:09.383005 22778 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 60e8868d100f4d4492c4eddcee10628a: Bootstrap starting.
I20250112 15:08:09.383299 22778 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 60e8868d100f4d4492c4eddcee10628a: Neither blocks nor log segments found. Creating new log.
I20250112 15:08:09.383791 22778 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 60e8868d100f4d4492c4eddcee10628a: No bootstrap required, opened a new log
I20250112 15:08:09.383901 22778 raft_consensus.cc:357] T 00000000000000000000000000000000 P 60e8868d100f4d4492c4eddcee10628a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "60e8868d100f4d4492c4eddcee10628a" member_type: VOTER }
I20250112 15:08:09.383947 22778 raft_consensus.cc:383] T 00000000000000000000000000000000 P 60e8868d100f4d4492c4eddcee10628a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 15:08:09.383960 22778 raft_consensus.cc:738] T 00000000000000000000000000000000 P 60e8868d100f4d4492c4eddcee10628a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 60e8868d100f4d4492c4eddcee10628a, State: Initialized, Role: FOLLOWER
I20250112 15:08:09.383991 22778 consensus_queue.cc:260] T 00000000000000000000000000000000 P 60e8868d100f4d4492c4eddcee10628a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "60e8868d100f4d4492c4eddcee10628a" member_type: VOTER }
I20250112 15:08:09.384019 22778 raft_consensus.cc:397] T 00000000000000000000000000000000 P 60e8868d100f4d4492c4eddcee10628a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250112 15:08:09.384037 22778 raft_consensus.cc:491] T 00000000000000000000000000000000 P 60e8868d100f4d4492c4eddcee10628a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250112 15:08:09.384063 22778 raft_consensus.cc:3054] T 00000000000000000000000000000000 P 60e8868d100f4d4492c4eddcee10628a [term 0 FOLLOWER]: Advancing to term 1
I20250112 15:08:09.384516 22778 raft_consensus.cc:513] T 00000000000000000000000000000000 P 60e8868d100f4d4492c4eddcee10628a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "60e8868d100f4d4492c4eddcee10628a" member_type: VOTER }
I20250112 15:08:09.384586 22778 leader_election.cc:304] T 00000000000000000000000000000000 P 60e8868d100f4d4492c4eddcee10628a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 60e8868d100f4d4492c4eddcee10628a; no voters:
I20250112 15:08:09.384696 22778 leader_election.cc:290] T 00000000000000000000000000000000 P 60e8868d100f4d4492c4eddcee10628a [CANDIDATE]: Term 1 election: Requested vote from peers
I20250112 15:08:09.384744 22781 raft_consensus.cc:2798] T 00000000000000000000000000000000 P 60e8868d100f4d4492c4eddcee10628a [term 1 FOLLOWER]: Leader election won for term 1
I20250112 15:08:09.384833 22778 sys_catalog.cc:564] T 00000000000000000000000000000000 P 60e8868d100f4d4492c4eddcee10628a [sys.catalog]: configured and running, proceeding with master startup.
I20250112 15:08:09.384900 22781 raft_consensus.cc:695] T 00000000000000000000000000000000 P 60e8868d100f4d4492c4eddcee10628a [term 1 LEADER]: Becoming Leader. State: Replica: 60e8868d100f4d4492c4eddcee10628a, State: Running, Role: LEADER
I20250112 15:08:09.384953 22781 consensus_queue.cc:237] T 00000000000000000000000000000000 P 60e8868d100f4d4492c4eddcee10628a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "60e8868d100f4d4492c4eddcee10628a" member_type: VOTER }
I20250112 15:08:09.385208 22782 sys_catalog.cc:455] T 00000000000000000000000000000000 P 60e8868d100f4d4492c4eddcee10628a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "60e8868d100f4d4492c4eddcee10628a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "60e8868d100f4d4492c4eddcee10628a" member_type: VOTER } }
I20250112 15:08:09.385238 22783 sys_catalog.cc:455] T 00000000000000000000000000000000 P 60e8868d100f4d4492c4eddcee10628a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 60e8868d100f4d4492c4eddcee10628a. Latest consensus state: current_term: 1 leader_uuid: "60e8868d100f4d4492c4eddcee10628a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "60e8868d100f4d4492c4eddcee10628a" member_type: VOTER } }
I20250112 15:08:09.385279 22782 sys_catalog.cc:458] T 00000000000000000000000000000000 P 60e8868d100f4d4492c4eddcee10628a [sys.catalog]: This master's current role is: LEADER
I20250112 15:08:09.385299 22783 sys_catalog.cc:458] T 00000000000000000000000000000000 P 60e8868d100f4d4492c4eddcee10628a [sys.catalog]: This master's current role is: LEADER
I20250112 15:08:09.385461 22787 catalog_manager.cc:1476] Loading table and tablet metadata into memory...
I20250112 15:08:09.385588 22787 catalog_manager.cc:1485] Initializing Kudu cluster ID...
I20250112 15:08:09.386102 17651 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20250112 15:08:09.386173 22787 catalog_manager.cc:1348] Generated new cluster ID: 966024c37fdf4944990962aeaaebefa3
I20250112 15:08:09.386210 22787 catalog_manager.cc:1496] Initializing Kudu internal certificate authority...
I20250112 15:08:09.392414 22787 catalog_manager.cc:1371] Generated new certificate authority record
I20250112 15:08:09.392843 22787 catalog_manager.cc:1505] Loading token signing keys...
I20250112 15:08:09.399673 22787 catalog_manager.cc:5899] T 00000000000000000000000000000000 P 60e8868d100f4d4492c4eddcee10628a: Generated new TSK 0
I20250112 15:08:09.399755 22787 catalog_manager.cc:1515] Initializing in-progress tserver states...
I20250112 15:08:09.401773 17651 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250112 15:08:09.403159 22800 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250112 15:08:09.403860 22802 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250112 15:08:09.404639 22799 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250112 15:08:09.405441 17651 server_base.cc:1034] running on GCE node
I20250112 15:08:09.405535 17651 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250112 15:08:09.405584 17651 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250112 15:08:09.405596 17651 hybrid_clock.cc:648] HybridClock initialized: now 1736694489405597 us; error 0 us; skew 500 ppm
I20250112 15:08:09.406232 17651 webserver.cc:458] Webserver started at http://127.17.60.193:42551/ using document root <none> and password file <none>
I20250112 15:08:09.406316 17651 fs_manager.cc:362] Metadata directory not provided
I20250112 15:08:09.406364 17651 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250112 15:08:09.406412 17651 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250112 15:08:09.406666 17651 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task95dLSD/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotMeetReplicationFactor.1736694351070047-17651-0/minicluster-data/ts-0-root/instance:
uuid: "1a8c110210714f978bf61828cc7eed39"
format_stamp: "Formatted at 2025-01-12 15:08:09 on dist-test-slave-b4nz"
I20250112 15:08:09.407505 17651 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20250112 15:08:09.408092 22807 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250112 15:08:09.408229 17651 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20250112 15:08:09.408273 17651 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task95dLSD/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotMeetReplicationFactor.1736694351070047-17651-0/minicluster-data/ts-0-root
uuid: "1a8c110210714f978bf61828cc7eed39"
format_stamp: "Formatted at 2025-01-12 15:08:09 on dist-test-slave-b4nz"
I20250112 15:08:09.408309 17651 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task95dLSD/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotMeetReplicationFactor.1736694351070047-17651-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-task95dLSD/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotMeetReplicationFactor.1736694351070047-17651-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-task95dLSD/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotMeetReplicationFactor.1736694351070047-17651-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250112 15:08:09.430714 17651 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250112 15:08:09.430956 17651 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250112 15:08:09.431159 17651 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250112 15:08:09.431392 17651 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250112 15:08:09.431434 17651 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250112 15:08:09.431473 17651 ts_tablet_manager.cc:610] Registered 0 tablets
I20250112 15:08:09.431497 17651 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250112 15:08:09.434595 17651 rpc_server.cc:307] RPC server started. Bound to: 127.17.60.193:42663
I20250112 15:08:09.434655 22870 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.17.60.193:42663 every 8 connection(s)
I20250112 15:08:09.434978 17651 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250112 15:08:09.436652 22875 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250112 15:08:09.436888 22876 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250112 15:08:09.437480 22878 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250112 15:08:09.438359 22871 heartbeater.cc:346] Connected to a master server at 127.17.60.254:36431
I20250112 15:08:09.438416 22871 heartbeater.cc:463] Registering TS with master...
I20250112 15:08:09.438529 22871 heartbeater.cc:510] Master 127.17.60.254:36431 requested a full tablet report, sending...
I20250112 15:08:09.438575 17651 server_base.cc:1034] running on GCE node
I20250112 15:08:09.438730 17651 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250112 15:08:09.438758 17651 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250112 15:08:09.438783 17651 hybrid_clock.cc:648] HybridClock initialized: now 1736694489438771 us; error 0 us; skew 500 ppm
I20250112 15:08:09.438820 22742 ts_manager.cc:194] Registered new tserver with Master: 1a8c110210714f978bf61828cc7eed39 (127.17.60.193:42663)
I20250112 15:08:09.439533 17651 webserver.cc:458] Webserver started at http://127.17.60.194:36907/ using document root <none> and password file <none>
I20250112 15:08:09.439612 17651 fs_manager.cc:362] Metadata directory not provided
I20250112 15:08:09.439638 22742 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:54856
I20250112 15:08:09.439664 17651 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250112 15:08:09.439738 17651 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250112 15:08:09.440037 17651 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task95dLSD/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotMeetReplicationFactor.1736694351070047-17651-0/minicluster-data/ts-1-root/instance:
uuid: "eedc143a1c994f1795bffdc6227d877e"
format_stamp: "Formatted at 2025-01-12 15:08:09 on dist-test-slave-b4nz"
I20250112 15:08:09.440929 17651 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20250112 15:08:09.441355 22883 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250112 15:08:09.441484 17651 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20250112 15:08:09.441524 17651 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task95dLSD/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotMeetReplicationFactor.1736694351070047-17651-0/minicluster-data/ts-1-root
uuid: "eedc143a1c994f1795bffdc6227d877e"
format_stamp: "Formatted at 2025-01-12 15:08:09 on dist-test-slave-b4nz"
I20250112 15:08:09.441570 17651 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task95dLSD/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotMeetReplicationFactor.1736694351070047-17651-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-task95dLSD/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotMeetReplicationFactor.1736694351070047-17651-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-task95dLSD/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotMeetReplicationFactor.1736694351070047-17651-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250112 15:08:09.450982 17651 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250112 15:08:09.451159 17651 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250112 15:08:09.451282 17651 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250112 15:08:09.451442 17651 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250112 15:08:09.451465 17651 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250112 15:08:09.451494 17651 ts_tablet_manager.cc:610] Registered 0 tablets
I20250112 15:08:09.451520 17651 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250112 15:08:09.454056 17651 rpc_server.cc:307] RPC server started. Bound to: 127.17.60.194:41707
I20250112 15:08:09.454107 22946 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.17.60.194:41707 every 8 connection(s)
I20250112 15:08:09.454418 17651 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250112 15:08:09.455958 22951 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250112 15:08:09.456630 22953 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250112 15:08:09.456995 22950 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250112 15:08:09.457293 22947 heartbeater.cc:346] Connected to a master server at 127.17.60.254:36431
I20250112 15:08:09.457343 22947 heartbeater.cc:463] Registering TS with master...
I20250112 15:08:09.457438 22947 heartbeater.cc:510] Master 127.17.60.254:36431 requested a full tablet report, sending...
I20250112 15:08:09.457662 22742 ts_manager.cc:194] Registered new tserver with Master: eedc143a1c994f1795bffdc6227d877e (127.17.60.194:41707)
I20250112 15:08:09.457741 17651 server_base.cc:1034] running on GCE node
I20250112 15:08:09.457863 17651 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250112 15:08:09.457901 17651 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250112 15:08:09.457914 17651 hybrid_clock.cc:648] HybridClock initialized: now 1736694489457914 us; error 0 us; skew 500 ppm
I20250112 15:08:09.458092 22742 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:54860
I20250112 15:08:09.458590 17651 webserver.cc:458] Webserver started at http://127.17.60.195:45433/ using document root <none> and password file <none>
I20250112 15:08:09.458670 17651 fs_manager.cc:362] Metadata directory not provided
I20250112 15:08:09.458719 17651 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250112 15:08:09.458765 17651 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250112 15:08:09.459043 17651 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task95dLSD/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotMeetReplicationFactor.1736694351070047-17651-0/minicluster-data/ts-2-root/instance:
uuid: "42b1a867dc5d4c3b83b3eafe300787cd"
format_stamp: "Formatted at 2025-01-12 15:08:09 on dist-test-slave-b4nz"
I20250112 15:08:09.459968 17651 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20250112 15:08:09.460419 22958 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250112 15:08:09.460538 17651 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20250112 15:08:09.460608 17651 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task95dLSD/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotMeetReplicationFactor.1736694351070047-17651-0/minicluster-data/ts-2-root
uuid: "42b1a867dc5d4c3b83b3eafe300787cd"
format_stamp: "Formatted at 2025-01-12 15:08:09 on dist-test-slave-b4nz"
I20250112 15:08:09.460649 17651 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task95dLSD/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotMeetReplicationFactor.1736694351070047-17651-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-task95dLSD/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotMeetReplicationFactor.1736694351070047-17651-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-task95dLSD/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotMeetReplicationFactor.1736694351070047-17651-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250112 15:08:09.471022 17651 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250112 15:08:09.471207 17651 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250112 15:08:09.471335 17651 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250112 15:08:09.471501 17651 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250112 15:08:09.471526 17651 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250112 15:08:09.471554 17651 ts_tablet_manager.cc:610] Registered 0 tablets
I20250112 15:08:09.471580 17651 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250112 15:08:09.474591 17651 rpc_server.cc:307] RPC server started. Bound to: 127.17.60.195:36587
I20250112 15:08:09.474648 23021 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.17.60.195:36587 every 8 connection(s)
I20250112 15:08:09.477164 23022 heartbeater.cc:346] Connected to a master server at 127.17.60.254:36431
I20250112 15:08:09.477214 23022 heartbeater.cc:463] Registering TS with master...
I20250112 15:08:09.477298 23022 heartbeater.cc:510] Master 127.17.60.254:36431 requested a full tablet report, sending...
I20250112 15:08:09.477501 22742 ts_manager.cc:194] Registered new tserver with Master: 42b1a867dc5d4c3b83b3eafe300787cd (127.17.60.195:36587)
I20250112 15:08:09.477841 22742 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:54870
I20250112 15:08:09.478072 17651 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.003333449s
I20250112 15:08:10.440161 22871 heartbeater.cc:502] Master 127.17.60.254:36431 was elected leader, sending a full tablet report...
I20250112 15:08:10.458600 22947 heartbeater.cc:502] Master 127.17.60.254:36431 was elected leader, sending a full tablet report...
I20250112 15:08:10.478410 23022 heartbeater.cc:502] Master 127.17.60.254:36431 was elected leader, sending a full tablet report...
I20250112 15:08:10.503235 17651 test_util.cc:274] Using random seed: -1305438144
I20250112 15:08:10.508287 22742 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:37536:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 15:08:10.508430 22742 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-workload in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 15:08:10.512988 22911 tablet_service.cc:1467] Processing CreateTablet for tablet dc1afd2385f64d25ac24031b32ff7d27 (DEFAULT_TABLE table=test-workload [id=aa55339e901f4ec0aa5837a9f6e70ad0]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 15:08:10.513125 22911 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet dc1afd2385f64d25ac24031b32ff7d27. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 15:08:10.513545 22986 tablet_service.cc:1467] Processing CreateTablet for tablet dc1afd2385f64d25ac24031b32ff7d27 (DEFAULT_TABLE table=test-workload [id=aa55339e901f4ec0aa5837a9f6e70ad0]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 15:08:10.513644 22986 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet dc1afd2385f64d25ac24031b32ff7d27. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 15:08:10.513743 22835 tablet_service.cc:1467] Processing CreateTablet for tablet dc1afd2385f64d25ac24031b32ff7d27 (DEFAULT_TABLE table=test-workload [id=aa55339e901f4ec0aa5837a9f6e70ad0]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 15:08:10.513841 22835 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet dc1afd2385f64d25ac24031b32ff7d27. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 15:08:10.514633 23042 tablet_bootstrap.cc:492] T dc1afd2385f64d25ac24031b32ff7d27 P eedc143a1c994f1795bffdc6227d877e: Bootstrap starting.
I20250112 15:08:10.514976 23043 tablet_bootstrap.cc:492] T dc1afd2385f64d25ac24031b32ff7d27 P 42b1a867dc5d4c3b83b3eafe300787cd: Bootstrap starting.
I20250112 15:08:10.515094 23042 tablet_bootstrap.cc:654] T dc1afd2385f64d25ac24031b32ff7d27 P eedc143a1c994f1795bffdc6227d877e: Neither blocks nor log segments found. Creating new log.
I20250112 15:08:10.515337 23044 tablet_bootstrap.cc:492] T dc1afd2385f64d25ac24031b32ff7d27 P 1a8c110210714f978bf61828cc7eed39: Bootstrap starting.
I20250112 15:08:10.515383 23043 tablet_bootstrap.cc:654] T dc1afd2385f64d25ac24031b32ff7d27 P 42b1a867dc5d4c3b83b3eafe300787cd: Neither blocks nor log segments found. Creating new log.
I20250112 15:08:10.515683 23044 tablet_bootstrap.cc:654] T dc1afd2385f64d25ac24031b32ff7d27 P 1a8c110210714f978bf61828cc7eed39: Neither blocks nor log segments found. Creating new log.
I20250112 15:08:10.515887 23043 tablet_bootstrap.cc:492] T dc1afd2385f64d25ac24031b32ff7d27 P 42b1a867dc5d4c3b83b3eafe300787cd: No bootstrap required, opened a new log
I20250112 15:08:10.515928 23043 ts_tablet_manager.cc:1397] T dc1afd2385f64d25ac24031b32ff7d27 P 42b1a867dc5d4c3b83b3eafe300787cd: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20250112 15:08:10.516072 23043 raft_consensus.cc:357] T dc1afd2385f64d25ac24031b32ff7d27 P 42b1a867dc5d4c3b83b3eafe300787cd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eedc143a1c994f1795bffdc6227d877e" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 41707 } } peers { permanent_uuid: "42b1a867dc5d4c3b83b3eafe300787cd" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 36587 } } peers { permanent_uuid: "1a8c110210714f978bf61828cc7eed39" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 42663 } }
I20250112 15:08:10.516148 23043 raft_consensus.cc:383] T dc1afd2385f64d25ac24031b32ff7d27 P 42b1a867dc5d4c3b83b3eafe300787cd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 15:08:10.516168 23043 raft_consensus.cc:738] T dc1afd2385f64d25ac24031b32ff7d27 P 42b1a867dc5d4c3b83b3eafe300787cd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 42b1a867dc5d4c3b83b3eafe300787cd, State: Initialized, Role: FOLLOWER
I20250112 15:08:10.516224 23043 consensus_queue.cc:260] T dc1afd2385f64d25ac24031b32ff7d27 P 42b1a867dc5d4c3b83b3eafe300787cd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eedc143a1c994f1795bffdc6227d877e" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 41707 } } peers { permanent_uuid: "42b1a867dc5d4c3b83b3eafe300787cd" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 36587 } } peers { permanent_uuid: "1a8c110210714f978bf61828cc7eed39" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 42663 } }
I20250112 15:08:10.516269 23044 tablet_bootstrap.cc:492] T dc1afd2385f64d25ac24031b32ff7d27 P 1a8c110210714f978bf61828cc7eed39: No bootstrap required, opened a new log
I20250112 15:08:10.516315 23044 ts_tablet_manager.cc:1397] T dc1afd2385f64d25ac24031b32ff7d27 P 1a8c110210714f978bf61828cc7eed39: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20250112 15:08:10.516378 23043 ts_tablet_manager.cc:1428] T dc1afd2385f64d25ac24031b32ff7d27 P 42b1a867dc5d4c3b83b3eafe300787cd: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20250112 15:08:10.516463 23044 raft_consensus.cc:357] T dc1afd2385f64d25ac24031b32ff7d27 P 1a8c110210714f978bf61828cc7eed39 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eedc143a1c994f1795bffdc6227d877e" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 41707 } } peers { permanent_uuid: "42b1a867dc5d4c3b83b3eafe300787cd" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 36587 } } peers { permanent_uuid: "1a8c110210714f978bf61828cc7eed39" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 42663 } }
I20250112 15:08:10.516583 23044 raft_consensus.cc:383] T dc1afd2385f64d25ac24031b32ff7d27 P 1a8c110210714f978bf61828cc7eed39 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 15:08:10.516618 23044 raft_consensus.cc:738] T dc1afd2385f64d25ac24031b32ff7d27 P 1a8c110210714f978bf61828cc7eed39 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1a8c110210714f978bf61828cc7eed39, State: Initialized, Role: FOLLOWER
I20250112 15:08:10.516624 23042 tablet_bootstrap.cc:492] T dc1afd2385f64d25ac24031b32ff7d27 P eedc143a1c994f1795bffdc6227d877e: No bootstrap required, opened a new log
I20250112 15:08:10.516680 23042 ts_tablet_manager.cc:1397] T dc1afd2385f64d25ac24031b32ff7d27 P eedc143a1c994f1795bffdc6227d877e: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20250112 15:08:10.516698 23044 consensus_queue.cc:260] T dc1afd2385f64d25ac24031b32ff7d27 P 1a8c110210714f978bf61828cc7eed39 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eedc143a1c994f1795bffdc6227d877e" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 41707 } } peers { permanent_uuid: "42b1a867dc5d4c3b83b3eafe300787cd" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 36587 } } peers { permanent_uuid: "1a8c110210714f978bf61828cc7eed39" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 42663 } }
I20250112 15:08:10.516851 23044 ts_tablet_manager.cc:1428] T dc1afd2385f64d25ac24031b32ff7d27 P 1a8c110210714f978bf61828cc7eed39: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20250112 15:08:10.516834 23042 raft_consensus.cc:357] T dc1afd2385f64d25ac24031b32ff7d27 P eedc143a1c994f1795bffdc6227d877e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eedc143a1c994f1795bffdc6227d877e" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 41707 } } peers { permanent_uuid: "42b1a867dc5d4c3b83b3eafe300787cd" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 36587 } } peers { permanent_uuid: "1a8c110210714f978bf61828cc7eed39" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 42663 } }
I20250112 15:08:10.516902 23042 raft_consensus.cc:383] T dc1afd2385f64d25ac24031b32ff7d27 P eedc143a1c994f1795bffdc6227d877e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 15:08:10.516927 23042 raft_consensus.cc:738] T dc1afd2385f64d25ac24031b32ff7d27 P eedc143a1c994f1795bffdc6227d877e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eedc143a1c994f1795bffdc6227d877e, State: Initialized, Role: FOLLOWER
I20250112 15:08:10.516973 23042 consensus_queue.cc:260] T dc1afd2385f64d25ac24031b32ff7d27 P eedc143a1c994f1795bffdc6227d877e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eedc143a1c994f1795bffdc6227d877e" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 41707 } } peers { permanent_uuid: "42b1a867dc5d4c3b83b3eafe300787cd" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 36587 } } peers { permanent_uuid: "1a8c110210714f978bf61828cc7eed39" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 42663 } }
I20250112 15:08:10.517130 23042 ts_tablet_manager.cc:1428] T dc1afd2385f64d25ac24031b32ff7d27 P eedc143a1c994f1795bffdc6227d877e: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20250112 15:08:10.649739 23048 raft_consensus.cc:491] T dc1afd2385f64d25ac24031b32ff7d27 P 42b1a867dc5d4c3b83b3eafe300787cd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 15:08:10.649835 23048 raft_consensus.cc:513] T dc1afd2385f64d25ac24031b32ff7d27 P 42b1a867dc5d4c3b83b3eafe300787cd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eedc143a1c994f1795bffdc6227d877e" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 41707 } } peers { permanent_uuid: "42b1a867dc5d4c3b83b3eafe300787cd" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 36587 } } peers { permanent_uuid: "1a8c110210714f978bf61828cc7eed39" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 42663 } }
I20250112 15:08:10.650081 23048 leader_election.cc:290] T dc1afd2385f64d25ac24031b32ff7d27 P 42b1a867dc5d4c3b83b3eafe300787cd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers eedc143a1c994f1795bffdc6227d877e (127.17.60.194:41707), 1a8c110210714f978bf61828cc7eed39 (127.17.60.193:42663)
I20250112 15:08:10.652585 22921 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc1afd2385f64d25ac24031b32ff7d27" candidate_uuid: "42b1a867dc5d4c3b83b3eafe300787cd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eedc143a1c994f1795bffdc6227d877e" is_pre_election: true
I20250112 15:08:10.652678 22921 raft_consensus.cc:2463] T dc1afd2385f64d25ac24031b32ff7d27 P eedc143a1c994f1795bffdc6227d877e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 42b1a867dc5d4c3b83b3eafe300787cd in term 0.
I20250112 15:08:10.652590 22845 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc1afd2385f64d25ac24031b32ff7d27" candidate_uuid: "42b1a867dc5d4c3b83b3eafe300787cd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1a8c110210714f978bf61828cc7eed39" is_pre_election: true
I20250112 15:08:10.652719 22845 raft_consensus.cc:2463] T dc1afd2385f64d25ac24031b32ff7d27 P 1a8c110210714f978bf61828cc7eed39 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 42b1a867dc5d4c3b83b3eafe300787cd in term 0.
I20250112 15:08:10.652814 22961 leader_election.cc:304] T dc1afd2385f64d25ac24031b32ff7d27 P 42b1a867dc5d4c3b83b3eafe300787cd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 42b1a867dc5d4c3b83b3eafe300787cd, eedc143a1c994f1795bffdc6227d877e; no voters:
I20250112 15:08:10.652895 23048 raft_consensus.cc:2798] T dc1afd2385f64d25ac24031b32ff7d27 P 42b1a867dc5d4c3b83b3eafe300787cd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250112 15:08:10.652937 23048 raft_consensus.cc:491] T dc1afd2385f64d25ac24031b32ff7d27 P 42b1a867dc5d4c3b83b3eafe300787cd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250112 15:08:10.652951 23048 raft_consensus.cc:3054] T dc1afd2385f64d25ac24031b32ff7d27 P 42b1a867dc5d4c3b83b3eafe300787cd [term 0 FOLLOWER]: Advancing to term 1
I20250112 15:08:10.653520 23048 raft_consensus.cc:513] T dc1afd2385f64d25ac24031b32ff7d27 P 42b1a867dc5d4c3b83b3eafe300787cd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eedc143a1c994f1795bffdc6227d877e" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 41707 } } peers { permanent_uuid: "42b1a867dc5d4c3b83b3eafe300787cd" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 36587 } } peers { permanent_uuid: "1a8c110210714f978bf61828cc7eed39" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 42663 } }
I20250112 15:08:10.653625 23048 leader_election.cc:290] T dc1afd2385f64d25ac24031b32ff7d27 P 42b1a867dc5d4c3b83b3eafe300787cd [CANDIDATE]: Term 1 election: Requested vote from peers eedc143a1c994f1795bffdc6227d877e (127.17.60.194:41707), 1a8c110210714f978bf61828cc7eed39 (127.17.60.193:42663)
I20250112 15:08:10.653748 22845 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc1afd2385f64d25ac24031b32ff7d27" candidate_uuid: "42b1a867dc5d4c3b83b3eafe300787cd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1a8c110210714f978bf61828cc7eed39"
I20250112 15:08:10.653754 22921 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc1afd2385f64d25ac24031b32ff7d27" candidate_uuid: "42b1a867dc5d4c3b83b3eafe300787cd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eedc143a1c994f1795bffdc6227d877e"
I20250112 15:08:10.653810 22845 raft_consensus.cc:3054] T dc1afd2385f64d25ac24031b32ff7d27 P 1a8c110210714f978bf61828cc7eed39 [term 0 FOLLOWER]: Advancing to term 1
I20250112 15:08:10.653827 22921 raft_consensus.cc:3054] T dc1afd2385f64d25ac24031b32ff7d27 P eedc143a1c994f1795bffdc6227d877e [term 0 FOLLOWER]: Advancing to term 1
I20250112 15:08:10.654451 22845 raft_consensus.cc:2463] T dc1afd2385f64d25ac24031b32ff7d27 P 1a8c110210714f978bf61828cc7eed39 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 42b1a867dc5d4c3b83b3eafe300787cd in term 1.
I20250112 15:08:10.654441 22921 raft_consensus.cc:2463] T dc1afd2385f64d25ac24031b32ff7d27 P eedc143a1c994f1795bffdc6227d877e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 42b1a867dc5d4c3b83b3eafe300787cd in term 1.
I20250112 15:08:10.654639 22962 leader_election.cc:304] T dc1afd2385f64d25ac24031b32ff7d27 P 42b1a867dc5d4c3b83b3eafe300787cd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1a8c110210714f978bf61828cc7eed39, 42b1a867dc5d4c3b83b3eafe300787cd; no voters:
I20250112 15:08:10.654744 23048 raft_consensus.cc:2798] T dc1afd2385f64d25ac24031b32ff7d27 P 42b1a867dc5d4c3b83b3eafe300787cd [term 1 FOLLOWER]: Leader election won for term 1
I20250112 15:08:10.654886 23048 raft_consensus.cc:695] T dc1afd2385f64d25ac24031b32ff7d27 P 42b1a867dc5d4c3b83b3eafe300787cd [term 1 LEADER]: Becoming Leader. State: Replica: 42b1a867dc5d4c3b83b3eafe300787cd, State: Running, Role: LEADER
I20250112 15:08:10.654942 23048 consensus_queue.cc:237] T dc1afd2385f64d25ac24031b32ff7d27 P 42b1a867dc5d4c3b83b3eafe300787cd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eedc143a1c994f1795bffdc6227d877e" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 41707 } } peers { permanent_uuid: "42b1a867dc5d4c3b83b3eafe300787cd" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 36587 } } peers { permanent_uuid: "1a8c110210714f978bf61828cc7eed39" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 42663 } }
I20250112 15:08:10.655462 22742 catalog_manager.cc:5526] T dc1afd2385f64d25ac24031b32ff7d27 P 42b1a867dc5d4c3b83b3eafe300787cd reported cstate change: term changed from 0 to 1, leader changed from <none> to 42b1a867dc5d4c3b83b3eafe300787cd (127.17.60.195). New cstate: current_term: 1 leader_uuid: "42b1a867dc5d4c3b83b3eafe300787cd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eedc143a1c994f1795bffdc6227d877e" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 41707 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "42b1a867dc5d4c3b83b3eafe300787cd" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 36587 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1a8c110210714f978bf61828cc7eed39" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 42663 } health_report { overall_health: UNKNOWN } } }
I20250112 15:08:10.657049 17651 tablet_server.cc:178] TabletServer@127.17.60.193:0 shutting down...
I20250112 15:08:10.659335 17651 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250112 15:08:10.659485 17651 tablet_replica.cc:331] T dc1afd2385f64d25ac24031b32ff7d27 P 1a8c110210714f978bf61828cc7eed39: stopping tablet replica
I20250112 15:08:10.659533 17651 raft_consensus.cc:2238] T dc1afd2385f64d25ac24031b32ff7d27 P 1a8c110210714f978bf61828cc7eed39 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250112 15:08:10.659576 17651 raft_consensus.cc:2267] T dc1afd2385f64d25ac24031b32ff7d27 P 1a8c110210714f978bf61828cc7eed39 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 15:08:10.671325 17651 tablet_server.cc:195] TabletServer@127.17.60.193:0 shutdown complete.
W20250112 15:08:11.051465 22962 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.17.60.193:42663: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20250112 15:08:11.051867 22962 consensus_peers.cc:487] T dc1afd2385f64d25ac24031b32ff7d27 P 42b1a867dc5d4c3b83b3eafe300787cd -> Peer 1a8c110210714f978bf61828cc7eed39 (127.17.60.193:42663): Couldn't send request to peer 1a8c110210714f978bf61828cc7eed39. Status: Network error: Client connection negotiation failed: client connection to 127.17.60.193:42663: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20250112 15:08:11.258111 23048 consensus_queue.cc:1035] T dc1afd2385f64d25ac24031b32ff7d27 P 42b1a867dc5d4c3b83b3eafe300787cd [LEADER]: Connected to new peer: Peer: permanent_uuid: "eedc143a1c994f1795bffdc6227d877e" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 41707 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250112 15:08:11.386150 22796 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250112 15:08:11.386267 22796 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20250112 15:08:11.386296 22796 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20250112 15:08:13.386440 22796 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250112 15:08:13.386576 22796 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20250112 15:08:13.386637 22796 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
W20250112 15:08:13.564798 22962 consensus_peers.cc:487] T dc1afd2385f64d25ac24031b32ff7d27 P 42b1a867dc5d4c3b83b3eafe300787cd -> Peer 1a8c110210714f978bf61828cc7eed39 (127.17.60.193:42663): Couldn't send request to peer 1a8c110210714f978bf61828cc7eed39. Status: Network error: Client connection negotiation failed: client connection to 127.17.60.193:42663: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20250112 15:08:15.386788 22796 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250112 15:08:15.386929 22796 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20250112 15:08:15.386969 22796 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
W20250112 15:08:16.022828 22962 consensus_peers.cc:487] T dc1afd2385f64d25ac24031b32ff7d27 P 42b1a867dc5d4c3b83b3eafe300787cd -> Peer 1a8c110210714f978bf61828cc7eed39 (127.17.60.193:42663): Couldn't send request to peer 1a8c110210714f978bf61828cc7eed39. Status: Network error: Client connection negotiation failed: client connection to 127.17.60.193:42663: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20250112 15:08:16.408522 22962 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.17.60.193:42663: connect: Connection refused (error 111) [suppressed 10 similar messages]
I20250112 15:08:17.387125 22796 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250112 15:08:17.387274 22796 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20250112 15:08:17.387320 22796 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
W20250112 15:08:18.357834 22962 consensus_peers.cc:487] T dc1afd2385f64d25ac24031b32ff7d27 P 42b1a867dc5d4c3b83b3eafe300787cd -> Peer 1a8c110210714f978bf61828cc7eed39 (127.17.60.193:42663): Couldn't send request to peer 1a8c110210714f978bf61828cc7eed39. Status: Network error: Client connection negotiation failed: client connection to 127.17.60.193:42663: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20250112 15:08:19.387472 22796 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250112 15:08:19.387619 22796 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20250112 15:08:19.387667 22796 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
W20250112 15:08:20.953846 22962 consensus_peers.cc:487] T dc1afd2385f64d25ac24031b32ff7d27 P 42b1a867dc5d4c3b83b3eafe300787cd -> Peer 1a8c110210714f978bf61828cc7eed39 (127.17.60.193:42663): Couldn't send request to peer 1a8c110210714f978bf61828cc7eed39. Status: Network error: Client connection negotiation failed: client connection to 127.17.60.193:42663: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20250112 15:08:21.387840 22796 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250112 15:08:21.387984 22796 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20250112 15:08:21.388034 22796 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
W20250112 15:08:21.550033 22962 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.17.60.193:42663: connect: Connection refused (error 111) [suppressed 9 similar messages]
I20250112 15:08:21.723170 17651 tablet_server.cc:178] TabletServer@127.17.60.194:0 shutting down...
I20250112 15:08:21.725889 17651 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250112 15:08:21.726009 17651 tablet_replica.cc:331] T dc1afd2385f64d25ac24031b32ff7d27 P eedc143a1c994f1795bffdc6227d877e: stopping tablet replica
I20250112 15:08:21.726073 17651 raft_consensus.cc:2238] T dc1afd2385f64d25ac24031b32ff7d27 P eedc143a1c994f1795bffdc6227d877e [term 1 FOLLOWER]: Raft consensus shutting down.
I20250112 15:08:21.726133 17651 raft_consensus.cc:2267] T dc1afd2385f64d25ac24031b32ff7d27 P eedc143a1c994f1795bffdc6227d877e [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 15:08:21.737659 17651 tablet_server.cc:195] TabletServer@127.17.60.194:0 shutdown complete.
I20250112 15:08:21.738520 17651 tablet_server.cc:178] TabletServer@127.17.60.195:0 shutting down...
I20250112 15:08:21.741283 17651 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250112 15:08:21.741439 17651 tablet_replica.cc:331] T dc1afd2385f64d25ac24031b32ff7d27 P 42b1a867dc5d4c3b83b3eafe300787cd: stopping tablet replica
I20250112 15:08:21.741510 17651 raft_consensus.cc:2238] T dc1afd2385f64d25ac24031b32ff7d27 P 42b1a867dc5d4c3b83b3eafe300787cd [term 1 LEADER]: Raft consensus shutting down.
I20250112 15:08:21.741580 17651 raft_consensus.cc:2267] T dc1afd2385f64d25ac24031b32ff7d27 P 42b1a867dc5d4c3b83b3eafe300787cd [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 15:08:21.753304 17651 tablet_server.cc:195] TabletServer@127.17.60.195:0 shutdown complete.
I20250112 15:08:21.754276 17651 master.cc:537] Master@127.17.60.254:36431 shutting down...
I20250112 15:08:21.756285 17651 raft_consensus.cc:2238] T 00000000000000000000000000000000 P 60e8868d100f4d4492c4eddcee10628a [term 1 LEADER]: Raft consensus shutting down.
I20250112 15:08:21.756371 17651 raft_consensus.cc:2267] T 00000000000000000000000000000000 P 60e8868d100f4d4492c4eddcee10628a [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 15:08:21.756405 17651 tablet_replica.cc:331] T 00000000000000000000000000000000 P 60e8868d100f4d4492c4eddcee10628a: stopping tablet replica
I20250112 15:08:21.768261 17651 master.cc:559] Master@127.17.60.254:36431 shutdown complete.
[ OK ] AutoRebalancerTest.NoReplicaMovesIfCannotMeetReplicationFactor (12407 ms)
[ RUN ] AutoRebalancerTest.NoRebalancingIfReplicasRecovering
I20250112 15:08:21.772378 17651 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.17.60.254:39133
I20250112 15:08:21.772511 17651 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250112 15:08:21.774165 23085 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250112 15:08:21.775174 17651 server_base.cc:1034] running on GCE node
W20250112 15:08:21.775831 23088 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250112 15:08:21.776522 23086 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250112 15:08:21.776675 17651 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250112 15:08:21.776707 17651 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250112 15:08:21.776719 17651 hybrid_clock.cc:648] HybridClock initialized: now 1736694501776720 us; error 0 us; skew 500 ppm
I20250112 15:08:21.777334 17651 webserver.cc:458] Webserver started at http://127.17.60.254:41171/ using document root <none> and password file <none>
I20250112 15:08:21.777401 17651 fs_manager.cc:362] Metadata directory not provided
I20250112 15:08:21.777437 17651 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250112 15:08:21.777470 17651 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250112 15:08:21.777715 17651 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task95dLSD/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoRebalancingIfReplicasRecovering.1736694351070047-17651-0/minicluster-data/master-0-root/instance:
uuid: "c53dc92ba0e24a21aec187711521e9c8"
format_stamp: "Formatted at 2025-01-12 15:08:21 on dist-test-slave-b4nz"
I20250112 15:08:21.778540 17651 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20250112 15:08:21.779078 23093 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250112 15:08:21.779204 17651 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20250112 15:08:21.779253 17651 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task95dLSD/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoRebalancingIfReplicasRecovering.1736694351070047-17651-0/minicluster-data/master-0-root
uuid: "c53dc92ba0e24a21aec187711521e9c8"
format_stamp: "Formatted at 2025-01-12 15:08:21 on dist-test-slave-b4nz"
I20250112 15:08:21.779292 17651 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task95dLSD/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoRebalancingIfReplicasRecovering.1736694351070047-17651-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-task95dLSD/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoRebalancingIfReplicasRecovering.1736694351070047-17651-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-task95dLSD/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoRebalancingIfReplicasRecovering.1736694351070047-17651-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250112 15:08:21.794834 17651 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250112 15:08:21.795027 17651 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250112 15:08:21.797474 17651 rpc_server.cc:307] RPC server started. Bound to: 127.17.60.254:39133
I20250112 15:08:21.800323 23145 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.17.60.254:39133 every 8 connection(s)
I20250112 15:08:21.800457 23146 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 15:08:21.801700 23146 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c53dc92ba0e24a21aec187711521e9c8: Bootstrap starting.
I20250112 15:08:21.802008 23146 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c53dc92ba0e24a21aec187711521e9c8: Neither blocks nor log segments found. Creating new log.
I20250112 15:08:21.802548 23146 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c53dc92ba0e24a21aec187711521e9c8: No bootstrap required, opened a new log
I20250112 15:08:21.802656 23146 raft_consensus.cc:357] T 00000000000000000000000000000000 P c53dc92ba0e24a21aec187711521e9c8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c53dc92ba0e24a21aec187711521e9c8" member_type: VOTER }
I20250112 15:08:21.802704 23146 raft_consensus.cc:383] T 00000000000000000000000000000000 P c53dc92ba0e24a21aec187711521e9c8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 15:08:21.802718 23146 raft_consensus.cc:738] T 00000000000000000000000000000000 P c53dc92ba0e24a21aec187711521e9c8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c53dc92ba0e24a21aec187711521e9c8, State: Initialized, Role: FOLLOWER
I20250112 15:08:21.802763 23146 consensus_queue.cc:260] T 00000000000000000000000000000000 P c53dc92ba0e24a21aec187711521e9c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c53dc92ba0e24a21aec187711521e9c8" member_type: VOTER }
I20250112 15:08:21.802806 23146 raft_consensus.cc:397] T 00000000000000000000000000000000 P c53dc92ba0e24a21aec187711521e9c8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250112 15:08:21.802830 23146 raft_consensus.cc:491] T 00000000000000000000000000000000 P c53dc92ba0e24a21aec187711521e9c8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250112 15:08:21.802851 23146 raft_consensus.cc:3054] T 00000000000000000000000000000000 P c53dc92ba0e24a21aec187711521e9c8 [term 0 FOLLOWER]: Advancing to term 1
I20250112 15:08:21.803321 23146 raft_consensus.cc:513] T 00000000000000000000000000000000 P c53dc92ba0e24a21aec187711521e9c8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c53dc92ba0e24a21aec187711521e9c8" member_type: VOTER }
I20250112 15:08:21.803375 23146 leader_election.cc:304] T 00000000000000000000000000000000 P c53dc92ba0e24a21aec187711521e9c8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c53dc92ba0e24a21aec187711521e9c8; no voters:
I20250112 15:08:21.803471 23146 leader_election.cc:290] T 00000000000000000000000000000000 P c53dc92ba0e24a21aec187711521e9c8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20250112 15:08:21.803524 23149 raft_consensus.cc:2798] T 00000000000000000000000000000000 P c53dc92ba0e24a21aec187711521e9c8 [term 1 FOLLOWER]: Leader election won for term 1
I20250112 15:08:21.803620 23146 sys_catalog.cc:564] T 00000000000000000000000000000000 P c53dc92ba0e24a21aec187711521e9c8 [sys.catalog]: configured and running, proceeding with master startup.
I20250112 15:08:21.803648 23149 raft_consensus.cc:695] T 00000000000000000000000000000000 P c53dc92ba0e24a21aec187711521e9c8 [term 1 LEADER]: Becoming Leader. State: Replica: c53dc92ba0e24a21aec187711521e9c8, State: Running, Role: LEADER
I20250112 15:08:21.803772 23149 consensus_queue.cc:237] T 00000000000000000000000000000000 P c53dc92ba0e24a21aec187711521e9c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c53dc92ba0e24a21aec187711521e9c8" member_type: VOTER }
I20250112 15:08:21.803995 23150 sys_catalog.cc:455] T 00000000000000000000000000000000 P c53dc92ba0e24a21aec187711521e9c8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c53dc92ba0e24a21aec187711521e9c8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c53dc92ba0e24a21aec187711521e9c8" member_type: VOTER } }
I20250112 15:08:21.804003 23151 sys_catalog.cc:455] T 00000000000000000000000000000000 P c53dc92ba0e24a21aec187711521e9c8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c53dc92ba0e24a21aec187711521e9c8. Latest consensus state: current_term: 1 leader_uuid: "c53dc92ba0e24a21aec187711521e9c8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c53dc92ba0e24a21aec187711521e9c8" member_type: VOTER } }
I20250112 15:08:21.804168 23151 sys_catalog.cc:458] T 00000000000000000000000000000000 P c53dc92ba0e24a21aec187711521e9c8 [sys.catalog]: This master's current role is: LEADER
I20250112 15:08:21.804153 23150 sys_catalog.cc:458] T 00000000000000000000000000000000 P c53dc92ba0e24a21aec187711521e9c8 [sys.catalog]: This master's current role is: LEADER
I20250112 15:08:21.804441 23155 catalog_manager.cc:1476] Loading table and tablet metadata into memory...
I20250112 15:08:21.804618 23155 catalog_manager.cc:1485] Initializing Kudu cluster ID...
I20250112 15:08:21.805125 17651 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20250112 15:08:21.805168 23155 catalog_manager.cc:1348] Generated new cluster ID: ea4a0e2eeb3442a081556cc5767c222b
I20250112 15:08:21.805214 23155 catalog_manager.cc:1496] Initializing Kudu internal certificate authority...
I20250112 15:08:21.821650 23155 catalog_manager.cc:1371] Generated new certificate authority record
I20250112 15:08:21.822077 23155 catalog_manager.cc:1505] Loading token signing keys...
I20250112 15:08:21.825228 23155 catalog_manager.cc:5899] T 00000000000000000000000000000000 P c53dc92ba0e24a21aec187711521e9c8: Generated new TSK 0
I20250112 15:08:21.825304 23155 catalog_manager.cc:1515] Initializing in-progress tserver states...
I20250112 15:08:21.837005 17651 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250112 15:08:21.838543 23167 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250112 15:08:21.839341 23170 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250112 15:08:21.840127 23168 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250112 15:08:21.840932 17651 server_base.cc:1034] running on GCE node
I20250112 15:08:21.841010 17651 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250112 15:08:21.841032 17651 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250112 15:08:21.841042 17651 hybrid_clock.cc:648] HybridClock initialized: now 1736694501841043 us; error 0 us; skew 500 ppm
I20250112 15:08:21.841688 17651 webserver.cc:458] Webserver started at http://127.17.60.193:34461/ using document root <none> and password file <none>
I20250112 15:08:21.841768 17651 fs_manager.cc:362] Metadata directory not provided
I20250112 15:08:21.841801 17651 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250112 15:08:21.841836 17651 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250112 15:08:21.842113 17651 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task95dLSD/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoRebalancingIfReplicasRecovering.1736694351070047-17651-0/minicluster-data/ts-0-root/instance:
uuid: "91de225bd4e84a4593199b49347b6d29"
format_stamp: "Formatted at 2025-01-12 15:08:21 on dist-test-slave-b4nz"
I20250112 15:08:21.842885 17651 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20250112 15:08:21.843319 23175 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250112 15:08:21.843452 17651 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20250112 15:08:21.843498 17651 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task95dLSD/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoRebalancingIfReplicasRecovering.1736694351070047-17651-0/minicluster-data/ts-0-root
uuid: "91de225bd4e84a4593199b49347b6d29"
format_stamp: "Formatted at 2025-01-12 15:08:21 on dist-test-slave-b4nz"
I20250112 15:08:21.843534 17651 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task95dLSD/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoRebalancingIfReplicasRecovering.1736694351070047-17651-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-task95dLSD/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoRebalancingIfReplicasRecovering.1736694351070047-17651-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-task95dLSD/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoRebalancingIfReplicasRecovering.1736694351070047-17651-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250112 15:08:21.853045 17651 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250112 15:08:21.853257 17651 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250112 15:08:21.853381 17651 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250112 15:08:21.853565 17651 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250112 15:08:21.853590 17651 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250112 15:08:21.853614 17651 ts_tablet_manager.cc:610] Registered 0 tablets
I20250112 15:08:21.853639 17651 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250112 15:08:21.856186 17651 rpc_server.cc:307] RPC server started. Bound to: 127.17.60.193:39577
I20250112 15:08:21.856251 23238 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.17.60.193:39577 every 8 connection(s)
I20250112 15:08:21.856536 17651 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250112 15:08:21.858117 23246 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250112 15:08:21.858438 23243 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250112 15:08:21.859170 23244 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250112 15:08:21.859709 23239 heartbeater.cc:346] Connected to a master server at 127.17.60.254:39133
I20250112 15:08:21.859771 23239 heartbeater.cc:463] Registering TS with master...
I20250112 15:08:21.859819 17651 server_base.cc:1034] running on GCE node
I20250112 15:08:21.859897 23239 heartbeater.cc:510] Master 127.17.60.254:39133 requested a full tablet report, sending...
I20250112 15:08:21.859934 17651 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250112 15:08:21.859954 17651 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250112 15:08:21.859968 17651 hybrid_clock.cc:648] HybridClock initialized: now 1736694501859968 us; error 0 us; skew 500 ppm
I20250112 15:08:21.860188 23110 ts_manager.cc:194] Registered new tserver with Master: 91de225bd4e84a4593199b49347b6d29 (127.17.60.193:39577)
I20250112 15:08:21.860742 17651 webserver.cc:458] Webserver started at http://127.17.60.194:37969/ using document root <none> and password file <none>
I20250112 15:08:21.860788 23110 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:49894
I20250112 15:08:21.860834 17651 fs_manager.cc:362] Metadata directory not provided
I20250112 15:08:21.860882 17651 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250112 15:08:21.860929 17651 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250112 15:08:21.861306 17651 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task95dLSD/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoRebalancingIfReplicasRecovering.1736694351070047-17651-0/minicluster-data/ts-1-root/instance:
uuid: "5b101dbed8f34e9281a33db399f65ed6"
format_stamp: "Formatted at 2025-01-12 15:08:21 on dist-test-slave-b4nz"
I20250112 15:08:21.862097 17651 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20250112 15:08:21.862473 23251 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250112 15:08:21.862589 17651 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20250112 15:08:21.862641 17651 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task95dLSD/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoRebalancingIfReplicasRecovering.1736694351070047-17651-0/minicluster-data/ts-1-root
uuid: "5b101dbed8f34e9281a33db399f65ed6"
format_stamp: "Formatted at 2025-01-12 15:08:21 on dist-test-slave-b4nz"
I20250112 15:08:21.862675 17651 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task95dLSD/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoRebalancingIfReplicasRecovering.1736694351070047-17651-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-task95dLSD/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoRebalancingIfReplicasRecovering.1736694351070047-17651-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-task95dLSD/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoRebalancingIfReplicasRecovering.1736694351070047-17651-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250112 15:08:21.874608 17651 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250112 15:08:21.874826 17651 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250112 15:08:21.874967 17651 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250112 15:08:21.875171 17651 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250112 15:08:21.875200 17651 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250112 15:08:21.875226 17651 ts_tablet_manager.cc:610] Registered 0 tablets
I20250112 15:08:21.875249 17651 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250112 15:08:21.878571 17651 rpc_server.cc:307] RPC server started. Bound to: 127.17.60.194:36341
I20250112 15:08:21.878625 23314 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.17.60.194:36341 every 8 connection(s)
I20250112 15:08:21.879009 17651 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250112 15:08:21.880722 23319 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250112 15:08:21.880901 23318 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250112 15:08:21.881981 23321 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250112 15:08:21.882290 23315 heartbeater.cc:346] Connected to a master server at 127.17.60.254:39133
I20250112 15:08:21.882344 23315 heartbeater.cc:463] Registering TS with master...
I20250112 15:08:21.882445 23315 heartbeater.cc:510] Master 127.17.60.254:39133 requested a full tablet report, sending...
I20250112 15:08:21.882612 17651 server_base.cc:1034] running on GCE node
I20250112 15:08:21.882707 23110 ts_manager.cc:194] Registered new tserver with Master: 5b101dbed8f34e9281a33db399f65ed6 (127.17.60.194:36341)
I20250112 15:08:21.882751 17651 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250112 15:08:21.882787 17651 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250112 15:08:21.882812 17651 hybrid_clock.cc:648] HybridClock initialized: now 1736694501882812 us; error 0 us; skew 500 ppm
I20250112 15:08:21.883142 23110 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:49906
I20250112 15:08:21.883527 17651 webserver.cc:458] Webserver started at http://127.17.60.195:41327/ using document root <none> and password file <none>
I20250112 15:08:21.883606 17651 fs_manager.cc:362] Metadata directory not provided
I20250112 15:08:21.883659 17651 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250112 15:08:21.883705 17651 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250112 15:08:21.883968 17651 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task95dLSD/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoRebalancingIfReplicasRecovering.1736694351070047-17651-0/minicluster-data/ts-2-root/instance:
uuid: "b3791458f5554d158fcb770980760849"
format_stamp: "Formatted at 2025-01-12 15:08:21 on dist-test-slave-b4nz"
I20250112 15:08:21.884872 17651 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20250112 15:08:21.885347 23326 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250112 15:08:21.885470 17651 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20250112 15:08:21.885509 17651 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task95dLSD/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoRebalancingIfReplicasRecovering.1736694351070047-17651-0/minicluster-data/ts-2-root
uuid: "b3791458f5554d158fcb770980760849"
format_stamp: "Formatted at 2025-01-12 15:08:21 on dist-test-slave-b4nz"
I20250112 15:08:21.885550 17651 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task95dLSD/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoRebalancingIfReplicasRecovering.1736694351070047-17651-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-task95dLSD/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoRebalancingIfReplicasRecovering.1736694351070047-17651-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-task95dLSD/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoRebalancingIfReplicasRecovering.1736694351070047-17651-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250112 15:08:21.907495 17651 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250112 15:08:21.907747 17651 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250112 15:08:21.907922 17651 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250112 15:08:21.908186 17651 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250112 15:08:21.908218 17651 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250112 15:08:21.908250 17651 ts_tablet_manager.cc:610] Registered 0 tablets
I20250112 15:08:21.908278 17651 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250112 15:08:21.911535 17651 rpc_server.cc:307] RPC server started. Bound to: 127.17.60.195:42069
I20250112 15:08:21.911602 23389 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.17.60.195:42069 every 8 connection(s)
I20250112 15:08:21.914721 23390 heartbeater.cc:346] Connected to a master server at 127.17.60.254:39133
I20250112 15:08:21.914783 23390 heartbeater.cc:463] Registering TS with master...
I20250112 15:08:21.914881 23390 heartbeater.cc:510] Master 127.17.60.254:39133 requested a full tablet report, sending...
I20250112 15:08:21.915105 23110 ts_manager.cc:194] Registered new tserver with Master: b3791458f5554d158fcb770980760849 (127.17.60.195:42069)
I20250112 15:08:21.915406 23110 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:49918
I20250112 15:08:21.916106 17651 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.004396672s
I20250112 15:08:22.861541 23239 heartbeater.cc:502] Master 127.17.60.254:39133 was elected leader, sending a full tablet report...
I20250112 15:08:22.883673 23315 heartbeater.cc:502] Master 127.17.60.254:39133 was elected leader, sending a full tablet report...
I20250112 15:08:22.915891 23390 heartbeater.cc:502] Master 127.17.60.254:39133 was elected leader, sending a full tablet report...
I20250112 15:08:22.941604 17651 test_util.cc:274] Using random seed: -1292999775
I20250112 15:08:22.946730 23110 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49934:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "\004\001\000\377\377\377\037\004\001\000\376\377\377?\004\001\000\375\377\377_""\004\001\000\377\377\377\037\004\001\000\376\377\377?\004\001\000\375\377\377_"
indirect_data: """"
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 15:08:22.946887 23110 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-workload in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 15:08:22.951745 23279 tablet_service.cc:1467] Processing CreateTablet for tablet a84082c6bef84e638159590ed624062e (DEFAULT_TABLE table=test-workload [id=2570d90deee448cea24e7125529fb8c9]), partition=RANGE (key) PARTITION VALUES < 536870911
I20250112 15:08:22.951907 23279 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a84082c6bef84e638159590ed624062e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 15:08:22.952286 23203 tablet_service.cc:1467] Processing CreateTablet for tablet a84082c6bef84e638159590ed624062e (DEFAULT_TABLE table=test-workload [id=2570d90deee448cea24e7125529fb8c9]), partition=RANGE (key) PARTITION VALUES < 536870911
I20250112 15:08:22.952337 23201 tablet_service.cc:1467] Processing CreateTablet for tablet 46c5820c43a64ba285e33ee2b607789b (DEFAULT_TABLE table=test-workload [id=2570d90deee448cea24e7125529fb8c9]), partition=RANGE (key) PARTITION 1073741822 <= VALUES < 1610612733
I20250112 15:08:22.952409 23203 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a84082c6bef84e638159590ed624062e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 15:08:22.952440 23200 tablet_service.cc:1467] Processing CreateTablet for tablet d5a633e37cc84a098547fa042708b1d3 (DEFAULT_TABLE table=test-workload [id=2570d90deee448cea24e7125529fb8c9]), partition=RANGE (key) PARTITION 1610612733 <= VALUES
I20250112 15:08:22.952517 23201 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 46c5820c43a64ba285e33ee2b607789b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 15:08:22.952294 23202 tablet_service.cc:1467] Processing CreateTablet for tablet 656d5437ee6a475a89af5e24b74f315a (DEFAULT_TABLE table=test-workload [id=2570d90deee448cea24e7125529fb8c9]), partition=RANGE (key) PARTITION 536870911 <= VALUES < 1073741822
I20250112 15:08:22.952682 23202 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 656d5437ee6a475a89af5e24b74f315a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 15:08:22.953478 23278 tablet_service.cc:1467] Processing CreateTablet for tablet 656d5437ee6a475a89af5e24b74f315a (DEFAULT_TABLE table=test-workload [id=2570d90deee448cea24e7125529fb8c9]), partition=RANGE (key) PARTITION 536870911 <= VALUES < 1073741822
I20250112 15:08:22.953562 23278 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 656d5437ee6a475a89af5e24b74f315a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 15:08:22.953941 23411 tablet_bootstrap.cc:492] T a84082c6bef84e638159590ed624062e P 91de225bd4e84a4593199b49347b6d29: Bootstrap starting.
I20250112 15:08:22.954042 23200 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d5a633e37cc84a098547fa042708b1d3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 15:08:22.954370 23411 tablet_bootstrap.cc:654] T a84082c6bef84e638159590ed624062e P 91de225bd4e84a4593199b49347b6d29: Neither blocks nor log segments found. Creating new log.
I20250112 15:08:22.954507 23277 tablet_service.cc:1467] Processing CreateTablet for tablet 46c5820c43a64ba285e33ee2b607789b (DEFAULT_TABLE table=test-workload [id=2570d90deee448cea24e7125529fb8c9]), partition=RANGE (key) PARTITION 1073741822 <= VALUES < 1610612733
I20250112 15:08:22.954569 23277 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 46c5820c43a64ba285e33ee2b607789b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 15:08:22.954953 23411 tablet_bootstrap.cc:492] T a84082c6bef84e638159590ed624062e P 91de225bd4e84a4593199b49347b6d29: No bootstrap required, opened a new log
I20250112 15:08:22.954993 23411 ts_tablet_manager.cc:1397] T a84082c6bef84e638159590ed624062e P 91de225bd4e84a4593199b49347b6d29: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20250112 15:08:22.955145 23411 raft_consensus.cc:357] T a84082c6bef84e638159590ed624062e P 91de225bd4e84a4593199b49347b6d29 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b101dbed8f34e9281a33db399f65ed6" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 36341 } } peers { permanent_uuid: "b3791458f5554d158fcb770980760849" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 42069 } } peers { permanent_uuid: "91de225bd4e84a4593199b49347b6d29" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 39577 } }
I20250112 15:08:22.955214 23411 raft_consensus.cc:383] T a84082c6bef84e638159590ed624062e P 91de225bd4e84a4593199b49347b6d29 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 15:08:22.955233 23411 raft_consensus.cc:738] T a84082c6bef84e638159590ed624062e P 91de225bd4e84a4593199b49347b6d29 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 91de225bd4e84a4593199b49347b6d29, State: Initialized, Role: FOLLOWER
I20250112 15:08:22.955272 23411 consensus_queue.cc:260] T a84082c6bef84e638159590ed624062e P 91de225bd4e84a4593199b49347b6d29 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b101dbed8f34e9281a33db399f65ed6" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 36341 } } peers { permanent_uuid: "b3791458f5554d158fcb770980760849" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 42069 } } peers { permanent_uuid: "91de225bd4e84a4593199b49347b6d29" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 39577 } }
I20250112 15:08:22.955430 23411 ts_tablet_manager.cc:1428] T a84082c6bef84e638159590ed624062e P 91de225bd4e84a4593199b49347b6d29: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20250112 15:08:22.955446 23410 tablet_bootstrap.cc:492] T a84082c6bef84e638159590ed624062e P 5b101dbed8f34e9281a33db399f65ed6: Bootstrap starting.
I20250112 15:08:22.955550 23276 tablet_service.cc:1467] Processing CreateTablet for tablet d5a633e37cc84a098547fa042708b1d3 (DEFAULT_TABLE table=test-workload [id=2570d90deee448cea24e7125529fb8c9]), partition=RANGE (key) PARTITION 1610612733 <= VALUES
I20250112 15:08:22.955632 23276 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d5a633e37cc84a098547fa042708b1d3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 15:08:22.955709 23411 tablet_bootstrap.cc:492] T 46c5820c43a64ba285e33ee2b607789b P 91de225bd4e84a4593199b49347b6d29: Bootstrap starting.
I20250112 15:08:22.955889 23410 tablet_bootstrap.cc:654] T a84082c6bef84e638159590ed624062e P 5b101dbed8f34e9281a33db399f65ed6: Neither blocks nor log segments found. Creating new log.
I20250112 15:08:22.956521 23410 tablet_bootstrap.cc:492] T a84082c6bef84e638159590ed624062e P 5b101dbed8f34e9281a33db399f65ed6: No bootstrap required, opened a new log
I20250112 15:08:22.956538 23411 tablet_bootstrap.cc:654] T 46c5820c43a64ba285e33ee2b607789b P 91de225bd4e84a4593199b49347b6d29: Neither blocks nor log segments found. Creating new log.
I20250112 15:08:22.956564 23410 ts_tablet_manager.cc:1397] T a84082c6bef84e638159590ed624062e P 5b101dbed8f34e9281a33db399f65ed6: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20250112 15:08:22.956743 23410 raft_consensus.cc:357] T a84082c6bef84e638159590ed624062e P 5b101dbed8f34e9281a33db399f65ed6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b101dbed8f34e9281a33db399f65ed6" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 36341 } } peers { permanent_uuid: "b3791458f5554d158fcb770980760849" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 42069 } } peers { permanent_uuid: "91de225bd4e84a4593199b49347b6d29" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 39577 } }
I20250112 15:08:22.956802 23410 raft_consensus.cc:383] T a84082c6bef84e638159590ed624062e P 5b101dbed8f34e9281a33db399f65ed6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 15:08:22.956828 23410 raft_consensus.cc:738] T a84082c6bef84e638159590ed624062e P 5b101dbed8f34e9281a33db399f65ed6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5b101dbed8f34e9281a33db399f65ed6, State: Initialized, Role: FOLLOWER
I20250112 15:08:22.956836 23354 tablet_service.cc:1467] Processing CreateTablet for tablet a84082c6bef84e638159590ed624062e (DEFAULT_TABLE table=test-workload [id=2570d90deee448cea24e7125529fb8c9]), partition=RANGE (key) PARTITION VALUES < 536870911
I20250112 15:08:22.956861 23353 tablet_service.cc:1467] Processing CreateTablet for tablet 656d5437ee6a475a89af5e24b74f315a (DEFAULT_TABLE table=test-workload [id=2570d90deee448cea24e7125529fb8c9]), partition=RANGE (key) PARTITION 536870911 <= VALUES < 1073741822
I20250112 15:08:22.956878 23410 consensus_queue.cc:260] T a84082c6bef84e638159590ed624062e P 5b101dbed8f34e9281a33db399f65ed6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b101dbed8f34e9281a33db399f65ed6" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 36341 } } peers { permanent_uuid: "b3791458f5554d158fcb770980760849" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 42069 } } peers { permanent_uuid: "91de225bd4e84a4593199b49347b6d29" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 39577 } }
I20250112 15:08:22.956943 23354 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a84082c6bef84e638159590ed624062e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 15:08:22.956990 23351 tablet_service.cc:1467] Processing CreateTablet for tablet d5a633e37cc84a098547fa042708b1d3 (DEFAULT_TABLE table=test-workload [id=2570d90deee448cea24e7125529fb8c9]), partition=RANGE (key) PARTITION 1610612733 <= VALUES
I20250112 15:08:22.957026 23410 ts_tablet_manager.cc:1428] T a84082c6bef84e638159590ed624062e P 5b101dbed8f34e9281a33db399f65ed6: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20250112 15:08:22.957087 23410 tablet_bootstrap.cc:492] T 656d5437ee6a475a89af5e24b74f315a P 5b101dbed8f34e9281a33db399f65ed6: Bootstrap starting.
I20250112 15:08:22.957409 23410 tablet_bootstrap.cc:654] T 656d5437ee6a475a89af5e24b74f315a P 5b101dbed8f34e9281a33db399f65ed6: Neither blocks nor log segments found. Creating new log.
I20250112 15:08:22.956943 23352 tablet_service.cc:1467] Processing CreateTablet for tablet 46c5820c43a64ba285e33ee2b607789b (DEFAULT_TABLE table=test-workload [id=2570d90deee448cea24e7125529fb8c9]), partition=RANGE (key) PARTITION 1073741822 <= VALUES < 1610612733
I20250112 15:08:22.957592 23351 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d5a633e37cc84a098547fa042708b1d3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 15:08:22.957835 23352 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 46c5820c43a64ba285e33ee2b607789b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 15:08:22.958726 23411 tablet_bootstrap.cc:492] T 46c5820c43a64ba285e33ee2b607789b P 91de225bd4e84a4593199b49347b6d29: No bootstrap required, opened a new log
I20250112 15:08:22.958755 23411 ts_tablet_manager.cc:1397] T 46c5820c43a64ba285e33ee2b607789b P 91de225bd4e84a4593199b49347b6d29: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20250112 15:08:22.958853 23411 raft_consensus.cc:357] T 46c5820c43a64ba285e33ee2b607789b P 91de225bd4e84a4593199b49347b6d29 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91de225bd4e84a4593199b49347b6d29" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 39577 } } peers { permanent_uuid: "5b101dbed8f34e9281a33db399f65ed6" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 36341 } } peers { permanent_uuid: "b3791458f5554d158fcb770980760849" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 42069 } }
I20250112 15:08:22.958935 23411 raft_consensus.cc:383] T 46c5820c43a64ba285e33ee2b607789b P 91de225bd4e84a4593199b49347b6d29 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 15:08:22.958979 23411 raft_consensus.cc:738] T 46c5820c43a64ba285e33ee2b607789b P 91de225bd4e84a4593199b49347b6d29 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 91de225bd4e84a4593199b49347b6d29, State: Initialized, Role: FOLLOWER
I20250112 15:08:22.958992 23353 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 656d5437ee6a475a89af5e24b74f315a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 15:08:22.959039 23411 consensus_queue.cc:260] T 46c5820c43a64ba285e33ee2b607789b P 91de225bd4e84a4593199b49347b6d29 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91de225bd4e84a4593199b49347b6d29" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 39577 } } peers { permanent_uuid: "5b101dbed8f34e9281a33db399f65ed6" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 36341 } } peers { permanent_uuid: "b3791458f5554d158fcb770980760849" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 42069 } }
I20250112 15:08:22.959136 23411 ts_tablet_manager.cc:1428] T 46c5820c43a64ba285e33ee2b607789b P 91de225bd4e84a4593199b49347b6d29: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20250112 15:08:22.959192 23411 tablet_bootstrap.cc:492] T 656d5437ee6a475a89af5e24b74f315a P 91de225bd4e84a4593199b49347b6d29: Bootstrap starting.
I20250112 15:08:22.959532 23411 tablet_bootstrap.cc:654] T 656d5437ee6a475a89af5e24b74f315a P 91de225bd4e84a4593199b49347b6d29: Neither blocks nor log segments found. Creating new log.
I20250112 15:08:22.959972 23411 tablet_bootstrap.cc:492] T 656d5437ee6a475a89af5e24b74f315a P 91de225bd4e84a4593199b49347b6d29: No bootstrap required, opened a new log
I20250112 15:08:22.960013 23411 ts_tablet_manager.cc:1397] T 656d5437ee6a475a89af5e24b74f315a P 91de225bd4e84a4593199b49347b6d29: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20250112 15:08:22.960142 23411 raft_consensus.cc:357] T 656d5437ee6a475a89af5e24b74f315a P 91de225bd4e84a4593199b49347b6d29 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91de225bd4e84a4593199b49347b6d29" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 39577 } } peers { permanent_uuid: "5b101dbed8f34e9281a33db399f65ed6" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 36341 } } peers { permanent_uuid: "b3791458f5554d158fcb770980760849" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 42069 } }
I20250112 15:08:22.960237 23411 raft_consensus.cc:383] T 656d5437ee6a475a89af5e24b74f315a P 91de225bd4e84a4593199b49347b6d29 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 15:08:22.960256 23411 raft_consensus.cc:738] T 656d5437ee6a475a89af5e24b74f315a P 91de225bd4e84a4593199b49347b6d29 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 91de225bd4e84a4593199b49347b6d29, State: Initialized, Role: FOLLOWER
I20250112 15:08:22.960300 23411 consensus_queue.cc:260] T 656d5437ee6a475a89af5e24b74f315a P 91de225bd4e84a4593199b49347b6d29 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91de225bd4e84a4593199b49347b6d29" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 39577 } } peers { permanent_uuid: "5b101dbed8f34e9281a33db399f65ed6" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 36341 } } peers { permanent_uuid: "b3791458f5554d158fcb770980760849" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 42069 } }
I20250112 15:08:22.960377 23411 ts_tablet_manager.cc:1428] T 656d5437ee6a475a89af5e24b74f315a P 91de225bd4e84a4593199b49347b6d29: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20250112 15:08:22.960417 23418 tablet_bootstrap.cc:492] T a84082c6bef84e638159590ed624062e P b3791458f5554d158fcb770980760849: Bootstrap starting.
I20250112 15:08:22.960422 23411 tablet_bootstrap.cc:492] T d5a633e37cc84a098547fa042708b1d3 P 91de225bd4e84a4593199b49347b6d29: Bootstrap starting.
I20250112 15:08:22.960731 23410 tablet_bootstrap.cc:492] T 656d5437ee6a475a89af5e24b74f315a P 5b101dbed8f34e9281a33db399f65ed6: No bootstrap required, opened a new log
I20250112 15:08:22.960780 23410 ts_tablet_manager.cc:1397] T 656d5437ee6a475a89af5e24b74f315a P 5b101dbed8f34e9281a33db399f65ed6: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.001s
I20250112 15:08:22.960858 23411 tablet_bootstrap.cc:654] T d5a633e37cc84a098547fa042708b1d3 P 91de225bd4e84a4593199b49347b6d29: Neither blocks nor log segments found. Creating new log.
I20250112 15:08:22.960922 23410 raft_consensus.cc:357] T 656d5437ee6a475a89af5e24b74f315a P 5b101dbed8f34e9281a33db399f65ed6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91de225bd4e84a4593199b49347b6d29" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 39577 } } peers { permanent_uuid: "5b101dbed8f34e9281a33db399f65ed6" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 36341 } } peers { permanent_uuid: "b3791458f5554d158fcb770980760849" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 42069 } }
I20250112 15:08:22.960985 23410 raft_consensus.cc:383] T 656d5437ee6a475a89af5e24b74f315a P 5b101dbed8f34e9281a33db399f65ed6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 15:08:22.961004 23410 raft_consensus.cc:738] T 656d5437ee6a475a89af5e24b74f315a P 5b101dbed8f34e9281a33db399f65ed6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5b101dbed8f34e9281a33db399f65ed6, State: Initialized, Role: FOLLOWER
I20250112 15:08:22.961047 23410 consensus_queue.cc:260] T 656d5437ee6a475a89af5e24b74f315a P 5b101dbed8f34e9281a33db399f65ed6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91de225bd4e84a4593199b49347b6d29" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 39577 } } peers { permanent_uuid: "5b101dbed8f34e9281a33db399f65ed6" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 36341 } } peers { permanent_uuid: "b3791458f5554d158fcb770980760849" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 42069 } }
I20250112 15:08:22.961130 23410 ts_tablet_manager.cc:1428] T 656d5437ee6a475a89af5e24b74f315a P 5b101dbed8f34e9281a33db399f65ed6: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20250112 15:08:22.961179 23410 tablet_bootstrap.cc:492] T 46c5820c43a64ba285e33ee2b607789b P 5b101dbed8f34e9281a33db399f65ed6: Bootstrap starting.
I20250112 15:08:22.961594 23410 tablet_bootstrap.cc:654] T 46c5820c43a64ba285e33ee2b607789b P 5b101dbed8f34e9281a33db399f65ed6: Neither blocks nor log segments found. Creating new log.
I20250112 15:08:22.961629 23411 tablet_bootstrap.cc:492] T d5a633e37cc84a098547fa042708b1d3 P 91de225bd4e84a4593199b49347b6d29: No bootstrap required, opened a new log
I20250112 15:08:22.961675 23411 ts_tablet_manager.cc:1397] T d5a633e37cc84a098547fa042708b1d3 P 91de225bd4e84a4593199b49347b6d29: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20250112 15:08:22.961823 23411 raft_consensus.cc:357] T d5a633e37cc84a098547fa042708b1d3 P 91de225bd4e84a4593199b49347b6d29 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91de225bd4e84a4593199b49347b6d29" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 39577 } } peers { permanent_uuid: "5b101dbed8f34e9281a33db399f65ed6" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 36341 } } peers { permanent_uuid: "b3791458f5554d158fcb770980760849" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 42069 } }
I20250112 15:08:22.961885 23411 raft_consensus.cc:383] T d5a633e37cc84a098547fa042708b1d3 P 91de225bd4e84a4593199b49347b6d29 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 15:08:22.961915 23411 raft_consensus.cc:738] T d5a633e37cc84a098547fa042708b1d3 P 91de225bd4e84a4593199b49347b6d29 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 91de225bd4e84a4593199b49347b6d29, State: Initialized, Role: FOLLOWER
I20250112 15:08:22.961958 23411 consensus_queue.cc:260] T d5a633e37cc84a098547fa042708b1d3 P 91de225bd4e84a4593199b49347b6d29 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91de225bd4e84a4593199b49347b6d29" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 39577 } } peers { permanent_uuid: "5b101dbed8f34e9281a33db399f65ed6" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 36341 } } peers { permanent_uuid: "b3791458f5554d158fcb770980760849" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 42069 } }
I20250112 15:08:22.962065 23411 ts_tablet_manager.cc:1428] T d5a633e37cc84a098547fa042708b1d3 P 91de225bd4e84a4593199b49347b6d29: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20250112 15:08:22.962138 23410 tablet_bootstrap.cc:492] T 46c5820c43a64ba285e33ee2b607789b P 5b101dbed8f34e9281a33db399f65ed6: No bootstrap required, opened a new log
I20250112 15:08:22.962178 23410 ts_tablet_manager.cc:1397] T 46c5820c43a64ba285e33ee2b607789b P 5b101dbed8f34e9281a33db399f65ed6: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20250112 15:08:22.962314 23410 raft_consensus.cc:357] T 46c5820c43a64ba285e33ee2b607789b P 5b101dbed8f34e9281a33db399f65ed6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91de225bd4e84a4593199b49347b6d29" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 39577 } } peers { permanent_uuid: "5b101dbed8f34e9281a33db399f65ed6" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 36341 } } peers { permanent_uuid: "b3791458f5554d158fcb770980760849" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 42069 } }
I20250112 15:08:22.962376 23410 raft_consensus.cc:383] T 46c5820c43a64ba285e33ee2b607789b P 5b101dbed8f34e9281a33db399f65ed6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 15:08:22.962399 23410 raft_consensus.cc:738] T 46c5820c43a64ba285e33ee2b607789b P 5b101dbed8f34e9281a33db399f65ed6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5b101dbed8f34e9281a33db399f65ed6, State: Initialized, Role: FOLLOWER
I20250112 15:08:22.962445 23410 consensus_queue.cc:260] T 46c5820c43a64ba285e33ee2b607789b P 5b101dbed8f34e9281a33db399f65ed6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91de225bd4e84a4593199b49347b6d29" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 39577 } } peers { permanent_uuid: "5b101dbed8f34e9281a33db399f65ed6" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 36341 } } peers { permanent_uuid: "b3791458f5554d158fcb770980760849" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 42069 } }
I20250112 15:08:22.962514 23410 ts_tablet_manager.cc:1428] T 46c5820c43a64ba285e33ee2b607789b P 5b101dbed8f34e9281a33db399f65ed6: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20250112 15:08:22.962567 23410 tablet_bootstrap.cc:492] T d5a633e37cc84a098547fa042708b1d3 P 5b101dbed8f34e9281a33db399f65ed6: Bootstrap starting.
I20250112 15:08:22.962893 23410 tablet_bootstrap.cc:654] T d5a633e37cc84a098547fa042708b1d3 P 5b101dbed8f34e9281a33db399f65ed6: Neither blocks nor log segments found. Creating new log.
I20250112 15:08:22.963541 23410 tablet_bootstrap.cc:492] T d5a633e37cc84a098547fa042708b1d3 P 5b101dbed8f34e9281a33db399f65ed6: No bootstrap required, opened a new log
I20250112 15:08:22.963593 23410 ts_tablet_manager.cc:1397] T d5a633e37cc84a098547fa042708b1d3 P 5b101dbed8f34e9281a33db399f65ed6: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20250112 15:08:22.963732 23410 raft_consensus.cc:357] T d5a633e37cc84a098547fa042708b1d3 P 5b101dbed8f34e9281a33db399f65ed6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91de225bd4e84a4593199b49347b6d29" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 39577 } } peers { permanent_uuid: "5b101dbed8f34e9281a33db399f65ed6" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 36341 } } peers { permanent_uuid: "b3791458f5554d158fcb770980760849" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 42069 } }
I20250112 15:08:22.963802 23410 raft_consensus.cc:383] T d5a633e37cc84a098547fa042708b1d3 P 5b101dbed8f34e9281a33db399f65ed6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 15:08:22.963824 23410 raft_consensus.cc:738] T d5a633e37cc84a098547fa042708b1d3 P 5b101dbed8f34e9281a33db399f65ed6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5b101dbed8f34e9281a33db399f65ed6, State: Initialized, Role: FOLLOWER
I20250112 15:08:22.963872 23410 consensus_queue.cc:260] T d5a633e37cc84a098547fa042708b1d3 P 5b101dbed8f34e9281a33db399f65ed6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91de225bd4e84a4593199b49347b6d29" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 39577 } } peers { permanent_uuid: "5b101dbed8f34e9281a33db399f65ed6" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 36341 } } peers { permanent_uuid: "b3791458f5554d158fcb770980760849" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 42069 } }
I20250112 15:08:22.963959 23410 ts_tablet_manager.cc:1428] T d5a633e37cc84a098547fa042708b1d3 P 5b101dbed8f34e9281a33db399f65ed6: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20250112 15:08:22.964877 23418 tablet_bootstrap.cc:654] T a84082c6bef84e638159590ed624062e P b3791458f5554d158fcb770980760849: Neither blocks nor log segments found. Creating new log.
I20250112 15:08:22.965456 23418 tablet_bootstrap.cc:492] T a84082c6bef84e638159590ed624062e P b3791458f5554d158fcb770980760849: No bootstrap required, opened a new log
I20250112 15:08:22.965508 23418 ts_tablet_manager.cc:1397] T a84082c6bef84e638159590ed624062e P b3791458f5554d158fcb770980760849: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.001s
I20250112 15:08:22.965672 23418 raft_consensus.cc:357] T a84082c6bef84e638159590ed624062e P b3791458f5554d158fcb770980760849 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b101dbed8f34e9281a33db399f65ed6" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 36341 } } peers { permanent_uuid: "b3791458f5554d158fcb770980760849" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 42069 } } peers { permanent_uuid: "91de225bd4e84a4593199b49347b6d29" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 39577 } }
I20250112 15:08:22.965731 23418 raft_consensus.cc:383] T a84082c6bef84e638159590ed624062e P b3791458f5554d158fcb770980760849 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 15:08:22.965759 23418 raft_consensus.cc:738] T a84082c6bef84e638159590ed624062e P b3791458f5554d158fcb770980760849 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b3791458f5554d158fcb770980760849, State: Initialized, Role: FOLLOWER
I20250112 15:08:22.965821 23418 consensus_queue.cc:260] T a84082c6bef84e638159590ed624062e P b3791458f5554d158fcb770980760849 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b101dbed8f34e9281a33db399f65ed6" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 36341 } } peers { permanent_uuid: "b3791458f5554d158fcb770980760849" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 42069 } } peers { permanent_uuid: "91de225bd4e84a4593199b49347b6d29" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 39577 } }
I20250112 15:08:22.965977 23418 ts_tablet_manager.cc:1428] T a84082c6bef84e638159590ed624062e P b3791458f5554d158fcb770980760849: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20250112 15:08:22.966128 23418 tablet_bootstrap.cc:492] T d5a633e37cc84a098547fa042708b1d3 P b3791458f5554d158fcb770980760849: Bootstrap starting.
I20250112 15:08:22.966558 23418 tablet_bootstrap.cc:654] T d5a633e37cc84a098547fa042708b1d3 P b3791458f5554d158fcb770980760849: Neither blocks nor log segments found. Creating new log.
I20250112 15:08:22.967140 23418 tablet_bootstrap.cc:492] T d5a633e37cc84a098547fa042708b1d3 P b3791458f5554d158fcb770980760849: No bootstrap required, opened a new log
I20250112 15:08:22.967185 23418 ts_tablet_manager.cc:1397] T d5a633e37cc84a098547fa042708b1d3 P b3791458f5554d158fcb770980760849: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20250112 15:08:22.967305 23418 raft_consensus.cc:357] T d5a633e37cc84a098547fa042708b1d3 P b3791458f5554d158fcb770980760849 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91de225bd4e84a4593199b49347b6d29" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 39577 } } peers { permanent_uuid: "5b101dbed8f34e9281a33db399f65ed6" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 36341 } } peers { permanent_uuid: "b3791458f5554d158fcb770980760849" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 42069 } }
I20250112 15:08:22.967350 23418 raft_consensus.cc:383] T d5a633e37cc84a098547fa042708b1d3 P b3791458f5554d158fcb770980760849 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 15:08:22.967372 23418 raft_consensus.cc:738] T d5a633e37cc84a098547fa042708b1d3 P b3791458f5554d158fcb770980760849 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b3791458f5554d158fcb770980760849, State: Initialized, Role: FOLLOWER
I20250112 15:08:22.967453 23418 consensus_queue.cc:260] T d5a633e37cc84a098547fa042708b1d3 P b3791458f5554d158fcb770980760849 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91de225bd4e84a4593199b49347b6d29" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 39577 } } peers { permanent_uuid: "5b101dbed8f34e9281a33db399f65ed6" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 36341 } } peers { permanent_uuid: "b3791458f5554d158fcb770980760849" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 42069 } }
I20250112 15:08:22.967535 23418 ts_tablet_manager.cc:1428] T d5a633e37cc84a098547fa042708b1d3 P b3791458f5554d158fcb770980760849: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20250112 15:08:22.967605 23418 tablet_bootstrap.cc:492] T 46c5820c43a64ba285e33ee2b607789b P b3791458f5554d158fcb770980760849: Bootstrap starting.
I20250112 15:08:22.968040 23418 tablet_bootstrap.cc:654] T 46c5820c43a64ba285e33ee2b607789b P b3791458f5554d158fcb770980760849: Neither blocks nor log segments found. Creating new log.
I20250112 15:08:22.968658 23418 tablet_bootstrap.cc:492] T 46c5820c43a64ba285e33ee2b607789b P b3791458f5554d158fcb770980760849: No bootstrap required, opened a new log
I20250112 15:08:22.968711 23418 ts_tablet_manager.cc:1397] T 46c5820c43a64ba285e33ee2b607789b P b3791458f5554d158fcb770980760849: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20250112 15:08:22.968833 23418 raft_consensus.cc:357] T 46c5820c43a64ba285e33ee2b607789b P b3791458f5554d158fcb770980760849 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91de225bd4e84a4593199b49347b6d29" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 39577 } } peers { permanent_uuid: "5b101dbed8f34e9281a33db399f65ed6" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 36341 } } peers { permanent_uuid: "b3791458f5554d158fcb770980760849" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 42069 } }
I20250112 15:08:22.968878 23418 raft_consensus.cc:383] T 46c5820c43a64ba285e33ee2b607789b P b3791458f5554d158fcb770980760849 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 15:08:22.968889 23418 raft_consensus.cc:738] T 46c5820c43a64ba285e33ee2b607789b P b3791458f5554d158fcb770980760849 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b3791458f5554d158fcb770980760849, State: Initialized, Role: FOLLOWER
I20250112 15:08:22.968947 23418 consensus_queue.cc:260] T 46c5820c43a64ba285e33ee2b607789b P b3791458f5554d158fcb770980760849 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91de225bd4e84a4593199b49347b6d29" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 39577 } } peers { permanent_uuid: "5b101dbed8f34e9281a33db399f65ed6" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 36341 } } peers { permanent_uuid: "b3791458f5554d158fcb770980760849" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 42069 } }
I20250112 15:08:22.969048 23418 ts_tablet_manager.cc:1428] T 46c5820c43a64ba285e33ee2b607789b P b3791458f5554d158fcb770980760849: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20250112 15:08:22.969151 23418 tablet_bootstrap.cc:492] T 656d5437ee6a475a89af5e24b74f315a P b3791458f5554d158fcb770980760849: Bootstrap starting.
I20250112 15:08:22.969576 23418 tablet_bootstrap.cc:654] T 656d5437ee6a475a89af5e24b74f315a P b3791458f5554d158fcb770980760849: Neither blocks nor log segments found. Creating new log.
I20250112 15:08:22.970180 23418 tablet_bootstrap.cc:492] T 656d5437ee6a475a89af5e24b74f315a P b3791458f5554d158fcb770980760849: No bootstrap required, opened a new log
I20250112 15:08:22.970242 23418 ts_tablet_manager.cc:1397] T 656d5437ee6a475a89af5e24b74f315a P b3791458f5554d158fcb770980760849: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20250112 15:08:22.970394 23418 raft_consensus.cc:357] T 656d5437ee6a475a89af5e24b74f315a P b3791458f5554d158fcb770980760849 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91de225bd4e84a4593199b49347b6d29" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 39577 } } peers { permanent_uuid: "5b101dbed8f34e9281a33db399f65ed6" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 36341 } } peers { permanent_uuid: "b3791458f5554d158fcb770980760849" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 42069 } }
I20250112 15:08:22.970440 23418 raft_consensus.cc:383] T 656d5437ee6a475a89af5e24b74f315a P b3791458f5554d158fcb770980760849 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 15:08:22.970461 23418 raft_consensus.cc:738] T 656d5437ee6a475a89af5e24b74f315a P b3791458f5554d158fcb770980760849 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b3791458f5554d158fcb770980760849, State: Initialized, Role: FOLLOWER
I20250112 15:08:22.970499 23418 consensus_queue.cc:260] T 656d5437ee6a475a89af5e24b74f315a P b3791458f5554d158fcb770980760849 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91de225bd4e84a4593199b49347b6d29" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 39577 } } peers { permanent_uuid: "5b101dbed8f34e9281a33db399f65ed6" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 36341 } } peers { permanent_uuid: "b3791458f5554d158fcb770980760849" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 42069 } }
I20250112 15:08:22.970582 23418 ts_tablet_manager.cc:1428] T 656d5437ee6a475a89af5e24b74f315a P b3791458f5554d158fcb770980760849: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20250112 15:08:22.984647 23413 raft_consensus.cc:491] T a84082c6bef84e638159590ed624062e P 91de225bd4e84a4593199b49347b6d29 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 15:08:22.984759 23413 raft_consensus.cc:513] T a84082c6bef84e638159590ed624062e P 91de225bd4e84a4593199b49347b6d29 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b101dbed8f34e9281a33db399f65ed6" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 36341 } } peers { permanent_uuid: "b3791458f5554d158fcb770980760849" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 42069 } } peers { permanent_uuid: "91de225bd4e84a4593199b49347b6d29" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 39577 } }
I20250112 15:08:22.984935 23413 leader_election.cc:290] T a84082c6bef84e638159590ed624062e P 91de225bd4e84a4593199b49347b6d29 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5b101dbed8f34e9281a33db399f65ed6 (127.17.60.194:36341), b3791458f5554d158fcb770980760849 (127.17.60.195:42069)
I20250112 15:08:22.987708 23364 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a84082c6bef84e638159590ed624062e" candidate_uuid: "91de225bd4e84a4593199b49347b6d29" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b3791458f5554d158fcb770980760849" is_pre_election: true
I20250112 15:08:22.987812 23364 raft_consensus.cc:2463] T a84082c6bef84e638159590ed624062e P b3791458f5554d158fcb770980760849 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 91de225bd4e84a4593199b49347b6d29 in term 0.
I20250112 15:08:22.987722 23289 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a84082c6bef84e638159590ed624062e" candidate_uuid: "91de225bd4e84a4593199b49347b6d29" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b101dbed8f34e9281a33db399f65ed6" is_pre_election: true
I20250112 15:08:22.987861 23289 raft_consensus.cc:2463] T a84082c6bef84e638159590ed624062e P 5b101dbed8f34e9281a33db399f65ed6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 91de225bd4e84a4593199b49347b6d29 in term 0.
I20250112 15:08:22.987965 23177 leader_election.cc:304] T a84082c6bef84e638159590ed624062e P 91de225bd4e84a4593199b49347b6d29 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 91de225bd4e84a4593199b49347b6d29, b3791458f5554d158fcb770980760849; no voters:
I20250112 15:08:22.988070 23413 raft_consensus.cc:2798] T a84082c6bef84e638159590ed624062e P 91de225bd4e84a4593199b49347b6d29 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250112 15:08:22.988106 23413 raft_consensus.cc:491] T a84082c6bef84e638159590ed624062e P 91de225bd4e84a4593199b49347b6d29 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250112 15:08:22.988129 23413 raft_consensus.cc:3054] T a84082c6bef84e638159590ed624062e P 91de225bd4e84a4593199b49347b6d29 [term 0 FOLLOWER]: Advancing to term 1
I20250112 15:08:22.988852 23413 raft_consensus.cc:513] T a84082c6bef84e638159590ed624062e P 91de225bd4e84a4593199b49347b6d29 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b101dbed8f34e9281a33db399f65ed6" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 36341 } } peers { permanent_uuid: "b3791458f5554d158fcb770980760849" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 42069 } } peers { permanent_uuid: "91de225bd4e84a4593199b49347b6d29" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 39577 } }
I20250112 15:08:22.988996 23413 leader_election.cc:290] T a84082c6bef84e638159590ed624062e P 91de225bd4e84a4593199b49347b6d29 [CANDIDATE]: Term 1 election: Requested vote from peers 5b101dbed8f34e9281a33db399f65ed6 (127.17.60.194:36341), b3791458f5554d158fcb770980760849 (127.17.60.195:42069)
I20250112 15:08:22.989162 23289 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a84082c6bef84e638159590ed624062e" candidate_uuid: "91de225bd4e84a4593199b49347b6d29" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b101dbed8f34e9281a33db399f65ed6"
I20250112 15:08:22.989218 23289 raft_consensus.cc:3054] T a84082c6bef84e638159590ed624062e P 5b101dbed8f34e9281a33db399f65ed6 [term 0 FOLLOWER]: Advancing to term 1
I20250112 15:08:22.989202 23364 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a84082c6bef84e638159590ed624062e" candidate_uuid: "91de225bd4e84a4593199b49347b6d29" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b3791458f5554d158fcb770980760849"
I20250112 15:08:22.989306 23364 raft_consensus.cc:3054] T a84082c6bef84e638159590ed624062e P b3791458f5554d158fcb770980760849 [term 0 FOLLOWER]: Advancing to term 1
I20250112 15:08:22.989863 23289 raft_consensus.cc:2463] T a84082c6bef84e638159590ed624062e P 5b101dbed8f34e9281a33db399f65ed6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 91de225bd4e84a4593199b49347b6d29 in term 1.
I20250112 15:08:22.989953 23364 raft_consensus.cc:2463] T a84082c6bef84e638159590ed624062e P b3791458f5554d158fcb770980760849 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 91de225bd4e84a4593199b49347b6d29 in term 1.
I20250112 15:08:22.990087 23177 leader_election.cc:304] T a84082c6bef84e638159590ed624062e P 91de225bd4e84a4593199b49347b6d29 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5b101dbed8f34e9281a33db399f65ed6, 91de225bd4e84a4593199b49347b6d29; no voters:
I20250112 15:08:22.990201 23413 raft_consensus.cc:2798] T a84082c6bef84e638159590ed624062e P 91de225bd4e84a4593199b49347b6d29 [term 1 FOLLOWER]: Leader election won for term 1
I20250112 15:08:22.990341 23413 raft_consensus.cc:695] T a84082c6bef84e638159590ed624062e P 91de225bd4e84a4593199b49347b6d29 [term 1 LEADER]: Becoming Leader. State: Replica: 91de225bd4e84a4593199b49347b6d29, State: Running, Role: LEADER
I20250112 15:08:22.990387 23413 consensus_queue.cc:237] T a84082c6bef84e638159590ed624062e P 91de225bd4e84a4593199b49347b6d29 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b101dbed8f34e9281a33db399f65ed6" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 36341 } } peers { permanent_uuid: "b3791458f5554d158fcb770980760849" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 42069 } } peers { permanent_uuid: "91de225bd4e84a4593199b49347b6d29" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 39577 } }
I20250112 15:08:22.990902 23110 catalog_manager.cc:5526] T a84082c6bef84e638159590ed624062e P 91de225bd4e84a4593199b49347b6d29 reported cstate change: term changed from 0 to 1, leader changed from <none> to 91de225bd4e84a4593199b49347b6d29 (127.17.60.193). New cstate: current_term: 1 leader_uuid: "91de225bd4e84a4593199b49347b6d29" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b101dbed8f34e9281a33db399f65ed6" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 36341 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b3791458f5554d158fcb770980760849" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 42069 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "91de225bd4e84a4593199b49347b6d29" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 39577 } health_report { overall_health: HEALTHY } } }
I20250112 15:08:23.166702 23416 raft_consensus.cc:491] T d5a633e37cc84a098547fa042708b1d3 P 5b101dbed8f34e9281a33db399f65ed6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 15:08:23.166818 23416 raft_consensus.cc:513] T d5a633e37cc84a098547fa042708b1d3 P 5b101dbed8f34e9281a33db399f65ed6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91de225bd4e84a4593199b49347b6d29" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 39577 } } peers { permanent_uuid: "5b101dbed8f34e9281a33db399f65ed6" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 36341 } } peers { permanent_uuid: "b3791458f5554d158fcb770980760849" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 42069 } }
I20250112 15:08:23.167057 23416 leader_election.cc:290] T d5a633e37cc84a098547fa042708b1d3 P 5b101dbed8f34e9281a33db399f65ed6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 91de225bd4e84a4593199b49347b6d29 (127.17.60.193:39577), b3791458f5554d158fcb770980760849 (127.17.60.195:42069)
I20250112 15:08:23.169661 23364 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d5a633e37cc84a098547fa042708b1d3" candidate_uuid: "5b101dbed8f34e9281a33db399f65ed6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b3791458f5554d158fcb770980760849" is_pre_election: true
I20250112 15:08:23.169680 23213 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d5a633e37cc84a098547fa042708b1d3" candidate_uuid: "5b101dbed8f34e9281a33db399f65ed6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "91de225bd4e84a4593199b49347b6d29" is_pre_election: true
I20250112 15:08:23.169767 23364 raft_consensus.cc:2463] T d5a633e37cc84a098547fa042708b1d3 P b3791458f5554d158fcb770980760849 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5b101dbed8f34e9281a33db399f65ed6 in term 0.
I20250112 15:08:23.169785 23213 raft_consensus.cc:2463] T d5a633e37cc84a098547fa042708b1d3 P 91de225bd4e84a4593199b49347b6d29 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5b101dbed8f34e9281a33db399f65ed6 in term 0.
I20250112 15:08:23.169930 23253 leader_election.cc:304] T d5a633e37cc84a098547fa042708b1d3 P 5b101dbed8f34e9281a33db399f65ed6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5b101dbed8f34e9281a33db399f65ed6, b3791458f5554d158fcb770980760849; no voters:
I20250112 15:08:23.170018 23416 raft_consensus.cc:2798] T d5a633e37cc84a098547fa042708b1d3 P 5b101dbed8f34e9281a33db399f65ed6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250112 15:08:23.170061 23416 raft_consensus.cc:491] T d5a633e37cc84a098547fa042708b1d3 P 5b101dbed8f34e9281a33db399f65ed6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250112 15:08:23.170078 23416 raft_consensus.cc:3054] T d5a633e37cc84a098547fa042708b1d3 P 5b101dbed8f34e9281a33db399f65ed6 [term 0 FOLLOWER]: Advancing to term 1
I20250112 15:08:23.170694 23416 raft_consensus.cc:513] T d5a633e37cc84a098547fa042708b1d3 P 5b101dbed8f34e9281a33db399f65ed6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91de225bd4e84a4593199b49347b6d29" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 39577 } } peers { permanent_uuid: "5b101dbed8f34e9281a33db399f65ed6" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 36341 } } peers { permanent_uuid: "b3791458f5554d158fcb770980760849" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 42069 } }
I20250112 15:08:23.170802 23416 leader_election.cc:290] T d5a633e37cc84a098547fa042708b1d3 P 5b101dbed8f34e9281a33db399f65ed6 [CANDIDATE]: Term 1 election: Requested vote from peers 91de225bd4e84a4593199b49347b6d29 (127.17.60.193:39577), b3791458f5554d158fcb770980760849 (127.17.60.195:42069)
I20250112 15:08:23.170984 23364 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d5a633e37cc84a098547fa042708b1d3" candidate_uuid: "5b101dbed8f34e9281a33db399f65ed6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b3791458f5554d158fcb770980760849"
I20250112 15:08:23.171034 23364 raft_consensus.cc:3054] T d5a633e37cc84a098547fa042708b1d3 P b3791458f5554d158fcb770980760849 [term 0 FOLLOWER]: Advancing to term 1
I20250112 15:08:23.171518 23364 raft_consensus.cc:2463] T d5a633e37cc84a098547fa042708b1d3 P b3791458f5554d158fcb770980760849 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5b101dbed8f34e9281a33db399f65ed6 in term 1.
I20250112 15:08:23.171597 23213 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d5a633e37cc84a098547fa042708b1d3" candidate_uuid: "5b101dbed8f34e9281a33db399f65ed6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "91de225bd4e84a4593199b49347b6d29"
I20250112 15:08:23.171648 23213 raft_consensus.cc:3054] T d5a633e37cc84a098547fa042708b1d3 P 91de225bd4e84a4593199b49347b6d29 [term 0 FOLLOWER]: Advancing to term 1
I20250112 15:08:23.171656 23253 leader_election.cc:304] T d5a633e37cc84a098547fa042708b1d3 P 5b101dbed8f34e9281a33db399f65ed6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5b101dbed8f34e9281a33db399f65ed6, b3791458f5554d158fcb770980760849; no voters:
I20250112 15:08:23.171746 23416 raft_consensus.cc:2798] T d5a633e37cc84a098547fa042708b1d3 P 5b101dbed8f34e9281a33db399f65ed6 [term 1 FOLLOWER]: Leader election won for term 1
I20250112 15:08:23.171891 23416 raft_consensus.cc:695] T d5a633e37cc84a098547fa042708b1d3 P 5b101dbed8f34e9281a33db399f65ed6 [term 1 LEADER]: Becoming Leader. State: Replica: 5b101dbed8f34e9281a33db399f65ed6, State: Running, Role: LEADER
I20250112 15:08:23.171973 23416 consensus_queue.cc:237] T d5a633e37cc84a098547fa042708b1d3 P 5b101dbed8f34e9281a33db399f65ed6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91de225bd4e84a4593199b49347b6d29" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 39577 } } peers { permanent_uuid: "5b101dbed8f34e9281a33db399f65ed6" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 36341 } } peers { permanent_uuid: "b3791458f5554d158fcb770980760849" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 42069 } }
I20250112 15:08:23.172315 23213 raft_consensus.cc:2463] T d5a633e37cc84a098547fa042708b1d3 P 91de225bd4e84a4593199b49347b6d29 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5b101dbed8f34e9281a33db399f65ed6 in term 1.
I20250112 15:08:23.172502 23110 catalog_manager.cc:5526] T d5a633e37cc84a098547fa042708b1d3 P 5b101dbed8f34e9281a33db399f65ed6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5b101dbed8f34e9281a33db399f65ed6 (127.17.60.194). New cstate: current_term: 1 leader_uuid: "5b101dbed8f34e9281a33db399f65ed6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91de225bd4e84a4593199b49347b6d29" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 39577 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5b101dbed8f34e9281a33db399f65ed6" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 36341 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b3791458f5554d158fcb770980760849" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 42069 } health_report { overall_health: UNKNOWN } } }
I20250112 15:08:23.261133 23416 raft_consensus.cc:491] T 656d5437ee6a475a89af5e24b74f315a P 5b101dbed8f34e9281a33db399f65ed6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 15:08:23.261241 23416 raft_consensus.cc:513] T 656d5437ee6a475a89af5e24b74f315a P 5b101dbed8f34e9281a33db399f65ed6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91de225bd4e84a4593199b49347b6d29" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 39577 } } peers { permanent_uuid: "5b101dbed8f34e9281a33db399f65ed6" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 36341 } } peers { permanent_uuid: "b3791458f5554d158fcb770980760849" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 42069 } }
I20250112 15:08:23.261420 23416 leader_election.cc:290] T 656d5437ee6a475a89af5e24b74f315a P 5b101dbed8f34e9281a33db399f65ed6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 91de225bd4e84a4593199b49347b6d29 (127.17.60.193:39577), b3791458f5554d158fcb770980760849 (127.17.60.195:42069)
I20250112 15:08:23.261651 23213 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "656d5437ee6a475a89af5e24b74f315a" candidate_uuid: "5b101dbed8f34e9281a33db399f65ed6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "91de225bd4e84a4593199b49347b6d29" is_pre_election: true
I20250112 15:08:23.261648 23364 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "656d5437ee6a475a89af5e24b74f315a" candidate_uuid: "5b101dbed8f34e9281a33db399f65ed6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b3791458f5554d158fcb770980760849" is_pre_election: true
I20250112 15:08:23.261744 23213 raft_consensus.cc:2463] T 656d5437ee6a475a89af5e24b74f315a P 91de225bd4e84a4593199b49347b6d29 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5b101dbed8f34e9281a33db399f65ed6 in term 0.
I20250112 15:08:23.261745 23364 raft_consensus.cc:2463] T 656d5437ee6a475a89af5e24b74f315a P b3791458f5554d158fcb770980760849 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5b101dbed8f34e9281a33db399f65ed6 in term 0.
I20250112 15:08:23.261924 23254 leader_election.cc:304] T 656d5437ee6a475a89af5e24b74f315a P 5b101dbed8f34e9281a33db399f65ed6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5b101dbed8f34e9281a33db399f65ed6, 91de225bd4e84a4593199b49347b6d29; no voters:
I20250112 15:08:23.262035 23416 raft_consensus.cc:2798] T 656d5437ee6a475a89af5e24b74f315a P 5b101dbed8f34e9281a33db399f65ed6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250112 15:08:23.262069 23416 raft_consensus.cc:491] T 656d5437ee6a475a89af5e24b74f315a P 5b101dbed8f34e9281a33db399f65ed6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250112 15:08:23.262096 23416 raft_consensus.cc:3054] T 656d5437ee6a475a89af5e24b74f315a P 5b101dbed8f34e9281a33db399f65ed6 [term 0 FOLLOWER]: Advancing to term 1
I20250112 15:08:23.262701 23416 raft_consensus.cc:513] T 656d5437ee6a475a89af5e24b74f315a P 5b101dbed8f34e9281a33db399f65ed6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91de225bd4e84a4593199b49347b6d29" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 39577 } } peers { permanent_uuid: "5b101dbed8f34e9281a33db399f65ed6" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 36341 } } peers { permanent_uuid: "b3791458f5554d158fcb770980760849" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 42069 } }
I20250112 15:08:23.262820 23416 leader_election.cc:290] T 656d5437ee6a475a89af5e24b74f315a P 5b101dbed8f34e9281a33db399f65ed6 [CANDIDATE]: Term 1 election: Requested vote from peers 91de225bd4e84a4593199b49347b6d29 (127.17.60.193:39577), b3791458f5554d158fcb770980760849 (127.17.60.195:42069)
I20250112 15:08:23.262959 23364 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "656d5437ee6a475a89af5e24b74f315a" candidate_uuid: "5b101dbed8f34e9281a33db399f65ed6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b3791458f5554d158fcb770980760849"
I20250112 15:08:23.262955 23213 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "656d5437ee6a475a89af5e24b74f315a" candidate_uuid: "5b101dbed8f34e9281a33db399f65ed6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "91de225bd4e84a4593199b49347b6d29"
I20250112 15:08:23.263089 23364 raft_consensus.cc:3054] T 656d5437ee6a475a89af5e24b74f315a P b3791458f5554d158fcb770980760849 [term 0 FOLLOWER]: Advancing to term 1
I20250112 15:08:23.263125 23213 raft_consensus.cc:3054] T 656d5437ee6a475a89af5e24b74f315a P 91de225bd4e84a4593199b49347b6d29 [term 0 FOLLOWER]: Advancing to term 1
I20250112 15:08:23.263661 23364 raft_consensus.cc:2463] T 656d5437ee6a475a89af5e24b74f315a P b3791458f5554d158fcb770980760849 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5b101dbed8f34e9281a33db399f65ed6 in term 1.
I20250112 15:08:23.263680 23213 raft_consensus.cc:2463] T 656d5437ee6a475a89af5e24b74f315a P 91de225bd4e84a4593199b49347b6d29 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5b101dbed8f34e9281a33db399f65ed6 in term 1.
I20250112 15:08:23.263806 23253 leader_election.cc:304] T 656d5437ee6a475a89af5e24b74f315a P 5b101dbed8f34e9281a33db399f65ed6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5b101dbed8f34e9281a33db399f65ed6, b3791458f5554d158fcb770980760849; no voters:
I20250112 15:08:23.263885 23416 raft_consensus.cc:2798] T 656d5437ee6a475a89af5e24b74f315a P 5b101dbed8f34e9281a33db399f65ed6 [term 1 FOLLOWER]: Leader election won for term 1
I20250112 15:08:23.263953 23416 raft_consensus.cc:695] T 656d5437ee6a475a89af5e24b74f315a P 5b101dbed8f34e9281a33db399f65ed6 [term 1 LEADER]: Becoming Leader. State: Replica: 5b101dbed8f34e9281a33db399f65ed6, State: Running, Role: LEADER
I20250112 15:08:23.264014 23416 consensus_queue.cc:237] T 656d5437ee6a475a89af5e24b74f315a P 5b101dbed8f34e9281a33db399f65ed6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91de225bd4e84a4593199b49347b6d29" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 39577 } } peers { permanent_uuid: "5b101dbed8f34e9281a33db399f65ed6" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 36341 } } peers { permanent_uuid: "b3791458f5554d158fcb770980760849" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 42069 } }
I20250112 15:08:23.264582 23110 catalog_manager.cc:5526] T 656d5437ee6a475a89af5e24b74f315a P 5b101dbed8f34e9281a33db399f65ed6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5b101dbed8f34e9281a33db399f65ed6 (127.17.60.194). New cstate: current_term: 1 leader_uuid: "5b101dbed8f34e9281a33db399f65ed6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91de225bd4e84a4593199b49347b6d29" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 39577 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5b101dbed8f34e9281a33db399f65ed6" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 36341 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b3791458f5554d158fcb770980760849" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 42069 } health_report { overall_health: UNKNOWN } } }
I20250112 15:08:23.301002 23413 raft_consensus.cc:491] T 46c5820c43a64ba285e33ee2b607789b P 91de225bd4e84a4593199b49347b6d29 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 15:08:23.301126 23413 raft_consensus.cc:513] T 46c5820c43a64ba285e33ee2b607789b P 91de225bd4e84a4593199b49347b6d29 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91de225bd4e84a4593199b49347b6d29" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 39577 } } peers { permanent_uuid: "5b101dbed8f34e9281a33db399f65ed6" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 36341 } } peers { permanent_uuid: "b3791458f5554d158fcb770980760849" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 42069 } }
I20250112 15:08:23.301290 23413 leader_election.cc:290] T 46c5820c43a64ba285e33ee2b607789b P 91de225bd4e84a4593199b49347b6d29 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5b101dbed8f34e9281a33db399f65ed6 (127.17.60.194:36341), b3791458f5554d158fcb770980760849 (127.17.60.195:42069)
I20250112 15:08:23.301559 23289 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "46c5820c43a64ba285e33ee2b607789b" candidate_uuid: "91de225bd4e84a4593199b49347b6d29" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b101dbed8f34e9281a33db399f65ed6" is_pre_election: true
I20250112 15:08:23.301559 23364 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "46c5820c43a64ba285e33ee2b607789b" candidate_uuid: "91de225bd4e84a4593199b49347b6d29" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b3791458f5554d158fcb770980760849" is_pre_election: true
I20250112 15:08:23.301676 23289 raft_consensus.cc:2463] T 46c5820c43a64ba285e33ee2b607789b P 5b101dbed8f34e9281a33db399f65ed6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 91de225bd4e84a4593199b49347b6d29 in term 0.
I20250112 15:08:23.301684 23364 raft_consensus.cc:2463] T 46c5820c43a64ba285e33ee2b607789b P b3791458f5554d158fcb770980760849 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 91de225bd4e84a4593199b49347b6d29 in term 0.
I20250112 15:08:23.301833 23177 leader_election.cc:304] T 46c5820c43a64ba285e33ee2b607789b P 91de225bd4e84a4593199b49347b6d29 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5b101dbed8f34e9281a33db399f65ed6, 91de225bd4e84a4593199b49347b6d29; no voters:
I20250112 15:08:23.301909 23413 raft_consensus.cc:2798] T 46c5820c43a64ba285e33ee2b607789b P 91de225bd4e84a4593199b49347b6d29 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250112 15:08:23.301947 23413 raft_consensus.cc:491] T 46c5820c43a64ba285e33ee2b607789b P 91de225bd4e84a4593199b49347b6d29 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250112 15:08:23.301966 23413 raft_consensus.cc:3054] T 46c5820c43a64ba285e33ee2b607789b P 91de225bd4e84a4593199b49347b6d29 [term 0 FOLLOWER]: Advancing to term 1
I20250112 15:08:23.302699 23413 raft_consensus.cc:513] T 46c5820c43a64ba285e33ee2b607789b P 91de225bd4e84a4593199b49347b6d29 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91de225bd4e84a4593199b49347b6d29" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 39577 } } peers { permanent_uuid: "5b101dbed8f34e9281a33db399f65ed6" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 36341 } } peers { permanent_uuid: "b3791458f5554d158fcb770980760849" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 42069 } }
I20250112 15:08:23.302809 23413 leader_election.cc:290] T 46c5820c43a64ba285e33ee2b607789b P 91de225bd4e84a4593199b49347b6d29 [CANDIDATE]: Term 1 election: Requested vote from peers 5b101dbed8f34e9281a33db399f65ed6 (127.17.60.194:36341), b3791458f5554d158fcb770980760849 (127.17.60.195:42069)
I20250112 15:08:23.302958 23289 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "46c5820c43a64ba285e33ee2b607789b" candidate_uuid: "91de225bd4e84a4593199b49347b6d29" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b101dbed8f34e9281a33db399f65ed6"
I20250112 15:08:23.302996 23364 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "46c5820c43a64ba285e33ee2b607789b" candidate_uuid: "91de225bd4e84a4593199b49347b6d29" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b3791458f5554d158fcb770980760849"
I20250112 15:08:23.303036 23289 raft_consensus.cc:3054] T 46c5820c43a64ba285e33ee2b607789b P 5b101dbed8f34e9281a33db399f65ed6 [term 0 FOLLOWER]: Advancing to term 1
I20250112 15:08:23.303056 23364 raft_consensus.cc:3054] T 46c5820c43a64ba285e33ee2b607789b P b3791458f5554d158fcb770980760849 [term 0 FOLLOWER]: Advancing to term 1
I20250112 15:08:23.303562 23289 raft_consensus.cc:2463] T 46c5820c43a64ba285e33ee2b607789b P 5b101dbed8f34e9281a33db399f65ed6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 91de225bd4e84a4593199b49347b6d29 in term 1.
I20250112 15:08:23.303568 23364 raft_consensus.cc:2463] T 46c5820c43a64ba285e33ee2b607789b P b3791458f5554d158fcb770980760849 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 91de225bd4e84a4593199b49347b6d29 in term 1.
I20250112 15:08:23.303725 23177 leader_election.cc:304] T 46c5820c43a64ba285e33ee2b607789b P 91de225bd4e84a4593199b49347b6d29 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5b101dbed8f34e9281a33db399f65ed6, 91de225bd4e84a4593199b49347b6d29; no voters:
I20250112 15:08:23.303813 23413 raft_consensus.cc:2798] T 46c5820c43a64ba285e33ee2b607789b P 91de225bd4e84a4593199b49347b6d29 [term 1 FOLLOWER]: Leader election won for term 1
I20250112 15:08:23.303862 23413 raft_consensus.cc:695] T 46c5820c43a64ba285e33ee2b607789b P 91de225bd4e84a4593199b49347b6d29 [term 1 LEADER]: Becoming Leader. State: Replica: 91de225bd4e84a4593199b49347b6d29, State: Running, Role: LEADER
I20250112 15:08:23.303903 23413 consensus_queue.cc:237] T 46c5820c43a64ba285e33ee2b607789b P 91de225bd4e84a4593199b49347b6d29 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91de225bd4e84a4593199b49347b6d29" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 39577 } } peers { permanent_uuid: "5b101dbed8f34e9281a33db399f65ed6" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 36341 } } peers { permanent_uuid: "b3791458f5554d158fcb770980760849" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 42069 } }
I20250112 15:08:23.304474 23110 catalog_manager.cc:5526] T 46c5820c43a64ba285e33ee2b607789b P 91de225bd4e84a4593199b49347b6d29 reported cstate change: term changed from 0 to 1, leader changed from <none> to 91de225bd4e84a4593199b49347b6d29 (127.17.60.193). New cstate: current_term: 1 leader_uuid: "91de225bd4e84a4593199b49347b6d29" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91de225bd4e84a4593199b49347b6d29" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 39577 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5b101dbed8f34e9281a33db399f65ed6" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 36341 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b3791458f5554d158fcb770980760849" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 42069 } health_report { overall_health: UNKNOWN } } }
I20250112 15:08:23.390151 17651 tablet_server.cc:178] TabletServer@127.17.60.193:0 shutting down...
I20250112 15:08:23.392490 17651 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250112 15:08:23.392653 17651 tablet_replica.cc:331] T 656d5437ee6a475a89af5e24b74f315a P 91de225bd4e84a4593199b49347b6d29: stopping tablet replica
I20250112 15:08:23.392753 17651 raft_consensus.cc:2238] T 656d5437ee6a475a89af5e24b74f315a P 91de225bd4e84a4593199b49347b6d29 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250112 15:08:23.392807 17651 raft_consensus.cc:2267] T 656d5437ee6a475a89af5e24b74f315a P 91de225bd4e84a4593199b49347b6d29 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 15:08:23.393016 17651 tablet_replica.cc:331] T a84082c6bef84e638159590ed624062e P 91de225bd4e84a4593199b49347b6d29: stopping tablet replica
I20250112 15:08:23.393056 17651 raft_consensus.cc:2238] T a84082c6bef84e638159590ed624062e P 91de225bd4e84a4593199b49347b6d29 [term 1 LEADER]: Raft consensus shutting down.
I20250112 15:08:23.393105 17651 pending_rounds.cc:62] T a84082c6bef84e638159590ed624062e P 91de225bd4e84a4593199b49347b6d29: Trying to abort 1 pending ops.
I20250112 15:08:23.393146 17651 pending_rounds.cc:69] T a84082c6bef84e638159590ed624062e P 91de225bd4e84a4593199b49347b6d29: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7113500684249206784 op_type: NO_OP noop_request { }
I20250112 15:08:23.393230 17651 raft_consensus.cc:2883] T a84082c6bef84e638159590ed624062e P 91de225bd4e84a4593199b49347b6d29 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20250112 15:08:23.393258 17651 raft_consensus.cc:2267] T a84082c6bef84e638159590ed624062e P 91de225bd4e84a4593199b49347b6d29 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 15:08:23.393512 17651 tablet_replica.cc:331] T d5a633e37cc84a098547fa042708b1d3 P 91de225bd4e84a4593199b49347b6d29: stopping tablet replica
I20250112 15:08:23.393564 17651 raft_consensus.cc:2238] T d5a633e37cc84a098547fa042708b1d3 P 91de225bd4e84a4593199b49347b6d29 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250112 15:08:23.393606 17651 raft_consensus.cc:2267] T d5a633e37cc84a098547fa042708b1d3 P 91de225bd4e84a4593199b49347b6d29 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 15:08:23.393779 17651 tablet_replica.cc:331] T 46c5820c43a64ba285e33ee2b607789b P 91de225bd4e84a4593199b49347b6d29: stopping tablet replica
I20250112 15:08:23.393813 17651 raft_consensus.cc:2238] T 46c5820c43a64ba285e33ee2b607789b P 91de225bd4e84a4593199b49347b6d29 [term 1 LEADER]: Raft consensus shutting down.
I20250112 15:08:23.393860 17651 pending_rounds.cc:62] T 46c5820c43a64ba285e33ee2b607789b P 91de225bd4e84a4593199b49347b6d29: Trying to abort 1 pending ops.
I20250112 15:08:23.393883 17651 pending_rounds.cc:69] T 46c5820c43a64ba285e33ee2b607789b P 91de225bd4e84a4593199b49347b6d29: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7113500685533282304 op_type: NO_OP noop_request { }
I20250112 15:08:23.393929 17651 raft_consensus.cc:2883] T 46c5820c43a64ba285e33ee2b607789b P 91de225bd4e84a4593199b49347b6d29 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20250112 15:08:23.393968 17651 raft_consensus.cc:2267] T 46c5820c43a64ba285e33ee2b607789b P 91de225bd4e84a4593199b49347b6d29 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 15:08:23.405943 17651 tablet_server.cc:195] TabletServer@127.17.60.193:0 shutdown complete.
I20250112 15:08:23.596416 23416 consensus_queue.cc:1035] T d5a633e37cc84a098547fa042708b1d3 P 5b101dbed8f34e9281a33db399f65ed6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b3791458f5554d158fcb770980760849" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 42069 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20250112 15:08:23.598384 23254 consensus_peers.cc:487] T d5a633e37cc84a098547fa042708b1d3 P 5b101dbed8f34e9281a33db399f65ed6 -> Peer 91de225bd4e84a4593199b49347b6d29 (127.17.60.193:39577): Couldn't send request to peer 91de225bd4e84a4593199b49347b6d29. Status: Network error: Client connection negotiation failed: client connection to 127.17.60.193:39577: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20250112 15:08:23.769311 23434 consensus_queue.cc:1035] T 656d5437ee6a475a89af5e24b74f315a P 5b101dbed8f34e9281a33db399f65ed6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b3791458f5554d158fcb770980760849" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 42069 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20250112 15:08:23.770960 23254 consensus_peers.cc:487] T 656d5437ee6a475a89af5e24b74f315a P 5b101dbed8f34e9281a33db399f65ed6 -> Peer 91de225bd4e84a4593199b49347b6d29 (127.17.60.193:39577): Couldn't send request to peer 91de225bd4e84a4593199b49347b6d29. Status: Network error: Client connection negotiation failed: client connection to 127.17.60.193:39577: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20250112 15:08:24.309942 23444 consensus_queue.cc:579] T 656d5437ee6a475a89af5e24b74f315a P 5b101dbed8f34e9281a33db399f65ed6 [LEADER]: Leader has been unable to successfully communicate with peer 91de225bd4e84a4593199b49347b6d29 for more than 1 seconds (1.046s)
W20250112 15:08:24.310724 23109 catalog_manager.cc:5204] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 656d5437ee6a475a89af5e24b74f315a with cas_config_opid_index -1: no extra replica candidate found for tablet 656d5437ee6a475a89af5e24b74f315a (table test-workload [id=2570d90deee448cea24e7125529fb8c9]): Not found: could not select location for extra replica: not enough tablet servers to satisfy replica placement policy: the total number of registered tablet servers (3) does not allow for adding an extra replica; consider bringing up more to have at least 4 tablet servers up and running
I20250112 15:08:24.407490 17651 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250112 15:08:24.409050 23447 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250112 15:08:24.409958 23448 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250112 15:08:24.410715 23450 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250112 15:08:24.411459 17651 server_base.cc:1034] running on GCE node
I20250112 15:08:24.411583 17651 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250112 15:08:24.411613 17651 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250112 15:08:24.411636 17651 hybrid_clock.cc:648] HybridClock initialized: now 1736694504411636 us; error 0 us; skew 500 ppm
I20250112 15:08:24.417004 17651 webserver.cc:458] Webserver started at http://127.17.60.196:42243/ using document root <none> and password file <none>
I20250112 15:08:24.417114 17651 fs_manager.cc:362] Metadata directory not provided
I20250112 15:08:24.417179 17651 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250112 15:08:24.417228 17651 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250112 15:08:24.417507 17651 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task95dLSD/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoRebalancingIfReplicasRecovering.1736694351070047-17651-0/minicluster-data/ts-3-root/instance:
uuid: "036d7a638abc417a9f4360ebfa5061f1"
format_stamp: "Formatted at 2025-01-12 15:08:24 on dist-test-slave-b4nz"
I20250112 15:08:24.418401 17651 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20250112 15:08:24.418860 23455 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250112 15:08:24.419008 17651 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20250112 15:08:24.419056 17651 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task95dLSD/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoRebalancingIfReplicasRecovering.1736694351070047-17651-0/minicluster-data/ts-3-root
uuid: "036d7a638abc417a9f4360ebfa5061f1"
format_stamp: "Formatted at 2025-01-12 15:08:24 on dist-test-slave-b4nz"
I20250112 15:08:24.419101 17651 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task95dLSD/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoRebalancingIfReplicasRecovering.1736694351070047-17651-0/minicluster-data/ts-3-root
metadata directory: /tmp/dist-test-task95dLSD/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoRebalancingIfReplicasRecovering.1736694351070047-17651-0/minicluster-data/ts-3-root
1 data directories: /tmp/dist-test-task95dLSD/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoRebalancingIfReplicasRecovering.1736694351070047-17651-0/minicluster-data/ts-3-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250112 15:08:24.426555 17651 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250112 15:08:24.426748 17651 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250112 15:08:24.426877 17651 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250112 15:08:24.427069 17651 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250112 15:08:24.427098 17651 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250112 15:08:24.427124 17651 ts_tablet_manager.cc:610] Registered 0 tablets
I20250112 15:08:24.427150 17651 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250112 15:08:24.429842 17651 rpc_server.cc:307] RPC server started. Bound to: 127.17.60.196:35337
I20250112 15:08:24.430485 23518 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.17.60.196:35337 every 8 connection(s)
I20250112 15:08:24.433476 23519 heartbeater.cc:346] Connected to a master server at 127.17.60.254:39133
I20250112 15:08:24.433533 23519 heartbeater.cc:463] Registering TS with master...
I20250112 15:08:24.433638 23519 heartbeater.cc:510] Master 127.17.60.254:39133 requested a full tablet report, sending...
I20250112 15:08:24.433862 23109 ts_manager.cc:194] Registered new tserver with Master: 036d7a638abc417a9f4360ebfa5061f1 (127.17.60.196:35337)
I20250112 15:08:24.434223 23109 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:49938
I20250112 15:08:24.500252 23289 consensus_queue.cc:237] T 656d5437ee6a475a89af5e24b74f315a P 5b101dbed8f34e9281a33db399f65ed6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "91de225bd4e84a4593199b49347b6d29" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 39577 } } peers { permanent_uuid: "5b101dbed8f34e9281a33db399f65ed6" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 36341 } } peers { permanent_uuid: "b3791458f5554d158fcb770980760849" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 42069 } } peers { permanent_uuid: "036d7a638abc417a9f4360ebfa5061f1" member_type: NON_VOTER last_known_addr { host: "127.17.60.196" port: 35337 } attrs { promote: true } }
I20250112 15:08:24.500897 23364 raft_consensus.cc:1270] T 656d5437ee6a475a89af5e24b74f315a P b3791458f5554d158fcb770980760849 [term 1 FOLLOWER]: Refusing update from remote peer 5b101dbed8f34e9281a33db399f65ed6: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250112 15:08:24.501158 23445 consensus_queue.cc:1035] T 656d5437ee6a475a89af5e24b74f315a P 5b101dbed8f34e9281a33db399f65ed6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b3791458f5554d158fcb770980760849" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 42069 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20250112 15:08:24.501670 23254 consensus_peers.cc:487] T 656d5437ee6a475a89af5e24b74f315a P 5b101dbed8f34e9281a33db399f65ed6 -> Peer 91de225bd4e84a4593199b49347b6d29 (127.17.60.193:39577): Couldn't send request to peer 91de225bd4e84a4593199b49347b6d29. Status: Network error: Client connection negotiation failed: client connection to 127.17.60.193:39577: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20250112 15:08:24.501825 23444 raft_consensus.cc:2949] T 656d5437ee6a475a89af5e24b74f315a P 5b101dbed8f34e9281a33db399f65ed6 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 036d7a638abc417a9f4360ebfa5061f1 (127.17.60.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "91de225bd4e84a4593199b49347b6d29" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 39577 } } peers { permanent_uuid: "5b101dbed8f34e9281a33db399f65ed6" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 36341 } } peers { permanent_uuid: "b3791458f5554d158fcb770980760849" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 42069 } } peers { permanent_uuid: "036d7a638abc417a9f4360ebfa5061f1" member_type: NON_VOTER last_known_addr { host: "127.17.60.196" port: 35337 } attrs { promote: true } } }
I20250112 15:08:24.501952 23364 raft_consensus.cc:2949] T 656d5437ee6a475a89af5e24b74f315a P b3791458f5554d158fcb770980760849 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 036d7a638abc417a9f4360ebfa5061f1 (127.17.60.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "91de225bd4e84a4593199b49347b6d29" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 39577 } } peers { permanent_uuid: "5b101dbed8f34e9281a33db399f65ed6" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 36341 } } peers { permanent_uuid: "b3791458f5554d158fcb770980760849" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 42069 } } peers { permanent_uuid: "036d7a638abc417a9f4360ebfa5061f1" member_type: NON_VOTER last_known_addr { host: "127.17.60.196" port: 35337 } attrs { promote: true } } }
I20250112 15:08:24.502975 23095 catalog_manager.cc:5039] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 656d5437ee6a475a89af5e24b74f315a with cas_config_opid_index -1: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 4)
I20250112 15:08:24.503086 23109 catalog_manager.cc:5526] T 656d5437ee6a475a89af5e24b74f315a P 5b101dbed8f34e9281a33db399f65ed6 reported cstate change: config changed from index -1 to 2, NON_VOTER 036d7a638abc417a9f4360ebfa5061f1 (127.17.60.196) added. New cstate: current_term: 1 leader_uuid: "5b101dbed8f34e9281a33db399f65ed6" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "91de225bd4e84a4593199b49347b6d29" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 39577 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5b101dbed8f34e9281a33db399f65ed6" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 36341 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b3791458f5554d158fcb770980760849" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 42069 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "036d7a638abc417a9f4360ebfa5061f1" member_type: NON_VOTER last_known_addr { host: "127.17.60.196" port: 35337 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20250112 15:08:24.504433 23254 consensus_peers.cc:487] T 656d5437ee6a475a89af5e24b74f315a P 5b101dbed8f34e9281a33db399f65ed6 -> Peer 036d7a638abc417a9f4360ebfa5061f1 (127.17.60.196:35337): Couldn't send request to peer 036d7a638abc417a9f4360ebfa5061f1. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 656d5437ee6a475a89af5e24b74f315a. This is attempt 1: this message will repeat every 5th retry.
I20250112 15:08:24.590294 23525 raft_consensus.cc:491] T a84082c6bef84e638159590ed624062e P b3791458f5554d158fcb770980760849 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 15:08:24.590421 23525 raft_consensus.cc:513] T a84082c6bef84e638159590ed624062e P b3791458f5554d158fcb770980760849 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b101dbed8f34e9281a33db399f65ed6" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 36341 } } peers { permanent_uuid: "b3791458f5554d158fcb770980760849" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 42069 } } peers { permanent_uuid: "91de225bd4e84a4593199b49347b6d29" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 39577 } }
I20250112 15:08:24.590658 23525 leader_election.cc:290] T a84082c6bef84e638159590ed624062e P b3791458f5554d158fcb770980760849 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 5b101dbed8f34e9281a33db399f65ed6 (127.17.60.194:36341), 91de225bd4e84a4593199b49347b6d29 (127.17.60.193:39577)
W20250112 15:08:24.591475 23329 leader_election.cc:336] T a84082c6bef84e638159590ed624062e P b3791458f5554d158fcb770980760849 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 91de225bd4e84a4593199b49347b6d29 (127.17.60.193:39577): Network error: Client connection negotiation failed: client connection to 127.17.60.193:39577: connect: Connection refused (error 111)
I20250112 15:08:24.593483 23289 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a84082c6bef84e638159590ed624062e" candidate_uuid: "b3791458f5554d158fcb770980760849" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b101dbed8f34e9281a33db399f65ed6" is_pre_election: true
I20250112 15:08:24.593590 23289 raft_consensus.cc:2463] T a84082c6bef84e638159590ed624062e P 5b101dbed8f34e9281a33db399f65ed6 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b3791458f5554d158fcb770980760849 in term 1.
I20250112 15:08:24.593720 23328 leader_election.cc:304] T a84082c6bef84e638159590ed624062e P b3791458f5554d158fcb770980760849 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5b101dbed8f34e9281a33db399f65ed6, b3791458f5554d158fcb770980760849; no voters: 91de225bd4e84a4593199b49347b6d29
I20250112 15:08:24.593839 23525 raft_consensus.cc:2798] T a84082c6bef84e638159590ed624062e P b3791458f5554d158fcb770980760849 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250112 15:08:24.593886 23525 raft_consensus.cc:491] T a84082c6bef84e638159590ed624062e P b3791458f5554d158fcb770980760849 [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250112 15:08:24.593901 23525 raft_consensus.cc:3054] T a84082c6bef84e638159590ed624062e P b3791458f5554d158fcb770980760849 [term 1 FOLLOWER]: Advancing to term 2
I20250112 15:08:24.594587 23525 raft_consensus.cc:513] T a84082c6bef84e638159590ed624062e P b3791458f5554d158fcb770980760849 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b101dbed8f34e9281a33db399f65ed6" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 36341 } } peers { permanent_uuid: "b3791458f5554d158fcb770980760849" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 42069 } } peers { permanent_uuid: "91de225bd4e84a4593199b49347b6d29" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 39577 } }
I20250112 15:08:24.594705 23525 leader_election.cc:290] T a84082c6bef84e638159590ed624062e P b3791458f5554d158fcb770980760849 [CANDIDATE]: Term 2 election: Requested vote from peers 5b101dbed8f34e9281a33db399f65ed6 (127.17.60.194:36341), 91de225bd4e84a4593199b49347b6d29 (127.17.60.193:39577)
I20250112 15:08:24.594846 23289 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a84082c6bef84e638159590ed624062e" candidate_uuid: "b3791458f5554d158fcb770980760849" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b101dbed8f34e9281a33db399f65ed6"
I20250112 15:08:24.594911 23289 raft_consensus.cc:3054] T a84082c6bef84e638159590ed624062e P 5b101dbed8f34e9281a33db399f65ed6 [term 1 FOLLOWER]: Advancing to term 2
W20250112 15:08:24.595116 23329 leader_election.cc:336] T a84082c6bef84e638159590ed624062e P b3791458f5554d158fcb770980760849 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 91de225bd4e84a4593199b49347b6d29 (127.17.60.193:39577): Network error: Client connection negotiation failed: client connection to 127.17.60.193:39577: connect: Connection refused (error 111)
I20250112 15:08:24.595432 23289 raft_consensus.cc:2463] T a84082c6bef84e638159590ed624062e P 5b101dbed8f34e9281a33db399f65ed6 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b3791458f5554d158fcb770980760849 in term 2.
I20250112 15:08:24.595574 23328 leader_election.cc:304] T a84082c6bef84e638159590ed624062e P b3791458f5554d158fcb770980760849 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5b101dbed8f34e9281a33db399f65ed6, b3791458f5554d158fcb770980760849; no voters: 91de225bd4e84a4593199b49347b6d29
I20250112 15:08:24.595664 23525 raft_consensus.cc:2798] T a84082c6bef84e638159590ed624062e P b3791458f5554d158fcb770980760849 [term 2 FOLLOWER]: Leader election won for term 2
I20250112 15:08:24.595794 23525 raft_consensus.cc:695] T a84082c6bef84e638159590ed624062e P b3791458f5554d158fcb770980760849 [term 2 LEADER]: Becoming Leader. State: Replica: b3791458f5554d158fcb770980760849, State: Running, Role: LEADER
I20250112 15:08:24.595845 23525 consensus_queue.cc:237] T a84082c6bef84e638159590ed624062e P b3791458f5554d158fcb770980760849 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b101dbed8f34e9281a33db399f65ed6" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 36341 } } peers { permanent_uuid: "b3791458f5554d158fcb770980760849" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 42069 } } peers { permanent_uuid: "91de225bd4e84a4593199b49347b6d29" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 39577 } }
I20250112 15:08:24.596372 23110 catalog_manager.cc:5526] T a84082c6bef84e638159590ed624062e P b3791458f5554d158fcb770980760849 reported cstate change: term changed from 1 to 2, leader changed from 91de225bd4e84a4593199b49347b6d29 (127.17.60.193) to b3791458f5554d158fcb770980760849 (127.17.60.195). New cstate: current_term: 2 leader_uuid: "b3791458f5554d158fcb770980760849" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b101dbed8f34e9281a33db399f65ed6" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 36341 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b3791458f5554d158fcb770980760849" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 42069 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "91de225bd4e84a4593199b49347b6d29" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 39577 } health_report { overall_health: UNKNOWN } } }
I20250112 15:08:24.732332 23445 consensus_queue.cc:579] T d5a633e37cc84a098547fa042708b1d3 P 5b101dbed8f34e9281a33db399f65ed6 [LEADER]: Leader has been unable to successfully communicate with peer 91de225bd4e84a4593199b49347b6d29 for more than 1 seconds (1.560s)
I20250112 15:08:24.733341 23289 consensus_queue.cc:237] T d5a633e37cc84a098547fa042708b1d3 P 5b101dbed8f34e9281a33db399f65ed6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "91de225bd4e84a4593199b49347b6d29" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 39577 } } peers { permanent_uuid: "5b101dbed8f34e9281a33db399f65ed6" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 36341 } } peers { permanent_uuid: "b3791458f5554d158fcb770980760849" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 42069 } } peers { permanent_uuid: "036d7a638abc417a9f4360ebfa5061f1" member_type: NON_VOTER last_known_addr { host: "127.17.60.196" port: 35337 } attrs { promote: true } }
I20250112 15:08:24.733832 23364 raft_consensus.cc:1270] T d5a633e37cc84a098547fa042708b1d3 P b3791458f5554d158fcb770980760849 [term 1 FOLLOWER]: Refusing update from remote peer 5b101dbed8f34e9281a33db399f65ed6: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20250112 15:08:24.733930 23254 consensus_peers.cc:487] T d5a633e37cc84a098547fa042708b1d3 P 5b101dbed8f34e9281a33db399f65ed6 -> Peer 036d7a638abc417a9f4360ebfa5061f1 (127.17.60.196:35337): Couldn't send request to peer 036d7a638abc417a9f4360ebfa5061f1. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d5a633e37cc84a098547fa042708b1d3. This is attempt 1: this message will repeat every 5th retry.
I20250112 15:08:24.734030 23446 consensus_queue.cc:1035] T d5a633e37cc84a098547fa042708b1d3 P 5b101dbed8f34e9281a33db399f65ed6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b3791458f5554d158fcb770980760849" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 42069 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20250112 15:08:24.734198 23254 consensus_peers.cc:487] T d5a633e37cc84a098547fa042708b1d3 P 5b101dbed8f34e9281a33db399f65ed6 -> Peer 91de225bd4e84a4593199b49347b6d29 (127.17.60.193:39577): Couldn't send request to peer 91de225bd4e84a4593199b49347b6d29. Status: Network error: Client connection negotiation failed: client connection to 127.17.60.193:39577: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20250112 15:08:24.734760 23444 raft_consensus.cc:2949] T d5a633e37cc84a098547fa042708b1d3 P 5b101dbed8f34e9281a33db399f65ed6 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 036d7a638abc417a9f4360ebfa5061f1 (127.17.60.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "91de225bd4e84a4593199b49347b6d29" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 39577 } } peers { permanent_uuid: "5b101dbed8f34e9281a33db399f65ed6" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 36341 } } peers { permanent_uuid: "b3791458f5554d158fcb770980760849" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 42069 } } peers { permanent_uuid: "036d7a638abc417a9f4360ebfa5061f1" member_type: NON_VOTER last_known_addr { host: "127.17.60.196" port: 35337 } attrs { promote: true } } }
I20250112 15:08:24.734930 23364 raft_consensus.cc:2949] T d5a633e37cc84a098547fa042708b1d3 P b3791458f5554d158fcb770980760849 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 036d7a638abc417a9f4360ebfa5061f1 (127.17.60.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "91de225bd4e84a4593199b49347b6d29" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 39577 } } peers { permanent_uuid: "5b101dbed8f34e9281a33db399f65ed6" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 36341 } } peers { permanent_uuid: "b3791458f5554d158fcb770980760849" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 42069 } } peers { permanent_uuid: "036d7a638abc417a9f4360ebfa5061f1" member_type: NON_VOTER last_known_addr { host: "127.17.60.196" port: 35337 } attrs { promote: true } } }
I20250112 15:08:24.736028 23095 catalog_manager.cc:5039] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet d5a633e37cc84a098547fa042708b1d3 with cas_config_opid_index -1: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20250112 15:08:24.736121 23110 catalog_manager.cc:5526] T d5a633e37cc84a098547fa042708b1d3 P b3791458f5554d158fcb770980760849 reported cstate change: config changed from index -1 to 2, NON_VOTER 036d7a638abc417a9f4360ebfa5061f1 (127.17.60.196) added. New cstate: current_term: 1 leader_uuid: "5b101dbed8f34e9281a33db399f65ed6" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "91de225bd4e84a4593199b49347b6d29" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 39577 } } peers { permanent_uuid: "5b101dbed8f34e9281a33db399f65ed6" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 36341 } } peers { permanent_uuid: "b3791458f5554d158fcb770980760849" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 42069 } } peers { permanent_uuid: "036d7a638abc417a9f4360ebfa5061f1" member_type: NON_VOTER last_known_addr { host: "127.17.60.196" port: 35337 } attrs { promote: true } } }
W20250112 15:08:24.805156 23159 auto_rebalancer.cc:227] Could not retrieve cluster info: Not found: tserver 91de225bd4e84a4593199b49347b6d29 not available for placement
I20250112 15:08:24.997016 23446 raft_consensus.cc:491] T 46c5820c43a64ba285e33ee2b607789b P 5b101dbed8f34e9281a33db399f65ed6 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 15:08:24.997145 23446 raft_consensus.cc:513] T 46c5820c43a64ba285e33ee2b607789b P 5b101dbed8f34e9281a33db399f65ed6 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91de225bd4e84a4593199b49347b6d29" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 39577 } } peers { permanent_uuid: "5b101dbed8f34e9281a33db399f65ed6" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 36341 } } peers { permanent_uuid: "b3791458f5554d158fcb770980760849" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 42069 } }
I20250112 15:08:24.997325 23446 leader_election.cc:290] T 46c5820c43a64ba285e33ee2b607789b P 5b101dbed8f34e9281a33db399f65ed6 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 91de225bd4e84a4593199b49347b6d29 (127.17.60.193:39577), b3791458f5554d158fcb770980760849 (127.17.60.195:42069)
I20250112 15:08:24.997501 23364 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "46c5820c43a64ba285e33ee2b607789b" candidate_uuid: "5b101dbed8f34e9281a33db399f65ed6" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b3791458f5554d158fcb770980760849" is_pre_election: true
I20250112 15:08:24.997602 23364 raft_consensus.cc:2463] T 46c5820c43a64ba285e33ee2b607789b P b3791458f5554d158fcb770980760849 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5b101dbed8f34e9281a33db399f65ed6 in term 1.
W20250112 15:08:24.997646 23254 leader_election.cc:336] T 46c5820c43a64ba285e33ee2b607789b P 5b101dbed8f34e9281a33db399f65ed6 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 91de225bd4e84a4593199b49347b6d29 (127.17.60.193:39577): Network error: Client connection negotiation failed: client connection to 127.17.60.193:39577: connect: Connection refused (error 111)
I20250112 15:08:24.997749 23253 leader_election.cc:304] T 46c5820c43a64ba285e33ee2b607789b P 5b101dbed8f34e9281a33db399f65ed6 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5b101dbed8f34e9281a33db399f65ed6, b3791458f5554d158fcb770980760849; no voters: 91de225bd4e84a4593199b49347b6d29
I20250112 15:08:24.997839 23446 raft_consensus.cc:2798] T 46c5820c43a64ba285e33ee2b607789b P 5b101dbed8f34e9281a33db399f65ed6 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250112 15:08:24.997881 23446 raft_consensus.cc:491] T 46c5820c43a64ba285e33ee2b607789b P 5b101dbed8f34e9281a33db399f65ed6 [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250112 15:08:24.997895 23446 raft_consensus.cc:3054] T 46c5820c43a64ba285e33ee2b607789b P 5b101dbed8f34e9281a33db399f65ed6 [term 1 FOLLOWER]: Advancing to term 2
I20250112 15:08:24.998567 23446 raft_consensus.cc:513] T 46c5820c43a64ba285e33ee2b607789b P 5b101dbed8f34e9281a33db399f65ed6 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91de225bd4e84a4593199b49347b6d29" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 39577 } } peers { permanent_uuid: "5b101dbed8f34e9281a33db399f65ed6" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 36341 } } peers { permanent_uuid: "b3791458f5554d158fcb770980760849" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 42069 } }
I20250112 15:08:24.998682 23446 leader_election.cc:290] T 46c5820c43a64ba285e33ee2b607789b P 5b101dbed8f34e9281a33db399f65ed6 [CANDIDATE]: Term 2 election: Requested vote from peers 91de225bd4e84a4593199b49347b6d29 (127.17.60.193:39577), b3791458f5554d158fcb770980760849 (127.17.60.195:42069)
I20250112 15:08:24.998824 23364 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "46c5820c43a64ba285e33ee2b607789b" candidate_uuid: "5b101dbed8f34e9281a33db399f65ed6" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b3791458f5554d158fcb770980760849"
I20250112 15:08:24.998885 23364 raft_consensus.cc:3054] T 46c5820c43a64ba285e33ee2b607789b P b3791458f5554d158fcb770980760849 [term 1 FOLLOWER]: Advancing to term 2
W20250112 15:08:24.999212 23254 leader_election.cc:336] T 46c5820c43a64ba285e33ee2b607789b P 5b101dbed8f34e9281a33db399f65ed6 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 91de225bd4e84a4593199b49347b6d29 (127.17.60.193:39577): Network error: Client connection negotiation failed: client connection to 127.17.60.193:39577: connect: Connection refused (error 111)
I20250112 15:08:24.999574 23364 raft_consensus.cc:2463] T 46c5820c43a64ba285e33ee2b607789b P b3791458f5554d158fcb770980760849 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5b101dbed8f34e9281a33db399f65ed6 in term 2.
I20250112 15:08:24.999734 23253 leader_election.cc:304] T 46c5820c43a64ba285e33ee2b607789b P 5b101dbed8f34e9281a33db399f65ed6 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5b101dbed8f34e9281a33db399f65ed6, b3791458f5554d158fcb770980760849; no voters: 91de225bd4e84a4593199b49347b6d29
I20250112 15:08:24.999821 23446 raft_consensus.cc:2798] T 46c5820c43a64ba285e33ee2b607789b P 5b101dbed8f34e9281a33db399f65ed6 [term 2 FOLLOWER]: Leader election won for term 2
I20250112 15:08:24.999876 23446 raft_consensus.cc:695] T 46c5820c43a64ba285e33ee2b607789b P 5b101dbed8f34e9281a33db399f65ed6 [term 2 LEADER]: Becoming Leader. State: Replica: 5b101dbed8f34e9281a33db399f65ed6, State: Running, Role: LEADER
I20250112 15:08:24.999923 23446 consensus_queue.cc:237] T 46c5820c43a64ba285e33ee2b607789b P 5b101dbed8f34e9281a33db399f65ed6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91de225bd4e84a4593199b49347b6d29" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 39577 } } peers { permanent_uuid: "5b101dbed8f34e9281a33db399f65ed6" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 36341 } } peers { permanent_uuid: "b3791458f5554d158fcb770980760849" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 42069 } }
I20250112 15:08:25.000454 23109 catalog_manager.cc:5526] T 46c5820c43a64ba285e33ee2b607789b P 5b101dbed8f34e9281a33db399f65ed6 reported cstate change: term changed from 1 to 2, leader changed from 91de225bd4e84a4593199b49347b6d29 (127.17.60.193) to 5b101dbed8f34e9281a33db399f65ed6 (127.17.60.194). New cstate: current_term: 2 leader_uuid: "5b101dbed8f34e9281a33db399f65ed6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91de225bd4e84a4593199b49347b6d29" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 39577 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5b101dbed8f34e9281a33db399f65ed6" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 36341 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b3791458f5554d158fcb770980760849" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 42069 } health_report { overall_health: UNKNOWN } } }
I20250112 15:08:25.044837 23541 ts_tablet_manager.cc:927] T 656d5437ee6a475a89af5e24b74f315a P 036d7a638abc417a9f4360ebfa5061f1: Initiating tablet copy from peer 5b101dbed8f34e9281a33db399f65ed6 (127.17.60.194:36341)
I20250112 15:08:25.045027 23541 tablet_copy_client.cc:323] T 656d5437ee6a475a89af5e24b74f315a P 036d7a638abc417a9f4360ebfa5061f1: tablet copy: Beginning tablet copy session from remote peer at address 127.17.60.194:36341
I20250112 15:08:25.047627 23299 tablet_copy_service.cc:140] P 5b101dbed8f34e9281a33db399f65ed6: Received BeginTabletCopySession request for tablet 656d5437ee6a475a89af5e24b74f315a from peer 036d7a638abc417a9f4360ebfa5061f1 ({username='slave'} at 127.0.0.1:39172)
I20250112 15:08:25.047695 23299 tablet_copy_service.cc:161] P 5b101dbed8f34e9281a33db399f65ed6: Beginning new tablet copy session on tablet 656d5437ee6a475a89af5e24b74f315a from peer 036d7a638abc417a9f4360ebfa5061f1 at {username='slave'} at 127.0.0.1:39172: session id = 036d7a638abc417a9f4360ebfa5061f1-656d5437ee6a475a89af5e24b74f315a
I20250112 15:08:25.048192 23299 tablet_copy_source_session.cc:215] T 656d5437ee6a475a89af5e24b74f315a P 5b101dbed8f34e9281a33db399f65ed6: Tablet Copy: opened 0 blocks and 1 log segments
I20250112 15:08:25.048514 23541 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 656d5437ee6a475a89af5e24b74f315a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 15:08:25.049671 23541 tablet_copy_client.cc:806] T 656d5437ee6a475a89af5e24b74f315a P 036d7a638abc417a9f4360ebfa5061f1: tablet copy: Starting download of 0 data blocks...
I20250112 15:08:25.049787 23541 tablet_copy_client.cc:670] T 656d5437ee6a475a89af5e24b74f315a P 036d7a638abc417a9f4360ebfa5061f1: tablet copy: Starting download of 1 WAL segments...
I20250112 15:08:25.050347 23541 tablet_copy_client.cc:538] T 656d5437ee6a475a89af5e24b74f315a P 036d7a638abc417a9f4360ebfa5061f1: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250112 15:08:25.050906 23541 tablet_bootstrap.cc:492] T 656d5437ee6a475a89af5e24b74f315a P 036d7a638abc417a9f4360ebfa5061f1: Bootstrap starting.
I20250112 15:08:25.051877 23541 tablet_bootstrap.cc:492] T 656d5437ee6a475a89af5e24b74f315a P 036d7a638abc417a9f4360ebfa5061f1: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250112 15:08:25.052004 23541 tablet_bootstrap.cc:492] T 656d5437ee6a475a89af5e24b74f315a P 036d7a638abc417a9f4360ebfa5061f1: Bootstrap complete.
I20250112 15:08:25.052068 23541 ts_tablet_manager.cc:1397] T 656d5437ee6a475a89af5e24b74f315a P 036d7a638abc417a9f4360ebfa5061f1: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20250112 15:08:25.052171 23541 raft_consensus.cc:357] T 656d5437ee6a475a89af5e24b74f315a P 036d7a638abc417a9f4360ebfa5061f1 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "91de225bd4e84a4593199b49347b6d29" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 39577 } } peers { permanent_uuid: "5b101dbed8f34e9281a33db399f65ed6" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 36341 } } peers { permanent_uuid: "b3791458f5554d158fcb770980760849" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 42069 } } peers { permanent_uuid: "036d7a638abc417a9f4360ebfa5061f1" member_type: NON_VOTER last_known_addr { host: "127.17.60.196" port: 35337 } attrs { promote: true } }
I20250112 15:08:25.052227 23541 raft_consensus.cc:738] T 656d5437ee6a475a89af5e24b74f315a P 036d7a638abc417a9f4360ebfa5061f1 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 036d7a638abc417a9f4360ebfa5061f1, State: Initialized, Role: LEARNER
I20250112 15:08:25.052248 23541 consensus_queue.cc:260] T 656d5437ee6a475a89af5e24b74f315a P 036d7a638abc417a9f4360ebfa5061f1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "91de225bd4e84a4593199b49347b6d29" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 39577 } } peers { permanent_uuid: "5b101dbed8f34e9281a33db399f65ed6" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 36341 } } peers { permanent_uuid: "b3791458f5554d158fcb770980760849" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 42069 } } peers { permanent_uuid: "036d7a638abc417a9f4360ebfa5061f1" member_type: NON_VOTER last_known_addr { host: "127.17.60.196" port: 35337 } attrs { promote: true } }
I20250112 15:08:25.052376 23541 ts_tablet_manager.cc:1428] T 656d5437ee6a475a89af5e24b74f315a P 036d7a638abc417a9f4360ebfa5061f1: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20250112 15:08:25.052431 23519 heartbeater.cc:502] Master 127.17.60.254:39133 was elected leader, sending a full tablet report...
I20250112 15:08:25.052772 23299 tablet_copy_service.cc:342] P 5b101dbed8f34e9281a33db399f65ed6: Request end of tablet copy session 036d7a638abc417a9f4360ebfa5061f1-656d5437ee6a475a89af5e24b74f315a received from {username='slave'} at 127.0.0.1:39172
I20250112 15:08:25.052819 23299 tablet_copy_service.cc:434] P 5b101dbed8f34e9281a33db399f65ed6: ending tablet copy session 036d7a638abc417a9f4360ebfa5061f1-656d5437ee6a475a89af5e24b74f315a on tablet 656d5437ee6a475a89af5e24b74f315a with peer 036d7a638abc417a9f4360ebfa5061f1
W20250112 15:08:25.119964 23329 consensus_peers.cc:487] T a84082c6bef84e638159590ed624062e P b3791458f5554d158fcb770980760849 -> Peer 91de225bd4e84a4593199b49347b6d29 (127.17.60.193:39577): Couldn't send request to peer 91de225bd4e84a4593199b49347b6d29. Status: Network error: Client connection negotiation failed: client connection to 127.17.60.193:39577: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20250112 15:08:25.128759 23289 raft_consensus.cc:1270] T a84082c6bef84e638159590ed624062e P 5b101dbed8f34e9281a33db399f65ed6 [term 2 FOLLOWER]: Refusing update from remote peer b3791458f5554d158fcb770980760849: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 2 index: 1. (index mismatch)
I20250112 15:08:25.129017 23525 consensus_queue.cc:1035] T a84082c6bef84e638159590ed624062e P b3791458f5554d158fcb770980760849 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5b101dbed8f34e9281a33db399f65ed6" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 36341 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250112 15:08:25.176486 23541 ts_tablet_manager.cc:927] T d5a633e37cc84a098547fa042708b1d3 P 036d7a638abc417a9f4360ebfa5061f1: Initiating tablet copy from peer 5b101dbed8f34e9281a33db399f65ed6 (127.17.60.194:36341)
I20250112 15:08:25.176738 23541 tablet_copy_client.cc:323] T d5a633e37cc84a098547fa042708b1d3 P 036d7a638abc417a9f4360ebfa5061f1: tablet copy: Beginning tablet copy session from remote peer at address 127.17.60.194:36341
I20250112 15:08:25.177002 23299 tablet_copy_service.cc:140] P 5b101dbed8f34e9281a33db399f65ed6: Received BeginTabletCopySession request for tablet d5a633e37cc84a098547fa042708b1d3 from peer 036d7a638abc417a9f4360ebfa5061f1 ({username='slave'} at 127.0.0.1:39172)
I20250112 15:08:25.177076 23299 tablet_copy_service.cc:161] P 5b101dbed8f34e9281a33db399f65ed6: Beginning new tablet copy session on tablet d5a633e37cc84a098547fa042708b1d3 from peer 036d7a638abc417a9f4360ebfa5061f1 at {username='slave'} at 127.0.0.1:39172: session id = 036d7a638abc417a9f4360ebfa5061f1-d5a633e37cc84a098547fa042708b1d3
I20250112 15:08:25.177525 23299 tablet_copy_source_session.cc:215] T d5a633e37cc84a098547fa042708b1d3 P 5b101dbed8f34e9281a33db399f65ed6: Tablet Copy: opened 0 blocks and 1 log segments
I20250112 15:08:25.177830 23541 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d5a633e37cc84a098547fa042708b1d3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 15:08:25.178742 23541 tablet_copy_client.cc:806] T d5a633e37cc84a098547fa042708b1d3 P 036d7a638abc417a9f4360ebfa5061f1: tablet copy: Starting download of 0 data blocks...
I20250112 15:08:25.178862 23541 tablet_copy_client.cc:670] T d5a633e37cc84a098547fa042708b1d3 P 036d7a638abc417a9f4360ebfa5061f1: tablet copy: Starting download of 1 WAL segments...
I20250112 15:08:25.179481 23541 tablet_copy_client.cc:538] T d5a633e37cc84a098547fa042708b1d3 P 036d7a638abc417a9f4360ebfa5061f1: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250112 15:08:25.180117 23541 tablet_bootstrap.cc:492] T d5a633e37cc84a098547fa042708b1d3 P 036d7a638abc417a9f4360ebfa5061f1: Bootstrap starting.
I20250112 15:08:25.181203 23541 tablet_bootstrap.cc:492] T d5a633e37cc84a098547fa042708b1d3 P 036d7a638abc417a9f4360ebfa5061f1: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250112 15:08:25.181365 23541 tablet_bootstrap.cc:492] T d5a633e37cc84a098547fa042708b1d3 P 036d7a638abc417a9f4360ebfa5061f1: Bootstrap complete.
I20250112 15:08:25.181459 23541 ts_tablet_manager.cc:1397] T d5a633e37cc84a098547fa042708b1d3 P 036d7a638abc417a9f4360ebfa5061f1: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20250112 15:08:25.181586 23541 raft_consensus.cc:357] T d5a633e37cc84a098547fa042708b1d3 P 036d7a638abc417a9f4360ebfa5061f1 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "91de225bd4e84a4593199b49347b6d29" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 39577 } } peers { permanent_uuid: "5b101dbed8f34e9281a33db399f65ed6" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 36341 } } peers { permanent_uuid: "b3791458f5554d158fcb770980760849" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 42069 } } peers { permanent_uuid: "036d7a638abc417a9f4360ebfa5061f1" member_type: NON_VOTER last_known_addr { host: "127.17.60.196" port: 35337 } attrs { promote: true } }
I20250112 15:08:25.181674 23541 raft_consensus.cc:738] T d5a633e37cc84a098547fa042708b1d3 P 036d7a638abc417a9f4360ebfa5061f1 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 036d7a638abc417a9f4360ebfa5061f1, State: Initialized, Role: LEARNER
I20250112 15:08:25.181725 23541 consensus_queue.cc:260] T d5a633e37cc84a098547fa042708b1d3 P 036d7a638abc417a9f4360ebfa5061f1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "91de225bd4e84a4593199b49347b6d29" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 39577 } } peers { permanent_uuid: "5b101dbed8f34e9281a33db399f65ed6" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 36341 } } peers { permanent_uuid: "b3791458f5554d158fcb770980760849" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 42069 } } peers { permanent_uuid: "036d7a638abc417a9f4360ebfa5061f1" member_type: NON_VOTER last_known_addr { host: "127.17.60.196" port: 35337 } attrs { promote: true } }
I20250112 15:08:25.181835 23541 ts_tablet_manager.cc:1428] T d5a633e37cc84a098547fa042708b1d3 P 036d7a638abc417a9f4360ebfa5061f1: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20250112 15:08:25.182041 23299 tablet_copy_service.cc:342] P 5b101dbed8f34e9281a33db399f65ed6: Request end of tablet copy session 036d7a638abc417a9f4360ebfa5061f1-d5a633e37cc84a098547fa042708b1d3 received from {username='slave'} at 127.0.0.1:39172
I20250112 15:08:25.182091 23299 tablet_copy_service.cc:434] P 5b101dbed8f34e9281a33db399f65ed6: ending tablet copy session 036d7a638abc417a9f4360ebfa5061f1-d5a633e37cc84a098547fa042708b1d3 on tablet d5a633e37cc84a098547fa042708b1d3 with peer 036d7a638abc417a9f4360ebfa5061f1
I20250112 15:08:25.455168 23364 raft_consensus.cc:1270] T 46c5820c43a64ba285e33ee2b607789b P b3791458f5554d158fcb770980760849 [term 2 FOLLOWER]: Refusing update from remote peer 5b101dbed8f34e9281a33db399f65ed6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 2 index: 1. (index mismatch)
I20250112 15:08:25.455523 23446 consensus_queue.cc:1035] T 46c5820c43a64ba285e33ee2b607789b P 5b101dbed8f34e9281a33db399f65ed6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b3791458f5554d158fcb770980760849" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 42069 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20250112 15:08:25.457139 23254 consensus_peers.cc:487] T 46c5820c43a64ba285e33ee2b607789b P 5b101dbed8f34e9281a33db399f65ed6 -> Peer 91de225bd4e84a4593199b49347b6d29 (127.17.60.193:39577): Couldn't send request to peer 91de225bd4e84a4593199b49347b6d29. Status: Network error: Client connection negotiation failed: client connection to 127.17.60.193:39577: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20250112 15:08:25.459100 23493 raft_consensus.cc:1212] T 656d5437ee6a475a89af5e24b74f315a P 036d7a638abc417a9f4360ebfa5061f1 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20250112 15:08:25.547273 23446 consensus_queue.cc:579] T 656d5437ee6a475a89af5e24b74f315a P 5b101dbed8f34e9281a33db399f65ed6 [LEADER]: Leader has been unable to successfully communicate with peer 91de225bd4e84a4593199b49347b6d29 for more than 1 seconds (1.047s)
I20250112 15:08:25.714998 23493 raft_consensus.cc:1212] T d5a633e37cc84a098547fa042708b1d3 P 036d7a638abc417a9f4360ebfa5061f1 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
W20250112 15:08:25.805414 23159 auto_rebalancer.cc:227] Could not retrieve cluster info: Not found: tserver 91de225bd4e84a4593199b49347b6d29 not available for placement
I20250112 15:08:25.834210 23446 consensus_queue.cc:579] T d5a633e37cc84a098547fa042708b1d3 P 5b101dbed8f34e9281a33db399f65ed6 [LEADER]: Leader has been unable to successfully communicate with peer 91de225bd4e84a4593199b49347b6d29 for more than 1 seconds (1.101s)
I20250112 15:08:26.011319 23538 raft_consensus.cc:1059] T 656d5437ee6a475a89af5e24b74f315a P 5b101dbed8f34e9281a33db399f65ed6: attempting to promote NON_VOTER 036d7a638abc417a9f4360ebfa5061f1 to VOTER
I20250112 15:08:26.011523 23538 consensus_queue.cc:237] T 656d5437ee6a475a89af5e24b74f315a P 5b101dbed8f34e9281a33db399f65ed6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "91de225bd4e84a4593199b49347b6d29" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 39577 } } peers { permanent_uuid: "5b101dbed8f34e9281a33db399f65ed6" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 36341 } } peers { permanent_uuid: "b3791458f5554d158fcb770980760849" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 42069 } } peers { permanent_uuid: "036d7a638abc417a9f4360ebfa5061f1" member_type: VOTER last_known_addr { host: "127.17.60.196" port: 35337 } attrs { promote: false } }
I20250112 15:08:26.011976 23493 raft_consensus.cc:1270] T 656d5437ee6a475a89af5e24b74f315a P 036d7a638abc417a9f4360ebfa5061f1 [term 1 LEARNER]: Refusing update from remote peer 5b101dbed8f34e9281a33db399f65ed6: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250112 15:08:26.012133 23364 raft_consensus.cc:1270] T 656d5437ee6a475a89af5e24b74f315a P b3791458f5554d158fcb770980760849 [term 1 FOLLOWER]: Refusing update from remote peer 5b101dbed8f34e9281a33db399f65ed6: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250112 15:08:26.012156 23555 consensus_queue.cc:1035] T 656d5437ee6a475a89af5e24b74f315a P 5b101dbed8f34e9281a33db399f65ed6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "036d7a638abc417a9f4360ebfa5061f1" member_type: VOTER last_known_addr { host: "127.17.60.196" port: 35337 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250112 15:08:26.012332 23555 consensus_queue.cc:1035] T 656d5437ee6a475a89af5e24b74f315a P 5b101dbed8f34e9281a33db399f65ed6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b3791458f5554d158fcb770980760849" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 42069 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
W20250112 15:08:26.012430 23254 consensus_peers.cc:487] T 656d5437ee6a475a89af5e24b74f315a P 5b101dbed8f34e9281a33db399f65ed6 -> Peer 91de225bd4e84a4593199b49347b6d29 (127.17.60.193:39577): Couldn't send request to peer 91de225bd4e84a4593199b49347b6d29. Status: Network error: Client connection negotiation failed: client connection to 127.17.60.193:39577: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20250112 15:08:26.013167 23557 raft_consensus.cc:2949] T 656d5437ee6a475a89af5e24b74f315a P 5b101dbed8f34e9281a33db399f65ed6 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 036d7a638abc417a9f4360ebfa5061f1 (127.17.60.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "91de225bd4e84a4593199b49347b6d29" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 39577 } } peers { permanent_uuid: "5b101dbed8f34e9281a33db399f65ed6" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 36341 } } peers { permanent_uuid: "b3791458f5554d158fcb770980760849" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 42069 } } peers { permanent_uuid: "036d7a638abc417a9f4360ebfa5061f1" member_type: VOTER last_known_addr { host: "127.17.60.196" port: 35337 } attrs { promote: false } } }
I20250112 15:08:26.013288 23364 raft_consensus.cc:2949] T 656d5437ee6a475a89af5e24b74f315a P b3791458f5554d158fcb770980760849 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 036d7a638abc417a9f4360ebfa5061f1 (127.17.60.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "91de225bd4e84a4593199b49347b6d29" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 39577 } } peers { permanent_uuid: "5b101dbed8f34e9281a33db399f65ed6" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 36341 } } peers { permanent_uuid: "b3791458f5554d158fcb770980760849" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 42069 } } peers { permanent_uuid: "036d7a638abc417a9f4360ebfa5061f1" member_type: VOTER last_known_addr { host: "127.17.60.196" port: 35337 } attrs { promote: false } } }
I20250112 15:08:26.014459 23109 catalog_manager.cc:5526] T 656d5437ee6a475a89af5e24b74f315a P b3791458f5554d158fcb770980760849 reported cstate change: config changed from index 2 to 3, 036d7a638abc417a9f4360ebfa5061f1 (127.17.60.196) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "5b101dbed8f34e9281a33db399f65ed6" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "91de225bd4e84a4593199b49347b6d29" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 39577 } } peers { permanent_uuid: "5b101dbed8f34e9281a33db399f65ed6" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 36341 } } peers { permanent_uuid: "b3791458f5554d158fcb770980760849" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 42069 } } peers { permanent_uuid: "036d7a638abc417a9f4360ebfa5061f1" member_type: VOTER last_known_addr { host: "127.17.60.196" port: 35337 } attrs { promote: false } } }
I20250112 15:08:26.014433 23493 raft_consensus.cc:2949] T 656d5437ee6a475a89af5e24b74f315a P 036d7a638abc417a9f4360ebfa5061f1 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 036d7a638abc417a9f4360ebfa5061f1 (127.17.60.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "91de225bd4e84a4593199b49347b6d29" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 39577 } } peers { permanent_uuid: "5b101dbed8f34e9281a33db399f65ed6" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 36341 } } peers { permanent_uuid: "b3791458f5554d158fcb770980760849" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 42069 } } peers { permanent_uuid: "036d7a638abc417a9f4360ebfa5061f1" member_type: VOTER last_known_addr { host: "127.17.60.196" port: 35337 } attrs { promote: false } } }
I20250112 15:08:26.015786 23289 consensus_queue.cc:237] T 656d5437ee6a475a89af5e24b74f315a P 5b101dbed8f34e9281a33db399f65ed6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5b101dbed8f34e9281a33db399f65ed6" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 36341 } } peers { permanent_uuid: "b3791458f5554d158fcb770980760849" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 42069 } } peers { permanent_uuid: "036d7a638abc417a9f4360ebfa5061f1" member_type: VOTER last_known_addr { host: "127.17.60.196" port: 35337 } attrs { promote: false } }
I20250112 15:08:26.016129 23493 raft_consensus.cc:1270] T 656d5437ee6a475a89af5e24b74f315a P 036d7a638abc417a9f4360ebfa5061f1 [term 1 FOLLOWER]: Refusing update from remote peer 5b101dbed8f34e9281a33db399f65ed6: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20250112 15:08:26.016160 23364 raft_consensus.cc:1270] T 656d5437ee6a475a89af5e24b74f315a P b3791458f5554d158fcb770980760849 [term 1 FOLLOWER]: Refusing update from remote peer 5b101dbed8f34e9281a33db399f65ed6: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20250112 15:08:26.016309 23446 consensus_queue.cc:1035] T 656d5437ee6a475a89af5e24b74f315a P 5b101dbed8f34e9281a33db399f65ed6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "036d7a638abc417a9f4360ebfa5061f1" member_type: VOTER last_known_addr { host: "127.17.60.196" port: 35337 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250112 15:08:26.016372 23555 consensus_queue.cc:1035] T 656d5437ee6a475a89af5e24b74f315a P 5b101dbed8f34e9281a33db399f65ed6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b3791458f5554d158fcb770980760849" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 42069 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250112 15:08:26.016927 23538 raft_consensus.cc:2949] T 656d5437ee6a475a89af5e24b74f315a P 5b101dbed8f34e9281a33db399f65ed6 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 91de225bd4e84a4593199b49347b6d29 (127.17.60.193) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5b101dbed8f34e9281a33db399f65ed6" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 36341 } } peers { permanent_uuid: "b3791458f5554d158fcb770980760849" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 42069 } } peers { permanent_uuid: "036d7a638abc417a9f4360ebfa5061f1" member_type: VOTER last_known_addr { host: "127.17.60.196" port: 35337 } attrs { promote: false } } }
I20250112 15:08:26.017095 23364 raft_consensus.cc:2949] T 656d5437ee6a475a89af5e24b74f315a P b3791458f5554d158fcb770980760849 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 91de225bd4e84a4593199b49347b6d29 (127.17.60.193) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5b101dbed8f34e9281a33db399f65ed6" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 36341 } } peers { permanent_uuid: "b3791458f5554d158fcb770980760849" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 42069 } } peers { permanent_uuid: "036d7a638abc417a9f4360ebfa5061f1" member_type: VOTER last_known_addr { host: "127.17.60.196" port: 35337 } attrs { promote: false } } }
I20250112 15:08:26.017099 23493 raft_consensus.cc:2949] T 656d5437ee6a475a89af5e24b74f315a P 036d7a638abc417a9f4360ebfa5061f1 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 91de225bd4e84a4593199b49347b6d29 (127.17.60.193) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5b101dbed8f34e9281a33db399f65ed6" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 36341 } } peers { permanent_uuid: "b3791458f5554d158fcb770980760849" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 42069 } } peers { permanent_uuid: "036d7a638abc417a9f4360ebfa5061f1" member_type: VOTER last_known_addr { host: "127.17.60.196" port: 35337 } attrs { promote: false } } }
I20250112 15:08:26.017954 23095 catalog_manager.cc:5039] ChangeConfig:REMOVE_PEER RPC for tablet 656d5437ee6a475a89af5e24b74f315a with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250112 15:08:26.018110 23108 catalog_manager.cc:5526] T 656d5437ee6a475a89af5e24b74f315a P 5b101dbed8f34e9281a33db399f65ed6 reported cstate change: config changed from index 3 to 4, VOTER 91de225bd4e84a4593199b49347b6d29 (127.17.60.193) evicted. New cstate: current_term: 1 leader_uuid: "5b101dbed8f34e9281a33db399f65ed6" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5b101dbed8f34e9281a33db399f65ed6" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 36341 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b3791458f5554d158fcb770980760849" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 42069 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "036d7a638abc417a9f4360ebfa5061f1" member_type: VOTER last_known_addr { host: "127.17.60.196" port: 35337 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
W20250112 15:08:26.019214 23096 catalog_manager.cc:4615] TS 91de225bd4e84a4593199b49347b6d29 (127.17.60.193:39577): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 656d5437ee6a475a89af5e24b74f315a: Network error: Client connection negotiation failed: client connection to 127.17.60.193:39577: connect: Connection refused (error 111)
I20250112 15:08:26.183151 23549 consensus_queue.cc:579] T a84082c6bef84e638159590ed624062e P b3791458f5554d158fcb770980760849 [LEADER]: Leader has been unable to successfully communicate with peer 91de225bd4e84a4593199b49347b6d29 for more than 1 seconds (1.587s)
I20250112 15:08:26.184223 23364 consensus_queue.cc:237] T a84082c6bef84e638159590ed624062e P b3791458f5554d158fcb770980760849 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.1, Last appended by leader: 0, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5b101dbed8f34e9281a33db399f65ed6" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 36341 } } peers { permanent_uuid: "b3791458f5554d158fcb770980760849" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 42069 } } peers { permanent_uuid: "91de225bd4e84a4593199b49347b6d29" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 39577 } } peers { permanent_uuid: "036d7a638abc417a9f4360ebfa5061f1" member_type: NON_VOTER last_known_addr { host: "127.17.60.196" port: 35337 } attrs { promote: true } }
I20250112 15:08:26.184772 23289 raft_consensus.cc:1270] T a84082c6bef84e638159590ed624062e P 5b101dbed8f34e9281a33db399f65ed6 [term 2 FOLLOWER]: Refusing update from remote peer b3791458f5554d158fcb770980760849: Log matching property violated. Preceding OpId in replica: term: 2 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250112 15:08:26.184985 23570 consensus_queue.cc:1035] T a84082c6bef84e638159590ed624062e P b3791458f5554d158fcb770980760849 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5b101dbed8f34e9281a33db399f65ed6" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 36341 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20250112 15:08:26.185432 23329 consensus_peers.cc:487] T a84082c6bef84e638159590ed624062e P b3791458f5554d158fcb770980760849 -> Peer 91de225bd4e84a4593199b49347b6d29 (127.17.60.193:39577): Couldn't send request to peer 91de225bd4e84a4593199b49347b6d29. Status: Network error: Client connection negotiation failed: client connection to 127.17.60.193:39577: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20250112 15:08:26.185823 23566 raft_consensus.cc:2949] T a84082c6bef84e638159590ed624062e P b3791458f5554d158fcb770980760849 [term 2 LEADER]: Committing config change with OpId 2.2: config changed from index -1 to 2, NON_VOTER 036d7a638abc417a9f4360ebfa5061f1 (127.17.60.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5b101dbed8f34e9281a33db399f65ed6" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 36341 } } peers { permanent_uuid: "b3791458f5554d158fcb770980760849" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 42069 } } peers { permanent_uuid: "91de225bd4e84a4593199b49347b6d29" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 39577 } } peers { permanent_uuid: "036d7a638abc417a9f4360ebfa5061f1" member_type: NON_VOTER last_known_addr { host: "127.17.60.196" port: 35337 } attrs { promote: true } } }
I20250112 15:08:26.185927 23289 raft_consensus.cc:2949] T a84082c6bef84e638159590ed624062e P 5b101dbed8f34e9281a33db399f65ed6 [term 2 FOLLOWER]: Committing config change with OpId 2.2: config changed from index -1 to 2, NON_VOTER 036d7a638abc417a9f4360ebfa5061f1 (127.17.60.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5b101dbed8f34e9281a33db399f65ed6" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 36341 } } peers { permanent_uuid: "b3791458f5554d158fcb770980760849" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 42069 } } peers { permanent_uuid: "91de225bd4e84a4593199b49347b6d29" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 39577 } } peers { permanent_uuid: "036d7a638abc417a9f4360ebfa5061f1" member_type: NON_VOTER last_known_addr { host: "127.17.60.196" port: 35337 } attrs { promote: true } } }
I20250112 15:08:26.186976 23095 catalog_manager.cc:5039] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet a84082c6bef84e638159590ed624062e with cas_config_opid_index -1: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20250112 15:08:26.187047 23110 catalog_manager.cc:5526] T a84082c6bef84e638159590ed624062e P 5b101dbed8f34e9281a33db399f65ed6 reported cstate change: config changed from index -1 to 2, NON_VOTER 036d7a638abc417a9f4360ebfa5061f1 (127.17.60.196) added. New cstate: current_term: 2 leader_uuid: "b3791458f5554d158fcb770980760849" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5b101dbed8f34e9281a33db399f65ed6" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 36341 } } peers { permanent_uuid: "b3791458f5554d158fcb770980760849" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 42069 } } peers { permanent_uuid: "91de225bd4e84a4593199b49347b6d29" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 39577 } } peers { permanent_uuid: "036d7a638abc417a9f4360ebfa5061f1" member_type: NON_VOTER last_known_addr { host: "127.17.60.196" port: 35337 } attrs { promote: true } } }
W20250112 15:08:26.187881 23329 consensus_peers.cc:487] T a84082c6bef84e638159590ed624062e P b3791458f5554d158fcb770980760849 -> Peer 036d7a638abc417a9f4360ebfa5061f1 (127.17.60.196:35337): Couldn't send request to peer 036d7a638abc417a9f4360ebfa5061f1. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: a84082c6bef84e638159590ed624062e. This is attempt 1: this message will repeat every 5th retry.
I20250112 15:08:26.298779 23557 raft_consensus.cc:1059] T d5a633e37cc84a098547fa042708b1d3 P 5b101dbed8f34e9281a33db399f65ed6: attempting to promote NON_VOTER 036d7a638abc417a9f4360ebfa5061f1 to VOTER
I20250112 15:08:26.298991 23557 consensus_queue.cc:237] T d5a633e37cc84a098547fa042708b1d3 P 5b101dbed8f34e9281a33db399f65ed6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "91de225bd4e84a4593199b49347b6d29" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 39577 } } peers { permanent_uuid: "5b101dbed8f34e9281a33db399f65ed6" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 36341 } } peers { permanent_uuid: "b3791458f5554d158fcb770980760849" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 42069 } } peers { permanent_uuid: "036d7a638abc417a9f4360ebfa5061f1" member_type: VOTER last_known_addr { host: "127.17.60.196" port: 35337 } attrs { promote: false } }
I20250112 15:08:26.299665 23364 raft_consensus.cc:1270] T d5a633e37cc84a098547fa042708b1d3 P b3791458f5554d158fcb770980760849 [term 1 FOLLOWER]: Refusing update from remote peer 5b101dbed8f34e9281a33db399f65ed6: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250112 15:08:26.299685 23493 raft_consensus.cc:1270] T d5a633e37cc84a098547fa042708b1d3 P 036d7a638abc417a9f4360ebfa5061f1 [term 1 LEARNER]: Refusing update from remote peer 5b101dbed8f34e9281a33db399f65ed6: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250112 15:08:26.299908 23555 consensus_queue.cc:1035] T d5a633e37cc84a098547fa042708b1d3 P 5b101dbed8f34e9281a33db399f65ed6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b3791458f5554d158fcb770980760849" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 42069 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
W20250112 15:08:26.299993 23254 consensus_peers.cc:487] T d5a633e37cc84a098547fa042708b1d3 P 5b101dbed8f34e9281a33db399f65ed6 -> Peer 91de225bd4e84a4593199b49347b6d29 (127.17.60.193:39577): Couldn't send request to peer 91de225bd4e84a4593199b49347b6d29. Status: Network error: Client connection negotiation failed: client connection to 127.17.60.193:39577: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20250112 15:08:26.300030 23538 consensus_queue.cc:1035] T d5a633e37cc84a098547fa042708b1d3 P 5b101dbed8f34e9281a33db399f65ed6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "036d7a638abc417a9f4360ebfa5061f1" member_type: VOTER last_known_addr { host: "127.17.60.196" port: 35337 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250112 15:08:26.300799 23555 raft_consensus.cc:2949] T d5a633e37cc84a098547fa042708b1d3 P 5b101dbed8f34e9281a33db399f65ed6 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 036d7a638abc417a9f4360ebfa5061f1 (127.17.60.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "91de225bd4e84a4593199b49347b6d29" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 39577 } } peers { permanent_uuid: "5b101dbed8f34e9281a33db399f65ed6" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 36341 } } peers { permanent_uuid: "b3791458f5554d158fcb770980760849" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 42069 } } peers { permanent_uuid: "036d7a638abc417a9f4360ebfa5061f1" member_type: VOTER last_known_addr { host: "127.17.60.196" port: 35337 } attrs { promote: false } } }
I20250112 15:08:26.300925 23493 raft_consensus.cc:2949] T d5a633e37cc84a098547fa042708b1d3 P 036d7a638abc417a9f4360ebfa5061f1 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 036d7a638abc417a9f4360ebfa5061f1 (127.17.60.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "91de225bd4e84a4593199b49347b6d29" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 39577 } } peers { permanent_uuid: "5b101dbed8f34e9281a33db399f65ed6" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 36341 } } peers { permanent_uuid: "b3791458f5554d158fcb770980760849" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 42069 } } peers { permanent_uuid: "036d7a638abc417a9f4360ebfa5061f1" member_type: VOTER last_known_addr { host: "127.17.60.196" port: 35337 } attrs { promote: false } } }
I20250112 15:08:26.301909 23364 raft_consensus.cc:2949] T d5a633e37cc84a098547fa042708b1d3 P b3791458f5554d158fcb770980760849 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 036d7a638abc417a9f4360ebfa5061f1 (127.17.60.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "91de225bd4e84a4593199b49347b6d29" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 39577 } } peers { permanent_uuid: "5b101dbed8f34e9281a33db399f65ed6" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 36341 } } peers { permanent_uuid: "b3791458f5554d158fcb770980760849" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 42069 } } peers { permanent_uuid: "036d7a638abc417a9f4360ebfa5061f1" member_type: VOTER last_known_addr { host: "127.17.60.196" port: 35337 } attrs { promote: false } } }
I20250112 15:08:26.302018 23108 catalog_manager.cc:5526] T d5a633e37cc84a098547fa042708b1d3 P 036d7a638abc417a9f4360ebfa5061f1 reported cstate change: config changed from index 2 to 3, 036d7a638abc417a9f4360ebfa5061f1 (127.17.60.196) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "5b101dbed8f34e9281a33db399f65ed6" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "91de225bd4e84a4593199b49347b6d29" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 39577 } } peers { permanent_uuid: "5b101dbed8f34e9281a33db399f65ed6" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 36341 } } peers { permanent_uuid: "b3791458f5554d158fcb770980760849" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 42069 } } peers { permanent_uuid: "036d7a638abc417a9f4360ebfa5061f1" member_type: VOTER last_known_addr { host: "127.17.60.196" port: 35337 } attrs { promote: false } } }
I20250112 15:08:26.302943 23289 consensus_queue.cc:237] T d5a633e37cc84a098547fa042708b1d3 P 5b101dbed8f34e9281a33db399f65ed6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5b101dbed8f34e9281a33db399f65ed6" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 36341 } } peers { permanent_uuid: "b3791458f5554d158fcb770980760849" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 42069 } } peers { permanent_uuid: "036d7a638abc417a9f4360ebfa5061f1" member_type: VOTER last_known_addr { host: "127.17.60.196" port: 35337 } attrs { promote: false } }
I20250112 15:08:26.303318 23493 raft_consensus.cc:1270] T d5a633e37cc84a098547fa042708b1d3 P 036d7a638abc417a9f4360ebfa5061f1 [term 1 FOLLOWER]: Refusing update from remote peer 5b101dbed8f34e9281a33db399f65ed6: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20250112 15:08:26.303344 23364 raft_consensus.cc:1270] T d5a633e37cc84a098547fa042708b1d3 P b3791458f5554d158fcb770980760849 [term 1 FOLLOWER]: Refusing update from remote peer 5b101dbed8f34e9281a33db399f65ed6: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20250112 15:08:26.303475 23538 consensus_queue.cc:1035] T d5a633e37cc84a098547fa042708b1d3 P 5b101dbed8f34e9281a33db399f65ed6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "036d7a638abc417a9f4360ebfa5061f1" member_type: VOTER last_known_addr { host: "127.17.60.196" port: 35337 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250112 15:08:26.303540 23557 consensus_queue.cc:1035] T d5a633e37cc84a098547fa042708b1d3 P 5b101dbed8f34e9281a33db399f65ed6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b3791458f5554d158fcb770980760849" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 42069 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250112 15:08:26.303984 23557 raft_consensus.cc:2949] T d5a633e37cc84a098547fa042708b1d3 P 5b101dbed8f34e9281a33db399f65ed6 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 91de225bd4e84a4593199b49347b6d29 (127.17.60.193) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5b101dbed8f34e9281a33db399f65ed6" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 36341 } } peers { permanent_uuid: "b3791458f5554d158fcb770980760849" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 42069 } } peers { permanent_uuid: "036d7a638abc417a9f4360ebfa5061f1" member_type: VOTER last_known_addr { host: "127.17.60.196" port: 35337 } attrs { promote: false } } }
I20250112 15:08:26.304167 23493 raft_consensus.cc:2949] T d5a633e37cc84a098547fa042708b1d3 P 036d7a638abc417a9f4360ebfa5061f1 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 91de225bd4e84a4593199b49347b6d29 (127.17.60.193) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5b101dbed8f34e9281a33db399f65ed6" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 36341 } } peers { permanent_uuid: "b3791458f5554d158fcb770980760849" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 42069 } } peers { permanent_uuid: "036d7a638abc417a9f4360ebfa5061f1" member_type: VOTER last_known_addr { host: "127.17.60.196" port: 35337 } attrs { promote: false } } }
I20250112 15:08:26.304172 23364 raft_consensus.cc:2949] T d5a633e37cc84a098547fa042708b1d3 P b3791458f5554d158fcb770980760849 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 91de225bd4e84a4593199b49347b6d29 (127.17.60.193) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5b101dbed8f34e9281a33db399f65ed6" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 36341 } } peers { permanent_uuid: "b3791458f5554d158fcb770980760849" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 42069 } } peers { permanent_uuid: "036d7a638abc417a9f4360ebfa5061f1" member_type: VOTER last_known_addr { host: "127.17.60.196" port: 35337 } attrs { promote: false } } }
I20250112 15:08:26.305020 23110 catalog_manager.cc:5526] T d5a633e37cc84a098547fa042708b1d3 P 036d7a638abc417a9f4360ebfa5061f1 reported cstate change: config changed from index 3 to 4, VOTER 91de225bd4e84a4593199b49347b6d29 (127.17.60.193) evicted. New cstate: current_term: 1 leader_uuid: "5b101dbed8f34e9281a33db399f65ed6" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5b101dbed8f34e9281a33db399f65ed6" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 36341 } } peers { permanent_uuid: "b3791458f5554d158fcb770980760849" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 42069 } } peers { permanent_uuid: "036d7a638abc417a9f4360ebfa5061f1" member_type: VOTER last_known_addr { host: "127.17.60.196" port: 35337 } attrs { promote: false } } }
I20250112 15:08:26.305085 23095 catalog_manager.cc:5039] ChangeConfig:REMOVE_PEER RPC for tablet d5a633e37cc84a098547fa042708b1d3 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250112 15:08:26.446744 23538 consensus_queue.cc:579] T 46c5820c43a64ba285e33ee2b607789b P 5b101dbed8f34e9281a33db399f65ed6 [LEADER]: Leader has been unable to successfully communicate with peer 91de225bd4e84a4593199b49347b6d29 for more than 1 seconds (1.447s)
I20250112 15:08:26.447625 23289 consensus_queue.cc:237] T 46c5820c43a64ba285e33ee2b607789b P 5b101dbed8f34e9281a33db399f65ed6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.1, Last appended by leader: 0, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "91de225bd4e84a4593199b49347b6d29" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 39577 } } peers { permanent_uuid: "5b101dbed8f34e9281a33db399f65ed6" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 36341 } } peers { permanent_uuid: "b3791458f5554d158fcb770980760849" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 42069 } } peers { permanent_uuid: "036d7a638abc417a9f4360ebfa5061f1" member_type: NON_VOTER last_known_addr { host: "127.17.60.196" port: 35337 } attrs { promote: true } }
W20250112 15:08:26.448158 23254 consensus_peers.cc:487] T 46c5820c43a64ba285e33ee2b607789b P 5b101dbed8f34e9281a33db399f65ed6 -> Peer 036d7a638abc417a9f4360ebfa5061f1 (127.17.60.196:35337): Couldn't send request to peer 036d7a638abc417a9f4360ebfa5061f1. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 46c5820c43a64ba285e33ee2b607789b. This is attempt 1: this message will repeat every 5th retry.
I20250112 15:08:26.448158 23364 raft_consensus.cc:1270] T 46c5820c43a64ba285e33ee2b607789b P b3791458f5554d158fcb770980760849 [term 2 FOLLOWER]: Refusing update from remote peer 5b101dbed8f34e9281a33db399f65ed6: Log matching property violated. Preceding OpId in replica: term: 2 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250112 15:08:26.448433 23446 consensus_queue.cc:1035] T 46c5820c43a64ba285e33ee2b607789b P 5b101dbed8f34e9281a33db399f65ed6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b3791458f5554d158fcb770980760849" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 42069 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20250112 15:08:26.448514 23254 consensus_peers.cc:487] T 46c5820c43a64ba285e33ee2b607789b P 5b101dbed8f34e9281a33db399f65ed6 -> Peer 91de225bd4e84a4593199b49347b6d29 (127.17.60.193:39577): Couldn't send request to peer 91de225bd4e84a4593199b49347b6d29. Status: Network error: Client connection negotiation failed: client connection to 127.17.60.193:39577: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20250112 15:08:26.450066 23555 raft_consensus.cc:2949] T 46c5820c43a64ba285e33ee2b607789b P 5b101dbed8f34e9281a33db399f65ed6 [term 2 LEADER]: Committing config change with OpId 2.2: config changed from index -1 to 2, NON_VOTER 036d7a638abc417a9f4360ebfa5061f1 (127.17.60.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "91de225bd4e84a4593199b49347b6d29" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 39577 } } peers { permanent_uuid: "5b101dbed8f34e9281a33db399f65ed6" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 36341 } } peers { permanent_uuid: "b3791458f5554d158fcb770980760849" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 42069 } } peers { permanent_uuid: "036d7a638abc417a9f4360ebfa5061f1" member_type: NON_VOTER last_known_addr { host: "127.17.60.196" port: 35337 } attrs { promote: true } } }
I20250112 15:08:26.450232 23364 raft_consensus.cc:2949] T 46c5820c43a64ba285e33ee2b607789b P b3791458f5554d158fcb770980760849 [term 2 FOLLOWER]: Committing config change with OpId 2.2: config changed from index -1 to 2, NON_VOTER 036d7a638abc417a9f4360ebfa5061f1 (127.17.60.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "91de225bd4e84a4593199b49347b6d29" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 39577 } } peers { permanent_uuid: "5b101dbed8f34e9281a33db399f65ed6" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 36341 } } peers { permanent_uuid: "b3791458f5554d158fcb770980760849" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 42069 } } peers { permanent_uuid: "036d7a638abc417a9f4360ebfa5061f1" member_type: NON_VOTER last_known_addr { host: "127.17.60.196" port: 35337 } attrs { promote: true } } }
I20250112 15:08:26.451010 23095 catalog_manager.cc:5039] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 46c5820c43a64ba285e33ee2b607789b with cas_config_opid_index -1: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20250112 15:08:26.451315 23108 catalog_manager.cc:5526] T 46c5820c43a64ba285e33ee2b607789b P b3791458f5554d158fcb770980760849 reported cstate change: config changed from index -1 to 2, NON_VOTER 036d7a638abc417a9f4360ebfa5061f1 (127.17.60.196) added. New cstate: current_term: 2 leader_uuid: "5b101dbed8f34e9281a33db399f65ed6" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "91de225bd4e84a4593199b49347b6d29" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 39577 } } peers { permanent_uuid: "5b101dbed8f34e9281a33db399f65ed6" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 36341 } } peers { permanent_uuid: "b3791458f5554d158fcb770980760849" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 42069 } } peers { permanent_uuid: "036d7a638abc417a9f4360ebfa5061f1" member_type: NON_VOTER last_known_addr { host: "127.17.60.196" port: 35337 } attrs { promote: true } } }
W20250112 15:08:26.578459 23096 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.17.60.193:39577: connect: Connection refused (error 111) [suppressed 35 similar messages]
I20250112 15:08:26.705070 23577 ts_tablet_manager.cc:927] T a84082c6bef84e638159590ed624062e P 036d7a638abc417a9f4360ebfa5061f1: Initiating tablet copy from peer b3791458f5554d158fcb770980760849 (127.17.60.195:42069)
I20250112 15:08:26.705255 23577 tablet_copy_client.cc:323] T a84082c6bef84e638159590ed624062e P 036d7a638abc417a9f4360ebfa5061f1: tablet copy: Beginning tablet copy session from remote peer at address 127.17.60.195:42069
I20250112 15:08:26.707813 23374 tablet_copy_service.cc:140] P b3791458f5554d158fcb770980760849: Received BeginTabletCopySession request for tablet a84082c6bef84e638159590ed624062e from peer 036d7a638abc417a9f4360ebfa5061f1 ({username='slave'} at 127.0.0.1:53680)
I20250112 15:08:26.707882 23374 tablet_copy_service.cc:161] P b3791458f5554d158fcb770980760849: Beginning new tablet copy session on tablet a84082c6bef84e638159590ed624062e from peer 036d7a638abc417a9f4360ebfa5061f1 at {username='slave'} at 127.0.0.1:53680: session id = 036d7a638abc417a9f4360ebfa5061f1-a84082c6bef84e638159590ed624062e
I20250112 15:08:26.708333 23374 tablet_copy_source_session.cc:215] T a84082c6bef84e638159590ed624062e P b3791458f5554d158fcb770980760849: Tablet Copy: opened 0 blocks and 1 log segments
I20250112 15:08:26.708644 23577 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a84082c6bef84e638159590ed624062e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 15:08:26.709673 23577 tablet_copy_client.cc:806] T a84082c6bef84e638159590ed624062e P 036d7a638abc417a9f4360ebfa5061f1: tablet copy: Starting download of 0 data blocks...
I20250112 15:08:26.709781 23577 tablet_copy_client.cc:670] T a84082c6bef84e638159590ed624062e P 036d7a638abc417a9f4360ebfa5061f1: tablet copy: Starting download of 1 WAL segments...
I20250112 15:08:26.710307 23577 tablet_copy_client.cc:538] T a84082c6bef84e638159590ed624062e P 036d7a638abc417a9f4360ebfa5061f1: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250112 15:08:26.710904 23577 tablet_bootstrap.cc:492] T a84082c6bef84e638159590ed624062e P 036d7a638abc417a9f4360ebfa5061f1: Bootstrap starting.
I20250112 15:08:26.711951 23577 tablet_bootstrap.cc:492] T a84082c6bef84e638159590ed624062e P 036d7a638abc417a9f4360ebfa5061f1: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250112 15:08:26.712105 23577 tablet_bootstrap.cc:492] T a84082c6bef84e638159590ed624062e P 036d7a638abc417a9f4360ebfa5061f1: Bootstrap complete.
I20250112 15:08:26.712189 23577 ts_tablet_manager.cc:1397] T a84082c6bef84e638159590ed624062e P 036d7a638abc417a9f4360ebfa5061f1: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20250112 15:08:26.712302 23577 raft_consensus.cc:357] T a84082c6bef84e638159590ed624062e P 036d7a638abc417a9f4360ebfa5061f1 [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5b101dbed8f34e9281a33db399f65ed6" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 36341 } } peers { permanent_uuid: "b3791458f5554d158fcb770980760849" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 42069 } } peers { permanent_uuid: "91de225bd4e84a4593199b49347b6d29" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 39577 } } peers { permanent_uuid: "036d7a638abc417a9f4360ebfa5061f1" member_type: NON_VOTER last_known_addr { host: "127.17.60.196" port: 35337 } attrs { promote: true } }
I20250112 15:08:26.712378 23577 raft_consensus.cc:738] T a84082c6bef84e638159590ed624062e P 036d7a638abc417a9f4360ebfa5061f1 [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: 036d7a638abc417a9f4360ebfa5061f1, State: Initialized, Role: LEARNER
I20250112 15:08:26.712406 23577 consensus_queue.cc:260] T a84082c6bef84e638159590ed624062e P 036d7a638abc417a9f4360ebfa5061f1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5b101dbed8f34e9281a33db399f65ed6" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 36341 } } peers { permanent_uuid: "b3791458f5554d158fcb770980760849" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 42069 } } peers { permanent_uuid: "91de225bd4e84a4593199b49347b6d29" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 39577 } } peers { permanent_uuid: "036d7a638abc417a9f4360ebfa5061f1" member_type: NON_VOTER last_known_addr { host: "127.17.60.196" port: 35337 } attrs { promote: true } }
I20250112 15:08:26.712543 23577 ts_tablet_manager.cc:1428] T a84082c6bef84e638159590ed624062e P 036d7a638abc417a9f4360ebfa5061f1: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20250112 15:08:26.712780 23374 tablet_copy_service.cc:342] P b3791458f5554d158fcb770980760849: Request end of tablet copy session 036d7a638abc417a9f4360ebfa5061f1-a84082c6bef84e638159590ed624062e received from {username='slave'} at 127.0.0.1:53680
I20250112 15:08:26.712836 23374 tablet_copy_service.cc:434] P b3791458f5554d158fcb770980760849: ending tablet copy session 036d7a638abc417a9f4360ebfa5061f1-a84082c6bef84e638159590ed624062e on tablet a84082c6bef84e638159590ed624062e with peer 036d7a638abc417a9f4360ebfa5061f1
W20250112 15:08:26.805683 23159 auto_rebalancer.cc:227] Could not retrieve cluster info: Not found: tserver 91de225bd4e84a4593199b49347b6d29 not available for placement
I20250112 15:08:26.849464 23577 ts_tablet_manager.cc:927] T 46c5820c43a64ba285e33ee2b607789b P 036d7a638abc417a9f4360ebfa5061f1: Initiating tablet copy from peer 5b101dbed8f34e9281a33db399f65ed6 (127.17.60.194:36341)
I20250112 15:08:26.849689 23577 tablet_copy_client.cc:323] T 46c5820c43a64ba285e33ee2b607789b P 036d7a638abc417a9f4360ebfa5061f1: tablet copy: Beginning tablet copy session from remote peer at address 127.17.60.194:36341
I20250112 15:08:26.849963 23299 tablet_copy_service.cc:140] P 5b101dbed8f34e9281a33db399f65ed6: Received BeginTabletCopySession request for tablet 46c5820c43a64ba285e33ee2b607789b from peer 036d7a638abc417a9f4360ebfa5061f1 ({username='slave'} at 127.0.0.1:39172)
I20250112 15:08:26.850033 23299 tablet_copy_service.cc:161] P 5b101dbed8f34e9281a33db399f65ed6: Beginning new tablet copy session on tablet 46c5820c43a64ba285e33ee2b607789b from peer 036d7a638abc417a9f4360ebfa5061f1 at {username='slave'} at 127.0.0.1:39172: session id = 036d7a638abc417a9f4360ebfa5061f1-46c5820c43a64ba285e33ee2b607789b
I20250112 15:08:26.850499 23299 tablet_copy_source_session.cc:215] T 46c5820c43a64ba285e33ee2b607789b P 5b101dbed8f34e9281a33db399f65ed6: Tablet Copy: opened 0 blocks and 1 log segments
I20250112 15:08:26.850865 23577 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 46c5820c43a64ba285e33ee2b607789b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 15:08:26.851795 23577 tablet_copy_client.cc:806] T 46c5820c43a64ba285e33ee2b607789b P 036d7a638abc417a9f4360ebfa5061f1: tablet copy: Starting download of 0 data blocks...
I20250112 15:08:26.851914 23577 tablet_copy_client.cc:670] T 46c5820c43a64ba285e33ee2b607789b P 036d7a638abc417a9f4360ebfa5061f1: tablet copy: Starting download of 1 WAL segments...
I20250112 15:08:26.852440 23577 tablet_copy_client.cc:538] T 46c5820c43a64ba285e33ee2b607789b P 036d7a638abc417a9f4360ebfa5061f1: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250112 15:08:26.853027 23577 tablet_bootstrap.cc:492] T 46c5820c43a64ba285e33ee2b607789b P 036d7a638abc417a9f4360ebfa5061f1: Bootstrap starting.
I20250112 15:08:26.854094 23577 tablet_bootstrap.cc:492] T 46c5820c43a64ba285e33ee2b607789b P 036d7a638abc417a9f4360ebfa5061f1: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250112 15:08:26.854239 23577 tablet_bootstrap.cc:492] T 46c5820c43a64ba285e33ee2b607789b P 036d7a638abc417a9f4360ebfa5061f1: Bootstrap complete.
I20250112 15:08:26.854317 23577 ts_tablet_manager.cc:1397] T 46c5820c43a64ba285e33ee2b607789b P 036d7a638abc417a9f4360ebfa5061f1: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20250112 15:08:26.854455 23577 raft_consensus.cc:357] T 46c5820c43a64ba285e33ee2b607789b P 036d7a638abc417a9f4360ebfa5061f1 [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "91de225bd4e84a4593199b49347b6d29" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 39577 } } peers { permanent_uuid: "5b101dbed8f34e9281a33db399f65ed6" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 36341 } } peers { permanent_uuid: "b3791458f5554d158fcb770980760849" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 42069 } } peers { permanent_uuid: "036d7a638abc417a9f4360ebfa5061f1" member_type: NON_VOTER last_known_addr { host: "127.17.60.196" port: 35337 } attrs { promote: true } }
I20250112 15:08:26.854516 23577 raft_consensus.cc:738] T 46c5820c43a64ba285e33ee2b607789b P 036d7a638abc417a9f4360ebfa5061f1 [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: 036d7a638abc417a9f4360ebfa5061f1, State: Initialized, Role: LEARNER
I20250112 15:08:26.854537 23577 consensus_queue.cc:260] T 46c5820c43a64ba285e33ee2b607789b P 036d7a638abc417a9f4360ebfa5061f1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "91de225bd4e84a4593199b49347b6d29" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 39577 } } peers { permanent_uuid: "5b101dbed8f34e9281a33db399f65ed6" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 36341 } } peers { permanent_uuid: "b3791458f5554d158fcb770980760849" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 42069 } } peers { permanent_uuid: "036d7a638abc417a9f4360ebfa5061f1" member_type: NON_VOTER last_known_addr { host: "127.17.60.196" port: 35337 } attrs { promote: true } }
I20250112 15:08:26.854645 23577 ts_tablet_manager.cc:1428] T 46c5820c43a64ba285e33ee2b607789b P 036d7a638abc417a9f4360ebfa5061f1: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20250112 15:08:26.854879 23299 tablet_copy_service.cc:342] P 5b101dbed8f34e9281a33db399f65ed6: Request end of tablet copy session 036d7a638abc417a9f4360ebfa5061f1-46c5820c43a64ba285e33ee2b607789b received from {username='slave'} at 127.0.0.1:39172
I20250112 15:08:26.854954 23299 tablet_copy_service.cc:434] P 5b101dbed8f34e9281a33db399f65ed6: ending tablet copy session 036d7a638abc417a9f4360ebfa5061f1-46c5820c43a64ba285e33ee2b607789b on tablet 46c5820c43a64ba285e33ee2b607789b with peer 036d7a638abc417a9f4360ebfa5061f1
W20250112 15:08:27.116139 23096 catalog_manager.cc:4615] TS 91de225bd4e84a4593199b49347b6d29 (127.17.60.193:39577): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 656d5437ee6a475a89af5e24b74f315a: Network error: Client connection negotiation failed: client connection to 127.17.60.193:39577: connect: Connection refused (error 111)
I20250112 15:08:27.164238 23493 raft_consensus.cc:1212] T a84082c6bef84e638159590ed624062e P 036d7a638abc417a9f4360ebfa5061f1 [term 2 LEARNER]: Deduplicated request from leader. Original: 2.1->[2.2-2.2] Dedup: 2.2->[]
I20250112 15:08:27.275173 23493 raft_consensus.cc:1212] T 46c5820c43a64ba285e33ee2b607789b P 036d7a638abc417a9f4360ebfa5061f1 [term 2 LEARNER]: Deduplicated request from leader. Original: 2.1->[2.2-2.2] Dedup: 2.2->[]
I20250112 15:08:27.366886 23570 consensus_queue.cc:579] T a84082c6bef84e638159590ed624062e P b3791458f5554d158fcb770980760849 [LEADER]: Leader has been unable to successfully communicate with peer 91de225bd4e84a4593199b49347b6d29 for more than 1 seconds (1.183s)
I20250112 15:08:27.538537 23587 consensus_queue.cc:579] T 46c5820c43a64ba285e33ee2b607789b P 5b101dbed8f34e9281a33db399f65ed6 [LEADER]: Leader has been unable to successfully communicate with peer 91de225bd4e84a4593199b49347b6d29 for more than 1 seconds (1.091s)
I20250112 15:08:27.659471 23588 raft_consensus.cc:1059] T a84082c6bef84e638159590ed624062e P b3791458f5554d158fcb770980760849: attempting to promote NON_VOTER 036d7a638abc417a9f4360ebfa5061f1 to VOTER
I20250112 15:08:27.659703 23588 consensus_queue.cc:237] T a84082c6bef84e638159590ed624062e P b3791458f5554d158fcb770980760849 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 0, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "5b101dbed8f34e9281a33db399f65ed6" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 36341 } } peers { permanent_uuid: "b3791458f5554d158fcb770980760849" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 42069 } } peers { permanent_uuid: "91de225bd4e84a4593199b49347b6d29" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 39577 } } peers { permanent_uuid: "036d7a638abc417a9f4360ebfa5061f1" member_type: VOTER last_known_addr { host: "127.17.60.196" port: 35337 } attrs { promote: false } }
I20250112 15:08:27.660272 23493 raft_consensus.cc:1270] T a84082c6bef84e638159590ed624062e P 036d7a638abc417a9f4360ebfa5061f1 [term 2 LEARNER]: Refusing update from remote peer b3791458f5554d158fcb770980760849: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20250112 15:08:27.660285 23289 raft_consensus.cc:1270] T a84082c6bef84e638159590ed624062e P 5b101dbed8f34e9281a33db399f65ed6 [term 2 FOLLOWER]: Refusing update from remote peer b3791458f5554d158fcb770980760849: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20250112 15:08:27.660490 23588 consensus_queue.cc:1035] T a84082c6bef84e638159590ed624062e P b3791458f5554d158fcb770980760849 [LEADER]: Connected to new peer: Peer: permanent_uuid: "036d7a638abc417a9f4360ebfa5061f1" member_type: VOTER last_known_addr { host: "127.17.60.196" port: 35337 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250112 15:08:27.660619 23595 consensus_queue.cc:1035] T a84082c6bef84e638159590ed624062e P b3791458f5554d158fcb770980760849 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5b101dbed8f34e9281a33db399f65ed6" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 36341 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
W20250112 15:08:27.660741 23329 consensus_peers.cc:487] T a84082c6bef84e638159590ed624062e P b3791458f5554d158fcb770980760849 -> Peer 91de225bd4e84a4593199b49347b6d29 (127.17.60.193:39577): Couldn't send request to peer 91de225bd4e84a4593199b49347b6d29. Status: Network error: Client connection negotiation failed: client connection to 127.17.60.193:39577: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20250112 15:08:27.661315 23570 raft_consensus.cc:2949] T a84082c6bef84e638159590ed624062e P b3791458f5554d158fcb770980760849 [term 2 LEADER]: Committing config change with OpId 2.3: config changed from index 2 to 3, 036d7a638abc417a9f4360ebfa5061f1 (127.17.60.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "5b101dbed8f34e9281a33db399f65ed6" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 36341 } } peers { permanent_uuid: "b3791458f5554d158fcb770980760849" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 42069 } } peers { permanent_uuid: "91de225bd4e84a4593199b49347b6d29" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 39577 } } peers { permanent_uuid: "036d7a638abc417a9f4360ebfa5061f1" member_type: VOTER last_known_addr { host: "127.17.60.196" port: 35337 } attrs { promote: false } } }
I20250112 15:08:27.661448 23493 raft_consensus.cc:2949] T a84082c6bef84e638159590ed624062e P 036d7a638abc417a9f4360ebfa5061f1 [term 2 FOLLOWER]: Committing config change with OpId 2.3: config changed from index 2 to 3, 036d7a638abc417a9f4360ebfa5061f1 (127.17.60.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "5b101dbed8f34e9281a33db399f65ed6" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 36341 } } peers { permanent_uuid: "b3791458f5554d158fcb770980760849" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 42069 } } peers { permanent_uuid: "91de225bd4e84a4593199b49347b6d29" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 39577 } } peers { permanent_uuid: "036d7a638abc417a9f4360ebfa5061f1" member_type: VOTER last_known_addr { host: "127.17.60.196" port: 35337 } attrs { promote: false } } }
I20250112 15:08:27.662477 23289 raft_consensus.cc:2949] T a84082c6bef84e638159590ed624062e P 5b101dbed8f34e9281a33db399f65ed6 [term 2 FOLLOWER]: Committing config change with OpId 2.3: config changed from index 2 to 3, 036d7a638abc417a9f4360ebfa5061f1 (127.17.60.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "5b101dbed8f34e9281a33db399f65ed6" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 36341 } } peers { permanent_uuid: "b3791458f5554d158fcb770980760849" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 42069 } } peers { permanent_uuid: "91de225bd4e84a4593199b49347b6d29" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 39577 } } peers { permanent_uuid: "036d7a638abc417a9f4360ebfa5061f1" member_type: VOTER last_known_addr { host: "127.17.60.196" port: 35337 } attrs { promote: false } } }
I20250112 15:08:27.662523 23108 catalog_manager.cc:5526] T a84082c6bef84e638159590ed624062e P b3791458f5554d158fcb770980760849 reported cstate change: config changed from index 2 to 3, 036d7a638abc417a9f4360ebfa5061f1 (127.17.60.196) changed from NON_VOTER to VOTER. New cstate: current_term: 2 leader_uuid: "b3791458f5554d158fcb770980760849" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "5b101dbed8f34e9281a33db399f65ed6" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 36341 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b3791458f5554d158fcb770980760849" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 42069 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "91de225bd4e84a4593199b49347b6d29" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 39577 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "036d7a638abc417a9f4360ebfa5061f1" member_type: VOTER last_known_addr { host: "127.17.60.196" port: 35337 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20250112 15:08:27.663919 23364 consensus_queue.cc:237] T a84082c6bef84e638159590ed624062e P b3791458f5554d158fcb770980760849 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.3, Last appended by leader: 0, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5b101dbed8f34e9281a33db399f65ed6" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 36341 } } peers { permanent_uuid: "b3791458f5554d158fcb770980760849" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 42069 } } peers { permanent_uuid: "036d7a638abc417a9f4360ebfa5061f1" member_type: VOTER last_known_addr { host: "127.17.60.196" port: 35337 } attrs { promote: false } }
I20250112 15:08:27.664238 23493 raft_consensus.cc:1270] T a84082c6bef84e638159590ed624062e P 036d7a638abc417a9f4360ebfa5061f1 [term 2 FOLLOWER]: Refusing update from remote peer b3791458f5554d158fcb770980760849: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20250112 15:08:27.664250 23289 raft_consensus.cc:1270] T a84082c6bef84e638159590ed624062e P 5b101dbed8f34e9281a33db399f65ed6 [term 2 FOLLOWER]: Refusing update from remote peer b3791458f5554d158fcb770980760849: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20250112 15:08:27.664405 23586 consensus_queue.cc:1035] T a84082c6bef84e638159590ed624062e P b3791458f5554d158fcb770980760849 [LEADER]: Connected to new peer: Peer: permanent_uuid: "036d7a638abc417a9f4360ebfa5061f1" member_type: VOTER last_known_addr { host: "127.17.60.196" port: 35337 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250112 15:08:27.664485 23588 consensus_queue.cc:1035] T a84082c6bef84e638159590ed624062e P b3791458f5554d158fcb770980760849 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5b101dbed8f34e9281a33db399f65ed6" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 36341 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250112 15:08:27.664973 23588 raft_consensus.cc:2949] T a84082c6bef84e638159590ed624062e P b3791458f5554d158fcb770980760849 [term 2 LEADER]: Committing config change with OpId 2.4: config changed from index 3 to 4, VOTER 91de225bd4e84a4593199b49347b6d29 (127.17.60.193) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5b101dbed8f34e9281a33db399f65ed6" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 36341 } } peers { permanent_uuid: "b3791458f5554d158fcb770980760849" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 42069 } } peers { permanent_uuid: "036d7a638abc417a9f4360ebfa5061f1" member_type: VOTER last_known_addr { host: "127.17.60.196" port: 35337 } attrs { promote: false } } }
I20250112 15:08:27.665063 23289 raft_consensus.cc:2949] T a84082c6bef84e638159590ed624062e P 5b101dbed8f34e9281a33db399f65ed6 [term 2 FOLLOWER]: Committing config change with OpId 2.4: config changed from index 3 to 4, VOTER 91de225bd4e84a4593199b49347b6d29 (127.17.60.193) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5b101dbed8f34e9281a33db399f65ed6" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 36341 } } peers { permanent_uuid: "b3791458f5554d158fcb770980760849" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 42069 } } peers { permanent_uuid: "036d7a638abc417a9f4360ebfa5061f1" member_type: VOTER last_known_addr { host: "127.17.60.196" port: 35337 } attrs { promote: false } } }
I20250112 15:08:27.665228 23493 raft_consensus.cc:2949] T a84082c6bef84e638159590ed624062e P 036d7a638abc417a9f4360ebfa5061f1 [term 2 FOLLOWER]: Committing config change with OpId 2.4: config changed from index 3 to 4, VOTER 91de225bd4e84a4593199b49347b6d29 (127.17.60.193) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5b101dbed8f34e9281a33db399f65ed6" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 36341 } } peers { permanent_uuid: "b3791458f5554d158fcb770980760849" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 42069 } } peers { permanent_uuid: "036d7a638abc417a9f4360ebfa5061f1" member_type: VOTER last_known_addr { host: "127.17.60.196" port: 35337 } attrs { promote: false } } }
I20250112 15:08:27.665962 23109 catalog_manager.cc:5526] T a84082c6bef84e638159590ed624062e P 5b101dbed8f34e9281a33db399f65ed6 reported cstate change: config changed from index 3 to 4, VOTER 91de225bd4e84a4593199b49347b6d29 (127.17.60.193) evicted. New cstate: current_term: 2 leader_uuid: "b3791458f5554d158fcb770980760849" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5b101dbed8f34e9281a33db399f65ed6" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 36341 } } peers { permanent_uuid: "b3791458f5554d158fcb770980760849" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 42069 } } peers { permanent_uuid: "036d7a638abc417a9f4360ebfa5061f1" member_type: VOTER last_known_addr { host: "127.17.60.196" port: 35337 } attrs { promote: false } } }
I20250112 15:08:27.666172 23095 catalog_manager.cc:5039] ChangeConfig:REMOVE_PEER RPC for tablet a84082c6bef84e638159590ed624062e with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
W20250112 15:08:27.805959 23159 auto_rebalancer.cc:227] Could not retrieve cluster info: Not found: tserver 91de225bd4e84a4593199b49347b6d29 not available for placement
I20250112 15:08:27.828866 23591 raft_consensus.cc:1059] T 46c5820c43a64ba285e33ee2b607789b P 5b101dbed8f34e9281a33db399f65ed6: attempting to promote NON_VOTER 036d7a638abc417a9f4360ebfa5061f1 to VOTER
I20250112 15:08:27.829083 23591 consensus_queue.cc:237] T 46c5820c43a64ba285e33ee2b607789b P 5b101dbed8f34e9281a33db399f65ed6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 0, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "91de225bd4e84a4593199b49347b6d29" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 39577 } } peers { permanent_uuid: "5b101dbed8f34e9281a33db399f65ed6" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 36341 } } peers { permanent_uuid: "b3791458f5554d158fcb770980760849" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 42069 } } peers { permanent_uuid: "036d7a638abc417a9f4360ebfa5061f1" member_type: VOTER last_known_addr { host: "127.17.60.196" port: 35337 } attrs { promote: false } }
I20250112 15:08:27.829715 23364 raft_consensus.cc:1270] T 46c5820c43a64ba285e33ee2b607789b P b3791458f5554d158fcb770980760849 [term 2 FOLLOWER]: Refusing update from remote peer 5b101dbed8f34e9281a33db399f65ed6: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20250112 15:08:27.829711 23493 raft_consensus.cc:1270] T 46c5820c43a64ba285e33ee2b607789b P 036d7a638abc417a9f4360ebfa5061f1 [term 2 LEARNER]: Refusing update from remote peer 5b101dbed8f34e9281a33db399f65ed6: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20250112 15:08:27.829998 23603 consensus_queue.cc:1035] T 46c5820c43a64ba285e33ee2b607789b P 5b101dbed8f34e9281a33db399f65ed6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b3791458f5554d158fcb770980760849" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 42069 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250112 15:08:27.830114 23587 consensus_queue.cc:1035] T 46c5820c43a64ba285e33ee2b607789b P 5b101dbed8f34e9281a33db399f65ed6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "036d7a638abc417a9f4360ebfa5061f1" member_type: VOTER last_known_addr { host: "127.17.60.196" port: 35337 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
W20250112 15:08:27.830201 23254 consensus_peers.cc:487] T 46c5820c43a64ba285e33ee2b607789b P 5b101dbed8f34e9281a33db399f65ed6 -> Peer 91de225bd4e84a4593199b49347b6d29 (127.17.60.193:39577): Couldn't send request to peer 91de225bd4e84a4593199b49347b6d29. Status: Network error: Client connection negotiation failed: client connection to 127.17.60.193:39577: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20250112 15:08:27.830956 23591 raft_consensus.cc:2949] T 46c5820c43a64ba285e33ee2b607789b P 5b101dbed8f34e9281a33db399f65ed6 [term 2 LEADER]: Committing config change with OpId 2.3: config changed from index 2 to 3, 036d7a638abc417a9f4360ebfa5061f1 (127.17.60.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "91de225bd4e84a4593199b49347b6d29" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 39577 } } peers { permanent_uuid: "5b101dbed8f34e9281a33db399f65ed6" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 36341 } } peers { permanent_uuid: "b3791458f5554d158fcb770980760849" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 42069 } } peers { permanent_uuid: "036d7a638abc417a9f4360ebfa5061f1" member_type: VOTER last_known_addr { host: "127.17.60.196" port: 35337 } attrs { promote: false } } }
I20250112 15:08:27.831159 23493 raft_consensus.cc:2949] T 46c5820c43a64ba285e33ee2b607789b P 036d7a638abc417a9f4360ebfa5061f1 [term 2 FOLLOWER]: Committing config change with OpId 2.3: config changed from index 2 to 3, 036d7a638abc417a9f4360ebfa5061f1 (127.17.60.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "91de225bd4e84a4593199b49347b6d29" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 39577 } } peers { permanent_uuid: "5b101dbed8f34e9281a33db399f65ed6" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 36341 } } peers { permanent_uuid: "b3791458f5554d158fcb770980760849" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 42069 } } peers { permanent_uuid: "036d7a638abc417a9f4360ebfa5061f1" member_type: VOTER last_known_addr { host: "127.17.60.196" port: 35337 } attrs { promote: false } } }
I20250112 15:08:27.832026 23364 raft_consensus.cc:2949] T 46c5820c43a64ba285e33ee2b607789b P b3791458f5554d158fcb770980760849 [term 2 FOLLOWER]: Committing config change with OpId 2.3: config changed from index 2 to 3, 036d7a638abc417a9f4360ebfa5061f1 (127.17.60.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "91de225bd4e84a4593199b49347b6d29" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 39577 } } peers { permanent_uuid: "5b101dbed8f34e9281a33db399f65ed6" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 36341 } } peers { permanent_uuid: "b3791458f5554d158fcb770980760849" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 42069 } } peers { permanent_uuid: "036d7a638abc417a9f4360ebfa5061f1" member_type: VOTER last_known_addr { host: "127.17.60.196" port: 35337 } attrs { promote: false } } }
I20250112 15:08:27.832055 23110 catalog_manager.cc:5526] T 46c5820c43a64ba285e33ee2b607789b P 5b101dbed8f34e9281a33db399f65ed6 reported cstate change: config changed from index 2 to 3, 036d7a638abc417a9f4360ebfa5061f1 (127.17.60.196) changed from NON_VOTER to VOTER. New cstate: current_term: 2 leader_uuid: "5b101dbed8f34e9281a33db399f65ed6" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "91de225bd4e84a4593199b49347b6d29" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 39577 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5b101dbed8f34e9281a33db399f65ed6" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 36341 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b3791458f5554d158fcb770980760849" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 42069 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "036d7a638abc417a9f4360ebfa5061f1" member_type: VOTER last_known_addr { host: "127.17.60.196" port: 35337 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20250112 15:08:27.833077 23289 consensus_queue.cc:237] T 46c5820c43a64ba285e33ee2b607789b P 5b101dbed8f34e9281a33db399f65ed6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.3, Last appended by leader: 0, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5b101dbed8f34e9281a33db399f65ed6" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 36341 } } peers { permanent_uuid: "b3791458f5554d158fcb770980760849" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 42069 } } peers { permanent_uuid: "036d7a638abc417a9f4360ebfa5061f1" member_type: VOTER last_known_addr { host: "127.17.60.196" port: 35337 } attrs { promote: false } }
I20250112 15:08:27.833422 23364 raft_consensus.cc:1270] T 46c5820c43a64ba285e33ee2b607789b P b3791458f5554d158fcb770980760849 [term 2 FOLLOWER]: Refusing update from remote peer 5b101dbed8f34e9281a33db399f65ed6: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20250112 15:08:27.833434 23493 raft_consensus.cc:1270] T 46c5820c43a64ba285e33ee2b607789b P 036d7a638abc417a9f4360ebfa5061f1 [term 2 FOLLOWER]: Refusing update from remote peer 5b101dbed8f34e9281a33db399f65ed6: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20250112 15:08:27.833585 23603 consensus_queue.cc:1035] T 46c5820c43a64ba285e33ee2b607789b P 5b101dbed8f34e9281a33db399f65ed6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b3791458f5554d158fcb770980760849" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 42069 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250112 15:08:27.833658 23587 consensus_queue.cc:1035] T 46c5820c43a64ba285e33ee2b607789b P 5b101dbed8f34e9281a33db399f65ed6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "036d7a638abc417a9f4360ebfa5061f1" member_type: VOTER last_known_addr { host: "127.17.60.196" port: 35337 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250112 15:08:27.834152 23587 raft_consensus.cc:2949] T 46c5820c43a64ba285e33ee2b607789b P 5b101dbed8f34e9281a33db399f65ed6 [term 2 LEADER]: Committing config change with OpId 2.4: config changed from index 3 to 4, VOTER 91de225bd4e84a4593199b49347b6d29 (127.17.60.193) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5b101dbed8f34e9281a33db399f65ed6" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 36341 } } peers { permanent_uuid: "b3791458f5554d158fcb770980760849" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 42069 } } peers { permanent_uuid: "036d7a638abc417a9f4360ebfa5061f1" member_type: VOTER last_known_addr { host: "127.17.60.196" port: 35337 } attrs { promote: false } } }
I20250112 15:08:27.834256 23364 raft_consensus.cc:2949] T 46c5820c43a64ba285e33ee2b607789b P b3791458f5554d158fcb770980760849 [term 2 FOLLOWER]: Committing config change with OpId 2.4: config changed from index 3 to 4, VOTER 91de225bd4e84a4593199b49347b6d29 (127.17.60.193) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5b101dbed8f34e9281a33db399f65ed6" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 36341 } } peers { permanent_uuid: "b3791458f5554d158fcb770980760849" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 42069 } } peers { permanent_uuid: "036d7a638abc417a9f4360ebfa5061f1" member_type: VOTER last_known_addr { host: "127.17.60.196" port: 35337 } attrs { promote: false } } }
I20250112 15:08:27.834314 23493 raft_consensus.cc:2949] T 46c5820c43a64ba285e33ee2b607789b P 036d7a638abc417a9f4360ebfa5061f1 [term 2 FOLLOWER]: Committing config change with OpId 2.4: config changed from index 3 to 4, VOTER 91de225bd4e84a4593199b49347b6d29 (127.17.60.193) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5b101dbed8f34e9281a33db399f65ed6" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 36341 } } peers { permanent_uuid: "b3791458f5554d158fcb770980760849" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 42069 } } peers { permanent_uuid: "036d7a638abc417a9f4360ebfa5061f1" member_type: VOTER last_known_addr { host: "127.17.60.196" port: 35337 } attrs { promote: false } } }
I20250112 15:08:27.835007 23095 catalog_manager.cc:5039] ChangeConfig:REMOVE_PEER RPC for tablet 46c5820c43a64ba285e33ee2b607789b with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250112 15:08:27.835150 23108 catalog_manager.cc:5526] T 46c5820c43a64ba285e33ee2b607789b P 5b101dbed8f34e9281a33db399f65ed6 reported cstate change: config changed from index 3 to 4, VOTER 91de225bd4e84a4593199b49347b6d29 (127.17.60.193) evicted. New cstate: current_term: 2 leader_uuid: "5b101dbed8f34e9281a33db399f65ed6" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5b101dbed8f34e9281a33db399f65ed6" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 36341 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b3791458f5554d158fcb770980760849" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 42069 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "036d7a638abc417a9f4360ebfa5061f1" member_type: VOTER last_known_addr { host: "127.17.60.196" port: 35337 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
W20250112 15:08:28.141932 23096 catalog_manager.cc:4615] TS 91de225bd4e84a4593199b49347b6d29 (127.17.60.193:39577): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 656d5437ee6a475a89af5e24b74f315a: Network error: Client connection negotiation failed: client connection to 127.17.60.193:39577: connect: Connection refused (error 111)
W20250112 15:08:29.832954 23096 catalog_manager.cc:4615] TS 91de225bd4e84a4593199b49347b6d29 (127.17.60.193:39577): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet a84082c6bef84e638159590ed624062e: Network error: Client connection negotiation failed: client connection to 127.17.60.193:39577: connect: Connection refused (error 111)
I20250112 15:08:31.455698 17651 tablet_server.cc:178] TabletServer@127.17.60.194:0 shutting down...
I20250112 15:08:31.458715 17651 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250112 15:08:31.458858 17651 tablet_replica.cc:331] T d5a633e37cc84a098547fa042708b1d3 P 5b101dbed8f34e9281a33db399f65ed6: stopping tablet replica
I20250112 15:08:31.458921 17651 raft_consensus.cc:2238] T d5a633e37cc84a098547fa042708b1d3 P 5b101dbed8f34e9281a33db399f65ed6 [term 1 LEADER]: Raft consensus shutting down.
I20250112 15:08:31.458989 17651 raft_consensus.cc:2267] T d5a633e37cc84a098547fa042708b1d3 P 5b101dbed8f34e9281a33db399f65ed6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 15:08:31.459161 17651 tablet_replica.cc:331] T 46c5820c43a64ba285e33ee2b607789b P 5b101dbed8f34e9281a33db399f65ed6: stopping tablet replica
I20250112 15:08:31.459194 17651 raft_consensus.cc:2238] T 46c5820c43a64ba285e33ee2b607789b P 5b101dbed8f34e9281a33db399f65ed6 [term 2 LEADER]: Raft consensus shutting down.
I20250112 15:08:31.459228 17651 raft_consensus.cc:2267] T 46c5820c43a64ba285e33ee2b607789b P 5b101dbed8f34e9281a33db399f65ed6 [term 2 FOLLOWER]: Raft consensus is shut down!
I20250112 15:08:31.459311 17651 tablet_replica.cc:331] T a84082c6bef84e638159590ed624062e P 5b101dbed8f34e9281a33db399f65ed6: stopping tablet replica
I20250112 15:08:31.459335 17651 raft_consensus.cc:2238] T a84082c6bef84e638159590ed624062e P 5b101dbed8f34e9281a33db399f65ed6 [term 2 FOLLOWER]: Raft consensus shutting down.
I20250112 15:08:31.459370 17651 raft_consensus.cc:2267] T a84082c6bef84e638159590ed624062e P 5b101dbed8f34e9281a33db399f65ed6 [term 2 FOLLOWER]: Raft consensus is shut down!
I20250112 15:08:31.460575 17651 tablet_replica.cc:331] T 656d5437ee6a475a89af5e24b74f315a P 5b101dbed8f34e9281a33db399f65ed6: stopping tablet replica
I20250112 15:08:31.460628 17651 raft_consensus.cc:2238] T 656d5437ee6a475a89af5e24b74f315a P 5b101dbed8f34e9281a33db399f65ed6 [term 1 LEADER]: Raft consensus shutting down.
I20250112 15:08:31.460669 17651 raft_consensus.cc:2267] T 656d5437ee6a475a89af5e24b74f315a P 5b101dbed8f34e9281a33db399f65ed6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 15:08:31.472536 17651 tablet_server.cc:195] TabletServer@127.17.60.194:0 shutdown complete.
I20250112 15:08:31.473850 17651 tablet_server.cc:178] TabletServer@127.17.60.195:0 shutting down...
I20250112 15:08:31.476349 17651 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250112 15:08:31.476454 17651 tablet_replica.cc:331] T 656d5437ee6a475a89af5e24b74f315a P b3791458f5554d158fcb770980760849: stopping tablet replica
I20250112 15:08:31.476516 17651 raft_consensus.cc:2238] T 656d5437ee6a475a89af5e24b74f315a P b3791458f5554d158fcb770980760849 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250112 15:08:31.476598 17651 raft_consensus.cc:2267] T 656d5437ee6a475a89af5e24b74f315a P b3791458f5554d158fcb770980760849 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 15:08:31.476760 17651 tablet_replica.cc:331] T a84082c6bef84e638159590ed624062e P b3791458f5554d158fcb770980760849: stopping tablet replica
I20250112 15:08:31.476794 17651 raft_consensus.cc:2238] T a84082c6bef84e638159590ed624062e P b3791458f5554d158fcb770980760849 [term 2 LEADER]: Raft consensus shutting down.
I20250112 15:08:31.476852 17651 raft_consensus.cc:2267] T a84082c6bef84e638159590ed624062e P b3791458f5554d158fcb770980760849 [term 2 FOLLOWER]: Raft consensus is shut down!
I20250112 15:08:31.477089 17651 tablet_replica.cc:331] T 46c5820c43a64ba285e33ee2b607789b P b3791458f5554d158fcb770980760849: stopping tablet replica
I20250112 15:08:31.477145 17651 raft_consensus.cc:2238] T 46c5820c43a64ba285e33ee2b607789b P b3791458f5554d158fcb770980760849 [term 2 FOLLOWER]: Raft consensus shutting down.
I20250112 15:08:31.477222 17651 raft_consensus.cc:2267] T 46c5820c43a64ba285e33ee2b607789b P b3791458f5554d158fcb770980760849 [term 2 FOLLOWER]: Raft consensus is shut down!
I20250112 15:08:31.477358 17651 tablet_replica.cc:331] T d5a633e37cc84a098547fa042708b1d3 P b3791458f5554d158fcb770980760849: stopping tablet replica
I20250112 15:08:31.477391 17651 raft_consensus.cc:2238] T d5a633e37cc84a098547fa042708b1d3 P b3791458f5554d158fcb770980760849 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250112 15:08:31.477438 17651 raft_consensus.cc:2267] T d5a633e37cc84a098547fa042708b1d3 P b3791458f5554d158fcb770980760849 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 15:08:31.489269 17651 tablet_server.cc:195] TabletServer@127.17.60.195:0 shutdown complete.
I20250112 15:08:31.490393 17651 tablet_server.cc:178] TabletServer@127.17.60.196:0 shutting down...
I20250112 15:08:31.492707 17651 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250112 15:08:31.492810 17651 tablet_replica.cc:331] T a84082c6bef84e638159590ed624062e P 036d7a638abc417a9f4360ebfa5061f1: stopping tablet replica
I20250112 15:08:31.492871 17651 raft_consensus.cc:2238] T a84082c6bef84e638159590ed624062e P 036d7a638abc417a9f4360ebfa5061f1 [term 2 FOLLOWER]: Raft consensus shutting down.
I20250112 15:08:31.492931 17651 raft_consensus.cc:2267] T a84082c6bef84e638159590ed624062e P 036d7a638abc417a9f4360ebfa5061f1 [term 2 FOLLOWER]: Raft consensus is shut down!
I20250112 15:08:31.493077 17651 tablet_replica.cc:331] T 656d5437ee6a475a89af5e24b74f315a P 036d7a638abc417a9f4360ebfa5061f1: stopping tablet replica
I20250112 15:08:31.493108 17651 raft_consensus.cc:2238] T 656d5437ee6a475a89af5e24b74f315a P 036d7a638abc417a9f4360ebfa5061f1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250112 15:08:31.493141 17651 raft_consensus.cc:2267] T 656d5437ee6a475a89af5e24b74f315a P 036d7a638abc417a9f4360ebfa5061f1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 15:08:31.493218 17651 tablet_replica.cc:331] T 46c5820c43a64ba285e33ee2b607789b P 036d7a638abc417a9f4360ebfa5061f1: stopping tablet replica
I20250112 15:08:31.493243 17651 raft_consensus.cc:2238] T 46c5820c43a64ba285e33ee2b607789b P 036d7a638abc417a9f4360ebfa5061f1 [term 2 FOLLOWER]: Raft consensus shutting down.
I20250112 15:08:31.493273 17651 raft_consensus.cc:2267] T 46c5820c43a64ba285e33ee2b607789b P 036d7a638abc417a9f4360ebfa5061f1 [term 2 FOLLOWER]: Raft consensus is shut down!
I20250112 15:08:31.493342 17651 tablet_replica.cc:331] T d5a633e37cc84a098547fa042708b1d3 P 036d7a638abc417a9f4360ebfa5061f1: stopping tablet replica
I20250112 15:08:31.493366 17651 raft_consensus.cc:2238] T d5a633e37cc84a098547fa042708b1d3 P 036d7a638abc417a9f4360ebfa5061f1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250112 15:08:31.493395 17651 raft_consensus.cc:2267] T d5a633e37cc84a098547fa042708b1d3 P 036d7a638abc417a9f4360ebfa5061f1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 15:08:31.504859 17651 tablet_server.cc:195] TabletServer@127.17.60.196:0 shutdown complete.
I20250112 15:08:31.505916 17651 master.cc:537] Master@127.17.60.254:39133 shutting down...
W20250112 15:08:31.902475 23096 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.17.60.193:39577: connect: Connection refused (error 111) [suppressed 30 similar messages]
W20250112 15:08:31.902882 23096 catalog_manager.cc:4615] TS 91de225bd4e84a4593199b49347b6d29 (127.17.60.193:39577): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet a84082c6bef84e638159590ed624062e: Network error: Client connection negotiation failed: client connection to 127.17.60.193:39577: connect: Connection refused (error 111)
W20250112 15:08:34.302186 23096 catalog_manager.cc:4615] TS 91de225bd4e84a4593199b49347b6d29 (127.17.60.193:39577): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 656d5437ee6a475a89af5e24b74f315a: Network error: Client connection negotiation failed: client connection to 127.17.60.193:39577: connect: Connection refused (error 111)
I20250112 15:08:35.100284 17651 raft_consensus.cc:2238] T 00000000000000000000000000000000 P c53dc92ba0e24a21aec187711521e9c8 [term 1 LEADER]: Raft consensus shutting down.
I20250112 15:08:35.100423 17651 raft_consensus.cc:2267] T 00000000000000000000000000000000 P c53dc92ba0e24a21aec187711521e9c8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 15:08:35.100450 17651 tablet_replica.cc:331] T 00000000000000000000000000000000 P c53dc92ba0e24a21aec187711521e9c8: stopping tablet replica
I20250112 15:08:35.112326 17651 master.cc:559] Master@127.17.60.254:39133 shutdown complete.
[ OK ] AutoRebalancerTest.NoRebalancingIfReplicasRecovering (13347 ms)
[ RUN ] AutoRebalancerTest.TestHandlingFailedTservers
I20250112 15:08:35.119971 17651 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.17.60.254:42077
I20250112 15:08:35.120206 17651 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250112 15:08:35.122426 23613 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250112 15:08:35.123188 23615 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250112 15:08:35.123553 23612 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250112 15:08:35.124183 17651 server_base.cc:1034] running on GCE node
I20250112 15:08:35.124289 17651 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250112 15:08:35.124332 17651 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250112 15:08:35.124346 17651 hybrid_clock.cc:648] HybridClock initialized: now 1736694515124346 us; error 0 us; skew 500 ppm
I20250112 15:08:35.132297 17651 webserver.cc:458] Webserver started at http://127.17.60.254:46619/ using document root <none> and password file <none>
I20250112 15:08:35.132400 17651 fs_manager.cc:362] Metadata directory not provided
I20250112 15:08:35.132463 17651 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250112 15:08:35.132511 17651 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250112 15:08:35.132823 17651 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task95dLSD/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestHandlingFailedTservers.1736694351070047-17651-0/minicluster-data/master-0-root/instance:
uuid: "d2b58b62446f43f69675af5ece903bf4"
format_stamp: "Formatted at 2025-01-12 15:08:35 on dist-test-slave-b4nz"
I20250112 15:08:35.133780 17651 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20250112 15:08:35.134289 23620 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250112 15:08:35.134397 17651 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20250112 15:08:35.134444 17651 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task95dLSD/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestHandlingFailedTservers.1736694351070047-17651-0/minicluster-data/master-0-root
uuid: "d2b58b62446f43f69675af5ece903bf4"
format_stamp: "Formatted at 2025-01-12 15:08:35 on dist-test-slave-b4nz"
I20250112 15:08:35.134493 17651 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task95dLSD/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestHandlingFailedTservers.1736694351070047-17651-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-task95dLSD/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestHandlingFailedTservers.1736694351070047-17651-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-task95dLSD/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestHandlingFailedTservers.1736694351070047-17651-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250112 15:08:35.142204 17651 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250112 15:08:35.142377 17651 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250112 15:08:35.144889 17651 rpc_server.cc:307] RPC server started. Bound to: 127.17.60.254:42077
I20250112 15:08:35.147809 23672 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.17.60.254:42077 every 8 connection(s)
I20250112 15:08:35.147974 23673 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 15:08:35.149127 23673 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d2b58b62446f43f69675af5ece903bf4: Bootstrap starting.
I20250112 15:08:35.149471 23673 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d2b58b62446f43f69675af5ece903bf4: Neither blocks nor log segments found. Creating new log.
I20250112 15:08:35.149961 23673 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d2b58b62446f43f69675af5ece903bf4: No bootstrap required, opened a new log
I20250112 15:08:35.150087 23673 raft_consensus.cc:357] T 00000000000000000000000000000000 P d2b58b62446f43f69675af5ece903bf4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d2b58b62446f43f69675af5ece903bf4" member_type: VOTER }
I20250112 15:08:35.150139 23673 raft_consensus.cc:383] T 00000000000000000000000000000000 P d2b58b62446f43f69675af5ece903bf4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 15:08:35.150153 23673 raft_consensus.cc:738] T 00000000000000000000000000000000 P d2b58b62446f43f69675af5ece903bf4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d2b58b62446f43f69675af5ece903bf4, State: Initialized, Role: FOLLOWER
I20250112 15:08:35.150208 23673 consensus_queue.cc:260] T 00000000000000000000000000000000 P d2b58b62446f43f69675af5ece903bf4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d2b58b62446f43f69675af5ece903bf4" member_type: VOTER }
I20250112 15:08:35.150247 23673 raft_consensus.cc:397] T 00000000000000000000000000000000 P d2b58b62446f43f69675af5ece903bf4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250112 15:08:35.150290 23673 raft_consensus.cc:491] T 00000000000000000000000000000000 P d2b58b62446f43f69675af5ece903bf4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250112 15:08:35.150317 23673 raft_consensus.cc:3054] T 00000000000000000000000000000000 P d2b58b62446f43f69675af5ece903bf4 [term 0 FOLLOWER]: Advancing to term 1
I20250112 15:08:35.150985 23673 raft_consensus.cc:513] T 00000000000000000000000000000000 P d2b58b62446f43f69675af5ece903bf4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d2b58b62446f43f69675af5ece903bf4" member_type: VOTER }
I20250112 15:08:35.151055 23673 leader_election.cc:304] T 00000000000000000000000000000000 P d2b58b62446f43f69675af5ece903bf4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d2b58b62446f43f69675af5ece903bf4; no voters:
I20250112 15:08:35.151165 23673 leader_election.cc:290] T 00000000000000000000000000000000 P d2b58b62446f43f69675af5ece903bf4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20250112 15:08:35.151206 23676 raft_consensus.cc:2798] T 00000000000000000000000000000000 P d2b58b62446f43f69675af5ece903bf4 [term 1 FOLLOWER]: Leader election won for term 1
I20250112 15:08:35.151337 23676 raft_consensus.cc:695] T 00000000000000000000000000000000 P d2b58b62446f43f69675af5ece903bf4 [term 1 LEADER]: Becoming Leader. State: Replica: d2b58b62446f43f69675af5ece903bf4, State: Running, Role: LEADER
I20250112 15:08:35.151337 23673 sys_catalog.cc:564] T 00000000000000000000000000000000 P d2b58b62446f43f69675af5ece903bf4 [sys.catalog]: configured and running, proceeding with master startup.
I20250112 15:08:35.151504 23676 consensus_queue.cc:237] T 00000000000000000000000000000000 P d2b58b62446f43f69675af5ece903bf4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d2b58b62446f43f69675af5ece903bf4" member_type: VOTER }
I20250112 15:08:35.151772 23677 sys_catalog.cc:455] T 00000000000000000000000000000000 P d2b58b62446f43f69675af5ece903bf4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d2b58b62446f43f69675af5ece903bf4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d2b58b62446f43f69675af5ece903bf4" member_type: VOTER } }
I20250112 15:08:35.151779 23678 sys_catalog.cc:455] T 00000000000000000000000000000000 P d2b58b62446f43f69675af5ece903bf4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d2b58b62446f43f69675af5ece903bf4. Latest consensus state: current_term: 1 leader_uuid: "d2b58b62446f43f69675af5ece903bf4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d2b58b62446f43f69675af5ece903bf4" member_type: VOTER } }
I20250112 15:08:35.151902 23677 sys_catalog.cc:458] T 00000000000000000000000000000000 P d2b58b62446f43f69675af5ece903bf4 [sys.catalog]: This master's current role is: LEADER
I20250112 15:08:35.151947 23678 sys_catalog.cc:458] T 00000000000000000000000000000000 P d2b58b62446f43f69675af5ece903bf4 [sys.catalog]: This master's current role is: LEADER
I20250112 15:08:35.152122 23682 catalog_manager.cc:1476] Loading table and tablet metadata into memory...
I20250112 15:08:35.152251 23682 catalog_manager.cc:1485] Initializing Kudu cluster ID...
I20250112 15:08:35.152822 23682 catalog_manager.cc:1348] Generated new cluster ID: 839459fb1106473bb87053ba00101f58
I20250112 15:08:35.152837 17651 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20250112 15:08:35.152894 23682 catalog_manager.cc:1496] Initializing Kudu internal certificate authority...
I20250112 15:08:35.165124 23682 catalog_manager.cc:1371] Generated new certificate authority record
I20250112 15:08:35.165596 23682 catalog_manager.cc:1505] Loading token signing keys...
I20250112 15:08:35.175484 23682 catalog_manager.cc:5899] T 00000000000000000000000000000000 P d2b58b62446f43f69675af5ece903bf4: Generated new TSK 0
I20250112 15:08:35.175582 23682 catalog_manager.cc:1515] Initializing in-progress tserver states...
I20250112 15:08:35.184703 17651 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250112 15:08:35.186681 23697 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250112 15:08:35.187340 23695 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250112 15:08:35.188102 17651 server_base.cc:1034] running on GCE node
W20250112 15:08:35.188511 23694 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250112 15:08:35.188668 17651 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250112 15:08:35.188702 17651 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250112 15:08:35.188715 17651 hybrid_clock.cc:648] HybridClock initialized: now 1736694515188716 us; error 0 us; skew 500 ppm
I20250112 15:08:35.189361 17651 webserver.cc:458] Webserver started at http://127.17.60.193:36285/ using document root <none> and password file <none>
I20250112 15:08:35.189435 17651 fs_manager.cc:362] Metadata directory not provided
I20250112 15:08:35.189473 17651 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250112 15:08:35.189508 17651 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250112 15:08:35.189779 17651 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task95dLSD/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestHandlingFailedTservers.1736694351070047-17651-0/minicluster-data/ts-0-root/instance:
uuid: "111d62a4d211430fa930ff5a4dfa99d0"
format_stamp: "Formatted at 2025-01-12 15:08:35 on dist-test-slave-b4nz"
I20250112 15:08:35.190634 17651 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20250112 15:08:35.191076 23702 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250112 15:08:35.191200 17651 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20250112 15:08:35.191246 17651 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task95dLSD/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestHandlingFailedTservers.1736694351070047-17651-0/minicluster-data/ts-0-root
uuid: "111d62a4d211430fa930ff5a4dfa99d0"
format_stamp: "Formatted at 2025-01-12 15:08:35 on dist-test-slave-b4nz"
I20250112 15:08:35.191293 17651 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task95dLSD/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestHandlingFailedTservers.1736694351070047-17651-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-task95dLSD/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestHandlingFailedTservers.1736694351070047-17651-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-task95dLSD/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestHandlingFailedTservers.1736694351070047-17651-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250112 15:08:35.212139 17651 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250112 15:08:35.212385 17651 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250112 15:08:35.212540 17651 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250112 15:08:35.212764 17651 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250112 15:08:35.212793 17651 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250112 15:08:35.212826 17651 ts_tablet_manager.cc:610] Registered 0 tablets
I20250112 15:08:35.212852 17651 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250112 15:08:35.215530 17651 rpc_server.cc:307] RPC server started. Bound to: 127.17.60.193:35133
I20250112 15:08:35.215587 23765 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.17.60.193:35133 every 8 connection(s)
I20250112 15:08:35.215904 17651 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250112 15:08:35.217294 23770 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250112 15:08:35.217693 23771 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250112 15:08:35.218401 23773 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250112 15:08:35.219098 17651 server_base.cc:1034] running on GCE node
I20250112 15:08:35.219226 17651 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250112 15:08:35.219276 17651 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250112 15:08:35.219309 17651 hybrid_clock.cc:648] HybridClock initialized: now 1736694515219309 us; error 0 us; skew 500 ppm
I20250112 15:08:35.219767 23766 heartbeater.cc:346] Connected to a master server at 127.17.60.254:42077
I20250112 15:08:35.219830 23766 heartbeater.cc:463] Registering TS with master...
I20250112 15:08:35.219959 23766 heartbeater.cc:510] Master 127.17.60.254:42077 requested a full tablet report, sending...
I20250112 15:08:35.220095 17651 webserver.cc:458] Webserver started at http://127.17.60.194:37787/ using document root <none> and password file <none>
I20250112 15:08:35.220216 17651 fs_manager.cc:362] Metadata directory not provided
I20250112 15:08:35.220223 23637 ts_manager.cc:194] Registered new tserver with Master: 111d62a4d211430fa930ff5a4dfa99d0 (127.17.60.193:35133)
I20250112 15:08:35.220290 17651 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250112 15:08:35.220360 17651 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250112 15:08:35.220695 17651 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task95dLSD/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestHandlingFailedTservers.1736694351070047-17651-0/minicluster-data/ts-1-root/instance:
uuid: "8586159698b04cde88062accfc5d1bf1"
format_stamp: "Formatted at 2025-01-12 15:08:35 on dist-test-slave-b4nz"
I20250112 15:08:35.220932 23637 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:40194
I20250112 15:08:35.221652 17651 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20250112 15:08:35.222110 23778 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250112 15:08:35.222239 17651 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20250112 15:08:35.222282 17651 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task95dLSD/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestHandlingFailedTservers.1736694351070047-17651-0/minicluster-data/ts-1-root
uuid: "8586159698b04cde88062accfc5d1bf1"
format_stamp: "Formatted at 2025-01-12 15:08:35 on dist-test-slave-b4nz"
I20250112 15:08:35.222340 17651 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task95dLSD/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestHandlingFailedTservers.1736694351070047-17651-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-task95dLSD/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestHandlingFailedTservers.1736694351070047-17651-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-task95dLSD/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestHandlingFailedTservers.1736694351070047-17651-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250112 15:08:35.244532 17651 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250112 15:08:35.244819 17651 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250112 15:08:35.244998 17651 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250112 15:08:35.245203 17651 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250112 15:08:35.245230 17651 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250112 15:08:35.245263 17651 ts_tablet_manager.cc:610] Registered 0 tablets
I20250112 15:08:35.245291 17651 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250112 15:08:35.247889 17651 rpc_server.cc:307] RPC server started. Bound to: 127.17.60.194:33901
I20250112 15:08:35.247968 23841 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.17.60.194:33901 every 8 connection(s)
I20250112 15:08:35.248273 17651 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250112 15:08:35.249620 23845 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250112 15:08:35.249998 23846 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250112 15:08:35.250787 23848 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250112 15:08:35.251385 17651 server_base.cc:1034] running on GCE node
I20250112 15:08:35.251534 17651 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250112 15:08:35.251574 17651 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250112 15:08:35.251592 17651 hybrid_clock.cc:648] HybridClock initialized: now 1736694515251592 us; error 0 us; skew 500 ppm
I20250112 15:08:35.251609 23842 heartbeater.cc:346] Connected to a master server at 127.17.60.254:42077
I20250112 15:08:35.251675 23842 heartbeater.cc:463] Registering TS with master...
I20250112 15:08:35.251787 23842 heartbeater.cc:510] Master 127.17.60.254:42077 requested a full tablet report, sending...
I20250112 15:08:35.252032 23637 ts_manager.cc:194] Registered new tserver with Master: 8586159698b04cde88062accfc5d1bf1 (127.17.60.194:33901)
I20250112 15:08:35.252382 17651 webserver.cc:458] Webserver started at http://127.17.60.195:41529/ using document root <none> and password file <none>
I20250112 15:08:35.252467 17651 fs_manager.cc:362] Metadata directory not provided
I20250112 15:08:35.252514 17651 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250112 15:08:35.252501 23637 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:40210
I20250112 15:08:35.252611 17651 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250112 15:08:35.252893 17651 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task95dLSD/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestHandlingFailedTservers.1736694351070047-17651-0/minicluster-data/ts-2-root/instance:
uuid: "05ca05df85b94bc985b549e68caa6c36"
format_stamp: "Formatted at 2025-01-12 15:08:35 on dist-test-slave-b4nz"
I20250112 15:08:35.253736 17651 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20250112 15:08:35.254154 23853 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250112 15:08:35.254254 17651 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20250112 15:08:35.254304 17651 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task95dLSD/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestHandlingFailedTservers.1736694351070047-17651-0/minicluster-data/ts-2-root
uuid: "05ca05df85b94bc985b549e68caa6c36"
format_stamp: "Formatted at 2025-01-12 15:08:35 on dist-test-slave-b4nz"
I20250112 15:08:35.254346 17651 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task95dLSD/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestHandlingFailedTservers.1736694351070047-17651-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-task95dLSD/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestHandlingFailedTservers.1736694351070047-17651-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-task95dLSD/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestHandlingFailedTservers.1736694351070047-17651-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250112 15:08:35.265565 17651 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250112 15:08:35.265821 17651 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250112 15:08:35.265975 17651 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250112 15:08:35.266180 17651 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250112 15:08:35.266207 17651 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250112 15:08:35.266243 17651 ts_tablet_manager.cc:610] Registered 0 tablets
I20250112 15:08:35.266275 17651 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250112 15:08:35.269517 17651 rpc_server.cc:307] RPC server started. Bound to: 127.17.60.195:44611
I20250112 15:08:35.269584 23916 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.17.60.195:44611 every 8 connection(s)
I20250112 15:08:35.272668 23917 heartbeater.cc:346] Connected to a master server at 127.17.60.254:42077
I20250112 15:08:35.272734 23917 heartbeater.cc:463] Registering TS with master...
I20250112 15:08:35.272840 23917 heartbeater.cc:510] Master 127.17.60.254:42077 requested a full tablet report, sending...
I20250112 15:08:35.273090 23637 ts_manager.cc:194] Registered new tserver with Master: 05ca05df85b94bc985b549e68caa6c36 (127.17.60.195:44611)
I20250112 15:08:35.273440 23637 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:40222
I20250112 15:08:35.274101 17651 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.004399076s
I20250112 15:08:36.221424 23766 heartbeater.cc:502] Master 127.17.60.254:42077 was elected leader, sending a full tablet report...
I20250112 15:08:36.253093 23842 heartbeater.cc:502] Master 127.17.60.254:42077 was elected leader, sending a full tablet report...
I20250112 15:08:36.273950 23917 heartbeater.cc:502] Master 127.17.60.254:42077 was elected leader, sending a full tablet report...
I20250112 15:08:36.299306 17651 test_util.cc:274] Using random seed: -1279642074
I20250112 15:08:36.304355 23637 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40230:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "\004\001\000\377\377\377\037\004\001\000\376\377\377?\004\001\000\375\377\377_""\004\001\000\377\377\377\037\004\001\000\376\377\377?\004\001\000\375\377\377_"
indirect_data: """"
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 15:08:36.304524 23637 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-workload in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 15:08:36.309583 23806 tablet_service.cc:1467] Processing CreateTablet for tablet 3e43528d2bb24563966ef89a76aa761e (DEFAULT_TABLE table=test-workload [id=f6f10825180e4ef88cec34111ab8cdcd]), partition=RANGE (key) PARTITION VALUES < 536870911
I20250112 15:08:36.309751 23806 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3e43528d2bb24563966ef89a76aa761e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 15:08:36.310002 23730 tablet_service.cc:1467] Processing CreateTablet for tablet 3e43528d2bb24563966ef89a76aa761e (DEFAULT_TABLE table=test-workload [id=f6f10825180e4ef88cec34111ab8cdcd]), partition=RANGE (key) PARTITION VALUES < 536870911
I20250112 15:08:36.310045 23729 tablet_service.cc:1467] Processing CreateTablet for tablet 92c129e062b945df8805bcc412f3a599 (DEFAULT_TABLE table=test-workload [id=f6f10825180e4ef88cec34111ab8cdcd]), partition=RANGE (key) PARTITION 536870911 <= VALUES < 1073741822
I20250112 15:08:36.310086 23727 tablet_service.cc:1467] Processing CreateTablet for tablet 6a1c881a80004842a1eb46bb488d3834 (DEFAULT_TABLE table=test-workload [id=f6f10825180e4ef88cec34111ab8cdcd]), partition=RANGE (key) PARTITION 1610612733 <= VALUES
I20250112 15:08:36.310138 23730 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3e43528d2bb24563966ef89a76aa761e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 15:08:36.310055 23728 tablet_service.cc:1467] Processing CreateTablet for tablet 483cfa86b848433bbc365bae489ed251 (DEFAULT_TABLE table=test-workload [id=f6f10825180e4ef88cec34111ab8cdcd]), partition=RANGE (key) PARTITION 1073741822 <= VALUES < 1610612733
I20250112 15:08:36.310238 23727 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6a1c881a80004842a1eb46bb488d3834. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 15:08:36.310326 23728 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 483cfa86b848433bbc365bae489ed251. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 15:08:36.311399 23805 tablet_service.cc:1467] Processing CreateTablet for tablet 92c129e062b945df8805bcc412f3a599 (DEFAULT_TABLE table=test-workload [id=f6f10825180e4ef88cec34111ab8cdcd]), partition=RANGE (key) PARTITION 536870911 <= VALUES < 1073741822
I20250112 15:08:36.311491 23805 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 92c129e062b945df8805bcc412f3a599. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 15:08:36.311770 23729 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 92c129e062b945df8805bcc412f3a599. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 15:08:36.311802 23937 tablet_bootstrap.cc:492] T 3e43528d2bb24563966ef89a76aa761e P 8586159698b04cde88062accfc5d1bf1: Bootstrap starting.
I20250112 15:08:36.312304 23937 tablet_bootstrap.cc:654] T 3e43528d2bb24563966ef89a76aa761e P 8586159698b04cde88062accfc5d1bf1: Neither blocks nor log segments found. Creating new log.
I20250112 15:08:36.312798 23804 tablet_service.cc:1467] Processing CreateTablet for tablet 483cfa86b848433bbc365bae489ed251 (DEFAULT_TABLE table=test-workload [id=f6f10825180e4ef88cec34111ab8cdcd]), partition=RANGE (key) PARTITION 1073741822 <= VALUES < 1610612733
I20250112 15:08:36.312871 23937 tablet_bootstrap.cc:492] T 3e43528d2bb24563966ef89a76aa761e P 8586159698b04cde88062accfc5d1bf1: No bootstrap required, opened a new log
I20250112 15:08:36.312886 23804 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 483cfa86b848433bbc365bae489ed251. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 15:08:36.312911 23937 ts_tablet_manager.cc:1397] T 3e43528d2bb24563966ef89a76aa761e P 8586159698b04cde88062accfc5d1bf1: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20250112 15:08:36.313057 23937 raft_consensus.cc:357] T 3e43528d2bb24563966ef89a76aa761e P 8586159698b04cde88062accfc5d1bf1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "111d62a4d211430fa930ff5a4dfa99d0" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 35133 } } peers { permanent_uuid: "05ca05df85b94bc985b549e68caa6c36" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 44611 } } peers { permanent_uuid: "8586159698b04cde88062accfc5d1bf1" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 33901 } }
I20250112 15:08:36.313133 23937 raft_consensus.cc:383] T 3e43528d2bb24563966ef89a76aa761e P 8586159698b04cde88062accfc5d1bf1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 15:08:36.313151 23937 raft_consensus.cc:738] T 3e43528d2bb24563966ef89a76aa761e P 8586159698b04cde88062accfc5d1bf1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8586159698b04cde88062accfc5d1bf1, State: Initialized, Role: FOLLOWER
I20250112 15:08:36.313202 23937 consensus_queue.cc:260] T 3e43528d2bb24563966ef89a76aa761e P 8586159698b04cde88062accfc5d1bf1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "111d62a4d211430fa930ff5a4dfa99d0" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 35133 } } peers { permanent_uuid: "05ca05df85b94bc985b549e68caa6c36" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 44611 } } peers { permanent_uuid: "8586159698b04cde88062accfc5d1bf1" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 33901 } }
I20250112 15:08:36.313369 23937 ts_tablet_manager.cc:1428] T 3e43528d2bb24563966ef89a76aa761e P 8586159698b04cde88062accfc5d1bf1: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20250112 15:08:36.313426 23937 tablet_bootstrap.cc:492] T 92c129e062b945df8805bcc412f3a599 P 8586159698b04cde88062accfc5d1bf1: Bootstrap starting.
I20250112 15:08:36.313798 23937 tablet_bootstrap.cc:654] T 92c129e062b945df8805bcc412f3a599 P 8586159698b04cde88062accfc5d1bf1: Neither blocks nor log segments found. Creating new log.
I20250112 15:08:36.314167 23803 tablet_service.cc:1467] Processing CreateTablet for tablet 6a1c881a80004842a1eb46bb488d3834 (DEFAULT_TABLE table=test-workload [id=f6f10825180e4ef88cec34111ab8cdcd]), partition=RANGE (key) PARTITION 1610612733 <= VALUES
I20250112 15:08:36.314253 23803 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6a1c881a80004842a1eb46bb488d3834. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 15:08:36.314340 23937 tablet_bootstrap.cc:492] T 92c129e062b945df8805bcc412f3a599 P 8586159698b04cde88062accfc5d1bf1: No bootstrap required, opened a new log
I20250112 15:08:36.314384 23937 ts_tablet_manager.cc:1397] T 92c129e062b945df8805bcc412f3a599 P 8586159698b04cde88062accfc5d1bf1: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20250112 15:08:36.314556 23937 raft_consensus.cc:357] T 92c129e062b945df8805bcc412f3a599 P 8586159698b04cde88062accfc5d1bf1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "111d62a4d211430fa930ff5a4dfa99d0" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 35133 } } peers { permanent_uuid: "8586159698b04cde88062accfc5d1bf1" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 33901 } } peers { permanent_uuid: "05ca05df85b94bc985b549e68caa6c36" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 44611 } }
I20250112 15:08:36.314641 23937 raft_consensus.cc:383] T 92c129e062b945df8805bcc412f3a599 P 8586159698b04cde88062accfc5d1bf1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 15:08:36.314678 23937 raft_consensus.cc:738] T 92c129e062b945df8805bcc412f3a599 P 8586159698b04cde88062accfc5d1bf1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8586159698b04cde88062accfc5d1bf1, State: Initialized, Role: FOLLOWER
I20250112 15:08:36.314720 23937 consensus_queue.cc:260] T 92c129e062b945df8805bcc412f3a599 P 8586159698b04cde88062accfc5d1bf1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "111d62a4d211430fa930ff5a4dfa99d0" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 35133 } } peers { permanent_uuid: "8586159698b04cde88062accfc5d1bf1" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 33901 } } peers { permanent_uuid: "05ca05df85b94bc985b549e68caa6c36" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 44611 } }
I20250112 15:08:36.314808 23937 ts_tablet_manager.cc:1428] T 92c129e062b945df8805bcc412f3a599 P 8586159698b04cde88062accfc5d1bf1: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20250112 15:08:36.314865 23937 tablet_bootstrap.cc:492] T 483cfa86b848433bbc365bae489ed251 P 8586159698b04cde88062accfc5d1bf1: Bootstrap starting.
I20250112 15:08:36.315151 23937 tablet_bootstrap.cc:654] T 483cfa86b848433bbc365bae489ed251 P 8586159698b04cde88062accfc5d1bf1: Neither blocks nor log segments found. Creating new log.
I20250112 15:08:36.315629 23937 tablet_bootstrap.cc:492] T 483cfa86b848433bbc365bae489ed251 P 8586159698b04cde88062accfc5d1bf1: No bootstrap required, opened a new log
I20250112 15:08:36.315670 23937 ts_tablet_manager.cc:1397] T 483cfa86b848433bbc365bae489ed251 P 8586159698b04cde88062accfc5d1bf1: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20250112 15:08:36.315765 23937 raft_consensus.cc:357] T 483cfa86b848433bbc365bae489ed251 P 8586159698b04cde88062accfc5d1bf1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "111d62a4d211430fa930ff5a4dfa99d0" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 35133 } } peers { permanent_uuid: "8586159698b04cde88062accfc5d1bf1" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 33901 } } peers { permanent_uuid: "05ca05df85b94bc985b549e68caa6c36" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 44611 } }
I20250112 15:08:36.315810 23937 raft_consensus.cc:383] T 483cfa86b848433bbc365bae489ed251 P 8586159698b04cde88062accfc5d1bf1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 15:08:36.315833 23937 raft_consensus.cc:738] T 483cfa86b848433bbc365bae489ed251 P 8586159698b04cde88062accfc5d1bf1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8586159698b04cde88062accfc5d1bf1, State: Initialized, Role: FOLLOWER
I20250112 15:08:36.315876 23937 consensus_queue.cc:260] T 483cfa86b848433bbc365bae489ed251 P 8586159698b04cde88062accfc5d1bf1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "111d62a4d211430fa930ff5a4dfa99d0" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 35133 } } peers { permanent_uuid: "8586159698b04cde88062accfc5d1bf1" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 33901 } } peers { permanent_uuid: "05ca05df85b94bc985b549e68caa6c36" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 44611 } }
I20250112 15:08:36.315958 23937 ts_tablet_manager.cc:1428] T 483cfa86b848433bbc365bae489ed251 P 8586159698b04cde88062accfc5d1bf1: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20250112 15:08:36.316007 23937 tablet_bootstrap.cc:492] T 6a1c881a80004842a1eb46bb488d3834 P 8586159698b04cde88062accfc5d1bf1: Bootstrap starting.
I20250112 15:08:36.316110 23938 tablet_bootstrap.cc:492] T 6a1c881a80004842a1eb46bb488d3834 P 111d62a4d211430fa930ff5a4dfa99d0: Bootstrap starting.
I20250112 15:08:36.316217 23881 tablet_service.cc:1467] Processing CreateTablet for tablet 3e43528d2bb24563966ef89a76aa761e (DEFAULT_TABLE table=test-workload [id=f6f10825180e4ef88cec34111ab8cdcd]), partition=RANGE (key) PARTITION VALUES < 536870911
I20250112 15:08:36.316311 23881 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3e43528d2bb24563966ef89a76aa761e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 15:08:36.316385 23937 tablet_bootstrap.cc:654] T 6a1c881a80004842a1eb46bb488d3834 P 8586159698b04cde88062accfc5d1bf1: Neither blocks nor log segments found. Creating new log.
I20250112 15:08:36.316625 23938 tablet_bootstrap.cc:654] T 6a1c881a80004842a1eb46bb488d3834 P 111d62a4d211430fa930ff5a4dfa99d0: Neither blocks nor log segments found. Creating new log.
I20250112 15:08:36.316908 23880 tablet_service.cc:1467] Processing CreateTablet for tablet 92c129e062b945df8805bcc412f3a599 (DEFAULT_TABLE table=test-workload [id=f6f10825180e4ef88cec34111ab8cdcd]), partition=RANGE (key) PARTITION 536870911 <= VALUES < 1073741822
I20250112 15:08:36.317001 23878 tablet_service.cc:1467] Processing CreateTablet for tablet 6a1c881a80004842a1eb46bb488d3834 (DEFAULT_TABLE table=test-workload [id=f6f10825180e4ef88cec34111ab8cdcd]), partition=RANGE (key) PARTITION 1610612733 <= VALUES
I20250112 15:08:36.317035 23880 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 92c129e062b945df8805bcc412f3a599. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 15:08:36.317096 23878 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6a1c881a80004842a1eb46bb488d3834. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 15:08:36.317126 23879 tablet_service.cc:1467] Processing CreateTablet for tablet 483cfa86b848433bbc365bae489ed251 (DEFAULT_TABLE table=test-workload [id=f6f10825180e4ef88cec34111ab8cdcd]), partition=RANGE (key) PARTITION 1073741822 <= VALUES < 1610612733
I20250112 15:08:36.317217 23879 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 483cfa86b848433bbc365bae489ed251. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 15:08:36.317374 23938 tablet_bootstrap.cc:492] T 6a1c881a80004842a1eb46bb488d3834 P 111d62a4d211430fa930ff5a4dfa99d0: No bootstrap required, opened a new log
I20250112 15:08:36.317409 23938 ts_tablet_manager.cc:1397] T 6a1c881a80004842a1eb46bb488d3834 P 111d62a4d211430fa930ff5a4dfa99d0: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20250112 15:08:36.317535 23938 raft_consensus.cc:357] T 6a1c881a80004842a1eb46bb488d3834 P 111d62a4d211430fa930ff5a4dfa99d0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "111d62a4d211430fa930ff5a4dfa99d0" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 35133 } } peers { permanent_uuid: "8586159698b04cde88062accfc5d1bf1" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 33901 } } peers { permanent_uuid: "05ca05df85b94bc985b549e68caa6c36" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 44611 } }
I20250112 15:08:36.317577 23938 raft_consensus.cc:383] T 6a1c881a80004842a1eb46bb488d3834 P 111d62a4d211430fa930ff5a4dfa99d0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 15:08:36.317593 23938 raft_consensus.cc:738] T 6a1c881a80004842a1eb46bb488d3834 P 111d62a4d211430fa930ff5a4dfa99d0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 111d62a4d211430fa930ff5a4dfa99d0, State: Initialized, Role: FOLLOWER
I20250112 15:08:36.317634 23938 consensus_queue.cc:260] T 6a1c881a80004842a1eb46bb488d3834 P 111d62a4d211430fa930ff5a4dfa99d0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "111d62a4d211430fa930ff5a4dfa99d0" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 35133 } } peers { permanent_uuid: "8586159698b04cde88062accfc5d1bf1" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 33901 } } peers { permanent_uuid: "05ca05df85b94bc985b549e68caa6c36" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 44611 } }
I20250112 15:08:36.317767 23938 ts_tablet_manager.cc:1428] T 6a1c881a80004842a1eb46bb488d3834 P 111d62a4d211430fa930ff5a4dfa99d0: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20250112 15:08:36.317827 23938 tablet_bootstrap.cc:492] T 3e43528d2bb24563966ef89a76aa761e P 111d62a4d211430fa930ff5a4dfa99d0: Bootstrap starting.
I20250112 15:08:36.318228 23938 tablet_bootstrap.cc:654] T 3e43528d2bb24563966ef89a76aa761e P 111d62a4d211430fa930ff5a4dfa99d0: Neither blocks nor log segments found. Creating new log.
I20250112 15:08:36.318540 23946 tablet_bootstrap.cc:492] T 3e43528d2bb24563966ef89a76aa761e P 05ca05df85b94bc985b549e68caa6c36: Bootstrap starting.
I20250112 15:08:36.318902 23938 tablet_bootstrap.cc:492] T 3e43528d2bb24563966ef89a76aa761e P 111d62a4d211430fa930ff5a4dfa99d0: No bootstrap required, opened a new log
I20250112 15:08:36.318962 23938 ts_tablet_manager.cc:1397] T 3e43528d2bb24563966ef89a76aa761e P 111d62a4d211430fa930ff5a4dfa99d0: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20250112 15:08:36.319046 23946 tablet_bootstrap.cc:654] T 3e43528d2bb24563966ef89a76aa761e P 05ca05df85b94bc985b549e68caa6c36: Neither blocks nor log segments found. Creating new log.
I20250112 15:08:36.319113 23938 raft_consensus.cc:357] T 3e43528d2bb24563966ef89a76aa761e P 111d62a4d211430fa930ff5a4dfa99d0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "111d62a4d211430fa930ff5a4dfa99d0" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 35133 } } peers { permanent_uuid: "05ca05df85b94bc985b549e68caa6c36" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 44611 } } peers { permanent_uuid: "8586159698b04cde88062accfc5d1bf1" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 33901 } }
I20250112 15:08:36.319168 23938 raft_consensus.cc:383] T 3e43528d2bb24563966ef89a76aa761e P 111d62a4d211430fa930ff5a4dfa99d0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 15:08:36.319191 23938 raft_consensus.cc:738] T 3e43528d2bb24563966ef89a76aa761e P 111d62a4d211430fa930ff5a4dfa99d0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 111d62a4d211430fa930ff5a4dfa99d0, State: Initialized, Role: FOLLOWER
I20250112 15:08:36.319243 23938 consensus_queue.cc:260] T 3e43528d2bb24563966ef89a76aa761e P 111d62a4d211430fa930ff5a4dfa99d0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "111d62a4d211430fa930ff5a4dfa99d0" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 35133 } } peers { permanent_uuid: "05ca05df85b94bc985b549e68caa6c36" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 44611 } } peers { permanent_uuid: "8586159698b04cde88062accfc5d1bf1" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 33901 } }
I20250112 15:08:36.319384 23938 ts_tablet_manager.cc:1428] T 3e43528d2bb24563966ef89a76aa761e P 111d62a4d211430fa930ff5a4dfa99d0: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20250112 15:08:36.319444 23938 tablet_bootstrap.cc:492] T 483cfa86b848433bbc365bae489ed251 P 111d62a4d211430fa930ff5a4dfa99d0: Bootstrap starting.
I20250112 15:08:36.319612 23937 tablet_bootstrap.cc:492] T 6a1c881a80004842a1eb46bb488d3834 P 8586159698b04cde88062accfc5d1bf1: No bootstrap required, opened a new log
I20250112 15:08:36.319612 23946 tablet_bootstrap.cc:492] T 3e43528d2bb24563966ef89a76aa761e P 05ca05df85b94bc985b549e68caa6c36: No bootstrap required, opened a new log
I20250112 15:08:36.319675 23937 ts_tablet_manager.cc:1397] T 6a1c881a80004842a1eb46bb488d3834 P 8586159698b04cde88062accfc5d1bf1: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.001s
I20250112 15:08:36.319684 23946 ts_tablet_manager.cc:1397] T 3e43528d2bb24563966ef89a76aa761e P 05ca05df85b94bc985b549e68caa6c36: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20250112 15:08:36.319784 23938 tablet_bootstrap.cc:654] T 483cfa86b848433bbc365bae489ed251 P 111d62a4d211430fa930ff5a4dfa99d0: Neither blocks nor log segments found. Creating new log.
I20250112 15:08:36.319833 23937 raft_consensus.cc:357] T 6a1c881a80004842a1eb46bb488d3834 P 8586159698b04cde88062accfc5d1bf1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "111d62a4d211430fa930ff5a4dfa99d0" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 35133 } } peers { permanent_uuid: "8586159698b04cde88062accfc5d1bf1" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 33901 } } peers { permanent_uuid: "05ca05df85b94bc985b549e68caa6c36" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 44611 } }
I20250112 15:08:36.319836 23946 raft_consensus.cc:357] T 3e43528d2bb24563966ef89a76aa761e P 05ca05df85b94bc985b549e68caa6c36 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "111d62a4d211430fa930ff5a4dfa99d0" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 35133 } } peers { permanent_uuid: "05ca05df85b94bc985b549e68caa6c36" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 44611 } } peers { permanent_uuid: "8586159698b04cde88062accfc5d1bf1" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 33901 } }
I20250112 15:08:36.319908 23937 raft_consensus.cc:383] T 6a1c881a80004842a1eb46bb488d3834 P 8586159698b04cde88062accfc5d1bf1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 15:08:36.319936 23937 raft_consensus.cc:738] T 6a1c881a80004842a1eb46bb488d3834 P 8586159698b04cde88062accfc5d1bf1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8586159698b04cde88062accfc5d1bf1, State: Initialized, Role: FOLLOWER
I20250112 15:08:36.319983 23937 consensus_queue.cc:260] T 6a1c881a80004842a1eb46bb488d3834 P 8586159698b04cde88062accfc5d1bf1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "111d62a4d211430fa930ff5a4dfa99d0" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 35133 } } peers { permanent_uuid: "8586159698b04cde88062accfc5d1bf1" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 33901 } } peers { permanent_uuid: "05ca05df85b94bc985b549e68caa6c36" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 44611 } }
I20250112 15:08:36.319916 23946 raft_consensus.cc:383] T 3e43528d2bb24563966ef89a76aa761e P 05ca05df85b94bc985b549e68caa6c36 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 15:08:36.320065 23937 ts_tablet_manager.cc:1428] T 6a1c881a80004842a1eb46bb488d3834 P 8586159698b04cde88062accfc5d1bf1: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20250112 15:08:36.320070 23946 raft_consensus.cc:738] T 3e43528d2bb24563966ef89a76aa761e P 05ca05df85b94bc985b549e68caa6c36 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 05ca05df85b94bc985b549e68caa6c36, State: Initialized, Role: FOLLOWER
I20250112 15:08:36.320173 23946 consensus_queue.cc:260] T 3e43528d2bb24563966ef89a76aa761e P 05ca05df85b94bc985b549e68caa6c36 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "111d62a4d211430fa930ff5a4dfa99d0" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 35133 } } peers { permanent_uuid: "05ca05df85b94bc985b549e68caa6c36" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 44611 } } peers { permanent_uuid: "8586159698b04cde88062accfc5d1bf1" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 33901 } }
I20250112 15:08:36.320323 23946 ts_tablet_manager.cc:1428] T 3e43528d2bb24563966ef89a76aa761e P 05ca05df85b94bc985b549e68caa6c36: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20250112 15:08:36.320394 23946 tablet_bootstrap.cc:492] T 6a1c881a80004842a1eb46bb488d3834 P 05ca05df85b94bc985b549e68caa6c36: Bootstrap starting.
I20250112 15:08:36.320853 23946 tablet_bootstrap.cc:654] T 6a1c881a80004842a1eb46bb488d3834 P 05ca05df85b94bc985b549e68caa6c36: Neither blocks nor log segments found. Creating new log.
I20250112 15:08:36.321175 23938 tablet_bootstrap.cc:492] T 483cfa86b848433bbc365bae489ed251 P 111d62a4d211430fa930ff5a4dfa99d0: No bootstrap required, opened a new log
I20250112 15:08:36.321233 23938 ts_tablet_manager.cc:1397] T 483cfa86b848433bbc365bae489ed251 P 111d62a4d211430fa930ff5a4dfa99d0: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20250112 15:08:36.321382 23938 raft_consensus.cc:357] T 483cfa86b848433bbc365bae489ed251 P 111d62a4d211430fa930ff5a4dfa99d0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "111d62a4d211430fa930ff5a4dfa99d0" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 35133 } } peers { permanent_uuid: "8586159698b04cde88062accfc5d1bf1" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 33901 } } peers { permanent_uuid: "05ca05df85b94bc985b549e68caa6c36" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 44611 } }
I20250112 15:08:36.321445 23938 raft_consensus.cc:383] T 483cfa86b848433bbc365bae489ed251 P 111d62a4d211430fa930ff5a4dfa99d0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 15:08:36.321460 23938 raft_consensus.cc:738] T 483cfa86b848433bbc365bae489ed251 P 111d62a4d211430fa930ff5a4dfa99d0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 111d62a4d211430fa930ff5a4dfa99d0, State: Initialized, Role: FOLLOWER
I20250112 15:08:36.321499 23938 consensus_queue.cc:260] T 483cfa86b848433bbc365bae489ed251 P 111d62a4d211430fa930ff5a4dfa99d0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "111d62a4d211430fa930ff5a4dfa99d0" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 35133 } } peers { permanent_uuid: "8586159698b04cde88062accfc5d1bf1" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 33901 } } peers { permanent_uuid: "05ca05df85b94bc985b549e68caa6c36" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 44611 } }
I20250112 15:08:36.321624 23938 ts_tablet_manager.cc:1428] T 483cfa86b848433bbc365bae489ed251 P 111d62a4d211430fa930ff5a4dfa99d0: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20250112 15:08:36.321689 23938 tablet_bootstrap.cc:492] T 92c129e062b945df8805bcc412f3a599 P 111d62a4d211430fa930ff5a4dfa99d0: Bootstrap starting.
I20250112 15:08:36.322037 23938 tablet_bootstrap.cc:654] T 92c129e062b945df8805bcc412f3a599 P 111d62a4d211430fa930ff5a4dfa99d0: Neither blocks nor log segments found. Creating new log.
I20250112 15:08:36.321486 23946 tablet_bootstrap.cc:492] T 6a1c881a80004842a1eb46bb488d3834 P 05ca05df85b94bc985b549e68caa6c36: No bootstrap required, opened a new log
I20250112 15:08:36.322331 23946 ts_tablet_manager.cc:1397] T 6a1c881a80004842a1eb46bb488d3834 P 05ca05df85b94bc985b549e68caa6c36: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20250112 15:08:36.322472 23946 raft_consensus.cc:357] T 6a1c881a80004842a1eb46bb488d3834 P 05ca05df85b94bc985b549e68caa6c36 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "111d62a4d211430fa930ff5a4dfa99d0" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 35133 } } peers { permanent_uuid: "8586159698b04cde88062accfc5d1bf1" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 33901 } } peers { permanent_uuid: "05ca05df85b94bc985b549e68caa6c36" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 44611 } }
I20250112 15:08:36.322520 23946 raft_consensus.cc:383] T 6a1c881a80004842a1eb46bb488d3834 P 05ca05df85b94bc985b549e68caa6c36 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 15:08:36.322543 23946 raft_consensus.cc:738] T 6a1c881a80004842a1eb46bb488d3834 P 05ca05df85b94bc985b549e68caa6c36 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 05ca05df85b94bc985b549e68caa6c36, State: Initialized, Role: FOLLOWER
I20250112 15:08:36.322566 23938 tablet_bootstrap.cc:492] T 92c129e062b945df8805bcc412f3a599 P 111d62a4d211430fa930ff5a4dfa99d0: No bootstrap required, opened a new log
I20250112 15:08:36.322595 23946 consensus_queue.cc:260] T 6a1c881a80004842a1eb46bb488d3834 P 05ca05df85b94bc985b549e68caa6c36 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "111d62a4d211430fa930ff5a4dfa99d0" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 35133 } } peers { permanent_uuid: "8586159698b04cde88062accfc5d1bf1" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 33901 } } peers { permanent_uuid: "05ca05df85b94bc985b549e68caa6c36" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 44611 } }
I20250112 15:08:36.322613 23938 ts_tablet_manager.cc:1397] T 92c129e062b945df8805bcc412f3a599 P 111d62a4d211430fa930ff5a4dfa99d0: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20250112 15:08:36.322695 23946 ts_tablet_manager.cc:1428] T 6a1c881a80004842a1eb46bb488d3834 P 05ca05df85b94bc985b549e68caa6c36: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20250112 15:08:36.322834 23946 tablet_bootstrap.cc:492] T 483cfa86b848433bbc365bae489ed251 P 05ca05df85b94bc985b549e68caa6c36: Bootstrap starting.
I20250112 15:08:36.322810 23938 raft_consensus.cc:357] T 92c129e062b945df8805bcc412f3a599 P 111d62a4d211430fa930ff5a4dfa99d0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "111d62a4d211430fa930ff5a4dfa99d0" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 35133 } } peers { permanent_uuid: "8586159698b04cde88062accfc5d1bf1" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 33901 } } peers { permanent_uuid: "05ca05df85b94bc985b549e68caa6c36" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 44611 } }
I20250112 15:08:36.322868 23938 raft_consensus.cc:383] T 92c129e062b945df8805bcc412f3a599 P 111d62a4d211430fa930ff5a4dfa99d0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 15:08:36.322887 23938 raft_consensus.cc:738] T 92c129e062b945df8805bcc412f3a599 P 111d62a4d211430fa930ff5a4dfa99d0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 111d62a4d211430fa930ff5a4dfa99d0, State: Initialized, Role: FOLLOWER
I20250112 15:08:36.322922 23938 consensus_queue.cc:260] T 92c129e062b945df8805bcc412f3a599 P 111d62a4d211430fa930ff5a4dfa99d0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "111d62a4d211430fa930ff5a4dfa99d0" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 35133 } } peers { permanent_uuid: "8586159698b04cde88062accfc5d1bf1" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 33901 } } peers { permanent_uuid: "05ca05df85b94bc985b549e68caa6c36" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 44611 } }
I20250112 15:08:36.323004 23938 ts_tablet_manager.cc:1428] T 92c129e062b945df8805bcc412f3a599 P 111d62a4d211430fa930ff5a4dfa99d0: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20250112 15:08:36.323292 23946 tablet_bootstrap.cc:654] T 483cfa86b848433bbc365bae489ed251 P 05ca05df85b94bc985b549e68caa6c36: Neither blocks nor log segments found. Creating new log.
I20250112 15:08:36.323763 23946 tablet_bootstrap.cc:492] T 483cfa86b848433bbc365bae489ed251 P 05ca05df85b94bc985b549e68caa6c36: No bootstrap required, opened a new log
I20250112 15:08:36.323800 23946 ts_tablet_manager.cc:1397] T 483cfa86b848433bbc365bae489ed251 P 05ca05df85b94bc985b549e68caa6c36: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20250112 15:08:36.323933 23946 raft_consensus.cc:357] T 483cfa86b848433bbc365bae489ed251 P 05ca05df85b94bc985b549e68caa6c36 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "111d62a4d211430fa930ff5a4dfa99d0" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 35133 } } peers { permanent_uuid: "8586159698b04cde88062accfc5d1bf1" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 33901 } } peers { permanent_uuid: "05ca05df85b94bc985b549e68caa6c36" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 44611 } }
I20250112 15:08:36.323984 23946 raft_consensus.cc:383] T 483cfa86b848433bbc365bae489ed251 P 05ca05df85b94bc985b549e68caa6c36 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 15:08:36.324007 23946 raft_consensus.cc:738] T 483cfa86b848433bbc365bae489ed251 P 05ca05df85b94bc985b549e68caa6c36 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 05ca05df85b94bc985b549e68caa6c36, State: Initialized, Role: FOLLOWER
I20250112 15:08:36.324055 23946 consensus_queue.cc:260] T 483cfa86b848433bbc365bae489ed251 P 05ca05df85b94bc985b549e68caa6c36 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "111d62a4d211430fa930ff5a4dfa99d0" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 35133 } } peers { permanent_uuid: "8586159698b04cde88062accfc5d1bf1" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 33901 } } peers { permanent_uuid: "05ca05df85b94bc985b549e68caa6c36" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 44611 } }
I20250112 15:08:36.324141 23946 ts_tablet_manager.cc:1428] T 483cfa86b848433bbc365bae489ed251 P 05ca05df85b94bc985b549e68caa6c36: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20250112 15:08:36.324195 23946 tablet_bootstrap.cc:492] T 92c129e062b945df8805bcc412f3a599 P 05ca05df85b94bc985b549e68caa6c36: Bootstrap starting.
I20250112 15:08:36.324529 23946 tablet_bootstrap.cc:654] T 92c129e062b945df8805bcc412f3a599 P 05ca05df85b94bc985b549e68caa6c36: Neither blocks nor log segments found. Creating new log.
I20250112 15:08:36.325026 23946 tablet_bootstrap.cc:492] T 92c129e062b945df8805bcc412f3a599 P 05ca05df85b94bc985b549e68caa6c36: No bootstrap required, opened a new log
I20250112 15:08:36.325065 23946 ts_tablet_manager.cc:1397] T 92c129e062b945df8805bcc412f3a599 P 05ca05df85b94bc985b549e68caa6c36: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20250112 15:08:36.325194 23946 raft_consensus.cc:357] T 92c129e062b945df8805bcc412f3a599 P 05ca05df85b94bc985b549e68caa6c36 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "111d62a4d211430fa930ff5a4dfa99d0" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 35133 } } peers { permanent_uuid: "8586159698b04cde88062accfc5d1bf1" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 33901 } } peers { permanent_uuid: "05ca05df85b94bc985b549e68caa6c36" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 44611 } }
I20250112 15:08:36.325240 23946 raft_consensus.cc:383] T 92c129e062b945df8805bcc412f3a599 P 05ca05df85b94bc985b549e68caa6c36 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 15:08:36.325263 23946 raft_consensus.cc:738] T 92c129e062b945df8805bcc412f3a599 P 05ca05df85b94bc985b549e68caa6c36 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 05ca05df85b94bc985b549e68caa6c36, State: Initialized, Role: FOLLOWER
I20250112 15:08:36.325313 23946 consensus_queue.cc:260] T 92c129e062b945df8805bcc412f3a599 P 05ca05df85b94bc985b549e68caa6c36 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "111d62a4d211430fa930ff5a4dfa99d0" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 35133 } } peers { permanent_uuid: "8586159698b04cde88062accfc5d1bf1" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 33901 } } peers { permanent_uuid: "05ca05df85b94bc985b549e68caa6c36" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 44611 } }
I20250112 15:08:36.325399 23946 ts_tablet_manager.cc:1428] T 92c129e062b945df8805bcc412f3a599 P 05ca05df85b94bc985b549e68caa6c36: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20250112 15:08:36.409976 23940 raft_consensus.cc:491] T 92c129e062b945df8805bcc412f3a599 P 8586159698b04cde88062accfc5d1bf1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 15:08:36.410095 23940 raft_consensus.cc:513] T 92c129e062b945df8805bcc412f3a599 P 8586159698b04cde88062accfc5d1bf1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "111d62a4d211430fa930ff5a4dfa99d0" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 35133 } } peers { permanent_uuid: "8586159698b04cde88062accfc5d1bf1" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 33901 } } peers { permanent_uuid: "05ca05df85b94bc985b549e68caa6c36" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 44611 } }
I20250112 15:08:36.410346 23940 leader_election.cc:290] T 92c129e062b945df8805bcc412f3a599 P 8586159698b04cde88062accfc5d1bf1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 111d62a4d211430fa930ff5a4dfa99d0 (127.17.60.193:35133), 05ca05df85b94bc985b549e68caa6c36 (127.17.60.195:44611)
I20250112 15:08:36.414049 23891 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "92c129e062b945df8805bcc412f3a599" candidate_uuid: "8586159698b04cde88062accfc5d1bf1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "05ca05df85b94bc985b549e68caa6c36" is_pre_election: true
I20250112 15:08:36.414049 23740 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "92c129e062b945df8805bcc412f3a599" candidate_uuid: "8586159698b04cde88062accfc5d1bf1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "111d62a4d211430fa930ff5a4dfa99d0" is_pre_election: true
I20250112 15:08:36.414213 23891 raft_consensus.cc:2463] T 92c129e062b945df8805bcc412f3a599 P 05ca05df85b94bc985b549e68caa6c36 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8586159698b04cde88062accfc5d1bf1 in term 0.
I20250112 15:08:36.414230 23740 raft_consensus.cc:2463] T 92c129e062b945df8805bcc412f3a599 P 111d62a4d211430fa930ff5a4dfa99d0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8586159698b04cde88062accfc5d1bf1 in term 0.
I20250112 15:08:36.414378 23782 leader_election.cc:304] T 92c129e062b945df8805bcc412f3a599 P 8586159698b04cde88062accfc5d1bf1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 111d62a4d211430fa930ff5a4dfa99d0, 8586159698b04cde88062accfc5d1bf1; no voters:
I20250112 15:08:36.414477 23940 raft_consensus.cc:2798] T 92c129e062b945df8805bcc412f3a599 P 8586159698b04cde88062accfc5d1bf1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250112 15:08:36.414520 23940 raft_consensus.cc:491] T 92c129e062b945df8805bcc412f3a599 P 8586159698b04cde88062accfc5d1bf1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250112 15:08:36.414538 23940 raft_consensus.cc:3054] T 92c129e062b945df8805bcc412f3a599 P 8586159698b04cde88062accfc5d1bf1 [term 0 FOLLOWER]: Advancing to term 1
I20250112 15:08:36.415155 23940 raft_consensus.cc:513] T 92c129e062b945df8805bcc412f3a599 P 8586159698b04cde88062accfc5d1bf1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "111d62a4d211430fa930ff5a4dfa99d0" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 35133 } } peers { permanent_uuid: "8586159698b04cde88062accfc5d1bf1" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 33901 } } peers { permanent_uuid: "05ca05df85b94bc985b549e68caa6c36" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 44611 } }
I20250112 15:08:36.415282 23940 leader_election.cc:290] T 92c129e062b945df8805bcc412f3a599 P 8586159698b04cde88062accfc5d1bf1 [CANDIDATE]: Term 1 election: Requested vote from peers 111d62a4d211430fa930ff5a4dfa99d0 (127.17.60.193:35133), 05ca05df85b94bc985b549e68caa6c36 (127.17.60.195:44611)
I20250112 15:08:36.415453 23740 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "92c129e062b945df8805bcc412f3a599" candidate_uuid: "8586159698b04cde88062accfc5d1bf1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "111d62a4d211430fa930ff5a4dfa99d0"
I20250112 15:08:36.415457 23891 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "92c129e062b945df8805bcc412f3a599" candidate_uuid: "8586159698b04cde88062accfc5d1bf1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "05ca05df85b94bc985b549e68caa6c36"
I20250112 15:08:36.415552 23740 raft_consensus.cc:3054] T 92c129e062b945df8805bcc412f3a599 P 111d62a4d211430fa930ff5a4dfa99d0 [term 0 FOLLOWER]: Advancing to term 1
I20250112 15:08:36.415557 23891 raft_consensus.cc:3054] T 92c129e062b945df8805bcc412f3a599 P 05ca05df85b94bc985b549e68caa6c36 [term 0 FOLLOWER]: Advancing to term 1
I20250112 15:08:36.416241 23891 raft_consensus.cc:2463] T 92c129e062b945df8805bcc412f3a599 P 05ca05df85b94bc985b549e68caa6c36 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8586159698b04cde88062accfc5d1bf1 in term 1.
I20250112 15:08:36.416250 23740 raft_consensus.cc:2463] T 92c129e062b945df8805bcc412f3a599 P 111d62a4d211430fa930ff5a4dfa99d0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8586159698b04cde88062accfc5d1bf1 in term 1.
I20250112 15:08:36.416432 23782 leader_election.cc:304] T 92c129e062b945df8805bcc412f3a599 P 8586159698b04cde88062accfc5d1bf1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 05ca05df85b94bc985b549e68caa6c36, 8586159698b04cde88062accfc5d1bf1; no voters:
I20250112 15:08:36.416519 23940 raft_consensus.cc:2798] T 92c129e062b945df8805bcc412f3a599 P 8586159698b04cde88062accfc5d1bf1 [term 1 FOLLOWER]: Leader election won for term 1
I20250112 15:08:36.416689 23940 raft_consensus.cc:695] T 92c129e062b945df8805bcc412f3a599 P 8586159698b04cde88062accfc5d1bf1 [term 1 LEADER]: Becoming Leader. State: Replica: 8586159698b04cde88062accfc5d1bf1, State: Running, Role: LEADER
I20250112 15:08:36.416740 23940 consensus_queue.cc:237] T 92c129e062b945df8805bcc412f3a599 P 8586159698b04cde88062accfc5d1bf1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "111d62a4d211430fa930ff5a4dfa99d0" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 35133 } } peers { permanent_uuid: "8586159698b04cde88062accfc5d1bf1" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 33901 } } peers { permanent_uuid: "05ca05df85b94bc985b549e68caa6c36" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 44611 } }
I20250112 15:08:36.417315 23636 catalog_manager.cc:5526] T 92c129e062b945df8805bcc412f3a599 P 8586159698b04cde88062accfc5d1bf1 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8586159698b04cde88062accfc5d1bf1 (127.17.60.194). New cstate: current_term: 1 leader_uuid: "8586159698b04cde88062accfc5d1bf1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "111d62a4d211430fa930ff5a4dfa99d0" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 35133 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8586159698b04cde88062accfc5d1bf1" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 33901 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "05ca05df85b94bc985b549e68caa6c36" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 44611 } health_report { overall_health: UNKNOWN } } }
W20250112 15:08:36.467080 23767 tablet.cc:2367] T 3e43528d2bb24563966ef89a76aa761e P 111d62a4d211430fa930ff5a4dfa99d0: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250112 15:08:36.544039 23950 raft_consensus.cc:491] T 6a1c881a80004842a1eb46bb488d3834 P 05ca05df85b94bc985b549e68caa6c36 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 15:08:36.544168 23950 raft_consensus.cc:513] T 6a1c881a80004842a1eb46bb488d3834 P 05ca05df85b94bc985b549e68caa6c36 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "111d62a4d211430fa930ff5a4dfa99d0" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 35133 } } peers { permanent_uuid: "8586159698b04cde88062accfc5d1bf1" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 33901 } } peers { permanent_uuid: "05ca05df85b94bc985b549e68caa6c36" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 44611 } }
I20250112 15:08:36.544409 23950 leader_election.cc:290] T 6a1c881a80004842a1eb46bb488d3834 P 05ca05df85b94bc985b549e68caa6c36 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 111d62a4d211430fa930ff5a4dfa99d0 (127.17.60.193:35133), 8586159698b04cde88062accfc5d1bf1 (127.17.60.194:33901)
I20250112 15:08:36.547024 23740 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6a1c881a80004842a1eb46bb488d3834" candidate_uuid: "05ca05df85b94bc985b549e68caa6c36" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "111d62a4d211430fa930ff5a4dfa99d0" is_pre_election: true
I20250112 15:08:36.547106 23740 raft_consensus.cc:2463] T 6a1c881a80004842a1eb46bb488d3834 P 111d62a4d211430fa930ff5a4dfa99d0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 05ca05df85b94bc985b549e68caa6c36 in term 0.
I20250112 15:08:36.547041 23816 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6a1c881a80004842a1eb46bb488d3834" candidate_uuid: "05ca05df85b94bc985b549e68caa6c36" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8586159698b04cde88062accfc5d1bf1" is_pre_election: true
I20250112 15:08:36.547145 23816 raft_consensus.cc:2463] T 6a1c881a80004842a1eb46bb488d3834 P 8586159698b04cde88062accfc5d1bf1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 05ca05df85b94bc985b549e68caa6c36 in term 0.
I20250112 15:08:36.547231 23857 leader_election.cc:304] T 6a1c881a80004842a1eb46bb488d3834 P 05ca05df85b94bc985b549e68caa6c36 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 05ca05df85b94bc985b549e68caa6c36, 111d62a4d211430fa930ff5a4dfa99d0; no voters:
I20250112 15:08:36.547331 23950 raft_consensus.cc:2798] T 6a1c881a80004842a1eb46bb488d3834 P 05ca05df85b94bc985b549e68caa6c36 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250112 15:08:36.547358 23950 raft_consensus.cc:491] T 6a1c881a80004842a1eb46bb488d3834 P 05ca05df85b94bc985b549e68caa6c36 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250112 15:08:36.547371 23950 raft_consensus.cc:3054] T 6a1c881a80004842a1eb46bb488d3834 P 05ca05df85b94bc985b549e68caa6c36 [term 0 FOLLOWER]: Advancing to term 1
I20250112 15:08:36.549432 23945 raft_consensus.cc:491] T 3e43528d2bb24563966ef89a76aa761e P 111d62a4d211430fa930ff5a4dfa99d0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 15:08:36.549492 23945 raft_consensus.cc:513] T 3e43528d2bb24563966ef89a76aa761e P 111d62a4d211430fa930ff5a4dfa99d0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "111d62a4d211430fa930ff5a4dfa99d0" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 35133 } } peers { permanent_uuid: "05ca05df85b94bc985b549e68caa6c36" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 44611 } } peers { permanent_uuid: "8586159698b04cde88062accfc5d1bf1" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 33901 } }
I20250112 15:08:36.549691 23945 leader_election.cc:290] T 3e43528d2bb24563966ef89a76aa761e P 111d62a4d211430fa930ff5a4dfa99d0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 05ca05df85b94bc985b549e68caa6c36 (127.17.60.195:44611), 8586159698b04cde88062accfc5d1bf1 (127.17.60.194:33901)
I20250112 15:08:36.549710 23950 raft_consensus.cc:513] T 6a1c881a80004842a1eb46bb488d3834 P 05ca05df85b94bc985b549e68caa6c36 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "111d62a4d211430fa930ff5a4dfa99d0" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 35133 } } peers { permanent_uuid: "8586159698b04cde88062accfc5d1bf1" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 33901 } } peers { permanent_uuid: "05ca05df85b94bc985b549e68caa6c36" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 44611 } }
I20250112 15:08:36.549860 23950 leader_election.cc:290] T 6a1c881a80004842a1eb46bb488d3834 P 05ca05df85b94bc985b549e68caa6c36 [CANDIDATE]: Term 1 election: Requested vote from peers 111d62a4d211430fa930ff5a4dfa99d0 (127.17.60.193:35133), 8586159698b04cde88062accfc5d1bf1 (127.17.60.194:33901)
I20250112 15:08:36.550073 23816 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6a1c881a80004842a1eb46bb488d3834" candidate_uuid: "05ca05df85b94bc985b549e68caa6c36" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8586159698b04cde88062accfc5d1bf1"
I20250112 15:08:36.550160 23816 raft_consensus.cc:3054] T 6a1c881a80004842a1eb46bb488d3834 P 8586159698b04cde88062accfc5d1bf1 [term 0 FOLLOWER]: Advancing to term 1
I20250112 15:08:36.550233 23740 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6a1c881a80004842a1eb46bb488d3834" candidate_uuid: "05ca05df85b94bc985b549e68caa6c36" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "111d62a4d211430fa930ff5a4dfa99d0"
I20250112 15:08:36.550308 23740 raft_consensus.cc:3054] T 6a1c881a80004842a1eb46bb488d3834 P 111d62a4d211430fa930ff5a4dfa99d0 [term 0 FOLLOWER]: Advancing to term 1
I20250112 15:08:36.550951 23816 raft_consensus.cc:2463] T 6a1c881a80004842a1eb46bb488d3834 P 8586159698b04cde88062accfc5d1bf1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 05ca05df85b94bc985b549e68caa6c36 in term 1.
I20250112 15:08:36.551045 23740 raft_consensus.cc:2463] T 6a1c881a80004842a1eb46bb488d3834 P 111d62a4d211430fa930ff5a4dfa99d0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 05ca05df85b94bc985b549e68caa6c36 in term 1.
I20250112 15:08:36.551113 23855 leader_election.cc:304] T 6a1c881a80004842a1eb46bb488d3834 P 05ca05df85b94bc985b549e68caa6c36 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 05ca05df85b94bc985b549e68caa6c36, 8586159698b04cde88062accfc5d1bf1; no voters:
I20250112 15:08:36.551200 23950 raft_consensus.cc:2798] T 6a1c881a80004842a1eb46bb488d3834 P 05ca05df85b94bc985b549e68caa6c36 [term 1 FOLLOWER]: Leader election won for term 1
I20250112 15:08:36.551331 23950 raft_consensus.cc:695] T 6a1c881a80004842a1eb46bb488d3834 P 05ca05df85b94bc985b549e68caa6c36 [term 1 LEADER]: Becoming Leader. State: Replica: 05ca05df85b94bc985b549e68caa6c36, State: Running, Role: LEADER
I20250112 15:08:36.551380 23950 consensus_queue.cc:237] T 6a1c881a80004842a1eb46bb488d3834 P 05ca05df85b94bc985b549e68caa6c36 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "111d62a4d211430fa930ff5a4dfa99d0" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 35133 } } peers { permanent_uuid: "8586159698b04cde88062accfc5d1bf1" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 33901 } } peers { permanent_uuid: "05ca05df85b94bc985b549e68caa6c36" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 44611 } }
I20250112 15:08:36.551923 23636 catalog_manager.cc:5526] T 6a1c881a80004842a1eb46bb488d3834 P 05ca05df85b94bc985b549e68caa6c36 reported cstate change: term changed from 0 to 1, leader changed from <none> to 05ca05df85b94bc985b549e68caa6c36 (127.17.60.195). New cstate: current_term: 1 leader_uuid: "05ca05df85b94bc985b549e68caa6c36" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "111d62a4d211430fa930ff5a4dfa99d0" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 35133 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8586159698b04cde88062accfc5d1bf1" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 33901 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "05ca05df85b94bc985b549e68caa6c36" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 44611 } health_report { overall_health: HEALTHY } } }
I20250112 15:08:36.552930 23891 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e43528d2bb24563966ef89a76aa761e" candidate_uuid: "111d62a4d211430fa930ff5a4dfa99d0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "05ca05df85b94bc985b549e68caa6c36" is_pre_election: true
I20250112 15:08:36.553049 23891 raft_consensus.cc:2463] T 3e43528d2bb24563966ef89a76aa761e P 05ca05df85b94bc985b549e68caa6c36 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 111d62a4d211430fa930ff5a4dfa99d0 in term 0.
I20250112 15:08:36.553206 23706 leader_election.cc:304] T 3e43528d2bb24563966ef89a76aa761e P 111d62a4d211430fa930ff5a4dfa99d0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 05ca05df85b94bc985b549e68caa6c36, 111d62a4d211430fa930ff5a4dfa99d0; no voters:
I20250112 15:08:36.553222 23816 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e43528d2bb24563966ef89a76aa761e" candidate_uuid: "111d62a4d211430fa930ff5a4dfa99d0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8586159698b04cde88062accfc5d1bf1" is_pre_election: true
I20250112 15:08:36.553299 23816 raft_consensus.cc:2463] T 3e43528d2bb24563966ef89a76aa761e P 8586159698b04cde88062accfc5d1bf1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 111d62a4d211430fa930ff5a4dfa99d0 in term 0.
I20250112 15:08:36.553323 23945 raft_consensus.cc:2798] T 3e43528d2bb24563966ef89a76aa761e P 111d62a4d211430fa930ff5a4dfa99d0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250112 15:08:36.553380 23945 raft_consensus.cc:491] T 3e43528d2bb24563966ef89a76aa761e P 111d62a4d211430fa930ff5a4dfa99d0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250112 15:08:36.553403 23945 raft_consensus.cc:3054] T 3e43528d2bb24563966ef89a76aa761e P 111d62a4d211430fa930ff5a4dfa99d0 [term 0 FOLLOWER]: Advancing to term 1
I20250112 15:08:36.553941 23945 raft_consensus.cc:513] T 3e43528d2bb24563966ef89a76aa761e P 111d62a4d211430fa930ff5a4dfa99d0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "111d62a4d211430fa930ff5a4dfa99d0" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 35133 } } peers { permanent_uuid: "05ca05df85b94bc985b549e68caa6c36" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 44611 } } peers { permanent_uuid: "8586159698b04cde88062accfc5d1bf1" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 33901 } }
I20250112 15:08:36.554065 23945 leader_election.cc:290] T 3e43528d2bb24563966ef89a76aa761e P 111d62a4d211430fa930ff5a4dfa99d0 [CANDIDATE]: Term 1 election: Requested vote from peers 05ca05df85b94bc985b549e68caa6c36 (127.17.60.195:44611), 8586159698b04cde88062accfc5d1bf1 (127.17.60.194:33901)
I20250112 15:08:36.554221 23891 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e43528d2bb24563966ef89a76aa761e" candidate_uuid: "111d62a4d211430fa930ff5a4dfa99d0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "05ca05df85b94bc985b549e68caa6c36"
I20250112 15:08:36.554248 23816 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e43528d2bb24563966ef89a76aa761e" candidate_uuid: "111d62a4d211430fa930ff5a4dfa99d0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8586159698b04cde88062accfc5d1bf1"
I20250112 15:08:36.554296 23891 raft_consensus.cc:3054] T 3e43528d2bb24563966ef89a76aa761e P 05ca05df85b94bc985b549e68caa6c36 [term 0 FOLLOWER]: Advancing to term 1
I20250112 15:08:36.554301 23816 raft_consensus.cc:3054] T 3e43528d2bb24563966ef89a76aa761e P 8586159698b04cde88062accfc5d1bf1 [term 0 FOLLOWER]: Advancing to term 1
I20250112 15:08:36.554912 23891 raft_consensus.cc:2463] T 3e43528d2bb24563966ef89a76aa761e P 05ca05df85b94bc985b549e68caa6c36 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 111d62a4d211430fa930ff5a4dfa99d0 in term 1.
I20250112 15:08:36.554912 23816 raft_consensus.cc:2463] T 3e43528d2bb24563966ef89a76aa761e P 8586159698b04cde88062accfc5d1bf1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 111d62a4d211430fa930ff5a4dfa99d0 in term 1.
I20250112 15:08:36.555119 23704 leader_election.cc:304] T 3e43528d2bb24563966ef89a76aa761e P 111d62a4d211430fa930ff5a4dfa99d0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 111d62a4d211430fa930ff5a4dfa99d0, 8586159698b04cde88062accfc5d1bf1; no voters:
I20250112 15:08:36.555213 23945 raft_consensus.cc:2798] T 3e43528d2bb24563966ef89a76aa761e P 111d62a4d211430fa930ff5a4dfa99d0 [term 1 FOLLOWER]: Leader election won for term 1
I20250112 15:08:36.555280 23945 raft_consensus.cc:695] T 3e43528d2bb24563966ef89a76aa761e P 111d62a4d211430fa930ff5a4dfa99d0 [term 1 LEADER]: Becoming Leader. State: Replica: 111d62a4d211430fa930ff5a4dfa99d0, State: Running, Role: LEADER
I20250112 15:08:36.555330 23945 consensus_queue.cc:237] T 3e43528d2bb24563966ef89a76aa761e P 111d62a4d211430fa930ff5a4dfa99d0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "111d62a4d211430fa930ff5a4dfa99d0" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 35133 } } peers { permanent_uuid: "05ca05df85b94bc985b549e68caa6c36" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 44611 } } peers { permanent_uuid: "8586159698b04cde88062accfc5d1bf1" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 33901 } }
I20250112 15:08:36.555838 23636 catalog_manager.cc:5526] T 3e43528d2bb24563966ef89a76aa761e P 111d62a4d211430fa930ff5a4dfa99d0 reported cstate change: term changed from 0 to 1, leader changed from <none> to 111d62a4d211430fa930ff5a4dfa99d0 (127.17.60.193). New cstate: current_term: 1 leader_uuid: "111d62a4d211430fa930ff5a4dfa99d0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "111d62a4d211430fa930ff5a4dfa99d0" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 35133 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "05ca05df85b94bc985b549e68caa6c36" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 44611 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8586159698b04cde88062accfc5d1bf1" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 33901 } health_report { overall_health: UNKNOWN } } }
I20250112 15:08:36.595853 23940 raft_consensus.cc:491] T 483cfa86b848433bbc365bae489ed251 P 8586159698b04cde88062accfc5d1bf1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 15:08:36.595970 23940 raft_consensus.cc:513] T 483cfa86b848433bbc365bae489ed251 P 8586159698b04cde88062accfc5d1bf1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "111d62a4d211430fa930ff5a4dfa99d0" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 35133 } } peers { permanent_uuid: "8586159698b04cde88062accfc5d1bf1" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 33901 } } peers { permanent_uuid: "05ca05df85b94bc985b549e68caa6c36" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 44611 } }
I20250112 15:08:36.596144 23940 leader_election.cc:290] T 483cfa86b848433bbc365bae489ed251 P 8586159698b04cde88062accfc5d1bf1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 111d62a4d211430fa930ff5a4dfa99d0 (127.17.60.193:35133), 05ca05df85b94bc985b549e68caa6c36 (127.17.60.195:44611)
I20250112 15:08:36.596356 23740 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "483cfa86b848433bbc365bae489ed251" candidate_uuid: "8586159698b04cde88062accfc5d1bf1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "111d62a4d211430fa930ff5a4dfa99d0" is_pre_election: true
I20250112 15:08:36.596426 23740 raft_consensus.cc:2463] T 483cfa86b848433bbc365bae489ed251 P 111d62a4d211430fa930ff5a4dfa99d0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8586159698b04cde88062accfc5d1bf1 in term 0.
I20250112 15:08:36.596386 23891 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "483cfa86b848433bbc365bae489ed251" candidate_uuid: "8586159698b04cde88062accfc5d1bf1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "05ca05df85b94bc985b549e68caa6c36" is_pre_election: true
I20250112 15:08:36.596482 23891 raft_consensus.cc:2463] T 483cfa86b848433bbc365bae489ed251 P 05ca05df85b94bc985b549e68caa6c36 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8586159698b04cde88062accfc5d1bf1 in term 0.
I20250112 15:08:36.596553 23782 leader_election.cc:304] T 483cfa86b848433bbc365bae489ed251 P 8586159698b04cde88062accfc5d1bf1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 111d62a4d211430fa930ff5a4dfa99d0, 8586159698b04cde88062accfc5d1bf1; no voters:
I20250112 15:08:36.596652 23940 raft_consensus.cc:2798] T 483cfa86b848433bbc365bae489ed251 P 8586159698b04cde88062accfc5d1bf1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250112 15:08:36.596681 23940 raft_consensus.cc:491] T 483cfa86b848433bbc365bae489ed251 P 8586159698b04cde88062accfc5d1bf1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250112 15:08:36.596694 23940 raft_consensus.cc:3054] T 483cfa86b848433bbc365bae489ed251 P 8586159698b04cde88062accfc5d1bf1 [term 0 FOLLOWER]: Advancing to term 1
I20250112 15:08:36.597335 23940 raft_consensus.cc:513] T 483cfa86b848433bbc365bae489ed251 P 8586159698b04cde88062accfc5d1bf1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "111d62a4d211430fa930ff5a4dfa99d0" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 35133 } } peers { permanent_uuid: "8586159698b04cde88062accfc5d1bf1" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 33901 } } peers { permanent_uuid: "05ca05df85b94bc985b549e68caa6c36" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 44611 } }
I20250112 15:08:36.597432 23940 leader_election.cc:290] T 483cfa86b848433bbc365bae489ed251 P 8586159698b04cde88062accfc5d1bf1 [CANDIDATE]: Term 1 election: Requested vote from peers 111d62a4d211430fa930ff5a4dfa99d0 (127.17.60.193:35133), 05ca05df85b94bc985b549e68caa6c36 (127.17.60.195:44611)
I20250112 15:08:36.597606 23740 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "483cfa86b848433bbc365bae489ed251" candidate_uuid: "8586159698b04cde88062accfc5d1bf1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "111d62a4d211430fa930ff5a4dfa99d0"
I20250112 15:08:36.597626 23891 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "483cfa86b848433bbc365bae489ed251" candidate_uuid: "8586159698b04cde88062accfc5d1bf1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "05ca05df85b94bc985b549e68caa6c36"
I20250112 15:08:36.597676 23740 raft_consensus.cc:3054] T 483cfa86b848433bbc365bae489ed251 P 111d62a4d211430fa930ff5a4dfa99d0 [term 0 FOLLOWER]: Advancing to term 1
I20250112 15:08:36.597684 23891 raft_consensus.cc:3054] T 483cfa86b848433bbc365bae489ed251 P 05ca05df85b94bc985b549e68caa6c36 [term 0 FOLLOWER]: Advancing to term 1
I20250112 15:08:36.598244 23891 raft_consensus.cc:2463] T 483cfa86b848433bbc365bae489ed251 P 05ca05df85b94bc985b549e68caa6c36 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8586159698b04cde88062accfc5d1bf1 in term 1.
I20250112 15:08:36.598234 23740 raft_consensus.cc:2463] T 483cfa86b848433bbc365bae489ed251 P 111d62a4d211430fa930ff5a4dfa99d0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8586159698b04cde88062accfc5d1bf1 in term 1.
I20250112 15:08:36.598425 23782 leader_election.cc:304] T 483cfa86b848433bbc365bae489ed251 P 8586159698b04cde88062accfc5d1bf1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 05ca05df85b94bc985b549e68caa6c36, 8586159698b04cde88062accfc5d1bf1; no voters:
I20250112 15:08:36.598511 23940 raft_consensus.cc:2798] T 483cfa86b848433bbc365bae489ed251 P 8586159698b04cde88062accfc5d1bf1 [term 1 FOLLOWER]: Leader election won for term 1
I20250112 15:08:36.598567 23940 raft_consensus.cc:695] T 483cfa86b848433bbc365bae489ed251 P 8586159698b04cde88062accfc5d1bf1 [term 1 LEADER]: Becoming Leader. State: Replica: 8586159698b04cde88062accfc5d1bf1, State: Running, Role: LEADER
I20250112 15:08:36.598625 23940 consensus_queue.cc:237] T 483cfa86b848433bbc365bae489ed251 P 8586159698b04cde88062accfc5d1bf1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "111d62a4d211430fa930ff5a4dfa99d0" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 35133 } } peers { permanent_uuid: "8586159698b04cde88062accfc5d1bf1" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 33901 } } peers { permanent_uuid: "05ca05df85b94bc985b549e68caa6c36" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 44611 } }
I20250112 15:08:36.599275 23636 catalog_manager.cc:5526] T 483cfa86b848433bbc365bae489ed251 P 8586159698b04cde88062accfc5d1bf1 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8586159698b04cde88062accfc5d1bf1 (127.17.60.194). New cstate: current_term: 1 leader_uuid: "8586159698b04cde88062accfc5d1bf1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "111d62a4d211430fa930ff5a4dfa99d0" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 35133 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8586159698b04cde88062accfc5d1bf1" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 33901 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "05ca05df85b94bc985b549e68caa6c36" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 44611 } health_report { overall_health: UNKNOWN } } }
I20250112 15:08:36.659699 17651 tablet_server.cc:178] TabletServer@127.17.60.193:0 shutting down...
I20250112 15:08:36.662390 17651 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250112 15:08:36.662531 17651 tablet_replica.cc:331] T 92c129e062b945df8805bcc412f3a599 P 111d62a4d211430fa930ff5a4dfa99d0: stopping tablet replica
I20250112 15:08:36.662606 17651 raft_consensus.cc:2238] T 92c129e062b945df8805bcc412f3a599 P 111d62a4d211430fa930ff5a4dfa99d0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250112 15:08:36.662657 17651 raft_consensus.cc:2267] T 92c129e062b945df8805bcc412f3a599 P 111d62a4d211430fa930ff5a4dfa99d0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 15:08:36.662861 17651 tablet_replica.cc:331] T 6a1c881a80004842a1eb46bb488d3834 P 111d62a4d211430fa930ff5a4dfa99d0: stopping tablet replica
I20250112 15:08:36.662894 17651 raft_consensus.cc:2238] T 6a1c881a80004842a1eb46bb488d3834 P 111d62a4d211430fa930ff5a4dfa99d0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250112 15:08:36.662918 17651 raft_consensus.cc:2267] T 6a1c881a80004842a1eb46bb488d3834 P 111d62a4d211430fa930ff5a4dfa99d0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 15:08:36.663074 17651 tablet_replica.cc:331] T 483cfa86b848433bbc365bae489ed251 P 111d62a4d211430fa930ff5a4dfa99d0: stopping tablet replica
I20250112 15:08:36.663107 17651 raft_consensus.cc:2238] T 483cfa86b848433bbc365bae489ed251 P 111d62a4d211430fa930ff5a4dfa99d0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250112 15:08:36.663130 17651 raft_consensus.cc:2267] T 483cfa86b848433bbc365bae489ed251 P 111d62a4d211430fa930ff5a4dfa99d0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 15:08:36.663269 17651 tablet_replica.cc:331] T 3e43528d2bb24563966ef89a76aa761e P 111d62a4d211430fa930ff5a4dfa99d0: stopping tablet replica
I20250112 15:08:36.663300 17651 raft_consensus.cc:2238] T 3e43528d2bb24563966ef89a76aa761e P 111d62a4d211430fa930ff5a4dfa99d0 [term 1 LEADER]: Raft consensus shutting down.
I20250112 15:08:36.663333 17651 pending_rounds.cc:62] T 3e43528d2bb24563966ef89a76aa761e P 111d62a4d211430fa930ff5a4dfa99d0: Trying to abort 1 pending ops.
I20250112 15:08:36.663357 17651 pending_rounds.cc:69] T 3e43528d2bb24563966ef89a76aa761e P 111d62a4d211430fa930ff5a4dfa99d0: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7113500739811098624 op_type: NO_OP noop_request { }
I20250112 15:08:36.663412 17651 raft_consensus.cc:2883] T 3e43528d2bb24563966ef89a76aa761e P 111d62a4d211430fa930ff5a4dfa99d0 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20250112 15:08:36.663434 17651 raft_consensus.cc:2267] T 3e43528d2bb24563966ef89a76aa761e P 111d62a4d211430fa930ff5a4dfa99d0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 15:08:36.675774 17651 tablet_server.cc:195] TabletServer@127.17.60.193:0 shutdown complete.
I20250112 15:08:36.676920 17651 tablet_server.cc:178] TabletServer@127.17.60.194:0 shutting down...
I20250112 15:08:36.679764 17651 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250112 15:08:36.679939 17651 tablet_replica.cc:331] T 6a1c881a80004842a1eb46bb488d3834 P 8586159698b04cde88062accfc5d1bf1: stopping tablet replica
I20250112 15:08:36.680011 17651 raft_consensus.cc:2238] T 6a1c881a80004842a1eb46bb488d3834 P 8586159698b04cde88062accfc5d1bf1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250112 15:08:36.680055 17651 raft_consensus.cc:2267] T 6a1c881a80004842a1eb46bb488d3834 P 8586159698b04cde88062accfc5d1bf1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 15:08:36.680279 17651 tablet_replica.cc:331] T 483cfa86b848433bbc365bae489ed251 P 8586159698b04cde88062accfc5d1bf1: stopping tablet replica
I20250112 15:08:36.680316 17651 raft_consensus.cc:2238] T 483cfa86b848433bbc365bae489ed251 P 8586159698b04cde88062accfc5d1bf1 [term 1 LEADER]: Raft consensus shutting down.
I20250112 15:08:36.680351 17651 pending_rounds.cc:62] T 483cfa86b848433bbc365bae489ed251 P 8586159698b04cde88062accfc5d1bf1: Trying to abort 1 pending ops.
I20250112 15:08:36.680367 17651 pending_rounds.cc:69] T 483cfa86b848433bbc365bae489ed251 P 8586159698b04cde88062accfc5d1bf1: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7113500739988496384 op_type: NO_OP noop_request { }
I20250112 15:08:36.680413 17651 raft_consensus.cc:2883] T 483cfa86b848433bbc365bae489ed251 P 8586159698b04cde88062accfc5d1bf1 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20250112 15:08:36.680428 17651 raft_consensus.cc:2267] T 483cfa86b848433bbc365bae489ed251 P 8586159698b04cde88062accfc5d1bf1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 15:08:36.680732 17651 tablet_replica.cc:331] T 3e43528d2bb24563966ef89a76aa761e P 8586159698b04cde88062accfc5d1bf1: stopping tablet replica
I20250112 15:08:36.680773 17651 raft_consensus.cc:2238] T 3e43528d2bb24563966ef89a76aa761e P 8586159698b04cde88062accfc5d1bf1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250112 15:08:36.680794 17651 raft_consensus.cc:2267] T 3e43528d2bb24563966ef89a76aa761e P 8586159698b04cde88062accfc5d1bf1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 15:08:36.680927 17651 tablet_replica.cc:331] T 92c129e062b945df8805bcc412f3a599 P 8586159698b04cde88062accfc5d1bf1: stopping tablet replica
I20250112 15:08:36.680961 17651 raft_consensus.cc:2238] T 92c129e062b945df8805bcc412f3a599 P 8586159698b04cde88062accfc5d1bf1 [term 1 LEADER]: Raft consensus shutting down.
I20250112 15:08:36.680994 17651 pending_rounds.cc:62] T 92c129e062b945df8805bcc412f3a599 P 8586159698b04cde88062accfc5d1bf1: Trying to abort 1 pending ops.
I20250112 15:08:36.681010 17651 pending_rounds.cc:69] T 92c129e062b945df8805bcc412f3a599 P 8586159698b04cde88062accfc5d1bf1: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7113500739243511808 op_type: NO_OP noop_request { }
I20250112 15:08:36.681042 17651 raft_consensus.cc:2883] T 92c129e062b945df8805bcc412f3a599 P 8586159698b04cde88062accfc5d1bf1 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20250112 15:08:36.681064 17651 raft_consensus.cc:2267] T 92c129e062b945df8805bcc412f3a599 P 8586159698b04cde88062accfc5d1bf1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 15:08:36.693092 17651 tablet_server.cc:195] TabletServer@127.17.60.194:0 shutdown complete.
I20250112 15:08:36.694077 17651 tablet_server.cc:178] TabletServer@127.17.60.195:0 shutting down...
I20250112 15:08:36.696292 17651 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250112 15:08:36.696440 17651 tablet_replica.cc:331] T 92c129e062b945df8805bcc412f3a599 P 05ca05df85b94bc985b549e68caa6c36: stopping tablet replica
I20250112 15:08:36.696509 17651 raft_consensus.cc:2238] T 92c129e062b945df8805bcc412f3a599 P 05ca05df85b94bc985b549e68caa6c36 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250112 15:08:36.696583 17651 raft_consensus.cc:2267] T 92c129e062b945df8805bcc412f3a599 P 05ca05df85b94bc985b549e68caa6c36 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 15:08:36.696784 17651 tablet_replica.cc:331] T 483cfa86b848433bbc365bae489ed251 P 05ca05df85b94bc985b549e68caa6c36: stopping tablet replica
I20250112 15:08:36.696823 17651 raft_consensus.cc:2238] T 483cfa86b848433bbc365bae489ed251 P 05ca05df85b94bc985b549e68caa6c36 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250112 15:08:36.696856 17651 raft_consensus.cc:2267] T 483cfa86b848433bbc365bae489ed251 P 05ca05df85b94bc985b549e68caa6c36 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 15:08:36.697010 17651 tablet_replica.cc:331] T 3e43528d2bb24563966ef89a76aa761e P 05ca05df85b94bc985b549e68caa6c36: stopping tablet replica
I20250112 15:08:36.697050 17651 raft_consensus.cc:2238] T 3e43528d2bb24563966ef89a76aa761e P 05ca05df85b94bc985b549e68caa6c36 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250112 15:08:36.697073 17651 raft_consensus.cc:2267] T 3e43528d2bb24563966ef89a76aa761e P 05ca05df85b94bc985b549e68caa6c36 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 15:08:36.697206 17651 tablet_replica.cc:331] T 6a1c881a80004842a1eb46bb488d3834 P 05ca05df85b94bc985b549e68caa6c36: stopping tablet replica
I20250112 15:08:36.697234 17651 raft_consensus.cc:2238] T 6a1c881a80004842a1eb46bb488d3834 P 05ca05df85b94bc985b549e68caa6c36 [term 1 LEADER]: Raft consensus shutting down.
I20250112 15:08:36.697276 17651 pending_rounds.cc:62] T 6a1c881a80004842a1eb46bb488d3834 P 05ca05df85b94bc985b549e68caa6c36: Trying to abort 1 pending ops.
I20250112 15:08:36.697301 17651 pending_rounds.cc:69] T 6a1c881a80004842a1eb46bb488d3834 P 05ca05df85b94bc985b549e68caa6c36: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7113500739794898944 op_type: NO_OP noop_request { }
I20250112 15:08:36.697374 17651 raft_consensus.cc:2883] T 6a1c881a80004842a1eb46bb488d3834 P 05ca05df85b94bc985b549e68caa6c36 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20250112 15:08:36.697405 17651 raft_consensus.cc:2267] T 6a1c881a80004842a1eb46bb488d3834 P 05ca05df85b94bc985b549e68caa6c36 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 15:08:36.709389 17651 tablet_server.cc:195] TabletServer@127.17.60.195:0 shutdown complete.
I20250112 15:08:36.710734 17651 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250112 15:08:36.712255 23968 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250112 15:08:36.712953 23969 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250112 15:08:36.713814 23971 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250112 15:08:36.714709 17651 server_base.cc:1034] running on GCE node
I20250112 15:08:36.714818 17651 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250112 15:08:36.714850 17651 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250112 15:08:36.714879 17651 hybrid_clock.cc:648] HybridClock initialized: now 1736694516714877 us; error 0 us; skew 500 ppm
I20250112 15:08:36.715540 17651 webserver.cc:458] Webserver started at http://127.17.60.196:45609/ using document root <none> and password file <none>
I20250112 15:08:36.715632 17651 fs_manager.cc:362] Metadata directory not provided
I20250112 15:08:36.715687 17651 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250112 15:08:36.715745 17651 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250112 15:08:36.716048 17651 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task95dLSD/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestHandlingFailedTservers.1736694351070047-17651-0/minicluster-data/ts-3-root/instance:
uuid: "932773f4da6d4c238b6963b863c59be9"
format_stamp: "Formatted at 2025-01-12 15:08:36 on dist-test-slave-b4nz"
I20250112 15:08:36.716993 17651 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20250112 15:08:36.717463 23976 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250112 15:08:36.717590 17651 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20250112 15:08:36.717641 17651 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task95dLSD/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestHandlingFailedTservers.1736694351070047-17651-0/minicluster-data/ts-3-root
uuid: "932773f4da6d4c238b6963b863c59be9"
format_stamp: "Formatted at 2025-01-12 15:08:36 on dist-test-slave-b4nz"
I20250112 15:08:36.717696 17651 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task95dLSD/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestHandlingFailedTservers.1736694351070047-17651-0/minicluster-data/ts-3-root
metadata directory: /tmp/dist-test-task95dLSD/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestHandlingFailedTservers.1736694351070047-17651-0/minicluster-data/ts-3-root
1 data directories: /tmp/dist-test-task95dLSD/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestHandlingFailedTservers.1736694351070047-17651-0/minicluster-data/ts-3-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250112 15:08:36.724377 17651 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250112 15:08:36.724606 17651 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250112 15:08:36.724756 17651 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250112 15:08:36.724938 17651 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250112 15:08:36.724969 17651 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250112 15:08:36.725003 17651 ts_tablet_manager.cc:610] Registered 0 tablets
I20250112 15:08:36.725031 17651 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250112 15:08:36.727586 17651 rpc_server.cc:307] RPC server started. Bound to: 127.17.60.196:44863
I20250112 15:08:36.729507 24039 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.17.60.196:44863 every 8 connection(s)
I20250112 15:08:36.732399 24040 heartbeater.cc:346] Connected to a master server at 127.17.60.254:42077
I20250112 15:08:36.732458 24040 heartbeater.cc:463] Registering TS with master...
I20250112 15:08:36.732543 24040 heartbeater.cc:510] Master 127.17.60.254:42077 requested a full tablet report, sending...
I20250112 15:08:36.732787 23636 ts_manager.cc:194] Registered new tserver with Master: 932773f4da6d4c238b6963b863c59be9 (127.17.60.196:44863)
I20250112 15:08:36.733114 23636 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:40232
W20250112 15:08:37.153424 23686 auto_rebalancer.cc:253] failed to send replica move request: Network error: Client connection negotiation failed: client connection to 127.17.60.195:44611: connect: Connection refused (error 111)
W20250112 15:08:37.153859 23624 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.17.60.195:44611: connect: Connection refused (error 111) [suppressed 3 similar messages]
W20250112 15:08:37.154248 23686 auto_rebalancer.cc:663] Could not move replica: Network error: Client connection negotiation failed: client connection to 127.17.60.195:44611: connect: Connection refused (error 111)
W20250112 15:08:37.154290 23686 auto_rebalancer.cc:263] scheduled replica move failed to complete: Network error: Client connection negotiation failed: client connection to 127.17.60.195:44611: connect: Connection refused (error 111)
W20250112 15:08:37.154712 23686 auto_rebalancer.cc:663] Could not move replica: Network error: Client connection negotiation failed: client connection to 127.17.60.194:33901: connect: Connection refused (error 111)
W20250112 15:08:37.154747 23686 auto_rebalancer.cc:263] scheduled replica move failed to complete: Network error: Client connection negotiation failed: client connection to 127.17.60.194:33901: connect: Connection refused (error 111)
W20250112 15:08:37.155143 23686 auto_rebalancer.cc:663] Could not move replica: Network error: Client connection negotiation failed: client connection to 127.17.60.193:35133: connect: Connection refused (error 111)
W20250112 15:08:37.155177 23686 auto_rebalancer.cc:263] scheduled replica move failed to complete: Network error: Client connection negotiation failed: client connection to 127.17.60.193:35133: connect: Connection refused (error 111)
I20250112 15:08:37.733754 24040 heartbeater.cc:502] Master 127.17.60.254:42077 was elected leader, sending a full tablet report...
W20250112 15:08:38.156245 23686 auto_rebalancer.cc:253] failed to send replica move request: Network error: Client connection negotiation failed: client connection to 127.17.60.194:33901: connect: Connection refused (error 111)
W20250112 15:08:38.157011 23686 auto_rebalancer.cc:663] Could not move replica: Network error: Client connection negotiation failed: client connection to 127.17.60.194:33901: connect: Connection refused (error 111)
W20250112 15:08:38.157049 23686 auto_rebalancer.cc:263] scheduled replica move failed to complete: Network error: Client connection negotiation failed: client connection to 127.17.60.194:33901: connect: Connection refused (error 111)
W20250112 15:08:38.157449 23686 auto_rebalancer.cc:663] Could not move replica: Network error: Client connection negotiation failed: client connection to 127.17.60.193:35133: connect: Connection refused (error 111)
W20250112 15:08:38.157505 23686 auto_rebalancer.cc:263] scheduled replica move failed to complete: Network error: Client connection negotiation failed: client connection to 127.17.60.193:35133: connect: Connection refused (error 111)
W20250112 15:08:38.157860 23686 auto_rebalancer.cc:663] Could not move replica: Network error: Client connection negotiation failed: client connection to 127.17.60.194:33901: connect: Connection refused (error 111)
W20250112 15:08:38.157899 23686 auto_rebalancer.cc:263] scheduled replica move failed to complete: Network error: Client connection negotiation failed: client connection to 127.17.60.194:33901: connect: Connection refused (error 111)
W20250112 15:08:39.158901 23686 auto_rebalancer.cc:253] failed to send replica move request: Network error: Client connection negotiation failed: client connection to 127.17.60.195:44611: connect: Connection refused (error 111)
W20250112 15:08:39.159616 23686 auto_rebalancer.cc:663] Could not move replica: Network error: Client connection negotiation failed: client connection to 127.17.60.195:44611: connect: Connection refused (error 111)
W20250112 15:08:39.159657 23686 auto_rebalancer.cc:263] scheduled replica move failed to complete: Network error: Client connection negotiation failed: client connection to 127.17.60.195:44611: connect: Connection refused (error 111)
W20250112 15:08:39.160100 23686 auto_rebalancer.cc:663] Could not move replica: Network error: Client connection negotiation failed: client connection to 127.17.60.193:35133: connect: Connection refused (error 111)
W20250112 15:08:39.160140 23686 auto_rebalancer.cc:263] scheduled replica move failed to complete: Network error: Client connection negotiation failed: client connection to 127.17.60.193:35133: connect: Connection refused (error 111)
W20250112 15:08:39.160537 23686 auto_rebalancer.cc:663] Could not move replica: Network error: Client connection negotiation failed: client connection to 127.17.60.194:33901: connect: Connection refused (error 111)
W20250112 15:08:39.160606 23686 auto_rebalancer.cc:263] scheduled replica move failed to complete: Network error: Client connection negotiation failed: client connection to 127.17.60.194:33901: connect: Connection refused (error 111)
W20250112 15:08:40.161466 23686 auto_rebalancer.cc:253] failed to send replica move request: Network error: Client connection negotiation failed: client connection to 127.17.60.193:35133: connect: Connection refused (error 111)
W20250112 15:08:40.162124 23686 auto_rebalancer.cc:663] Could not move replica: Network error: Client connection negotiation failed: client connection to 127.17.60.193:35133: connect: Connection refused (error 111)
W20250112 15:08:40.162174 23686 auto_rebalancer.cc:263] scheduled replica move failed to complete: Network error: Client connection negotiation failed: client connection to 127.17.60.193:35133: connect: Connection refused (error 111)
W20250112 15:08:40.162520 23686 auto_rebalancer.cc:663] Could not move replica: Network error: Client connection negotiation failed: client connection to 127.17.60.195:44611: connect: Connection refused (error 111)
W20250112 15:08:40.162559 23686 auto_rebalancer.cc:263] scheduled replica move failed to complete: Network error: Client connection negotiation failed: client connection to 127.17.60.195:44611: connect: Connection refused (error 111)
W20250112 15:08:40.162895 23686 auto_rebalancer.cc:663] Could not move replica: Network error: Client connection negotiation failed: client connection to 127.17.60.194:33901: connect: Connection refused (error 111)
W20250112 15:08:40.162930 23686 auto_rebalancer.cc:263] scheduled replica move failed to complete: Network error: Client connection negotiation failed: client connection to 127.17.60.194:33901: connect: Connection refused (error 111)
W20250112 15:08:41.163947 23686 auto_rebalancer.cc:253] failed to send replica move request: Network error: Client connection negotiation failed: client connection to 127.17.60.195:44611: connect: Connection refused (error 111)
W20250112 15:08:41.164588 23686 auto_rebalancer.cc:663] Could not move replica: Network error: Client connection negotiation failed: client connection to 127.17.60.195:44611: connect: Connection refused (error 111)
W20250112 15:08:41.164639 23686 auto_rebalancer.cc:263] scheduled replica move failed to complete: Network error: Client connection negotiation failed: client connection to 127.17.60.195:44611: connect: Connection refused (error 111)
W20250112 15:08:41.165010 23686 auto_rebalancer.cc:663] Could not move replica: Network error: Client connection negotiation failed: client connection to 127.17.60.193:35133: connect: Connection refused (error 111)
W20250112 15:08:41.165055 23686 auto_rebalancer.cc:263] scheduled replica move failed to complete: Network error: Client connection negotiation failed: client connection to 127.17.60.193:35133: connect: Connection refused (error 111)
W20250112 15:08:41.165477 23686 auto_rebalancer.cc:663] Could not move replica: Network error: Client connection negotiation failed: client connection to 127.17.60.194:33901: connect: Connection refused (error 111)
W20250112 15:08:41.165515 23686 auto_rebalancer.cc:263] scheduled replica move failed to complete: Network error: Client connection negotiation failed: client connection to 127.17.60.194:33901: connect: Connection refused (error 111)
W20250112 15:08:42.165839 23686 auto_rebalancer.cc:227] Could not retrieve cluster info: Not found: tserver 111d62a4d211430fa930ff5a4dfa99d0 not available for placement
I20250112 15:08:42.241835 17651 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250112 15:08:42.243722 24059 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250112 15:08:42.244397 24061 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250112 15:08:42.244733 24058 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250112 15:08:42.245538 17651 server_base.cc:1034] running on GCE node
I20250112 15:08:42.245628 17651 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250112 15:08:42.245651 17651 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250112 15:08:42.245661 17651 hybrid_clock.cc:648] HybridClock initialized: now 1736694522245662 us; error 0 us; skew 500 ppm
I20250112 15:08:42.251157 17651 webserver.cc:458] Webserver started at http://127.17.60.193:36285/ using document root <none> and password file <none>
I20250112 15:08:42.251260 17651 fs_manager.cc:362] Metadata directory not provided
I20250112 15:08:42.251307 17651 fs_manager.cc:365] Using existing metadata directory in first data directory
I20250112 15:08:42.251888 17651 fs_manager.cc:714] Time spent opening directory manager: real 0.000s user 0.000s sys 0.000s
I20250112 15:08:42.252475 24066 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250112 15:08:42.252609 17651 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20250112 15:08:42.252657 17651 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task95dLSD/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestHandlingFailedTservers.1736694351070047-17651-0/minicluster-data/ts-0-root
uuid: "111d62a4d211430fa930ff5a4dfa99d0"
format_stamp: "Formatted at 2025-01-12 15:08:35 on dist-test-slave-b4nz"
I20250112 15:08:42.252740 17651 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task95dLSD/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestHandlingFailedTservers.1736694351070047-17651-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-task95dLSD/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestHandlingFailedTservers.1736694351070047-17651-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-task95dLSD/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestHandlingFailedTservers.1736694351070047-17651-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250112 15:08:42.256731 17651 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250112 15:08:42.256897 17651 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250112 15:08:42.257026 17651 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250112 15:08:42.257314 24073 ts_tablet_manager.cc:536] Loading tablet metadata (0/4 complete)
I20250112 15:08:42.258556 17651 ts_tablet_manager.cc:579] Loaded tablet metadata (4 total tablets, 4 live tablets)
I20250112 15:08:42.258612 17651 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.001s user 0.000s sys 0.000s
I20250112 15:08:42.258648 17651 ts_tablet_manager.cc:594] Registering tablets (0/4 complete)
I20250112 15:08:42.259119 24073 tablet_bootstrap.cc:492] T 6a1c881a80004842a1eb46bb488d3834 P 111d62a4d211430fa930ff5a4dfa99d0: Bootstrap starting.
I20250112 15:08:42.259614 17651 ts_tablet_manager.cc:610] Registered 4 tablets
I20250112 15:08:42.259660 17651 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20250112 15:08:42.260057 24073 tablet_bootstrap.cc:492] T 6a1c881a80004842a1eb46bb488d3834 P 111d62a4d211430fa930ff5a4dfa99d0: Bootstrap replayed 1/1 log segments. Stats: ops{read=0 overwritten=0 applied=0 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250112 15:08:42.260229 24073 tablet_bootstrap.cc:492] T 6a1c881a80004842a1eb46bb488d3834 P 111d62a4d211430fa930ff5a4dfa99d0: Bootstrap complete.
I20250112 15:08:42.260336 24073 ts_tablet_manager.cc:1397] T 6a1c881a80004842a1eb46bb488d3834 P 111d62a4d211430fa930ff5a4dfa99d0: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20250112 15:08:42.260465 24073 raft_consensus.cc:357] T 6a1c881a80004842a1eb46bb488d3834 P 111d62a4d211430fa930ff5a4dfa99d0 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "111d62a4d211430fa930ff5a4dfa99d0" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 35133 } } peers { permanent_uuid: "8586159698b04cde88062accfc5d1bf1" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 33901 } } peers { permanent_uuid: "05ca05df85b94bc985b549e68caa6c36" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 44611 } }
I20250112 15:08:42.260550 24073 raft_consensus.cc:738] T 6a1c881a80004842a1eb46bb488d3834 P 111d62a4d211430fa930ff5a4dfa99d0 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 111d62a4d211430fa930ff5a4dfa99d0, State: Initialized, Role: FOLLOWER
I20250112 15:08:42.260655 24073 consensus_queue.cc:260] T 6a1c881a80004842a1eb46bb488d3834 P 111d62a4d211430fa930ff5a4dfa99d0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "111d62a4d211430fa930ff5a4dfa99d0" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 35133 } } peers { permanent_uuid: "8586159698b04cde88062accfc5d1bf1" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 33901 } } peers { permanent_uuid: "05ca05df85b94bc985b549e68caa6c36" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 44611 } }
I20250112 15:08:42.260820 24073 ts_tablet_manager.cc:1428] T 6a1c881a80004842a1eb46bb488d3834 P 111d62a4d211430fa930ff5a4dfa99d0: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20250112 15:08:42.260877 24073 tablet_bootstrap.cc:492] T 483cfa86b848433bbc365bae489ed251 P 111d62a4d211430fa930ff5a4dfa99d0: Bootstrap starting.
I20250112 15:08:42.261780 24073 tablet_bootstrap.cc:492] T 483cfa86b848433bbc365bae489ed251 P 111d62a4d211430fa930ff5a4dfa99d0: Bootstrap replayed 1/1 log segments. Stats: ops{read=0 overwritten=0 applied=0 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250112 15:08:42.261945 24073 tablet_bootstrap.cc:492] T 483cfa86b848433bbc365bae489ed251 P 111d62a4d211430fa930ff5a4dfa99d0: Bootstrap complete.
I20250112 15:08:42.262051 24073 ts_tablet_manager.cc:1397] T 483cfa86b848433bbc365bae489ed251 P 111d62a4d211430fa930ff5a4dfa99d0: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20250112 15:08:42.262166 24073 raft_consensus.cc:357] T 483cfa86b848433bbc365bae489ed251 P 111d62a4d211430fa930ff5a4dfa99d0 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "111d62a4d211430fa930ff5a4dfa99d0" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 35133 } } peers { permanent_uuid: "8586159698b04cde88062accfc5d1bf1" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 33901 } } peers { permanent_uuid: "05ca05df85b94bc985b549e68caa6c36" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 44611 } }
I20250112 15:08:42.262221 24073 raft_consensus.cc:738] T 483cfa86b848433bbc365bae489ed251 P 111d62a4d211430fa930ff5a4dfa99d0 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 111d62a4d211430fa930ff5a4dfa99d0, State: Initialized, Role: FOLLOWER
I20250112 15:08:42.262267 24073 consensus_queue.cc:260] T 483cfa86b848433bbc365bae489ed251 P 111d62a4d211430fa930ff5a4dfa99d0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "111d62a4d211430fa930ff5a4dfa99d0" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 35133 } } peers { permanent_uuid: "8586159698b04cde88062accfc5d1bf1" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 33901 } } peers { permanent_uuid: "05ca05df85b94bc985b549e68caa6c36" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 44611 } }
I20250112 15:08:42.262354 24073 ts_tablet_manager.cc:1428] T 483cfa86b848433bbc365bae489ed251 P 111d62a4d211430fa930ff5a4dfa99d0: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20250112 15:08:42.262408 24073 tablet_bootstrap.cc:492] T 3e43528d2bb24563966ef89a76aa761e P 111d62a4d211430fa930ff5a4dfa99d0: Bootstrap starting.
I20250112 15:08:42.263214 17651 rpc_server.cc:307] RPC server started. Bound to: 127.17.60.193:35133
I20250112 15:08:42.263279 24073 tablet_bootstrap.cc:492] T 3e43528d2bb24563966ef89a76aa761e P 111d62a4d211430fa930ff5a4dfa99d0: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=0 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20250112 15:08:42.263486 17651 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20250112 15:08:42.263491 24073 tablet_bootstrap.cc:492] T 3e43528d2bb24563966ef89a76aa761e P 111d62a4d211430fa930ff5a4dfa99d0: Bootstrap complete.
I20250112 15:08:42.263628 24073 ts_tablet_manager.cc:1397] T 3e43528d2bb24563966ef89a76aa761e P 111d62a4d211430fa930ff5a4dfa99d0: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20250112 15:08:42.263729 24073 raft_consensus.cc:357] T 3e43528d2bb24563966ef89a76aa761e P 111d62a4d211430fa930ff5a4dfa99d0 [term 1 FOLLOWER]: Replica starting. Triggering 1 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "111d62a4d211430fa930ff5a4dfa99d0" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 35133 } } peers { permanent_uuid: "05ca05df85b94bc985b549e68caa6c36" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 44611 } } peers { permanent_uuid: "8586159698b04cde88062accfc5d1bf1" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 33901 } }
I20250112 15:08:42.263810 24073 raft_consensus.cc:738] T 3e43528d2bb24563966ef89a76aa761e P 111d62a4d211430fa930ff5a4dfa99d0 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 111d62a4d211430fa930ff5a4dfa99d0, State: Initialized, Role: FOLLOWER
I20250112 15:08:42.263865 24073 consensus_queue.cc:260] T 3e43528d2bb24563966ef89a76aa761e P 111d62a4d211430fa930ff5a4dfa99d0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "111d62a4d211430fa930ff5a4dfa99d0" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 35133 } } peers { permanent_uuid: "05ca05df85b94bc985b549e68caa6c36" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 44611 } } peers { permanent_uuid: "8586159698b04cde88062accfc5d1bf1" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 33901 } }
I20250112 15:08:42.263891 24134 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.17.60.193:35133 every 8 connection(s)
I20250112 15:08:42.263981 24073 ts_tablet_manager.cc:1428] T 3e43528d2bb24563966ef89a76aa761e P 111d62a4d211430fa930ff5a4dfa99d0: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20250112 15:08:42.264042 24073 tablet_bootstrap.cc:492] T 92c129e062b945df8805bcc412f3a599 P 111d62a4d211430fa930ff5a4dfa99d0: Bootstrap starting.
W20250112 15:08:42.265033 24142 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250112 15:08:42.265053 24073 tablet_bootstrap.cc:492] T 92c129e062b945df8805bcc412f3a599 P 111d62a4d211430fa930ff5a4dfa99d0: Bootstrap replayed 1/1 log segments. Stats: ops{read=0 overwritten=0 applied=0 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250112 15:08:42.265234 24073 tablet_bootstrap.cc:492] T 92c129e062b945df8805bcc412f3a599 P 111d62a4d211430fa930ff5a4dfa99d0: Bootstrap complete.
I20250112 15:08:42.265339 24073 ts_tablet_manager.cc:1397] T 92c129e062b945df8805bcc412f3a599 P 111d62a4d211430fa930ff5a4dfa99d0: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20250112 15:08:42.265447 24073 raft_consensus.cc:357] T 92c129e062b945df8805bcc412f3a599 P 111d62a4d211430fa930ff5a4dfa99d0 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "111d62a4d211430fa930ff5a4dfa99d0" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 35133 } } peers { permanent_uuid: "8586159698b04cde88062accfc5d1bf1" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 33901 } } peers { permanent_uuid: "05ca05df85b94bc985b549e68caa6c36" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 44611 } }
I20250112 15:08:42.265499 24073 raft_consensus.cc:738] T 92c129e062b945df8805bcc412f3a599 P 111d62a4d211430fa930ff5a4dfa99d0 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 111d62a4d211430fa930ff5a4dfa99d0, State: Initialized, Role: FOLLOWER
I20250112 15:08:42.265538 24073 consensus_queue.cc:260] T 92c129e062b945df8805bcc412f3a599 P 111d62a4d211430fa930ff5a4dfa99d0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "111d62a4d211430fa930ff5a4dfa99d0" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 35133 } } peers { permanent_uuid: "8586159698b04cde88062accfc5d1bf1" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 33901 } } peers { permanent_uuid: "05ca05df85b94bc985b549e68caa6c36" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 44611 } }
I20250112 15:08:42.265625 24073 ts_tablet_manager.cc:1428] T 92c129e062b945df8805bcc412f3a599 P 111d62a4d211430fa930ff5a4dfa99d0: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
W20250112 15:08:42.265642 24140 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250112 15:08:42.266266 17651 server_base.cc:1034] running on GCE node
W20250112 15:08:42.267302 24138 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250112 15:08:42.267491 17651 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250112 15:08:42.267536 17651 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250112 15:08:42.267550 17651 hybrid_clock.cc:648] HybridClock initialized: now 1736694522267550 us; error 0 us; skew 500 ppm
I20250112 15:08:42.267844 24135 heartbeater.cc:346] Connected to a master server at 127.17.60.254:42077
I20250112 15:08:42.267886 24135 heartbeater.cc:463] Registering TS with master...
I20250112 15:08:42.267994 24135 heartbeater.cc:510] Master 127.17.60.254:42077 requested a full tablet report, sending...
I20250112 15:08:42.268349 23636 ts_manager.cc:194] Re-registered known tserver with Master: 111d62a4d211430fa930ff5a4dfa99d0 (127.17.60.193:35133)
I20250112 15:08:42.268389 17651 webserver.cc:458] Webserver started at http://127.17.60.194:37787/ using document root <none> and password file <none>
I20250112 15:08:42.268453 17651 fs_manager.cc:362] Metadata directory not provided
I20250112 15:08:42.268488 17651 fs_manager.cc:365] Using existing metadata directory in first data directory
I20250112 15:08:42.268875 23636 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:49264
I20250112 15:08:42.269091 17651 fs_manager.cc:714] Time spent opening directory manager: real 0.000s user 0.000s sys 0.000s
I20250112 15:08:42.269320 24135 heartbeater.cc:502] Master 127.17.60.254:42077 was elected leader, sending a full tablet report...
I20250112 15:08:42.269678 24148 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250112 15:08:42.269812 17651 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20250112 15:08:42.269855 17651 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task95dLSD/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestHandlingFailedTservers.1736694351070047-17651-0/minicluster-data/ts-1-root
uuid: "8586159698b04cde88062accfc5d1bf1"
format_stamp: "Formatted at 2025-01-12 15:08:35 on dist-test-slave-b4nz"
I20250112 15:08:42.269899 17651 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task95dLSD/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestHandlingFailedTservers.1736694351070047-17651-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-task95dLSD/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestHandlingFailedTservers.1736694351070047-17651-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-task95dLSD/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestHandlingFailedTservers.1736694351070047-17651-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250112 15:08:42.275259 17651 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250112 15:08:42.275431 17651 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250112 15:08:42.275543 17651 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250112 15:08:42.277021 17651 ts_tablet_manager.cc:579] Loaded tablet metadata (4 total tablets, 4 live tablets)
I20250112 15:08:42.277077 17651 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.001s user 0.000s sys 0.000s
I20250112 15:08:42.277494 24155 tablet_bootstrap.cc:492] T 6a1c881a80004842a1eb46bb488d3834 P 8586159698b04cde88062accfc5d1bf1: Bootstrap starting.
I20250112 15:08:42.278017 17651 ts_tablet_manager.cc:610] Registered 4 tablets
I20250112 15:08:42.278060 17651 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20250112 15:08:42.278438 24155 tablet_bootstrap.cc:492] T 6a1c881a80004842a1eb46bb488d3834 P 8586159698b04cde88062accfc5d1bf1: Bootstrap replayed 1/1 log segments. Stats: ops{read=0 overwritten=0 applied=0 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250112 15:08:42.278599 24155 tablet_bootstrap.cc:492] T 6a1c881a80004842a1eb46bb488d3834 P 8586159698b04cde88062accfc5d1bf1: Bootstrap complete.
I20250112 15:08:42.278703 24155 ts_tablet_manager.cc:1397] T 6a1c881a80004842a1eb46bb488d3834 P 8586159698b04cde88062accfc5d1bf1: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20250112 15:08:42.278820 24155 raft_consensus.cc:357] T 6a1c881a80004842a1eb46bb488d3834 P 8586159698b04cde88062accfc5d1bf1 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "111d62a4d211430fa930ff5a4dfa99d0" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 35133 } } peers { permanent_uuid: "8586159698b04cde88062accfc5d1bf1" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 33901 } } peers { permanent_uuid: "05ca05df85b94bc985b549e68caa6c36" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 44611 } }
I20250112 15:08:42.278884 24155 raft_consensus.cc:738] T 6a1c881a80004842a1eb46bb488d3834 P 8586159698b04cde88062accfc5d1bf1 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8586159698b04cde88062accfc5d1bf1, State: Initialized, Role: FOLLOWER
I20250112 15:08:42.278982 24155 consensus_queue.cc:260] T 6a1c881a80004842a1eb46bb488d3834 P 8586159698b04cde88062accfc5d1bf1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "111d62a4d211430fa930ff5a4dfa99d0" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 35133 } } peers { permanent_uuid: "8586159698b04cde88062accfc5d1bf1" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 33901 } } peers { permanent_uuid: "05ca05df85b94bc985b549e68caa6c36" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 44611 } }
I20250112 15:08:42.279140 24155 ts_tablet_manager.cc:1428] T 6a1c881a80004842a1eb46bb488d3834 P 8586159698b04cde88062accfc5d1bf1: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20250112 15:08:42.279206 24155 tablet_bootstrap.cc:492] T 483cfa86b848433bbc365bae489ed251 P 8586159698b04cde88062accfc5d1bf1: Bootstrap starting.
I20250112 15:08:42.280184 24155 tablet_bootstrap.cc:492] T 483cfa86b848433bbc365bae489ed251 P 8586159698b04cde88062accfc5d1bf1: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=0 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20250112 15:08:42.280411 24155 tablet_bootstrap.cc:492] T 483cfa86b848433bbc365bae489ed251 P 8586159698b04cde88062accfc5d1bf1: Bootstrap complete.
I20250112 15:08:42.280508 24155 ts_tablet_manager.cc:1397] T 483cfa86b848433bbc365bae489ed251 P 8586159698b04cde88062accfc5d1bf1: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20250112 15:08:42.280674 24155 raft_consensus.cc:357] T 483cfa86b848433bbc365bae489ed251 P 8586159698b04cde88062accfc5d1bf1 [term 1 FOLLOWER]: Replica starting. Triggering 1 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "111d62a4d211430fa930ff5a4dfa99d0" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 35133 } } peers { permanent_uuid: "8586159698b04cde88062accfc5d1bf1" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 33901 } } peers { permanent_uuid: "05ca05df85b94bc985b549e68caa6c36" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 44611 } }
I20250112 15:08:42.280735 24155 raft_consensus.cc:738] T 483cfa86b848433bbc365bae489ed251 P 8586159698b04cde88062accfc5d1bf1 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8586159698b04cde88062accfc5d1bf1, State: Initialized, Role: FOLLOWER
I20250112 15:08:42.280786 24155 consensus_queue.cc:260] T 483cfa86b848433bbc365bae489ed251 P 8586159698b04cde88062accfc5d1bf1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "111d62a4d211430fa930ff5a4dfa99d0" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 35133 } } peers { permanent_uuid: "8586159698b04cde88062accfc5d1bf1" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 33901 } } peers { permanent_uuid: "05ca05df85b94bc985b549e68caa6c36" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 44611 } }
I20250112 15:08:42.280872 24155 ts_tablet_manager.cc:1428] T 483cfa86b848433bbc365bae489ed251 P 8586159698b04cde88062accfc5d1bf1: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20250112 15:08:42.280923 24155 tablet_bootstrap.cc:492] T 3e43528d2bb24563966ef89a76aa761e P 8586159698b04cde88062accfc5d1bf1: Bootstrap starting.
I20250112 15:08:42.281965 17651 rpc_server.cc:307] RPC server started. Bound to: 127.17.60.194:33901
I20250112 15:08:42.281997 24155 tablet_bootstrap.cc:492] T 3e43528d2bb24563966ef89a76aa761e P 8586159698b04cde88062accfc5d1bf1: Bootstrap replayed 1/1 log segments. Stats: ops{read=0 overwritten=0 applied=0 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250112 15:08:42.282006 24216 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.17.60.194:33901 every 8 connection(s)
I20250112 15:08:42.282202 24155 tablet_bootstrap.cc:492] T 3e43528d2bb24563966ef89a76aa761e P 8586159698b04cde88062accfc5d1bf1: Bootstrap complete.
I20250112 15:08:42.282312 24155 ts_tablet_manager.cc:1397] T 3e43528d2bb24563966ef89a76aa761e P 8586159698b04cde88062accfc5d1bf1: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20250112 15:08:42.282332 17651 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20250112 15:08:42.282464 24155 raft_consensus.cc:357] T 3e43528d2bb24563966ef89a76aa761e P 8586159698b04cde88062accfc5d1bf1 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "111d62a4d211430fa930ff5a4dfa99d0" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 35133 } } peers { permanent_uuid: "05ca05df85b94bc985b549e68caa6c36" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 44611 } } peers { permanent_uuid: "8586159698b04cde88062accfc5d1bf1" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 33901 } }
I20250112 15:08:42.282545 24155 raft_consensus.cc:738] T 3e43528d2bb24563966ef89a76aa761e P 8586159698b04cde88062accfc5d1bf1 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8586159698b04cde88062accfc5d1bf1, State: Initialized, Role: FOLLOWER
I20250112 15:08:42.282634 24155 consensus_queue.cc:260] T 3e43528d2bb24563966ef89a76aa761e P 8586159698b04cde88062accfc5d1bf1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "111d62a4d211430fa930ff5a4dfa99d0" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 35133 } } peers { permanent_uuid: "05ca05df85b94bc985b549e68caa6c36" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 44611 } } peers { permanent_uuid: "8586159698b04cde88062accfc5d1bf1" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 33901 } }
I20250112 15:08:42.282778 24155 ts_tablet_manager.cc:1428] T 3e43528d2bb24563966ef89a76aa761e P 8586159698b04cde88062accfc5d1bf1: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20250112 15:08:42.282860 24155 tablet_bootstrap.cc:492] T 92c129e062b945df8805bcc412f3a599 P 8586159698b04cde88062accfc5d1bf1: Bootstrap starting.
W20250112 15:08:42.283862 24220 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250112 15:08:42.284077 24155 tablet_bootstrap.cc:492] T 92c129e062b945df8805bcc412f3a599 P 8586159698b04cde88062accfc5d1bf1: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=0 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20250112 15:08:42.284284 24155 tablet_bootstrap.cc:492] T 92c129e062b945df8805bcc412f3a599 P 8586159698b04cde88062accfc5d1bf1: Bootstrap complete.
I20250112 15:08:42.284374 24155 ts_tablet_manager.cc:1397] T 92c129e062b945df8805bcc412f3a599 P 8586159698b04cde88062accfc5d1bf1: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
W20250112 15:08:42.284453 24221 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250112 15:08:42.284514 24155 raft_consensus.cc:357] T 92c129e062b945df8805bcc412f3a599 P 8586159698b04cde88062accfc5d1bf1 [term 1 FOLLOWER]: Replica starting. Triggering 1 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "111d62a4d211430fa930ff5a4dfa99d0" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 35133 } } peers { permanent_uuid: "8586159698b04cde88062accfc5d1bf1" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 33901 } } peers { permanent_uuid: "05ca05df85b94bc985b549e68caa6c36" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 44611 } }
I20250112 15:08:42.284668 24155 raft_consensus.cc:738] T 92c129e062b945df8805bcc412f3a599 P 8586159698b04cde88062accfc5d1bf1 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8586159698b04cde88062accfc5d1bf1, State: Initialized, Role: FOLLOWER
I20250112 15:08:42.284731 24155 consensus_queue.cc:260] T 92c129e062b945df8805bcc412f3a599 P 8586159698b04cde88062accfc5d1bf1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "111d62a4d211430fa930ff5a4dfa99d0" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 35133 } } peers { permanent_uuid: "8586159698b04cde88062accfc5d1bf1" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 33901 } } peers { permanent_uuid: "05ca05df85b94bc985b549e68caa6c36" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 44611 } }
I20250112 15:08:42.284864 24155 ts_tablet_manager.cc:1428] T 92c129e062b945df8805bcc412f3a599 P 8586159698b04cde88062accfc5d1bf1: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
W20250112 15:08:42.285019 24223 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250112 15:08:42.285806 17651 server_base.cc:1034] running on GCE node
I20250112 15:08:42.285872 17651 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
I20250112 15:08:42.285878 24217 heartbeater.cc:346] Connected to a master server at 127.17.60.254:42077
W20250112 15:08:42.285915 17651 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250112 15:08:42.285972 24217 heartbeater.cc:463] Registering TS with master...
I20250112 15:08:42.285974 17651 hybrid_clock.cc:648] HybridClock initialized: now 1736694522285974 us; error 0 us; skew 500 ppm
I20250112 15:08:42.286091 24217 heartbeater.cc:510] Master 127.17.60.254:42077 requested a full tablet report, sending...
I20250112 15:08:42.286437 23636 ts_manager.cc:194] Re-registered known tserver with Master: 8586159698b04cde88062accfc5d1bf1 (127.17.60.194:33901)
I20250112 15:08:42.286708 17651 webserver.cc:458] Webserver started at http://127.17.60.195:41529/ using document root <none> and password file <none>
I20250112 15:08:42.286787 17651 fs_manager.cc:362] Metadata directory not provided
I20250112 15:08:42.286827 17651 fs_manager.cc:365] Using existing metadata directory in first data directory
I20250112 15:08:42.286872 23636 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:49268
I20250112 15:08:42.287349 24217 heartbeater.cc:502] Master 127.17.60.254:42077 was elected leader, sending a full tablet report...
I20250112 15:08:42.287503 17651 fs_manager.cc:714] Time spent opening directory manager: real 0.000s user 0.000s sys 0.001s
I20250112 15:08:42.288136 24229 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250112 15:08:42.288264 17651 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20250112 15:08:42.288312 17651 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task95dLSD/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestHandlingFailedTservers.1736694351070047-17651-0/minicluster-data/ts-2-root
uuid: "05ca05df85b94bc985b549e68caa6c36"
format_stamp: "Formatted at 2025-01-12 15:08:35 on dist-test-slave-b4nz"
I20250112 15:08:42.288357 17651 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task95dLSD/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestHandlingFailedTservers.1736694351070047-17651-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-task95dLSD/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestHandlingFailedTservers.1736694351070047-17651-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-task95dLSD/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestHandlingFailedTservers.1736694351070047-17651-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250112 15:08:42.293427 17651 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250112 15:08:42.293653 17651 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250112 15:08:42.293792 17651 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250112 15:08:42.295105 17651 ts_tablet_manager.cc:579] Loaded tablet metadata (4 total tablets, 4 live tablets)
I20250112 15:08:42.295136 17651 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.001s user 0.000s sys 0.000s
I20250112 15:08:42.295524 24236 tablet_bootstrap.cc:492] T 6a1c881a80004842a1eb46bb488d3834 P 05ca05df85b94bc985b549e68caa6c36: Bootstrap starting.
I20250112 15:08:42.295967 17651 ts_tablet_manager.cc:610] Registered 4 tablets
I20250112 15:08:42.296002 17651 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20250112 15:08:42.296715 24236 tablet_bootstrap.cc:492] T 6a1c881a80004842a1eb46bb488d3834 P 05ca05df85b94bc985b549e68caa6c36: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=0 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20250112 15:08:42.296931 24236 tablet_bootstrap.cc:492] T 6a1c881a80004842a1eb46bb488d3834 P 05ca05df85b94bc985b549e68caa6c36: Bootstrap complete.
I20250112 15:08:42.297066 24236 ts_tablet_manager.cc:1397] T 6a1c881a80004842a1eb46bb488d3834 P 05ca05df85b94bc985b549e68caa6c36: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20250112 15:08:42.297237 24236 raft_consensus.cc:357] T 6a1c881a80004842a1eb46bb488d3834 P 05ca05df85b94bc985b549e68caa6c36 [term 1 FOLLOWER]: Replica starting. Triggering 1 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "111d62a4d211430fa930ff5a4dfa99d0" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 35133 } } peers { permanent_uuid: "8586159698b04cde88062accfc5d1bf1" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 33901 } } peers { permanent_uuid: "05ca05df85b94bc985b549e68caa6c36" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 44611 } }
I20250112 15:08:42.297325 24236 raft_consensus.cc:738] T 6a1c881a80004842a1eb46bb488d3834 P 05ca05df85b94bc985b549e68caa6c36 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 05ca05df85b94bc985b549e68caa6c36, State: Initialized, Role: FOLLOWER
I20250112 15:08:42.297398 24236 consensus_queue.cc:260] T 6a1c881a80004842a1eb46bb488d3834 P 05ca05df85b94bc985b549e68caa6c36 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "111d62a4d211430fa930ff5a4dfa99d0" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 35133 } } peers { permanent_uuid: "8586159698b04cde88062accfc5d1bf1" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 33901 } } peers { permanent_uuid: "05ca05df85b94bc985b549e68caa6c36" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 44611 } }
I20250112 15:08:42.297572 24236 ts_tablet_manager.cc:1428] T 6a1c881a80004842a1eb46bb488d3834 P 05ca05df85b94bc985b549e68caa6c36: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20250112 15:08:42.297641 24236 tablet_bootstrap.cc:492] T 483cfa86b848433bbc365bae489ed251 P 05ca05df85b94bc985b549e68caa6c36: Bootstrap starting.
I20250112 15:08:42.298611 24236 tablet_bootstrap.cc:492] T 483cfa86b848433bbc365bae489ed251 P 05ca05df85b94bc985b549e68caa6c36: Bootstrap replayed 1/1 log segments. Stats: ops{read=0 overwritten=0 applied=0 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250112 15:08:42.298766 24236 tablet_bootstrap.cc:492] T 483cfa86b848433bbc365bae489ed251 P 05ca05df85b94bc985b549e68caa6c36: Bootstrap complete.
I20250112 15:08:42.298859 24236 ts_tablet_manager.cc:1397] T 483cfa86b848433bbc365bae489ed251 P 05ca05df85b94bc985b549e68caa6c36: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20250112 15:08:42.299036 24236 raft_consensus.cc:357] T 483cfa86b848433bbc365bae489ed251 P 05ca05df85b94bc985b549e68caa6c36 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "111d62a4d211430fa930ff5a4dfa99d0" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 35133 } } peers { permanent_uuid: "8586159698b04cde88062accfc5d1bf1" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 33901 } } peers { permanent_uuid: "05ca05df85b94bc985b549e68caa6c36" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 44611 } }
I20250112 15:08:42.299088 24236 raft_consensus.cc:738] T 483cfa86b848433bbc365bae489ed251 P 05ca05df85b94bc985b549e68caa6c36 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 05ca05df85b94bc985b549e68caa6c36, State: Initialized, Role: FOLLOWER
I20250112 15:08:42.299136 24236 consensus_queue.cc:260] T 483cfa86b848433bbc365bae489ed251 P 05ca05df85b94bc985b549e68caa6c36 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "111d62a4d211430fa930ff5a4dfa99d0" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 35133 } } peers { permanent_uuid: "8586159698b04cde88062accfc5d1bf1" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 33901 } } peers { permanent_uuid: "05ca05df85b94bc985b549e68caa6c36" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 44611 } }
I20250112 15:08:42.299221 24236 ts_tablet_manager.cc:1428] T 483cfa86b848433bbc365bae489ed251 P 05ca05df85b94bc985b549e68caa6c36: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20250112 15:08:42.299268 24236 tablet_bootstrap.cc:492] T 3e43528d2bb24563966ef89a76aa761e P 05ca05df85b94bc985b549e68caa6c36: Bootstrap starting.
I20250112 15:08:42.299758 17651 rpc_server.cc:307] RPC server started. Bound to: 127.17.60.195:44611
I20250112 15:08:42.299829 24296 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.17.60.195:44611 every 8 connection(s)
I20250112 15:08:42.300324 24236 tablet_bootstrap.cc:492] T 3e43528d2bb24563966ef89a76aa761e P 05ca05df85b94bc985b549e68caa6c36: Bootstrap replayed 1/1 log segments. Stats: ops{read=0 overwritten=0 applied=0 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250112 15:08:42.300503 24236 tablet_bootstrap.cc:492] T 3e43528d2bb24563966ef89a76aa761e P 05ca05df85b94bc985b549e68caa6c36: Bootstrap complete.
I20250112 15:08:42.300616 24236 ts_tablet_manager.cc:1397] T 3e43528d2bb24563966ef89a76aa761e P 05ca05df85b94bc985b549e68caa6c36: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20250112 15:08:42.300770 24236 raft_consensus.cc:357] T 3e43528d2bb24563966ef89a76aa761e P 05ca05df85b94bc985b549e68caa6c36 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "111d62a4d211430fa930ff5a4dfa99d0" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 35133 } } peers { permanent_uuid: "05ca05df85b94bc985b549e68caa6c36" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 44611 } } peers { permanent_uuid: "8586159698b04cde88062accfc5d1bf1" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 33901 } }
I20250112 15:08:42.300825 24236 raft_consensus.cc:738] T 3e43528d2bb24563966ef89a76aa761e P 05ca05df85b94bc985b549e68caa6c36 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 05ca05df85b94bc985b549e68caa6c36, State: Initialized, Role: FOLLOWER
I20250112 15:08:42.300876 24236 consensus_queue.cc:260] T 3e43528d2bb24563966ef89a76aa761e P 05ca05df85b94bc985b549e68caa6c36 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "111d62a4d211430fa930ff5a4dfa99d0" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 35133 } } peers { permanent_uuid: "05ca05df85b94bc985b549e68caa6c36" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 44611 } } peers { permanent_uuid: "8586159698b04cde88062accfc5d1bf1" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 33901 } }
I20250112 15:08:42.300971 24236 ts_tablet_manager.cc:1428] T 3e43528d2bb24563966ef89a76aa761e P 05ca05df85b94bc985b549e68caa6c36: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20250112 15:08:42.301025 24236 tablet_bootstrap.cc:492] T 92c129e062b945df8805bcc412f3a599 P 05ca05df85b94bc985b549e68caa6c36: Bootstrap starting.
I20250112 15:08:42.302011 24236 tablet_bootstrap.cc:492] T 92c129e062b945df8805bcc412f3a599 P 05ca05df85b94bc985b549e68caa6c36: Bootstrap replayed 1/1 log segments. Stats: ops{read=0 overwritten=0 applied=0 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250112 15:08:42.302162 24236 tablet_bootstrap.cc:492] T 92c129e062b945df8805bcc412f3a599 P 05ca05df85b94bc985b549e68caa6c36: Bootstrap complete.
I20250112 15:08:42.302263 24236 ts_tablet_manager.cc:1397] T 92c129e062b945df8805bcc412f3a599 P 05ca05df85b94bc985b549e68caa6c36: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20250112 15:08:42.302434 24236 raft_consensus.cc:357] T 92c129e062b945df8805bcc412f3a599 P 05ca05df85b94bc985b549e68caa6c36 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "111d62a4d211430fa930ff5a4dfa99d0" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 35133 } } peers { permanent_uuid: "8586159698b04cde88062accfc5d1bf1" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 33901 } } peers { permanent_uuid: "05ca05df85b94bc985b549e68caa6c36" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 44611 } }
I20250112 15:08:42.302510 24236 raft_consensus.cc:738] T 92c129e062b945df8805bcc412f3a599 P 05ca05df85b94bc985b549e68caa6c36 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 05ca05df85b94bc985b549e68caa6c36, State: Initialized, Role: FOLLOWER
I20250112 15:08:42.302557 24236 consensus_queue.cc:260] T 92c129e062b945df8805bcc412f3a599 P 05ca05df85b94bc985b549e68caa6c36 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "111d62a4d211430fa930ff5a4dfa99d0" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 35133 } } peers { permanent_uuid: "8586159698b04cde88062accfc5d1bf1" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 33901 } } peers { permanent_uuid: "05ca05df85b94bc985b549e68caa6c36" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 44611 } }
I20250112 15:08:42.302699 24236 ts_tablet_manager.cc:1428] T 92c129e062b945df8805bcc412f3a599 P 05ca05df85b94bc985b549e68caa6c36: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20250112 15:08:42.303292 24297 heartbeater.cc:346] Connected to a master server at 127.17.60.254:42077
I20250112 15:08:42.303341 24297 heartbeater.cc:463] Registering TS with master...
I20250112 15:08:42.303436 24297 heartbeater.cc:510] Master 127.17.60.254:42077 requested a full tablet report, sending...
I20250112 15:08:42.303694 23636 ts_manager.cc:194] Re-registered known tserver with Master: 05ca05df85b94bc985b549e68caa6c36 (127.17.60.195:44611)
I20250112 15:08:42.304062 23636 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:49282
I20250112 15:08:42.304400 24297 heartbeater.cc:502] Master 127.17.60.254:42077 was elected leader, sending a full tablet report...
W20250112 15:08:43.169471 23686 auto_rebalancer.cc:253] failed to send replica move request: Illegal state: Replica 111d62a4d211430fa930ff5a4dfa99d0 is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "111d62a4d211430fa930ff5a4dfa99d0" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 35133 } } peers { permanent_uuid: "05ca05df85b94bc985b549e68caa6c36" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 44611 } } peers { permanent_uuid: "8586159698b04cde88062accfc5d1bf1" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 33901 } } }
W20250112 15:08:43.169857 23686 auto_rebalancer.cc:663] Could not move replica: Incomplete: tablet 3e43528d2bb24563966ef89a76aa761e, TS 8586159698b04cde88062accfc5d1bf1 -> TS 932773f4da6d4c238b6963b863c59be9 move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "111d62a4d211430fa930ff5a4dfa99d0" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 35133 } } peers { permanent_uuid: "05ca05df85b94bc985b549e68caa6c36" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 44611 } } peers { permanent_uuid: "8586159698b04cde88062accfc5d1bf1" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 33901 } }
W20250112 15:08:43.169898 23686 auto_rebalancer.cc:263] scheduled replica move failed to complete: Incomplete: tablet 3e43528d2bb24563966ef89a76aa761e, TS 8586159698b04cde88062accfc5d1bf1 -> TS 932773f4da6d4c238b6963b863c59be9 move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "111d62a4d211430fa930ff5a4dfa99d0" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 35133 } } peers { permanent_uuid: "05ca05df85b94bc985b549e68caa6c36" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 44611 } } peers { permanent_uuid: "8586159698b04cde88062accfc5d1bf1" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 33901 } }
W20250112 15:08:43.172534 23686 auto_rebalancer.cc:663] Could not move replica: Incomplete: tablet 483cfa86b848433bbc365bae489ed251, TS 05ca05df85b94bc985b549e68caa6c36 -> TS 932773f4da6d4c238b6963b863c59be9 move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "111d62a4d211430fa930ff5a4dfa99d0" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 35133 } } peers { permanent_uuid: "8586159698b04cde88062accfc5d1bf1" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 33901 } } peers { permanent_uuid: "05ca05df85b94bc985b549e68caa6c36" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 44611 } }
W20250112 15:08:43.172612 23686 auto_rebalancer.cc:263] scheduled replica move failed to complete: Incomplete: tablet 483cfa86b848433bbc365bae489ed251, TS 05ca05df85b94bc985b549e68caa6c36 -> TS 932773f4da6d4c238b6963b863c59be9 move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "111d62a4d211430fa930ff5a4dfa99d0" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 35133 } } peers { permanent_uuid: "8586159698b04cde88062accfc5d1bf1" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 33901 } } peers { permanent_uuid: "05ca05df85b94bc985b549e68caa6c36" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 44611 } }
W20250112 15:08:43.172952 23686 auto_rebalancer.cc:663] Could not move replica: Incomplete: tablet 92c129e062b945df8805bcc412f3a599, TS 111d62a4d211430fa930ff5a4dfa99d0 -> TS 932773f4da6d4c238b6963b863c59be9 move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "111d62a4d211430fa930ff5a4dfa99d0" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 35133 } } peers { permanent_uuid: "8586159698b04cde88062accfc5d1bf1" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 33901 } } peers { permanent_uuid: "05ca05df85b94bc985b549e68caa6c36" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 44611 } }
W20250112 15:08:43.172989 23686 auto_rebalancer.cc:263] scheduled replica move failed to complete: Incomplete: tablet 92c129e062b945df8805bcc412f3a599, TS 111d62a4d211430fa930ff5a4dfa99d0 -> TS 932773f4da6d4c238b6963b863c59be9 move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "111d62a4d211430fa930ff5a4dfa99d0" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 35133 } } peers { permanent_uuid: "8586159698b04cde88062accfc5d1bf1" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 33901 } } peers { permanent_uuid: "05ca05df85b94bc985b549e68caa6c36" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 44611 } }
I20250112 15:08:43.429001 24305 raft_consensus.cc:491] T 92c129e062b945df8805bcc412f3a599 P 05ca05df85b94bc985b549e68caa6c36 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 15:08:43.429136 24305 raft_consensus.cc:513] T 92c129e062b945df8805bcc412f3a599 P 05ca05df85b94bc985b549e68caa6c36 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "111d62a4d211430fa930ff5a4dfa99d0" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 35133 } } peers { permanent_uuid: "8586159698b04cde88062accfc5d1bf1" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 33901 } } peers { permanent_uuid: "05ca05df85b94bc985b549e68caa6c36" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 44611 } }
I20250112 15:08:43.429352 24305 leader_election.cc:290] T 92c129e062b945df8805bcc412f3a599 P 05ca05df85b94bc985b549e68caa6c36 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 111d62a4d211430fa930ff5a4dfa99d0 (127.17.60.193:35133), 8586159698b04cde88062accfc5d1bf1 (127.17.60.194:33901)
I20250112 15:08:43.432080 24108 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "92c129e062b945df8805bcc412f3a599" candidate_uuid: "05ca05df85b94bc985b549e68caa6c36" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "111d62a4d211430fa930ff5a4dfa99d0" is_pre_election: true
I20250112 15:08:43.432178 24190 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "92c129e062b945df8805bcc412f3a599" candidate_uuid: "05ca05df85b94bc985b549e68caa6c36" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8586159698b04cde88062accfc5d1bf1" is_pre_election: true
I20250112 15:08:43.432185 24108 raft_consensus.cc:2463] T 92c129e062b945df8805bcc412f3a599 P 111d62a4d211430fa930ff5a4dfa99d0 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 05ca05df85b94bc985b549e68caa6c36 in term 1.
I20250112 15:08:43.432307 24190 raft_consensus.cc:2405] T 92c129e062b945df8805bcc412f3a599 P 8586159698b04cde88062accfc5d1bf1 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 05ca05df85b94bc985b549e68caa6c36 for term 2 because replica has last-logged OpId of term: 1 index: 1, which is greater than that of the candidate, which has last-logged OpId of term: 0 index: 0.
I20250112 15:08:43.432443 24233 leader_election.cc:304] T 92c129e062b945df8805bcc412f3a599 P 05ca05df85b94bc985b549e68caa6c36 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 05ca05df85b94bc985b549e68caa6c36, 111d62a4d211430fa930ff5a4dfa99d0; no voters:
I20250112 15:08:43.432538 24305 raft_consensus.cc:2798] T 92c129e062b945df8805bcc412f3a599 P 05ca05df85b94bc985b549e68caa6c36 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250112 15:08:43.432592 24305 raft_consensus.cc:491] T 92c129e062b945df8805bcc412f3a599 P 05ca05df85b94bc985b549e68caa6c36 [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250112 15:08:43.432616 24305 raft_consensus.cc:3054] T 92c129e062b945df8805bcc412f3a599 P 05ca05df85b94bc985b549e68caa6c36 [term 1 FOLLOWER]: Advancing to term 2
I20250112 15:08:43.433234 24305 raft_consensus.cc:513] T 92c129e062b945df8805bcc412f3a599 P 05ca05df85b94bc985b549e68caa6c36 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "111d62a4d211430fa930ff5a4dfa99d0" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 35133 } } peers { permanent_uuid: "8586159698b04cde88062accfc5d1bf1" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 33901 } } peers { permanent_uuid: "05ca05df85b94bc985b549e68caa6c36" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 44611 } }
I20250112 15:08:43.433344 24305 leader_election.cc:290] T 92c129e062b945df8805bcc412f3a599 P 05ca05df85b94bc985b549e68caa6c36 [CANDIDATE]: Term 2 election: Requested vote from peers 111d62a4d211430fa930ff5a4dfa99d0 (127.17.60.193:35133), 8586159698b04cde88062accfc5d1bf1 (127.17.60.194:33901)
I20250112 15:08:43.433483 24108 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "92c129e062b945df8805bcc412f3a599" candidate_uuid: "05ca05df85b94bc985b549e68caa6c36" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "111d62a4d211430fa930ff5a4dfa99d0"
I20250112 15:08:43.433508 24190 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "92c129e062b945df8805bcc412f3a599" candidate_uuid: "05ca05df85b94bc985b549e68caa6c36" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8586159698b04cde88062accfc5d1bf1"
I20250112 15:08:43.433568 24108 raft_consensus.cc:3054] T 92c129e062b945df8805bcc412f3a599 P 111d62a4d211430fa930ff5a4dfa99d0 [term 1 FOLLOWER]: Advancing to term 2
I20250112 15:08:43.433591 24190 raft_consensus.cc:3054] T 92c129e062b945df8805bcc412f3a599 P 8586159698b04cde88062accfc5d1bf1 [term 1 FOLLOWER]: Advancing to term 2
I20250112 15:08:43.434293 24108 raft_consensus.cc:2463] T 92c129e062b945df8805bcc412f3a599 P 111d62a4d211430fa930ff5a4dfa99d0 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 05ca05df85b94bc985b549e68caa6c36 in term 2.
I20250112 15:08:43.434309 24190 raft_consensus.cc:2405] T 92c129e062b945df8805bcc412f3a599 P 8586159698b04cde88062accfc5d1bf1 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 05ca05df85b94bc985b549e68caa6c36 for term 2 because replica has last-logged OpId of term: 1 index: 1, which is greater than that of the candidate, which has last-logged OpId of term: 0 index: 0.
I20250112 15:08:43.434453 24233 leader_election.cc:304] T 92c129e062b945df8805bcc412f3a599 P 05ca05df85b94bc985b549e68caa6c36 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 05ca05df85b94bc985b549e68caa6c36, 111d62a4d211430fa930ff5a4dfa99d0; no voters:
I20250112 15:08:43.434526 24305 raft_consensus.cc:2798] T 92c129e062b945df8805bcc412f3a599 P 05ca05df85b94bc985b549e68caa6c36 [term 2 FOLLOWER]: Leader election won for term 2
I20250112 15:08:43.434641 24305 raft_consensus.cc:695] T 92c129e062b945df8805bcc412f3a599 P 05ca05df85b94bc985b549e68caa6c36 [term 2 LEADER]: Becoming Leader. State: Replica: 05ca05df85b94bc985b549e68caa6c36, State: Running, Role: LEADER
I20250112 15:08:43.434681 24305 consensus_queue.cc:237] T 92c129e062b945df8805bcc412f3a599 P 05ca05df85b94bc985b549e68caa6c36 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "111d62a4d211430fa930ff5a4dfa99d0" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 35133 } } peers { permanent_uuid: "8586159698b04cde88062accfc5d1bf1" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 33901 } } peers { permanent_uuid: "05ca05df85b94bc985b549e68caa6c36" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 44611 } }
I20250112 15:08:43.435222 23636 catalog_manager.cc:5526] T 92c129e062b945df8805bcc412f3a599 P 05ca05df85b94bc985b549e68caa6c36 reported cstate change: term changed from 1 to 2, leader changed from 8586159698b04cde88062accfc5d1bf1 (127.17.60.194) to 05ca05df85b94bc985b549e68caa6c36 (127.17.60.195). New cstate: current_term: 2 leader_uuid: "05ca05df85b94bc985b549e68caa6c36" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "111d62a4d211430fa930ff5a4dfa99d0" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 35133 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8586159698b04cde88062accfc5d1bf1" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 33901 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "05ca05df85b94bc985b549e68caa6c36" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 44611 } health_report { overall_health: HEALTHY } } }
I20250112 15:08:43.469262 24314 raft_consensus.cc:491] T 6a1c881a80004842a1eb46bb488d3834 P 8586159698b04cde88062accfc5d1bf1 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 15:08:43.469378 24314 raft_consensus.cc:513] T 6a1c881a80004842a1eb46bb488d3834 P 8586159698b04cde88062accfc5d1bf1 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "111d62a4d211430fa930ff5a4dfa99d0" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 35133 } } peers { permanent_uuid: "8586159698b04cde88062accfc5d1bf1" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 33901 } } peers { permanent_uuid: "05ca05df85b94bc985b549e68caa6c36" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 44611 } }
I20250112 15:08:43.469584 24314 leader_election.cc:290] T 6a1c881a80004842a1eb46bb488d3834 P 8586159698b04cde88062accfc5d1bf1 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 111d62a4d211430fa930ff5a4dfa99d0 (127.17.60.193:35133), 05ca05df85b94bc985b549e68caa6c36 (127.17.60.195:44611)
I20250112 15:08:43.473078 24108 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6a1c881a80004842a1eb46bb488d3834" candidate_uuid: "8586159698b04cde88062accfc5d1bf1" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "111d62a4d211430fa930ff5a4dfa99d0" is_pre_election: true
I20250112 15:08:43.473090 24270 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6a1c881a80004842a1eb46bb488d3834" candidate_uuid: "8586159698b04cde88062accfc5d1bf1" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "05ca05df85b94bc985b549e68caa6c36" is_pre_election: true
I20250112 15:08:43.473199 24108 raft_consensus.cc:2463] T 6a1c881a80004842a1eb46bb488d3834 P 111d62a4d211430fa930ff5a4dfa99d0 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8586159698b04cde88062accfc5d1bf1 in term 1.
I20250112 15:08:43.473244 24270 raft_consensus.cc:2405] T 6a1c881a80004842a1eb46bb488d3834 P 05ca05df85b94bc985b549e68caa6c36 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 8586159698b04cde88062accfc5d1bf1 for term 2 because replica has last-logged OpId of term: 1 index: 1, which is greater than that of the candidate, which has last-logged OpId of term: 0 index: 0.
I20250112 15:08:43.473384 24152 leader_election.cc:304] T 6a1c881a80004842a1eb46bb488d3834 P 8586159698b04cde88062accfc5d1bf1 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 111d62a4d211430fa930ff5a4dfa99d0, 8586159698b04cde88062accfc5d1bf1; no voters:
I20250112 15:08:43.473474 24314 raft_consensus.cc:2798] T 6a1c881a80004842a1eb46bb488d3834 P 8586159698b04cde88062accfc5d1bf1 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250112 15:08:43.473515 24314 raft_consensus.cc:491] T 6a1c881a80004842a1eb46bb488d3834 P 8586159698b04cde88062accfc5d1bf1 [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250112 15:08:43.473538 24314 raft_consensus.cc:3054] T 6a1c881a80004842a1eb46bb488d3834 P 8586159698b04cde88062accfc5d1bf1 [term 1 FOLLOWER]: Advancing to term 2
I20250112 15:08:43.474143 24314 raft_consensus.cc:513] T 6a1c881a80004842a1eb46bb488d3834 P 8586159698b04cde88062accfc5d1bf1 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "111d62a4d211430fa930ff5a4dfa99d0" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 35133 } } peers { permanent_uuid: "8586159698b04cde88062accfc5d1bf1" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 33901 } } peers { permanent_uuid: "05ca05df85b94bc985b549e68caa6c36" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 44611 } }
I20250112 15:08:43.474248 24314 leader_election.cc:290] T 6a1c881a80004842a1eb46bb488d3834 P 8586159698b04cde88062accfc5d1bf1 [CANDIDATE]: Term 2 election: Requested vote from peers 111d62a4d211430fa930ff5a4dfa99d0 (127.17.60.193:35133), 05ca05df85b94bc985b549e68caa6c36 (127.17.60.195:44611)
I20250112 15:08:43.474402 24108 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6a1c881a80004842a1eb46bb488d3834" candidate_uuid: "8586159698b04cde88062accfc5d1bf1" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "111d62a4d211430fa930ff5a4dfa99d0"
I20250112 15:08:43.474407 24270 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6a1c881a80004842a1eb46bb488d3834" candidate_uuid: "8586159698b04cde88062accfc5d1bf1" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "05ca05df85b94bc985b549e68caa6c36"
I20250112 15:08:43.474457 24108 raft_consensus.cc:3054] T 6a1c881a80004842a1eb46bb488d3834 P 111d62a4d211430fa930ff5a4dfa99d0 [term 1 FOLLOWER]: Advancing to term 2
I20250112 15:08:43.474473 24270 raft_consensus.cc:3054] T 6a1c881a80004842a1eb46bb488d3834 P 05ca05df85b94bc985b549e68caa6c36 [term 1 FOLLOWER]: Advancing to term 2
I20250112 15:08:43.475010 24108 raft_consensus.cc:2463] T 6a1c881a80004842a1eb46bb488d3834 P 111d62a4d211430fa930ff5a4dfa99d0 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8586159698b04cde88062accfc5d1bf1 in term 2.
I20250112 15:08:43.475138 24270 raft_consensus.cc:2405] T 6a1c881a80004842a1eb46bb488d3834 P 05ca05df85b94bc985b549e68caa6c36 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 8586159698b04cde88062accfc5d1bf1 for term 2 because replica has last-logged OpId of term: 1 index: 1, which is greater than that of the candidate, which has last-logged OpId of term: 0 index: 0.
I20250112 15:08:43.475193 24152 leader_election.cc:304] T 6a1c881a80004842a1eb46bb488d3834 P 8586159698b04cde88062accfc5d1bf1 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 111d62a4d211430fa930ff5a4dfa99d0, 8586159698b04cde88062accfc5d1bf1; no voters:
I20250112 15:08:43.475270 24314 raft_consensus.cc:2798] T 6a1c881a80004842a1eb46bb488d3834 P 8586159698b04cde88062accfc5d1bf1 [term 2 FOLLOWER]: Leader election won for term 2
I20250112 15:08:43.475396 24314 raft_consensus.cc:695] T 6a1c881a80004842a1eb46bb488d3834 P 8586159698b04cde88062accfc5d1bf1 [term 2 LEADER]: Becoming Leader. State: Replica: 8586159698b04cde88062accfc5d1bf1, State: Running, Role: LEADER
I20250112 15:08:43.475438 24314 consensus_queue.cc:237] T 6a1c881a80004842a1eb46bb488d3834 P 8586159698b04cde88062accfc5d1bf1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "111d62a4d211430fa930ff5a4dfa99d0" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 35133 } } peers { permanent_uuid: "8586159698b04cde88062accfc5d1bf1" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 33901 } } peers { permanent_uuid: "05ca05df85b94bc985b549e68caa6c36" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 44611 } }
I20250112 15:08:43.475967 23636 catalog_manager.cc:5526] T 6a1c881a80004842a1eb46bb488d3834 P 8586159698b04cde88062accfc5d1bf1 reported cstate change: term changed from 1 to 2, leader changed from 05ca05df85b94bc985b549e68caa6c36 (127.17.60.195) to 8586159698b04cde88062accfc5d1bf1 (127.17.60.194). New cstate: current_term: 2 leader_uuid: "8586159698b04cde88062accfc5d1bf1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "111d62a4d211430fa930ff5a4dfa99d0" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 35133 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8586159698b04cde88062accfc5d1bf1" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 33901 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "05ca05df85b94bc985b549e68caa6c36" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 44611 } health_report { overall_health: UNKNOWN } } }
I20250112 15:08:43.497346 24305 raft_consensus.cc:491] T 483cfa86b848433bbc365bae489ed251 P 05ca05df85b94bc985b549e68caa6c36 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 15:08:43.497424 24305 raft_consensus.cc:513] T 483cfa86b848433bbc365bae489ed251 P 05ca05df85b94bc985b549e68caa6c36 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "111d62a4d211430fa930ff5a4dfa99d0" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 35133 } } peers { permanent_uuid: "8586159698b04cde88062accfc5d1bf1" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 33901 } } peers { permanent_uuid: "05ca05df85b94bc985b549e68caa6c36" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 44611 } }
I20250112 15:08:43.497583 24305 leader_election.cc:290] T 483cfa86b848433bbc365bae489ed251 P 05ca05df85b94bc985b549e68caa6c36 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 111d62a4d211430fa930ff5a4dfa99d0 (127.17.60.193:35133), 8586159698b04cde88062accfc5d1bf1 (127.17.60.194:33901)
I20250112 15:08:43.497761 24108 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "483cfa86b848433bbc365bae489ed251" candidate_uuid: "05ca05df85b94bc985b549e68caa6c36" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "111d62a4d211430fa930ff5a4dfa99d0" is_pre_election: true
I20250112 15:08:43.497818 24108 raft_consensus.cc:2463] T 483cfa86b848433bbc365bae489ed251 P 111d62a4d211430fa930ff5a4dfa99d0 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 05ca05df85b94bc985b549e68caa6c36 in term 1.
I20250112 15:08:43.497776 24190 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "483cfa86b848433bbc365bae489ed251" candidate_uuid: "05ca05df85b94bc985b549e68caa6c36" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8586159698b04cde88062accfc5d1bf1" is_pre_election: true
I20250112 15:08:43.497867 24190 raft_consensus.cc:2405] T 483cfa86b848433bbc365bae489ed251 P 8586159698b04cde88062accfc5d1bf1 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 05ca05df85b94bc985b549e68caa6c36 for term 2 because replica has last-logged OpId of term: 1 index: 1, which is greater than that of the candidate, which has last-logged OpId of term: 0 index: 0.
I20250112 15:08:43.497936 24233 leader_election.cc:304] T 483cfa86b848433bbc365bae489ed251 P 05ca05df85b94bc985b549e68caa6c36 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 05ca05df85b94bc985b549e68caa6c36, 111d62a4d211430fa930ff5a4dfa99d0; no voters:
I20250112 15:08:43.498041 24305 raft_consensus.cc:2798] T 483cfa86b848433bbc365bae489ed251 P 05ca05df85b94bc985b549e68caa6c36 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250112 15:08:43.498080 24305 raft_consensus.cc:491] T 483cfa86b848433bbc365bae489ed251 P 05ca05df85b94bc985b549e68caa6c36 [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250112 15:08:43.498102 24305 raft_consensus.cc:3054] T 483cfa86b848433bbc365bae489ed251 P 05ca05df85b94bc985b549e68caa6c36 [term 1 FOLLOWER]: Advancing to term 2
I20250112 15:08:43.498695 24305 raft_consensus.cc:513] T 483cfa86b848433bbc365bae489ed251 P 05ca05df85b94bc985b549e68caa6c36 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "111d62a4d211430fa930ff5a4dfa99d0" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 35133 } } peers { permanent_uuid: "8586159698b04cde88062accfc5d1bf1" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 33901 } } peers { permanent_uuid: "05ca05df85b94bc985b549e68caa6c36" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 44611 } }
I20250112 15:08:43.498814 24305 leader_election.cc:290] T 483cfa86b848433bbc365bae489ed251 P 05ca05df85b94bc985b549e68caa6c36 [CANDIDATE]: Term 2 election: Requested vote from peers 111d62a4d211430fa930ff5a4dfa99d0 (127.17.60.193:35133), 8586159698b04cde88062accfc5d1bf1 (127.17.60.194:33901)
I20250112 15:08:43.498937 24108 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "483cfa86b848433bbc365bae489ed251" candidate_uuid: "05ca05df85b94bc985b549e68caa6c36" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "111d62a4d211430fa930ff5a4dfa99d0"
I20250112 15:08:43.498977 24190 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "483cfa86b848433bbc365bae489ed251" candidate_uuid: "05ca05df85b94bc985b549e68caa6c36" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8586159698b04cde88062accfc5d1bf1"
I20250112 15:08:43.499032 24108 raft_consensus.cc:3054] T 483cfa86b848433bbc365bae489ed251 P 111d62a4d211430fa930ff5a4dfa99d0 [term 1 FOLLOWER]: Advancing to term 2
I20250112 15:08:43.499042 24190 raft_consensus.cc:3054] T 483cfa86b848433bbc365bae489ed251 P 8586159698b04cde88062accfc5d1bf1 [term 1 FOLLOWER]: Advancing to term 2
I20250112 15:08:43.499699 24108 raft_consensus.cc:2463] T 483cfa86b848433bbc365bae489ed251 P 111d62a4d211430fa930ff5a4dfa99d0 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 05ca05df85b94bc985b549e68caa6c36 in term 2.
I20250112 15:08:43.499706 24190 raft_consensus.cc:2405] T 483cfa86b848433bbc365bae489ed251 P 8586159698b04cde88062accfc5d1bf1 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 05ca05df85b94bc985b549e68caa6c36 for term 2 because replica has last-logged OpId of term: 1 index: 1, which is greater than that of the candidate, which has last-logged OpId of term: 0 index: 0.
I20250112 15:08:43.499918 24233 leader_election.cc:304] T 483cfa86b848433bbc365bae489ed251 P 05ca05df85b94bc985b549e68caa6c36 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 05ca05df85b94bc985b549e68caa6c36, 111d62a4d211430fa930ff5a4dfa99d0; no voters: 8586159698b04cde88062accfc5d1bf1
I20250112 15:08:43.500003 24305 raft_consensus.cc:2798] T 483cfa86b848433bbc365bae489ed251 P 05ca05df85b94bc985b549e68caa6c36 [term 2 FOLLOWER]: Leader election won for term 2
I20250112 15:08:43.500059 24305 raft_consensus.cc:695] T 483cfa86b848433bbc365bae489ed251 P 05ca05df85b94bc985b549e68caa6c36 [term 2 LEADER]: Becoming Leader. State: Replica: 05ca05df85b94bc985b549e68caa6c36, State: Running, Role: LEADER
I20250112 15:08:43.500115 24305 consensus_queue.cc:237] T 483cfa86b848433bbc365bae489ed251 P 05ca05df85b94bc985b549e68caa6c36 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "111d62a4d211430fa930ff5a4dfa99d0" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 35133 } } peers { permanent_uuid: "8586159698b04cde88062accfc5d1bf1" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 33901 } } peers { permanent_uuid: "05ca05df85b94bc985b549e68caa6c36" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 44611 } }
I20250112 15:08:43.500682 23636 catalog_manager.cc:5526] T 483cfa86b848433bbc365bae489ed251 P 05ca05df85b94bc985b549e68caa6c36 reported cstate change: term changed from 1 to 2, leader changed from 8586159698b04cde88062accfc5d1bf1 (127.17.60.194) to 05ca05df85b94bc985b549e68caa6c36 (127.17.60.195). New cstate: current_term: 2 leader_uuid: "05ca05df85b94bc985b549e68caa6c36" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "111d62a4d211430fa930ff5a4dfa99d0" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 35133 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8586159698b04cde88062accfc5d1bf1" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 33901 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "05ca05df85b94bc985b549e68caa6c36" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 44611 } health_report { overall_health: HEALTHY } } }
I20250112 15:08:43.578131 24314 raft_consensus.cc:491] T 92c129e062b945df8805bcc412f3a599 P 8586159698b04cde88062accfc5d1bf1 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 15:08:43.578248 24314 raft_consensus.cc:513] T 92c129e062b945df8805bcc412f3a599 P 8586159698b04cde88062accfc5d1bf1 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "111d62a4d211430fa930ff5a4dfa99d0" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 35133 } } peers { permanent_uuid: "8586159698b04cde88062accfc5d1bf1" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 33901 } } peers { permanent_uuid: "05ca05df85b94bc985b549e68caa6c36" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 44611 } }
I20250112 15:08:43.578415 24314 leader_election.cc:290] T 92c129e062b945df8805bcc412f3a599 P 8586159698b04cde88062accfc5d1bf1 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 111d62a4d211430fa930ff5a4dfa99d0 (127.17.60.193:35133), 05ca05df85b94bc985b549e68caa6c36 (127.17.60.195:44611)
I20250112 15:08:43.578684 24108 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "92c129e062b945df8805bcc412f3a599" candidate_uuid: "8586159698b04cde88062accfc5d1bf1" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "111d62a4d211430fa930ff5a4dfa99d0" is_pre_election: true
I20250112 15:08:43.578706 24270 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "92c129e062b945df8805bcc412f3a599" candidate_uuid: "8586159698b04cde88062accfc5d1bf1" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "05ca05df85b94bc985b549e68caa6c36" is_pre_election: true
I20250112 15:08:43.578768 24108 raft_consensus.cc:2463] T 92c129e062b945df8805bcc412f3a599 P 111d62a4d211430fa930ff5a4dfa99d0 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8586159698b04cde88062accfc5d1bf1 in term 2.
I20250112 15:08:43.578886 24152 leader_election.cc:304] T 92c129e062b945df8805bcc412f3a599 P 8586159698b04cde88062accfc5d1bf1 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 111d62a4d211430fa930ff5a4dfa99d0, 8586159698b04cde88062accfc5d1bf1; no voters:
I20250112 15:08:43.579010 24314 raft_consensus.cc:2798] T 92c129e062b945df8805bcc412f3a599 P 8586159698b04cde88062accfc5d1bf1 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250112 15:08:43.579053 24314 raft_consensus.cc:491] T 92c129e062b945df8805bcc412f3a599 P 8586159698b04cde88062accfc5d1bf1 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250112 15:08:43.579083 24314 raft_consensus.cc:3054] T 92c129e062b945df8805bcc412f3a599 P 8586159698b04cde88062accfc5d1bf1 [term 2 FOLLOWER]: Advancing to term 3
I20250112 15:08:43.579732 24314 raft_consensus.cc:513] T 92c129e062b945df8805bcc412f3a599 P 8586159698b04cde88062accfc5d1bf1 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "111d62a4d211430fa930ff5a4dfa99d0" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 35133 } } peers { permanent_uuid: "8586159698b04cde88062accfc5d1bf1" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 33901 } } peers { permanent_uuid: "05ca05df85b94bc985b549e68caa6c36" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 44611 } }
I20250112 15:08:43.579846 24314 leader_election.cc:290] T 92c129e062b945df8805bcc412f3a599 P 8586159698b04cde88062accfc5d1bf1 [CANDIDATE]: Term 3 election: Requested vote from peers 111d62a4d211430fa930ff5a4dfa99d0 (127.17.60.193:35133), 05ca05df85b94bc985b549e68caa6c36 (127.17.60.195:44611)
I20250112 15:08:43.579993 24108 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "92c129e062b945df8805bcc412f3a599" candidate_uuid: "8586159698b04cde88062accfc5d1bf1" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "111d62a4d211430fa930ff5a4dfa99d0"
I20250112 15:08:43.580049 24270 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "92c129e062b945df8805bcc412f3a599" candidate_uuid: "8586159698b04cde88062accfc5d1bf1" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "05ca05df85b94bc985b549e68caa6c36"
I20250112 15:08:43.580053 24108 raft_consensus.cc:3054] T 92c129e062b945df8805bcc412f3a599 P 111d62a4d211430fa930ff5a4dfa99d0 [term 2 FOLLOWER]: Advancing to term 3
I20250112 15:08:43.580732 24108 raft_consensus.cc:2463] T 92c129e062b945df8805bcc412f3a599 P 111d62a4d211430fa930ff5a4dfa99d0 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8586159698b04cde88062accfc5d1bf1 in term 3.
I20250112 15:08:43.580886 24152 leader_election.cc:304] T 92c129e062b945df8805bcc412f3a599 P 8586159698b04cde88062accfc5d1bf1 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 111d62a4d211430fa930ff5a4dfa99d0, 8586159698b04cde88062accfc5d1bf1; no voters: 05ca05df85b94bc985b549e68caa6c36
I20250112 15:08:43.580968 24314 raft_consensus.cc:2798] T 92c129e062b945df8805bcc412f3a599 P 8586159698b04cde88062accfc5d1bf1 [term 3 FOLLOWER]: Leader election won for term 3
I20250112 15:08:43.581028 24314 raft_consensus.cc:695] T 92c129e062b945df8805bcc412f3a599 P 8586159698b04cde88062accfc5d1bf1 [term 3 LEADER]: Becoming Leader. State: Replica: 8586159698b04cde88062accfc5d1bf1, State: Running, Role: LEADER
I20250112 15:08:43.581089 24314 consensus_queue.cc:237] T 92c129e062b945df8805bcc412f3a599 P 8586159698b04cde88062accfc5d1bf1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "111d62a4d211430fa930ff5a4dfa99d0" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 35133 } } peers { permanent_uuid: "8586159698b04cde88062accfc5d1bf1" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 33901 } } peers { permanent_uuid: "05ca05df85b94bc985b549e68caa6c36" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 44611 } }
I20250112 15:08:43.581671 23636 catalog_manager.cc:5526] T 92c129e062b945df8805bcc412f3a599 P 8586159698b04cde88062accfc5d1bf1 reported cstate change: term changed from 2 to 3, leader changed from 05ca05df85b94bc985b549e68caa6c36 (127.17.60.195) to 8586159698b04cde88062accfc5d1bf1 (127.17.60.194). New cstate: current_term: 3 leader_uuid: "8586159698b04cde88062accfc5d1bf1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "111d62a4d211430fa930ff5a4dfa99d0" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 35133 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8586159698b04cde88062accfc5d1bf1" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 33901 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "05ca05df85b94bc985b549e68caa6c36" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 44611 } health_report { overall_health: UNKNOWN } } }
I20250112 15:08:43.731709 24305 raft_consensus.cc:491] T 6a1c881a80004842a1eb46bb488d3834 P 05ca05df85b94bc985b549e68caa6c36 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 15:08:43.731823 24305 raft_consensus.cc:513] T 6a1c881a80004842a1eb46bb488d3834 P 05ca05df85b94bc985b549e68caa6c36 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "111d62a4d211430fa930ff5a4dfa99d0" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 35133 } } peers { permanent_uuid: "8586159698b04cde88062accfc5d1bf1" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 33901 } } peers { permanent_uuid: "05ca05df85b94bc985b549e68caa6c36" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 44611 } }
I20250112 15:08:43.732005 24305 leader_election.cc:290] T 6a1c881a80004842a1eb46bb488d3834 P 05ca05df85b94bc985b549e68caa6c36 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 111d62a4d211430fa930ff5a4dfa99d0 (127.17.60.193:35133), 8586159698b04cde88062accfc5d1bf1 (127.17.60.194:33901)
I20250112 15:08:43.732237 24108 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6a1c881a80004842a1eb46bb488d3834" candidate_uuid: "05ca05df85b94bc985b549e68caa6c36" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "111d62a4d211430fa930ff5a4dfa99d0" is_pre_election: true
I20250112 15:08:43.732249 24190 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6a1c881a80004842a1eb46bb488d3834" candidate_uuid: "05ca05df85b94bc985b549e68caa6c36" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8586159698b04cde88062accfc5d1bf1" is_pre_election: true
I20250112 15:08:43.732313 24108 raft_consensus.cc:2463] T 6a1c881a80004842a1eb46bb488d3834 P 111d62a4d211430fa930ff5a4dfa99d0 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 05ca05df85b94bc985b549e68caa6c36 in term 2.
I20250112 15:08:43.732468 24233 leader_election.cc:304] T 6a1c881a80004842a1eb46bb488d3834 P 05ca05df85b94bc985b549e68caa6c36 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 05ca05df85b94bc985b549e68caa6c36, 111d62a4d211430fa930ff5a4dfa99d0; no voters:
I20250112 15:08:43.732580 24305 raft_consensus.cc:2798] T 6a1c881a80004842a1eb46bb488d3834 P 05ca05df85b94bc985b549e68caa6c36 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250112 15:08:43.732618 24305 raft_consensus.cc:491] T 6a1c881a80004842a1eb46bb488d3834 P 05ca05df85b94bc985b549e68caa6c36 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250112 15:08:43.732642 24305 raft_consensus.cc:3054] T 6a1c881a80004842a1eb46bb488d3834 P 05ca05df85b94bc985b549e68caa6c36 [term 2 FOLLOWER]: Advancing to term 3
I20250112 15:08:43.733263 24305 raft_consensus.cc:513] T 6a1c881a80004842a1eb46bb488d3834 P 05ca05df85b94bc985b549e68caa6c36 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "111d62a4d211430fa930ff5a4dfa99d0" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 35133 } } peers { permanent_uuid: "8586159698b04cde88062accfc5d1bf1" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 33901 } } peers { permanent_uuid: "05ca05df85b94bc985b549e68caa6c36" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 44611 } }
I20250112 15:08:43.733381 24305 leader_election.cc:290] T 6a1c881a80004842a1eb46bb488d3834 P 05ca05df85b94bc985b549e68caa6c36 [CANDIDATE]: Term 3 election: Requested vote from peers 111d62a4d211430fa930ff5a4dfa99d0 (127.17.60.193:35133), 8586159698b04cde88062accfc5d1bf1 (127.17.60.194:33901)
I20250112 15:08:43.733547 24108 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6a1c881a80004842a1eb46bb488d3834" candidate_uuid: "05ca05df85b94bc985b549e68caa6c36" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "111d62a4d211430fa930ff5a4dfa99d0"
I20250112 15:08:43.733567 24190 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6a1c881a80004842a1eb46bb488d3834" candidate_uuid: "05ca05df85b94bc985b549e68caa6c36" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8586159698b04cde88062accfc5d1bf1"
I20250112 15:08:43.733608 24108 raft_consensus.cc:3054] T 6a1c881a80004842a1eb46bb488d3834 P 111d62a4d211430fa930ff5a4dfa99d0 [term 2 FOLLOWER]: Advancing to term 3
I20250112 15:08:43.734205 24108 raft_consensus.cc:2463] T 6a1c881a80004842a1eb46bb488d3834 P 111d62a4d211430fa930ff5a4dfa99d0 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 05ca05df85b94bc985b549e68caa6c36 in term 3.
I20250112 15:08:43.734359 24233 leader_election.cc:304] T 6a1c881a80004842a1eb46bb488d3834 P 05ca05df85b94bc985b549e68caa6c36 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 05ca05df85b94bc985b549e68caa6c36, 111d62a4d211430fa930ff5a4dfa99d0; no voters: 8586159698b04cde88062accfc5d1bf1
I20250112 15:08:43.734439 24305 raft_consensus.cc:2798] T 6a1c881a80004842a1eb46bb488d3834 P 05ca05df85b94bc985b549e68caa6c36 [term 3 FOLLOWER]: Leader election won for term 3
I20250112 15:08:43.734495 24305 raft_consensus.cc:695] T 6a1c881a80004842a1eb46bb488d3834 P 05ca05df85b94bc985b549e68caa6c36 [term 3 LEADER]: Becoming Leader. State: Replica: 05ca05df85b94bc985b549e68caa6c36, State: Running, Role: LEADER
I20250112 15:08:43.734536 24305 consensus_queue.cc:237] T 6a1c881a80004842a1eb46bb488d3834 P 05ca05df85b94bc985b549e68caa6c36 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "111d62a4d211430fa930ff5a4dfa99d0" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 35133 } } peers { permanent_uuid: "8586159698b04cde88062accfc5d1bf1" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 33901 } } peers { permanent_uuid: "05ca05df85b94bc985b549e68caa6c36" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 44611 } }
I20250112 15:08:43.735069 23636 catalog_manager.cc:5526] T 6a1c881a80004842a1eb46bb488d3834 P 05ca05df85b94bc985b549e68caa6c36 reported cstate change: term changed from 2 to 3, leader changed from 8586159698b04cde88062accfc5d1bf1 (127.17.60.194) to 05ca05df85b94bc985b549e68caa6c36 (127.17.60.195). New cstate: current_term: 3 leader_uuid: "05ca05df85b94bc985b549e68caa6c36" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "111d62a4d211430fa930ff5a4dfa99d0" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 35133 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8586159698b04cde88062accfc5d1bf1" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 33901 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "05ca05df85b94bc985b549e68caa6c36" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 44611 } health_report { overall_health: HEALTHY } } }
I20250112 15:08:43.866515 24190 raft_consensus.cc:1235] T 92c129e062b945df8805bcc412f3a599 P 8586159698b04cde88062accfc5d1bf1 [term 3 LEADER]: Rejecting Update request from peer 05ca05df85b94bc985b549e68caa6c36 for earlier term 2. Current term is 3. Ops: []
I20250112 15:08:43.866792 24305 consensus_queue.cc:1046] T 92c129e062b945df8805bcc412f3a599 P 05ca05df85b94bc985b549e68caa6c36 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "8586159698b04cde88062accfc5d1bf1" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 33901 }, Status: INVALID_TERM, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250112 15:08:43.866896 24305 consensus_queue.cc:1225] T 92c129e062b945df8805bcc412f3a599 P 05ca05df85b94bc985b549e68caa6c36 [LEADER]: Peer 8586159698b04cde88062accfc5d1bf1 log is divergent from this leader: its last log entry 3.2 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 0
I20250112 15:08:43.866967 24308 raft_consensus.cc:3049] T 92c129e062b945df8805bcc412f3a599 P 05ca05df85b94bc985b549e68caa6c36 [term 2 LEADER]: Stepping down as leader of term 2
I20250112 15:08:43.867041 24308 raft_consensus.cc:738] T 92c129e062b945df8805bcc412f3a599 P 05ca05df85b94bc985b549e68caa6c36 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 05ca05df85b94bc985b549e68caa6c36, State: Running, Role: LEADER
I20250112 15:08:43.867115 24308 consensus_queue.cc:260] T 92c129e062b945df8805bcc412f3a599 P 05ca05df85b94bc985b549e68caa6c36 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 2.1, Last appended by leader: 1, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "111d62a4d211430fa930ff5a4dfa99d0" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 35133 } } peers { permanent_uuid: "8586159698b04cde88062accfc5d1bf1" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 33901 } } peers { permanent_uuid: "05ca05df85b94bc985b549e68caa6c36" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 44611 } }
I20250112 15:08:43.867192 24308 raft_consensus.cc:3054] T 92c129e062b945df8805bcc412f3a599 P 05ca05df85b94bc985b549e68caa6c36 [term 2 FOLLOWER]: Advancing to term 3
I20250112 15:08:43.893548 24108 raft_consensus.cc:1235] T 6a1c881a80004842a1eb46bb488d3834 P 111d62a4d211430fa930ff5a4dfa99d0 [term 3 FOLLOWER]: Rejecting Update request from peer 8586159698b04cde88062accfc5d1bf1 for earlier term 2. Current term is 3. Ops: []
I20250112 15:08:43.893810 24314 consensus_queue.cc:1046] T 6a1c881a80004842a1eb46bb488d3834 P 8586159698b04cde88062accfc5d1bf1 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "111d62a4d211430fa930ff5a4dfa99d0" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 35133 }, Status: INVALID_TERM, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250112 15:08:43.893913 24314 raft_consensus.cc:3049] T 6a1c881a80004842a1eb46bb488d3834 P 8586159698b04cde88062accfc5d1bf1 [term 2 LEADER]: Stepping down as leader of term 2
I20250112 15:08:43.893947 24314 raft_consensus.cc:738] T 6a1c881a80004842a1eb46bb488d3834 P 8586159698b04cde88062accfc5d1bf1 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 8586159698b04cde88062accfc5d1bf1, State: Running, Role: LEADER
I20250112 15:08:43.894003 24314 consensus_queue.cc:260] T 6a1c881a80004842a1eb46bb488d3834 P 8586159698b04cde88062accfc5d1bf1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 2.1, Last appended by leader: 1, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "111d62a4d211430fa930ff5a4dfa99d0" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 35133 } } peers { permanent_uuid: "8586159698b04cde88062accfc5d1bf1" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 33901 } } peers { permanent_uuid: "05ca05df85b94bc985b549e68caa6c36" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 44611 } }
I20250112 15:08:43.894064 24314 raft_consensus.cc:3054] T 6a1c881a80004842a1eb46bb488d3834 P 8586159698b04cde88062accfc5d1bf1 [term 2 FOLLOWER]: Advancing to term 3
I20250112 15:08:43.944108 24323 raft_consensus.cc:491] T 3e43528d2bb24563966ef89a76aa761e P 111d62a4d211430fa930ff5a4dfa99d0 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 15:08:43.944216 24323 raft_consensus.cc:513] T 3e43528d2bb24563966ef89a76aa761e P 111d62a4d211430fa930ff5a4dfa99d0 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "111d62a4d211430fa930ff5a4dfa99d0" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 35133 } } peers { permanent_uuid: "05ca05df85b94bc985b549e68caa6c36" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 44611 } } peers { permanent_uuid: "8586159698b04cde88062accfc5d1bf1" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 33901 } }
I20250112 15:08:43.944389 24323 leader_election.cc:290] T 3e43528d2bb24563966ef89a76aa761e P 111d62a4d211430fa930ff5a4dfa99d0 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 05ca05df85b94bc985b549e68caa6c36 (127.17.60.195:44611), 8586159698b04cde88062accfc5d1bf1 (127.17.60.194:33901)
I20250112 15:08:43.947793 24190 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e43528d2bb24563966ef89a76aa761e" candidate_uuid: "111d62a4d211430fa930ff5a4dfa99d0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8586159698b04cde88062accfc5d1bf1" is_pre_election: true
I20250112 15:08:43.947793 24270 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e43528d2bb24563966ef89a76aa761e" candidate_uuid: "111d62a4d211430fa930ff5a4dfa99d0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "05ca05df85b94bc985b549e68caa6c36" is_pre_election: true
I20250112 15:08:43.947933 24190 raft_consensus.cc:2463] T 3e43528d2bb24563966ef89a76aa761e P 8586159698b04cde88062accfc5d1bf1 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 111d62a4d211430fa930ff5a4dfa99d0 in term 1.
I20250112 15:08:43.947973 24270 raft_consensus.cc:2463] T 3e43528d2bb24563966ef89a76aa761e P 05ca05df85b94bc985b549e68caa6c36 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 111d62a4d211430fa930ff5a4dfa99d0 in term 1.
I20250112 15:08:43.948127 24068 leader_election.cc:304] T 3e43528d2bb24563966ef89a76aa761e P 111d62a4d211430fa930ff5a4dfa99d0 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 111d62a4d211430fa930ff5a4dfa99d0, 8586159698b04cde88062accfc5d1bf1; no voters:
I20250112 15:08:43.948221 24323 raft_consensus.cc:2798] T 3e43528d2bb24563966ef89a76aa761e P 111d62a4d211430fa930ff5a4dfa99d0 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250112 15:08:43.948264 24323 raft_consensus.cc:491] T 3e43528d2bb24563966ef89a76aa761e P 111d62a4d211430fa930ff5a4dfa99d0 [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250112 15:08:43.948278 24323 raft_consensus.cc:3054] T 3e43528d2bb24563966ef89a76aa761e P 111d62a4d211430fa930ff5a4dfa99d0 [term 1 FOLLOWER]: Advancing to term 2
I20250112 15:08:43.948948 24323 raft_consensus.cc:513] T 3e43528d2bb24563966ef89a76aa761e P 111d62a4d211430fa930ff5a4dfa99d0 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "111d62a4d211430fa930ff5a4dfa99d0" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 35133 } } peers { permanent_uuid: "05ca05df85b94bc985b549e68caa6c36" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 44611 } } peers { permanent_uuid: "8586159698b04cde88062accfc5d1bf1" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 33901 } }
I20250112 15:08:43.949074 24323 leader_election.cc:290] T 3e43528d2bb24563966ef89a76aa761e P 111d62a4d211430fa930ff5a4dfa99d0 [CANDIDATE]: Term 2 election: Requested vote from peers 05ca05df85b94bc985b549e68caa6c36 (127.17.60.195:44611), 8586159698b04cde88062accfc5d1bf1 (127.17.60.194:33901)
I20250112 15:08:43.949219 24270 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e43528d2bb24563966ef89a76aa761e" candidate_uuid: "111d62a4d211430fa930ff5a4dfa99d0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "05ca05df85b94bc985b549e68caa6c36"
I20250112 15:08:43.949283 24190 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e43528d2bb24563966ef89a76aa761e" candidate_uuid: "111d62a4d211430fa930ff5a4dfa99d0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8586159698b04cde88062accfc5d1bf1"
I20250112 15:08:43.949301 24270 raft_consensus.cc:3054] T 3e43528d2bb24563966ef89a76aa761e P 05ca05df85b94bc985b549e68caa6c36 [term 1 FOLLOWER]: Advancing to term 2
I20250112 15:08:43.949339 24190 raft_consensus.cc:3054] T 3e43528d2bb24563966ef89a76aa761e P 8586159698b04cde88062accfc5d1bf1 [term 1 FOLLOWER]: Advancing to term 2
I20250112 15:08:43.949970 24270 raft_consensus.cc:2463] T 3e43528d2bb24563966ef89a76aa761e P 05ca05df85b94bc985b549e68caa6c36 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 111d62a4d211430fa930ff5a4dfa99d0 in term 2.
I20250112 15:08:43.949972 24190 raft_consensus.cc:2463] T 3e43528d2bb24563966ef89a76aa761e P 8586159698b04cde88062accfc5d1bf1 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 111d62a4d211430fa930ff5a4dfa99d0 in term 2.
I20250112 15:08:43.950160 24070 leader_election.cc:304] T 3e43528d2bb24563966ef89a76aa761e P 111d62a4d211430fa930ff5a4dfa99d0 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 05ca05df85b94bc985b549e68caa6c36, 111d62a4d211430fa930ff5a4dfa99d0; no voters:
I20250112 15:08:43.950261 24323 raft_consensus.cc:2798] T 3e43528d2bb24563966ef89a76aa761e P 111d62a4d211430fa930ff5a4dfa99d0 [term 2 FOLLOWER]: Leader election won for term 2
I20250112 15:08:43.950404 24323 raft_consensus.cc:695] T 3e43528d2bb24563966ef89a76aa761e P 111d62a4d211430fa930ff5a4dfa99d0 [term 2 LEADER]: Becoming Leader. State: Replica: 111d62a4d211430fa930ff5a4dfa99d0, State: Running, Role: LEADER
I20250112 15:08:43.950456 24323 consensus_queue.cc:237] T 3e43528d2bb24563966ef89a76aa761e P 111d62a4d211430fa930ff5a4dfa99d0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "111d62a4d211430fa930ff5a4dfa99d0" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 35133 } } peers { permanent_uuid: "05ca05df85b94bc985b549e68caa6c36" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 44611 } } peers { permanent_uuid: "8586159698b04cde88062accfc5d1bf1" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 33901 } }
I20250112 15:08:43.950976 23636 catalog_manager.cc:5526] T 3e43528d2bb24563966ef89a76aa761e P 111d62a4d211430fa930ff5a4dfa99d0 reported cstate change: term changed from 1 to 2. New cstate: current_term: 2 leader_uuid: "111d62a4d211430fa930ff5a4dfa99d0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "111d62a4d211430fa930ff5a4dfa99d0" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 35133 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "05ca05df85b94bc985b549e68caa6c36" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 44611 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8586159698b04cde88062accfc5d1bf1" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 33901 } health_report { overall_health: UNKNOWN } } }
I20250112 15:08:43.972920 24190 raft_consensus.cc:1270] T 483cfa86b848433bbc365bae489ed251 P 8586159698b04cde88062accfc5d1bf1 [term 2 FOLLOWER]: Refusing update from remote peer 05ca05df85b94bc985b549e68caa6c36: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 1. (term mismatch)
I20250112 15:08:43.973001 24190 pending_rounds.cc:77] T 483cfa86b848433bbc365bae489ed251 P 8586159698b04cde88062accfc5d1bf1: Aborting all ops after (but not including) 0
I20250112 15:08:43.973016 24190 pending_rounds.cc:99] T 483cfa86b848433bbc365bae489ed251 P 8586159698b04cde88062accfc5d1bf1: Aborting uncommitted NO_OP operation due to leader change: 1.1
I20250112 15:08:43.973038 24190 raft_consensus.cc:2883] T 483cfa86b848433bbc365bae489ed251 P 8586159698b04cde88062accfc5d1bf1 [term 2 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20250112 15:08:43.973258 24308 consensus_queue.cc:1035] T 483cfa86b848433bbc365bae489ed251 P 05ca05df85b94bc985b549e68caa6c36 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8586159698b04cde88062accfc5d1bf1" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 33901 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250112 15:08:43.974279 24108 raft_consensus.cc:1270] T 483cfa86b848433bbc365bae489ed251 P 111d62a4d211430fa930ff5a4dfa99d0 [term 2 FOLLOWER]: Refusing update from remote peer 05ca05df85b94bc985b549e68caa6c36: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 2 index: 1. (index mismatch)
I20250112 15:08:43.974568 24308 consensus_queue.cc:1035] T 483cfa86b848433bbc365bae489ed251 P 05ca05df85b94bc985b549e68caa6c36 [LEADER]: Connected to new peer: Peer: permanent_uuid: "111d62a4d211430fa930ff5a4dfa99d0" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 35133 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250112 15:08:44.061357 24270 raft_consensus.cc:1270] T 92c129e062b945df8805bcc412f3a599 P 05ca05df85b94bc985b549e68caa6c36 [term 3 FOLLOWER]: Refusing update from remote peer 8586159698b04cde88062accfc5d1bf1: Log matching property violated. Preceding OpId in replica: term: 2 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20250112 15:08:44.061628 24314 consensus_queue.cc:1035] T 92c129e062b945df8805bcc412f3a599 P 8586159698b04cde88062accfc5d1bf1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "05ca05df85b94bc985b549e68caa6c36" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 44611 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 0, Time since last communication: 0.000s
I20250112 15:08:44.061739 24314 consensus_queue.cc:1225] T 92c129e062b945df8805bcc412f3a599 P 8586159698b04cde88062accfc5d1bf1 [LEADER]: Peer 05ca05df85b94bc985b549e68caa6c36 log is divergent from this leader: its last log entry 2.1 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 0
I20250112 15:08:44.062040 24270 pending_rounds.cc:77] T 92c129e062b945df8805bcc412f3a599 P 05ca05df85b94bc985b549e68caa6c36: Aborting all ops after (but not including) 0
I20250112 15:08:44.062081 24270 pending_rounds.cc:99] T 92c129e062b945df8805bcc412f3a599 P 05ca05df85b94bc985b549e68caa6c36: Aborting uncommitted NO_OP operation due to leader change: 2.1
I20250112 15:08:44.062098 24270 raft_consensus.cc:2883] T 92c129e062b945df8805bcc412f3a599 P 05ca05df85b94bc985b549e68caa6c36 [term 3 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20250112 15:08:44.062978 24108 raft_consensus.cc:1270] T 92c129e062b945df8805bcc412f3a599 P 111d62a4d211430fa930ff5a4dfa99d0 [term 3 FOLLOWER]: Refusing update from remote peer 8586159698b04cde88062accfc5d1bf1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20250112 15:08:44.063184 24334 consensus_queue.cc:1035] T 92c129e062b945df8805bcc412f3a599 P 8586159698b04cde88062accfc5d1bf1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "111d62a4d211430fa930ff5a4dfa99d0" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 35133 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 0, Time since last communication: 0.000s
W20250112 15:08:44.173909 23686 auto_rebalancer.cc:253] failed to send replica move request: Illegal state: Leader has not yet committed an operation in its own term
W20250112 15:08:44.174336 23686 auto_rebalancer.cc:663] Could not move replica: Incomplete: tablet 3e43528d2bb24563966ef89a76aa761e, TS 8586159698b04cde88062accfc5d1bf1 -> TS 932773f4da6d4c238b6963b863c59be9 move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "111d62a4d211430fa930ff5a4dfa99d0" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 35133 } } peers { permanent_uuid: "05ca05df85b94bc985b549e68caa6c36" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 44611 } } peers { permanent_uuid: "8586159698b04cde88062accfc5d1bf1" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 33901 } }
W20250112 15:08:44.174373 23686 auto_rebalancer.cc:263] scheduled replica move failed to complete: Incomplete: tablet 3e43528d2bb24563966ef89a76aa761e, TS 8586159698b04cde88062accfc5d1bf1 -> TS 932773f4da6d4c238b6963b863c59be9 move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "111d62a4d211430fa930ff5a4dfa99d0" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 35133 } } peers { permanent_uuid: "05ca05df85b94bc985b549e68caa6c36" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 44611 } } peers { permanent_uuid: "8586159698b04cde88062accfc5d1bf1" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 33901 } }
W20250112 15:08:44.174698 23686 auto_rebalancer.cc:663] Could not move replica: Incomplete: tablet 92c129e062b945df8805bcc412f3a599, TS 05ca05df85b94bc985b549e68caa6c36 -> TS 932773f4da6d4c238b6963b863c59be9 move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "111d62a4d211430fa930ff5a4dfa99d0" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 35133 } } peers { permanent_uuid: "8586159698b04cde88062accfc5d1bf1" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 33901 } } peers { permanent_uuid: "05ca05df85b94bc985b549e68caa6c36" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 44611 } }
W20250112 15:08:44.174731 23686 auto_rebalancer.cc:263] scheduled replica move failed to complete: Incomplete: tablet 92c129e062b945df8805bcc412f3a599, TS 05ca05df85b94bc985b549e68caa6c36 -> TS 932773f4da6d4c238b6963b863c59be9 move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "111d62a4d211430fa930ff5a4dfa99d0" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 35133 } } peers { permanent_uuid: "8586159698b04cde88062accfc5d1bf1" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 33901 } } peers { permanent_uuid: "05ca05df85b94bc985b549e68caa6c36" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 44611 } }
W20250112 15:08:44.177335 23686 auto_rebalancer.cc:663] Could not move replica: Incomplete: tablet 6a1c881a80004842a1eb46bb488d3834, TS 111d62a4d211430fa930ff5a4dfa99d0 -> TS 932773f4da6d4c238b6963b863c59be9 move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "111d62a4d211430fa930ff5a4dfa99d0" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 35133 } } peers { permanent_uuid: "8586159698b04cde88062accfc5d1bf1" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 33901 } } peers { permanent_uuid: "05ca05df85b94bc985b549e68caa6c36" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 44611 } }
W20250112 15:08:44.177373 23686 auto_rebalancer.cc:263] scheduled replica move failed to complete: Incomplete: tablet 6a1c881a80004842a1eb46bb488d3834, TS 111d62a4d211430fa930ff5a4dfa99d0 -> TS 932773f4da6d4c238b6963b863c59be9 move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "111d62a4d211430fa930ff5a4dfa99d0" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 35133 } } peers { permanent_uuid: "8586159698b04cde88062accfc5d1bf1" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 33901 } } peers { permanent_uuid: "05ca05df85b94bc985b549e68caa6c36" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 44611 } }
I20250112 15:08:44.229771 24108 raft_consensus.cc:1270] T 6a1c881a80004842a1eb46bb488d3834 P 111d62a4d211430fa930ff5a4dfa99d0 [term 3 FOLLOWER]: Refusing update from remote peer 05ca05df85b94bc985b549e68caa6c36: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20250112 15:08:44.230036 24308 consensus_queue.cc:1035] T 6a1c881a80004842a1eb46bb488d3834 P 05ca05df85b94bc985b549e68caa6c36 [LEADER]: Connected to new peer: Peer: permanent_uuid: "111d62a4d211430fa930ff5a4dfa99d0" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 35133 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 0, Time since last communication: 0.000s
I20250112 15:08:44.231493 24190 raft_consensus.cc:1270] T 6a1c881a80004842a1eb46bb488d3834 P 8586159698b04cde88062accfc5d1bf1 [term 3 FOLLOWER]: Refusing update from remote peer 05ca05df85b94bc985b549e68caa6c36: Log matching property violated. Preceding OpId in replica: term: 2 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20250112 15:08:44.231698 24338 consensus_queue.cc:1035] T 6a1c881a80004842a1eb46bb488d3834 P 05ca05df85b94bc985b549e68caa6c36 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8586159698b04cde88062accfc5d1bf1" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 33901 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 0, Time since last communication: 0.000s
I20250112 15:08:44.231793 24338 consensus_queue.cc:1225] T 6a1c881a80004842a1eb46bb488d3834 P 05ca05df85b94bc985b549e68caa6c36 [LEADER]: Peer 8586159698b04cde88062accfc5d1bf1 log is divergent from this leader: its last log entry 2.1 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 0
I20250112 15:08:44.231999 24190 pending_rounds.cc:77] T 6a1c881a80004842a1eb46bb488d3834 P 8586159698b04cde88062accfc5d1bf1: Aborting all ops after (but not including) 0
I20250112 15:08:44.232034 24190 pending_rounds.cc:99] T 6a1c881a80004842a1eb46bb488d3834 P 8586159698b04cde88062accfc5d1bf1: Aborting uncommitted NO_OP operation due to leader change: 2.1
I20250112 15:08:44.232057 24190 raft_consensus.cc:2883] T 6a1c881a80004842a1eb46bb488d3834 P 8586159698b04cde88062accfc5d1bf1 [term 3 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20250112 15:08:44.553167 24190 raft_consensus.cc:1270] T 3e43528d2bb24563966ef89a76aa761e P 8586159698b04cde88062accfc5d1bf1 [term 2 FOLLOWER]: Refusing update from remote peer 111d62a4d211430fa930ff5a4dfa99d0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250112 15:08:44.553444 24323 consensus_queue.cc:1035] T 3e43528d2bb24563966ef89a76aa761e P 111d62a4d211430fa930ff5a4dfa99d0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8586159698b04cde88062accfc5d1bf1" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 33901 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 0, Time since last communication: 0.000s
I20250112 15:08:44.554723 24270 raft_consensus.cc:1270] T 3e43528d2bb24563966ef89a76aa761e P 05ca05df85b94bc985b549e68caa6c36 [term 2 FOLLOWER]: Refusing update from remote peer 111d62a4d211430fa930ff5a4dfa99d0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250112 15:08:44.554935 24340 consensus_queue.cc:1035] T 3e43528d2bb24563966ef89a76aa761e P 111d62a4d211430fa930ff5a4dfa99d0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "05ca05df85b94bc985b549e68caa6c36" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 44611 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 0, Time since last communication: 0.000s
I20250112 15:08:45.178184 24270 consensus_queue.cc:237] T 6a1c881a80004842a1eb46bb488d3834 P 05ca05df85b94bc985b549e68caa6c36 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "111d62a4d211430fa930ff5a4dfa99d0" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 35133 } } peers { permanent_uuid: "8586159698b04cde88062accfc5d1bf1" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 33901 } attrs { replace: true } } peers { permanent_uuid: "05ca05df85b94bc985b549e68caa6c36" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 44611 } } peers { permanent_uuid: "932773f4da6d4c238b6963b863c59be9" member_type: NON_VOTER last_known_addr { host: "127.17.60.196" port: 44863 } attrs { promote: true } }
I20250112 15:08:45.178797 24190 raft_consensus.cc:1270] T 6a1c881a80004842a1eb46bb488d3834 P 8586159698b04cde88062accfc5d1bf1 [term 3 FOLLOWER]: Refusing update from remote peer 05ca05df85b94bc985b549e68caa6c36: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20250112 15:08:45.178947 24108 raft_consensus.cc:1270] T 6a1c881a80004842a1eb46bb488d3834 P 111d62a4d211430fa930ff5a4dfa99d0 [term 3 FOLLOWER]: Refusing update from remote peer 05ca05df85b94bc985b549e68caa6c36: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20250112 15:08:45.179147 24338 consensus_queue.cc:1035] T 6a1c881a80004842a1eb46bb488d3834 P 05ca05df85b94bc985b549e68caa6c36 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8586159698b04cde88062accfc5d1bf1" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 33901 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250112 15:08:45.179284 24305 consensus_queue.cc:1035] T 6a1c881a80004842a1eb46bb488d3834 P 05ca05df85b94bc985b549e68caa6c36 [LEADER]: Connected to new peer: Peer: permanent_uuid: "111d62a4d211430fa930ff5a4dfa99d0" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 35133 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250112 15:08:45.179889 24338 raft_consensus.cc:2949] T 6a1c881a80004842a1eb46bb488d3834 P 05ca05df85b94bc985b549e68caa6c36 [term 3 LEADER]: Committing config change with OpId 3.3: config changed from index -1 to 3, NON_VOTER 932773f4da6d4c238b6963b863c59be9 (127.17.60.196) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "111d62a4d211430fa930ff5a4dfa99d0" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 35133 } } peers { permanent_uuid: "8586159698b04cde88062accfc5d1bf1" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 33901 } attrs { replace: true } } peers { permanent_uuid: "05ca05df85b94bc985b549e68caa6c36" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 44611 } } peers { permanent_uuid: "932773f4da6d4c238b6963b863c59be9" member_type: NON_VOTER last_known_addr { host: "127.17.60.196" port: 44863 } attrs { promote: true } } }
I20250112 15:08:45.180135 24190 raft_consensus.cc:2949] T 6a1c881a80004842a1eb46bb488d3834 P 8586159698b04cde88062accfc5d1bf1 [term 3 FOLLOWER]: Committing config change with OpId 3.3: config changed from index -1 to 3, NON_VOTER 932773f4da6d4c238b6963b863c59be9 (127.17.60.196) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "111d62a4d211430fa930ff5a4dfa99d0" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 35133 } } peers { permanent_uuid: "8586159698b04cde88062accfc5d1bf1" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 33901 } attrs { replace: true } } peers { permanent_uuid: "05ca05df85b94bc985b549e68caa6c36" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 44611 } } peers { permanent_uuid: "932773f4da6d4c238b6963b863c59be9" member_type: NON_VOTER last_known_addr { host: "127.17.60.196" port: 44863 } attrs { promote: true } } }
I20250112 15:08:45.180142 24108 raft_consensus.cc:2949] T 6a1c881a80004842a1eb46bb488d3834 P 111d62a4d211430fa930ff5a4dfa99d0 [term 3 FOLLOWER]: Committing config change with OpId 3.3: config changed from index -1 to 3, NON_VOTER 932773f4da6d4c238b6963b863c59be9 (127.17.60.196) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "111d62a4d211430fa930ff5a4dfa99d0" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 35133 } } peers { permanent_uuid: "8586159698b04cde88062accfc5d1bf1" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 33901 } attrs { replace: true } } peers { permanent_uuid: "05ca05df85b94bc985b549e68caa6c36" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 44611 } } peers { permanent_uuid: "932773f4da6d4c238b6963b863c59be9" member_type: NON_VOTER last_known_addr { host: "127.17.60.196" port: 44863 } attrs { promote: true } } }
I20250112 15:08:45.181237 23637 catalog_manager.cc:5526] T 6a1c881a80004842a1eb46bb488d3834 P 8586159698b04cde88062accfc5d1bf1 reported cstate change: config changed from index -1 to 3, NON_VOTER 932773f4da6d4c238b6963b863c59be9 (127.17.60.196) added. New cstate: current_term: 3 leader_uuid: "05ca05df85b94bc985b549e68caa6c36" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "111d62a4d211430fa930ff5a4dfa99d0" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 35133 } } peers { permanent_uuid: "8586159698b04cde88062accfc5d1bf1" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 33901 } attrs { replace: true } } peers { permanent_uuid: "05ca05df85b94bc985b549e68caa6c36" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 44611 } } peers { permanent_uuid: "932773f4da6d4c238b6963b863c59be9" member_type: NON_VOTER last_known_addr { host: "127.17.60.196" port: 44863 } attrs { promote: true } } }
I20250112 15:08:45.181478 24190 consensus_queue.cc:237] T 92c129e062b945df8805bcc412f3a599 P 8586159698b04cde88062accfc5d1bf1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "111d62a4d211430fa930ff5a4dfa99d0" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 35133 } attrs { replace: true } } peers { permanent_uuid: "8586159698b04cde88062accfc5d1bf1" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 33901 } } peers { permanent_uuid: "05ca05df85b94bc985b549e68caa6c36" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 44611 } } peers { permanent_uuid: "932773f4da6d4c238b6963b863c59be9" member_type: NON_VOTER last_known_addr { host: "127.17.60.196" port: 44863 } attrs { promote: true } }
I20250112 15:08:45.182173 24108 raft_consensus.cc:1270] T 92c129e062b945df8805bcc412f3a599 P 111d62a4d211430fa930ff5a4dfa99d0 [term 3 FOLLOWER]: Refusing update from remote peer 8586159698b04cde88062accfc5d1bf1: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20250112 15:08:45.182199 24270 raft_consensus.cc:1270] T 92c129e062b945df8805bcc412f3a599 P 05ca05df85b94bc985b549e68caa6c36 [term 3 FOLLOWER]: Refusing update from remote peer 8586159698b04cde88062accfc5d1bf1: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
W20250112 15:08:45.182363 24233 consensus_peers.cc:487] T 6a1c881a80004842a1eb46bb488d3834 P 05ca05df85b94bc985b549e68caa6c36 -> Peer 932773f4da6d4c238b6963b863c59be9 (127.17.60.196:44863): Couldn't send request to peer 932773f4da6d4c238b6963b863c59be9. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 6a1c881a80004842a1eb46bb488d3834. This is attempt 1: this message will repeat every 5th retry.
I20250112 15:08:45.182384 24334 consensus_queue.cc:1035] T 92c129e062b945df8805bcc412f3a599 P 8586159698b04cde88062accfc5d1bf1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "111d62a4d211430fa930ff5a4dfa99d0" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 35133 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250112 15:08:45.182518 24346 consensus_queue.cc:1035] T 92c129e062b945df8805bcc412f3a599 P 8586159698b04cde88062accfc5d1bf1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "05ca05df85b94bc985b549e68caa6c36" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 44611 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250112 15:08:45.183374 24346 raft_consensus.cc:2949] T 92c129e062b945df8805bcc412f3a599 P 8586159698b04cde88062accfc5d1bf1 [term 3 LEADER]: Committing config change with OpId 3.3: config changed from index -1 to 3, NON_VOTER 932773f4da6d4c238b6963b863c59be9 (127.17.60.196) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "111d62a4d211430fa930ff5a4dfa99d0" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 35133 } attrs { replace: true } } peers { permanent_uuid: "8586159698b04cde88062accfc5d1bf1" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 33901 } } peers { permanent_uuid: "05ca05df85b94bc985b549e68caa6c36" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 44611 } } peers { permanent_uuid: "932773f4da6d4c238b6963b863c59be9" member_type: NON_VOTER last_known_addr { host: "127.17.60.196" port: 44863 } attrs { promote: true } } }
I20250112 15:08:45.183478 24270 raft_consensus.cc:2949] T 92c129e062b945df8805bcc412f3a599 P 05ca05df85b94bc985b549e68caa6c36 [term 3 FOLLOWER]: Committing config change with OpId 3.3: config changed from index -1 to 3, NON_VOTER 932773f4da6d4c238b6963b863c59be9 (127.17.60.196) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "111d62a4d211430fa930ff5a4dfa99d0" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 35133 } attrs { replace: true } } peers { permanent_uuid: "8586159698b04cde88062accfc5d1bf1" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 33901 } } peers { permanent_uuid: "05ca05df85b94bc985b549e68caa6c36" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 44611 } } peers { permanent_uuid: "932773f4da6d4c238b6963b863c59be9" member_type: NON_VOTER last_known_addr { host: "127.17.60.196" port: 44863 } attrs { promote: true } } }
I20250112 15:08:45.183564 24108 raft_consensus.cc:2949] T 92c129e062b945df8805bcc412f3a599 P 111d62a4d211430fa930ff5a4dfa99d0 [term 3 FOLLOWER]: Committing config change with OpId 3.3: config changed from index -1 to 3, NON_VOTER 932773f4da6d4c238b6963b863c59be9 (127.17.60.196) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "111d62a4d211430fa930ff5a4dfa99d0" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 35133 } attrs { replace: true } } peers { permanent_uuid: "8586159698b04cde88062accfc5d1bf1" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 33901 } } peers { permanent_uuid: "05ca05df85b94bc985b549e68caa6c36" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 44611 } } peers { permanent_uuid: "932773f4da6d4c238b6963b863c59be9" member_type: NON_VOTER last_known_addr { host: "127.17.60.196" port: 44863 } attrs { promote: true } } }
I20250112 15:08:45.184414 23636 catalog_manager.cc:5526] T 92c129e062b945df8805bcc412f3a599 P 8586159698b04cde88062accfc5d1bf1 reported cstate change: config changed from index -1 to 3, NON_VOTER 932773f4da6d4c238b6963b863c59be9 (127.17.60.196) added. New cstate: current_term: 3 leader_uuid: "8586159698b04cde88062accfc5d1bf1" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "111d62a4d211430fa930ff5a4dfa99d0" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 35133 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8586159698b04cde88062accfc5d1bf1" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 33901 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "05ca05df85b94bc985b549e68caa6c36" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 44611 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "932773f4da6d4c238b6963b863c59be9" member_type: NON_VOTER last_known_addr { host: "127.17.60.196" port: 44863 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20250112 15:08:45.184909 24108 consensus_queue.cc:237] T 3e43528d2bb24563966ef89a76aa761e P 111d62a4d211430fa930ff5a4dfa99d0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "111d62a4d211430fa930ff5a4dfa99d0" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 35133 } } peers { permanent_uuid: "05ca05df85b94bc985b549e68caa6c36" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 44611 } attrs { replace: true } } peers { permanent_uuid: "8586159698b04cde88062accfc5d1bf1" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 33901 } } peers { permanent_uuid: "932773f4da6d4c238b6963b863c59be9" member_type: NON_VOTER last_known_addr { host: "127.17.60.196" port: 44863 } attrs { promote: true } }
I20250112 15:08:45.185365 24190 raft_consensus.cc:1270] T 3e43528d2bb24563966ef89a76aa761e P 8586159698b04cde88062accfc5d1bf1 [term 2 FOLLOWER]: Refusing update from remote peer 111d62a4d211430fa930ff5a4dfa99d0: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
W20250112 15:08:45.185444 24152 consensus_peers.cc:487] T 92c129e062b945df8805bcc412f3a599 P 8586159698b04cde88062accfc5d1bf1 -> Peer 932773f4da6d4c238b6963b863c59be9 (127.17.60.196:44863): Couldn't send request to peer 932773f4da6d4c238b6963b863c59be9. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 92c129e062b945df8805bcc412f3a599. This is attempt 1: this message will repeat every 5th retry.
I20250112 15:08:45.185523 24270 raft_consensus.cc:1270] T 3e43528d2bb24563966ef89a76aa761e P 05ca05df85b94bc985b549e68caa6c36 [term 2 FOLLOWER]: Refusing update from remote peer 111d62a4d211430fa930ff5a4dfa99d0: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20250112 15:08:45.185683 24327 consensus_queue.cc:1035] T 3e43528d2bb24563966ef89a76aa761e P 111d62a4d211430fa930ff5a4dfa99d0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "05ca05df85b94bc985b549e68caa6c36" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 44611 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250112 15:08:45.185776 24323 consensus_queue.cc:1035] T 3e43528d2bb24563966ef89a76aa761e P 111d62a4d211430fa930ff5a4dfa99d0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8586159698b04cde88062accfc5d1bf1" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 33901 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250112 15:08:45.186265 24340 raft_consensus.cc:2949] T 3e43528d2bb24563966ef89a76aa761e P 111d62a4d211430fa930ff5a4dfa99d0 [term 2 LEADER]: Committing config change with OpId 2.3: config changed from index -1 to 3, NON_VOTER 932773f4da6d4c238b6963b863c59be9 (127.17.60.196) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "111d62a4d211430fa930ff5a4dfa99d0" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 35133 } } peers { permanent_uuid: "05ca05df85b94bc985b549e68caa6c36" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 44611 } attrs { replace: true } } peers { permanent_uuid: "8586159698b04cde88062accfc5d1bf1" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 33901 } } peers { permanent_uuid: "932773f4da6d4c238b6963b863c59be9" member_type: NON_VOTER last_known_addr { host: "127.17.60.196" port: 44863 } attrs { promote: true } } }
I20250112 15:08:45.186383 24190 raft_consensus.cc:2949] T 3e43528d2bb24563966ef89a76aa761e P 8586159698b04cde88062accfc5d1bf1 [term 2 FOLLOWER]: Committing config change with OpId 2.3: config changed from index -1 to 3, NON_VOTER 932773f4da6d4c238b6963b863c59be9 (127.17.60.196) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "111d62a4d211430fa930ff5a4dfa99d0" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 35133 } } peers { permanent_uuid: "05ca05df85b94bc985b549e68caa6c36" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 44611 } attrs { replace: true } } peers { permanent_uuid: "8586159698b04cde88062accfc5d1bf1" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 33901 } } peers { permanent_uuid: "932773f4da6d4c238b6963b863c59be9" member_type: NON_VOTER last_known_addr { host: "127.17.60.196" port: 44863 } attrs { promote: true } } }
I20250112 15:08:45.187373 24270 raft_consensus.cc:2949] T 3e43528d2bb24563966ef89a76aa761e P 05ca05df85b94bc985b549e68caa6c36 [term 2 FOLLOWER]: Committing config change with OpId 2.3: config changed from index -1 to 3, NON_VOTER 932773f4da6d4c238b6963b863c59be9 (127.17.60.196) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "111d62a4d211430fa930ff5a4dfa99d0" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 35133 } } peers { permanent_uuid: "05ca05df85b94bc985b549e68caa6c36" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 44611 } attrs { replace: true } } peers { permanent_uuid: "8586159698b04cde88062accfc5d1bf1" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 33901 } } peers { permanent_uuid: "932773f4da6d4c238b6963b863c59be9" member_type: NON_VOTER last_known_addr { host: "127.17.60.196" port: 44863 } attrs { promote: true } } }
I20250112 15:08:45.187407 23637 catalog_manager.cc:5526] T 3e43528d2bb24563966ef89a76aa761e P 111d62a4d211430fa930ff5a4dfa99d0 reported cstate change: config changed from index -1 to 3, NON_VOTER 932773f4da6d4c238b6963b863c59be9 (127.17.60.196) added. New cstate: current_term: 2 leader_uuid: "111d62a4d211430fa930ff5a4dfa99d0" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "111d62a4d211430fa930ff5a4dfa99d0" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 35133 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "05ca05df85b94bc985b549e68caa6c36" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 44611 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8586159698b04cde88062accfc5d1bf1" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 33901 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "932773f4da6d4c238b6963b863c59be9" member_type: NON_VOTER last_known_addr { host: "127.17.60.196" port: 44863 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20250112 15:08:45.189182 24070 consensus_peers.cc:487] T 3e43528d2bb24563966ef89a76aa761e P 111d62a4d211430fa930ff5a4dfa99d0 -> Peer 932773f4da6d4c238b6963b863c59be9 (127.17.60.196:44863): Couldn't send request to peer 932773f4da6d4c238b6963b863c59be9. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 3e43528d2bb24563966ef89a76aa761e. This is attempt 1: this message will repeat every 5th retry.
I20250112 15:08:45.637002 24354 ts_tablet_manager.cc:927] T 92c129e062b945df8805bcc412f3a599 P 932773f4da6d4c238b6963b863c59be9: Initiating tablet copy from peer 8586159698b04cde88062accfc5d1bf1 (127.17.60.194:33901)
I20250112 15:08:45.637230 24354 tablet_copy_client.cc:323] T 92c129e062b945df8805bcc412f3a599 P 932773f4da6d4c238b6963b863c59be9: tablet copy: Beginning tablet copy session from remote peer at address 127.17.60.194:33901
I20250112 15:08:45.640157 24200 tablet_copy_service.cc:140] P 8586159698b04cde88062accfc5d1bf1: Received BeginTabletCopySession request for tablet 92c129e062b945df8805bcc412f3a599 from peer 932773f4da6d4c238b6963b863c59be9 ({username='slave'} at 127.0.0.1:39690)
I20250112 15:08:45.640230 24200 tablet_copy_service.cc:161] P 8586159698b04cde88062accfc5d1bf1: Beginning new tablet copy session on tablet 92c129e062b945df8805bcc412f3a599 from peer 932773f4da6d4c238b6963b863c59be9 at {username='slave'} at 127.0.0.1:39690: session id = 932773f4da6d4c238b6963b863c59be9-92c129e062b945df8805bcc412f3a599
I20250112 15:08:45.640810 24200 tablet_copy_source_session.cc:215] T 92c129e062b945df8805bcc412f3a599 P 8586159698b04cde88062accfc5d1bf1: Tablet Copy: opened 0 blocks and 1 log segments
I20250112 15:08:45.641093 24354 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 92c129e062b945df8805bcc412f3a599. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 15:08:45.642344 24354 tablet_copy_client.cc:806] T 92c129e062b945df8805bcc412f3a599 P 932773f4da6d4c238b6963b863c59be9: tablet copy: Starting download of 0 data blocks...
I20250112 15:08:45.642454 24354 tablet_copy_client.cc:670] T 92c129e062b945df8805bcc412f3a599 P 932773f4da6d4c238b6963b863c59be9: tablet copy: Starting download of 1 WAL segments...
I20250112 15:08:45.642956 24354 tablet_copy_client.cc:538] T 92c129e062b945df8805bcc412f3a599 P 932773f4da6d4c238b6963b863c59be9: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250112 15:08:45.643595 24354 tablet_bootstrap.cc:492] T 92c129e062b945df8805bcc412f3a599 P 932773f4da6d4c238b6963b863c59be9: Bootstrap starting.
I20250112 15:08:45.644948 24354 tablet_bootstrap.cc:492] T 92c129e062b945df8805bcc412f3a599 P 932773f4da6d4c238b6963b863c59be9: Bootstrap replayed 1/1 log segments. Stats: ops{read=3 overwritten=0 applied=3 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250112 15:08:45.645182 24354 tablet_bootstrap.cc:492] T 92c129e062b945df8805bcc412f3a599 P 932773f4da6d4c238b6963b863c59be9: Bootstrap complete.
I20250112 15:08:45.645270 24354 ts_tablet_manager.cc:1397] T 92c129e062b945df8805bcc412f3a599 P 932773f4da6d4c238b6963b863c59be9: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.001s
I20250112 15:08:45.645421 24354 raft_consensus.cc:357] T 92c129e062b945df8805bcc412f3a599 P 932773f4da6d4c238b6963b863c59be9 [term 3 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "111d62a4d211430fa930ff5a4dfa99d0" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 35133 } attrs { replace: true } } peers { permanent_uuid: "8586159698b04cde88062accfc5d1bf1" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 33901 } } peers { permanent_uuid: "05ca05df85b94bc985b549e68caa6c36" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 44611 } } peers { permanent_uuid: "932773f4da6d4c238b6963b863c59be9" member_type: NON_VOTER last_known_addr { host: "127.17.60.196" port: 44863 } attrs { promote: true } }
I20250112 15:08:45.645498 24354 raft_consensus.cc:738] T 92c129e062b945df8805bcc412f3a599 P 932773f4da6d4c238b6963b863c59be9 [term 3 LEARNER]: Becoming Follower/Learner. State: Replica: 932773f4da6d4c238b6963b863c59be9, State: Initialized, Role: LEARNER
I20250112 15:08:45.645536 24354 consensus_queue.cc:260] T 92c129e062b945df8805bcc412f3a599 P 932773f4da6d4c238b6963b863c59be9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "111d62a4d211430fa930ff5a4dfa99d0" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 35133 } attrs { replace: true } } peers { permanent_uuid: "8586159698b04cde88062accfc5d1bf1" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 33901 } } peers { permanent_uuid: "05ca05df85b94bc985b549e68caa6c36" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 44611 } } peers { permanent_uuid: "932773f4da6d4c238b6963b863c59be9" member_type: NON_VOTER last_known_addr { host: "127.17.60.196" port: 44863 } attrs { promote: true } }
I20250112 15:08:45.645720 24354 ts_tablet_manager.cc:1428] T 92c129e062b945df8805bcc412f3a599 P 932773f4da6d4c238b6963b863c59be9: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20250112 15:08:45.645884 24200 tablet_copy_service.cc:342] P 8586159698b04cde88062accfc5d1bf1: Request end of tablet copy session 932773f4da6d4c238b6963b863c59be9-92c129e062b945df8805bcc412f3a599 received from {username='slave'} at 127.0.0.1:39690
I20250112 15:08:45.645924 24200 tablet_copy_service.cc:434] P 8586159698b04cde88062accfc5d1bf1: ending tablet copy session 932773f4da6d4c238b6963b863c59be9-92c129e062b945df8805bcc412f3a599 on tablet 92c129e062b945df8805bcc412f3a599 with peer 932773f4da6d4c238b6963b863c59be9
I20250112 15:08:45.677102 24354 ts_tablet_manager.cc:927] T 3e43528d2bb24563966ef89a76aa761e P 932773f4da6d4c238b6963b863c59be9: Initiating tablet copy from peer 111d62a4d211430fa930ff5a4dfa99d0 (127.17.60.193:35133)
I20250112 15:08:45.677317 24354 tablet_copy_client.cc:323] T 3e43528d2bb24563966ef89a76aa761e P 932773f4da6d4c238b6963b863c59be9: tablet copy: Beginning tablet copy session from remote peer at address 127.17.60.193:35133
I20250112 15:08:45.680011 24118 tablet_copy_service.cc:140] P 111d62a4d211430fa930ff5a4dfa99d0: Received BeginTabletCopySession request for tablet 3e43528d2bb24563966ef89a76aa761e from peer 932773f4da6d4c238b6963b863c59be9 ({username='slave'} at 127.0.0.1:49430)
I20250112 15:08:45.680095 24118 tablet_copy_service.cc:161] P 111d62a4d211430fa930ff5a4dfa99d0: Beginning new tablet copy session on tablet 3e43528d2bb24563966ef89a76aa761e from peer 932773f4da6d4c238b6963b863c59be9 at {username='slave'} at 127.0.0.1:49430: session id = 932773f4da6d4c238b6963b863c59be9-3e43528d2bb24563966ef89a76aa761e
I20250112 15:08:45.680619 24118 tablet_copy_source_session.cc:215] T 3e43528d2bb24563966ef89a76aa761e P 111d62a4d211430fa930ff5a4dfa99d0: Tablet Copy: opened 0 blocks and 1 log segments
I20250112 15:08:45.680914 24354 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3e43528d2bb24563966ef89a76aa761e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 15:08:45.682090 24354 tablet_copy_client.cc:806] T 3e43528d2bb24563966ef89a76aa761e P 932773f4da6d4c238b6963b863c59be9: tablet copy: Starting download of 0 data blocks...
I20250112 15:08:45.682215 24354 tablet_copy_client.cc:670] T 3e43528d2bb24563966ef89a76aa761e P 932773f4da6d4c238b6963b863c59be9: tablet copy: Starting download of 1 WAL segments...
I20250112 15:08:45.682675 24354 tablet_copy_client.cc:538] T 3e43528d2bb24563966ef89a76aa761e P 932773f4da6d4c238b6963b863c59be9: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250112 15:08:45.683310 24354 tablet_bootstrap.cc:492] T 3e43528d2bb24563966ef89a76aa761e P 932773f4da6d4c238b6963b863c59be9: Bootstrap starting.
I20250112 15:08:45.684741 24354 tablet_bootstrap.cc:492] T 3e43528d2bb24563966ef89a76aa761e P 932773f4da6d4c238b6963b863c59be9: Bootstrap replayed 1/1 log segments. Stats: ops{read=3 overwritten=0 applied=3 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250112 15:08:45.684918 24354 tablet_bootstrap.cc:492] T 3e43528d2bb24563966ef89a76aa761e P 932773f4da6d4c238b6963b863c59be9: Bootstrap complete.
I20250112 15:08:45.685012 24354 ts_tablet_manager.cc:1397] T 3e43528d2bb24563966ef89a76aa761e P 932773f4da6d4c238b6963b863c59be9: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20250112 15:08:45.685150 24354 raft_consensus.cc:357] T 3e43528d2bb24563966ef89a76aa761e P 932773f4da6d4c238b6963b863c59be9 [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "111d62a4d211430fa930ff5a4dfa99d0" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 35133 } } peers { permanent_uuid: "05ca05df85b94bc985b549e68caa6c36" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 44611 } attrs { replace: true } } peers { permanent_uuid: "8586159698b04cde88062accfc5d1bf1" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 33901 } } peers { permanent_uuid: "932773f4da6d4c238b6963b863c59be9" member_type: NON_VOTER last_known_addr { host: "127.17.60.196" port: 44863 } attrs { promote: true } }
I20250112 15:08:45.685253 24354 raft_consensus.cc:738] T 3e43528d2bb24563966ef89a76aa761e P 932773f4da6d4c238b6963b863c59be9 [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: 932773f4da6d4c238b6963b863c59be9, State: Initialized, Role: LEARNER
I20250112 15:08:45.685298 24354 consensus_queue.cc:260] T 3e43528d2bb24563966ef89a76aa761e P 932773f4da6d4c238b6963b863c59be9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 2.3, Last appended by leader: 3, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "111d62a4d211430fa930ff5a4dfa99d0" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 35133 } } peers { permanent_uuid: "05ca05df85b94bc985b549e68caa6c36" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 44611 } attrs { replace: true } } peers { permanent_uuid: "8586159698b04cde88062accfc5d1bf1" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 33901 } } peers { permanent_uuid: "932773f4da6d4c238b6963b863c59be9" member_type: NON_VOTER last_known_addr { host: "127.17.60.196" port: 44863 } attrs { promote: true } }
I20250112 15:08:45.685392 24354 ts_tablet_manager.cc:1428] T 3e43528d2bb24563966ef89a76aa761e P 932773f4da6d4c238b6963b863c59be9: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20250112 15:08:45.685565 24118 tablet_copy_service.cc:342] P 111d62a4d211430fa930ff5a4dfa99d0: Request end of tablet copy session 932773f4da6d4c238b6963b863c59be9-3e43528d2bb24563966ef89a76aa761e received from {username='slave'} at 127.0.0.1:49430
I20250112 15:08:45.685621 24118 tablet_copy_service.cc:434] P 111d62a4d211430fa930ff5a4dfa99d0: ending tablet copy session 932773f4da6d4c238b6963b863c59be9-3e43528d2bb24563966ef89a76aa761e on tablet 3e43528d2bb24563966ef89a76aa761e with peer 932773f4da6d4c238b6963b863c59be9
I20250112 15:08:45.702898 24354 ts_tablet_manager.cc:927] T 6a1c881a80004842a1eb46bb488d3834 P 932773f4da6d4c238b6963b863c59be9: Initiating tablet copy from peer 05ca05df85b94bc985b549e68caa6c36 (127.17.60.195:44611)
I20250112 15:08:45.703073 24354 tablet_copy_client.cc:323] T 6a1c881a80004842a1eb46bb488d3834 P 932773f4da6d4c238b6963b863c59be9: tablet copy: Beginning tablet copy session from remote peer at address 127.17.60.195:44611
I20250112 15:08:45.706199 24277 tablet_copy_service.cc:140] P 05ca05df85b94bc985b549e68caa6c36: Received BeginTabletCopySession request for tablet 6a1c881a80004842a1eb46bb488d3834 from peer 932773f4da6d4c238b6963b863c59be9 ({username='slave'} at 127.0.0.1:53326)
I20250112 15:08:45.706277 24277 tablet_copy_service.cc:161] P 05ca05df85b94bc985b549e68caa6c36: Beginning new tablet copy session on tablet 6a1c881a80004842a1eb46bb488d3834 from peer 932773f4da6d4c238b6963b863c59be9 at {username='slave'} at 127.0.0.1:53326: session id = 932773f4da6d4c238b6963b863c59be9-6a1c881a80004842a1eb46bb488d3834
I20250112 15:08:45.706910 24277 tablet_copy_source_session.cc:215] T 6a1c881a80004842a1eb46bb488d3834 P 05ca05df85b94bc985b549e68caa6c36: Tablet Copy: opened 0 blocks and 1 log segments
I20250112 15:08:45.707190 24354 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6a1c881a80004842a1eb46bb488d3834. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 15:08:45.708552 24354 tablet_copy_client.cc:806] T 6a1c881a80004842a1eb46bb488d3834 P 932773f4da6d4c238b6963b863c59be9: tablet copy: Starting download of 0 data blocks...
I20250112 15:08:45.708725 24354 tablet_copy_client.cc:670] T 6a1c881a80004842a1eb46bb488d3834 P 932773f4da6d4c238b6963b863c59be9: tablet copy: Starting download of 1 WAL segments...
I20250112 15:08:45.709226 24354 tablet_copy_client.cc:538] T 6a1c881a80004842a1eb46bb488d3834 P 932773f4da6d4c238b6963b863c59be9: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250112 15:08:45.709986 24354 tablet_bootstrap.cc:492] T 6a1c881a80004842a1eb46bb488d3834 P 932773f4da6d4c238b6963b863c59be9: Bootstrap starting.
I20250112 15:08:45.711475 24354 tablet_bootstrap.cc:492] T 6a1c881a80004842a1eb46bb488d3834 P 932773f4da6d4c238b6963b863c59be9: Bootstrap replayed 1/1 log segments. Stats: ops{read=3 overwritten=0 applied=3 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250112 15:08:45.711697 24354 tablet_bootstrap.cc:492] T 6a1c881a80004842a1eb46bb488d3834 P 932773f4da6d4c238b6963b863c59be9: Bootstrap complete.
I20250112 15:08:45.711851 24354 ts_tablet_manager.cc:1397] T 6a1c881a80004842a1eb46bb488d3834 P 932773f4da6d4c238b6963b863c59be9: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20250112 15:08:45.712010 24354 raft_consensus.cc:357] T 6a1c881a80004842a1eb46bb488d3834 P 932773f4da6d4c238b6963b863c59be9 [term 3 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "111d62a4d211430fa930ff5a4dfa99d0" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 35133 } } peers { permanent_uuid: "8586159698b04cde88062accfc5d1bf1" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 33901 } attrs { replace: true } } peers { permanent_uuid: "05ca05df85b94bc985b549e68caa6c36" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 44611 } } peers { permanent_uuid: "932773f4da6d4c238b6963b863c59be9" member_type: NON_VOTER last_known_addr { host: "127.17.60.196" port: 44863 } attrs { promote: true } }
I20250112 15:08:45.712107 24354 raft_consensus.cc:738] T 6a1c881a80004842a1eb46bb488d3834 P 932773f4da6d4c238b6963b863c59be9 [term 3 LEARNER]: Becoming Follower/Learner. State: Replica: 932773f4da6d4c238b6963b863c59be9, State: Initialized, Role: LEARNER
I20250112 15:08:45.712141 24354 consensus_queue.cc:260] T 6a1c881a80004842a1eb46bb488d3834 P 932773f4da6d4c238b6963b863c59be9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "111d62a4d211430fa930ff5a4dfa99d0" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 35133 } } peers { permanent_uuid: "8586159698b04cde88062accfc5d1bf1" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 33901 } attrs { replace: true } } peers { permanent_uuid: "05ca05df85b94bc985b549e68caa6c36" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 44611 } } peers { permanent_uuid: "932773f4da6d4c238b6963b863c59be9" member_type: NON_VOTER last_known_addr { host: "127.17.60.196" port: 44863 } attrs { promote: true } }
I20250112 15:08:45.712247 24354 ts_tablet_manager.cc:1428] T 6a1c881a80004842a1eb46bb488d3834 P 932773f4da6d4c238b6963b863c59be9: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20250112 15:08:45.712445 24277 tablet_copy_service.cc:342] P 05ca05df85b94bc985b549e68caa6c36: Request end of tablet copy session 932773f4da6d4c238b6963b863c59be9-6a1c881a80004842a1eb46bb488d3834 received from {username='slave'} at 127.0.0.1:53326
I20250112 15:08:45.712484 24277 tablet_copy_service.cc:434] P 05ca05df85b94bc985b549e68caa6c36: ending tablet copy session 932773f4da6d4c238b6963b863c59be9-6a1c881a80004842a1eb46bb488d3834 on tablet 6a1c881a80004842a1eb46bb488d3834 with peer 932773f4da6d4c238b6963b863c59be9
I20250112 15:08:46.032723 24014 raft_consensus.cc:1212] T 92c129e062b945df8805bcc412f3a599 P 932773f4da6d4c238b6963b863c59be9 [term 3 LEARNER]: Deduplicated request from leader. Original: 3.2->[3.3-3.3] Dedup: 3.3->[]
I20250112 15:08:46.108093 24014 raft_consensus.cc:1212] T 6a1c881a80004842a1eb46bb488d3834 P 932773f4da6d4c238b6963b863c59be9 [term 3 LEARNER]: Deduplicated request from leader. Original: 3.2->[3.3-3.3] Dedup: 3.3->[]
I20250112 15:08:46.227105 24014 raft_consensus.cc:1212] T 3e43528d2bb24563966ef89a76aa761e P 932773f4da6d4c238b6963b863c59be9 [term 2 LEARNER]: Deduplicated request from leader. Original: 2.2->[2.3-2.3] Dedup: 2.3->[]
I20250112 15:08:46.325264 17651 tablet_server.cc:178] TabletServer@127.17.60.193:35133 shutting down...
W20250112 15:08:46.327174 23685 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer [suppressed 14 similar messages]
W20250112 15:08:46.327857 23686 auto_rebalancer.cc:663] Could not move replica: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20250112 15:08:46.327912 23686 auto_rebalancer.cc:263] scheduled replica move failed to complete: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20250112 15:08:46.328532 17651 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250112 15:08:46.328675 17651 tablet_replica.cc:331] stopping tablet replica
I20250112 15:08:46.328753 17651 raft_consensus.cc:2238] T 92c129e062b945df8805bcc412f3a599 P 111d62a4d211430fa930ff5a4dfa99d0 [term 3 FOLLOWER]: Raft consensus shutting down.
I20250112 15:08:46.328812 17651 raft_consensus.cc:2267] T 92c129e062b945df8805bcc412f3a599 P 111d62a4d211430fa930ff5a4dfa99d0 [term 3 FOLLOWER]: Raft consensus is shut down!
I20250112 15:08:46.328999 17651 tablet_replica.cc:331] stopping tablet replica
I20250112 15:08:46.329059 17651 raft_consensus.cc:2238] T 6a1c881a80004842a1eb46bb488d3834 P 111d62a4d211430fa930ff5a4dfa99d0 [term 3 FOLLOWER]: Raft consensus shutting down.
I20250112 15:08:46.329111 17651 raft_consensus.cc:2267] T 6a1c881a80004842a1eb46bb488d3834 P 111d62a4d211430fa930ff5a4dfa99d0 [term 3 FOLLOWER]: Raft consensus is shut down!
I20250112 15:08:46.329236 17651 tablet_replica.cc:331] stopping tablet replica
I20250112 15:08:46.329270 17651 raft_consensus.cc:2238] T 3e43528d2bb24563966ef89a76aa761e P 111d62a4d211430fa930ff5a4dfa99d0 [term 2 LEADER]: Raft consensus shutting down.
I20250112 15:08:46.329331 17651 raft_consensus.cc:2267] T 3e43528d2bb24563966ef89a76aa761e P 111d62a4d211430fa930ff5a4dfa99d0 [term 2 FOLLOWER]: Raft consensus is shut down!
I20250112 15:08:46.329458 17651 tablet_replica.cc:331] stopping tablet replica
I20250112 15:08:46.329491 17651 raft_consensus.cc:2238] T 483cfa86b848433bbc365bae489ed251 P 111d62a4d211430fa930ff5a4dfa99d0 [term 2 FOLLOWER]: Raft consensus shutting down.
I20250112 15:08:46.329516 17651 raft_consensus.cc:2267] T 483cfa86b848433bbc365bae489ed251 P 111d62a4d211430fa930ff5a4dfa99d0 [term 2 FOLLOWER]: Raft consensus is shut down!
I20250112 15:08:46.342494 17651 tablet_server.cc:195] TabletServer@127.17.60.193:35133 shutdown complete.
I20250112 15:08:46.343804 17651 tablet_server.cc:178] TabletServer@127.17.60.194:33901 shutting down...
W20250112 15:08:46.344656 23686 auto_rebalancer.cc:663] Could not move replica: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20250112 15:08:46.344697 23686 auto_rebalancer.cc:263] scheduled replica move failed to complete: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20250112 15:08:46.348547 17651 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250112 15:08:46.348727 17651 tablet_replica.cc:331] stopping tablet replica
I20250112 15:08:46.348789 17651 raft_consensus.cc:2238] T 92c129e062b945df8805bcc412f3a599 P 8586159698b04cde88062accfc5d1bf1 [term 3 LEADER]: Raft consensus shutting down.
I20250112 15:08:46.348876 17651 raft_consensus.cc:2267] T 92c129e062b945df8805bcc412f3a599 P 8586159698b04cde88062accfc5d1bf1 [term 3 FOLLOWER]: Raft consensus is shut down!
I20250112 15:08:46.349046 17651 tablet_replica.cc:331] stopping tablet replica
I20250112 15:08:46.349099 17651 raft_consensus.cc:2238] T 6a1c881a80004842a1eb46bb488d3834 P 8586159698b04cde88062accfc5d1bf1 [term 3 FOLLOWER]: Raft consensus shutting down.
I20250112 15:08:46.349181 17651 raft_consensus.cc:2267] T 6a1c881a80004842a1eb46bb488d3834 P 8586159698b04cde88062accfc5d1bf1 [term 3 FOLLOWER]: Raft consensus is shut down!
I20250112 15:08:46.349329 17651 tablet_replica.cc:331] stopping tablet replica
I20250112 15:08:46.349376 17651 raft_consensus.cc:2238] T 3e43528d2bb24563966ef89a76aa761e P 8586159698b04cde88062accfc5d1bf1 [term 2 FOLLOWER]: Raft consensus shutting down.
I20250112 15:08:46.349422 17651 raft_consensus.cc:2267] T 3e43528d2bb24563966ef89a76aa761e P 8586159698b04cde88062accfc5d1bf1 [term 2 FOLLOWER]: Raft consensus is shut down!
I20250112 15:08:46.349545 17651 tablet_replica.cc:331] stopping tablet replica
I20250112 15:08:46.349586 17651 raft_consensus.cc:2238] T 483cfa86b848433bbc365bae489ed251 P 8586159698b04cde88062accfc5d1bf1 [term 2 FOLLOWER]: Raft consensus shutting down.
I20250112 15:08:46.349653 17651 raft_consensus.cc:2267] T 483cfa86b848433bbc365bae489ed251 P 8586159698b04cde88062accfc5d1bf1 [term 2 FOLLOWER]: Raft consensus is shut down!
W20250112 15:08:46.361567 24231 consensus_peers.cc:487] T 483cfa86b848433bbc365bae489ed251 P 05ca05df85b94bc985b549e68caa6c36 -> Peer 8586159698b04cde88062accfc5d1bf1 (127.17.60.194:33901): Couldn't send request to peer 8586159698b04cde88062accfc5d1bf1. Status: Network error: Client connection negotiation failed: client connection to 127.17.60.194:33901: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20250112 15:08:46.361976 17651 tablet_server.cc:195] TabletServer@127.17.60.194:33901 shutdown complete.
I20250112 15:08:46.363399 17651 tablet_server.cc:178] TabletServer@127.17.60.195:44611 shutting down...
W20250112 15:08:46.363694 23686 auto_rebalancer.cc:663] Could not move replica: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20250112 15:08:46.363729 23686 auto_rebalancer.cc:263] scheduled replica move failed to complete: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20250112 15:08:46.366812 17651 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250112 15:08:46.366911 17651 tablet_replica.cc:331] stopping tablet replica
I20250112 15:08:46.366977 17651 raft_consensus.cc:2238] T 92c129e062b945df8805bcc412f3a599 P 05ca05df85b94bc985b549e68caa6c36 [term 3 FOLLOWER]: Raft consensus shutting down.
I20250112 15:08:46.367030 17651 raft_consensus.cc:2267] T 92c129e062b945df8805bcc412f3a599 P 05ca05df85b94bc985b549e68caa6c36 [term 3 FOLLOWER]: Raft consensus is shut down!
I20250112 15:08:46.367168 17651 tablet_replica.cc:331] stopping tablet replica
I20250112 15:08:46.367197 17651 raft_consensus.cc:2238] T 6a1c881a80004842a1eb46bb488d3834 P 05ca05df85b94bc985b549e68caa6c36 [term 3 LEADER]: Raft consensus shutting down.
I20250112 15:08:46.367250 17651 raft_consensus.cc:2267] T 6a1c881a80004842a1eb46bb488d3834 P 05ca05df85b94bc985b549e68caa6c36 [term 3 FOLLOWER]: Raft consensus is shut down!
I20250112 15:08:46.367328 17651 tablet_replica.cc:331] stopping tablet replica
I20250112 15:08:46.367352 17651 raft_consensus.cc:2238] T 3e43528d2bb24563966ef89a76aa761e P 05ca05df85b94bc985b549e68caa6c36 [term 2 FOLLOWER]: Raft consensus shutting down.
I20250112 15:08:46.367388 17651 raft_consensus.cc:2267] T 3e43528d2bb24563966ef89a76aa761e P 05ca05df85b94bc985b549e68caa6c36 [term 2 FOLLOWER]: Raft consensus is shut down!
I20250112 15:08:46.367455 17651 tablet_replica.cc:331] stopping tablet replica
I20250112 15:08:46.367480 17651 raft_consensus.cc:2238] T 483cfa86b848433bbc365bae489ed251 P 05ca05df85b94bc985b549e68caa6c36 [term 2 LEADER]: Raft consensus shutting down.
I20250112 15:08:46.367511 17651 raft_consensus.cc:2267] T 483cfa86b848433bbc365bae489ed251 P 05ca05df85b94bc985b549e68caa6c36 [term 2 FOLLOWER]: Raft consensus is shut down!
I20250112 15:08:46.379422 17651 tablet_server.cc:195] TabletServer@127.17.60.195:44611 shutdown complete.
I20250112 15:08:46.380535 17651 tablet_server.cc:178] TabletServer@127.17.60.196:0 shutting down...
I20250112 15:08:46.383976 17651 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250112 15:08:46.384093 17651 tablet_replica.cc:331] T 6a1c881a80004842a1eb46bb488d3834 P 932773f4da6d4c238b6963b863c59be9: stopping tablet replica
I20250112 15:08:46.384156 17651 raft_consensus.cc:2238] T 6a1c881a80004842a1eb46bb488d3834 P 932773f4da6d4c238b6963b863c59be9 [term 3 LEARNER]: Raft consensus shutting down.
I20250112 15:08:46.384208 17651 raft_consensus.cc:2267] T 6a1c881a80004842a1eb46bb488d3834 P 932773f4da6d4c238b6963b863c59be9 [term 3 LEARNER]: Raft consensus is shut down!
I20250112 15:08:46.384378 17651 tablet_replica.cc:331] T 92c129e062b945df8805bcc412f3a599 P 932773f4da6d4c238b6963b863c59be9: stopping tablet replica
I20250112 15:08:46.384423 17651 raft_consensus.cc:2238] T 92c129e062b945df8805bcc412f3a599 P 932773f4da6d4c238b6963b863c59be9 [term 3 LEARNER]: Raft consensus shutting down.
I20250112 15:08:46.384459 17651 raft_consensus.cc:2267] T 92c129e062b945df8805bcc412f3a599 P 932773f4da6d4c238b6963b863c59be9 [term 3 LEARNER]: Raft consensus is shut down!
I20250112 15:08:46.384590 17651 tablet_replica.cc:331] T 3e43528d2bb24563966ef89a76aa761e P 932773f4da6d4c238b6963b863c59be9: stopping tablet replica
I20250112 15:08:46.384627 17651 raft_consensus.cc:2238] T 3e43528d2bb24563966ef89a76aa761e P 932773f4da6d4c238b6963b863c59be9 [term 2 LEARNER]: Raft consensus shutting down.
I20250112 15:08:46.384662 17651 raft_consensus.cc:2267] T 3e43528d2bb24563966ef89a76aa761e P 932773f4da6d4c238b6963b863c59be9 [term 2 LEARNER]: Raft consensus is shut down!
I20250112 15:08:46.396342 17651 tablet_server.cc:195] TabletServer@127.17.60.196:0 shutdown complete.
I20250112 15:08:46.397564 17651 master.cc:537] Master@127.17.60.254:42077 shutting down...
I20250112 15:08:46.400358 17651 raft_consensus.cc:2238] T 00000000000000000000000000000000 P d2b58b62446f43f69675af5ece903bf4 [term 1 LEADER]: Raft consensus shutting down.
I20250112 15:08:46.400434 17651 raft_consensus.cc:2267] T 00000000000000000000000000000000 P d2b58b62446f43f69675af5ece903bf4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 15:08:46.400470 17651 tablet_replica.cc:331] T 00000000000000000000000000000000 P d2b58b62446f43f69675af5ece903bf4: stopping tablet replica
I20250112 15:08:46.412217 17651 master.cc:559] Master@127.17.60.254:42077 shutdown complete.
[ OK ] AutoRebalancerTest.TestHandlingFailedTservers (11299 ms)
[ RUN ] AutoRebalancerTest.TestDeletedTables
I20250112 15:08:46.419175 17651 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.17.60.254:37937
I20250112 15:08:46.419333 17651 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250112 15:08:46.420670 24372 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250112 15:08:46.421492 24375 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250112 15:08:46.422114 24373 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250112 15:08:46.422916 17651 server_base.cc:1034] running on GCE node
I20250112 15:08:46.423010 17651 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250112 15:08:46.423043 17651 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250112 15:08:46.423063 17651 hybrid_clock.cc:648] HybridClock initialized: now 1736694526423063 us; error 0 us; skew 500 ppm
I20250112 15:08:46.423696 17651 webserver.cc:458] Webserver started at http://127.17.60.254:41973/ using document root <none> and password file <none>
I20250112 15:08:46.423779 17651 fs_manager.cc:362] Metadata directory not provided
I20250112 15:08:46.423827 17651 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250112 15:08:46.423882 17651 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250112 15:08:46.424150 17651 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task95dLSD/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestDeletedTables.1736694351070047-17651-0/minicluster-data/master-0-root/instance:
uuid: "ea845a3e3060484f9ece4069e440908c"
format_stamp: "Formatted at 2025-01-12 15:08:46 on dist-test-slave-b4nz"
I20250112 15:08:46.425107 17651 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20250112 15:08:46.425657 24380 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250112 15:08:46.425843 17651 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20250112 15:08:46.425899 17651 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task95dLSD/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestDeletedTables.1736694351070047-17651-0/minicluster-data/master-0-root
uuid: "ea845a3e3060484f9ece4069e440908c"
format_stamp: "Formatted at 2025-01-12 15:08:46 on dist-test-slave-b4nz"
I20250112 15:08:46.425948 17651 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task95dLSD/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestDeletedTables.1736694351070047-17651-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-task95dLSD/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestDeletedTables.1736694351070047-17651-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-task95dLSD/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestDeletedTables.1736694351070047-17651-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250112 15:08:46.441598 17651 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250112 15:08:46.441797 17651 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250112 15:08:46.444346 17651 rpc_server.cc:307] RPC server started. Bound to: 127.17.60.254:37937
I20250112 15:08:46.447398 24432 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.17.60.254:37937 every 8 connection(s)
I20250112 15:08:46.447528 24433 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 15:08:46.448611 24433 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ea845a3e3060484f9ece4069e440908c: Bootstrap starting.
I20250112 15:08:46.448946 24433 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ea845a3e3060484f9ece4069e440908c: Neither blocks nor log segments found. Creating new log.
I20250112 15:08:46.449435 24433 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ea845a3e3060484f9ece4069e440908c: No bootstrap required, opened a new log
I20250112 15:08:46.449550 24433 raft_consensus.cc:357] T 00000000000000000000000000000000 P ea845a3e3060484f9ece4069e440908c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ea845a3e3060484f9ece4069e440908c" member_type: VOTER }
I20250112 15:08:46.449604 24433 raft_consensus.cc:383] T 00000000000000000000000000000000 P ea845a3e3060484f9ece4069e440908c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 15:08:46.449617 24433 raft_consensus.cc:738] T 00000000000000000000000000000000 P ea845a3e3060484f9ece4069e440908c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ea845a3e3060484f9ece4069e440908c, State: Initialized, Role: FOLLOWER
I20250112 15:08:46.449669 24433 consensus_queue.cc:260] T 00000000000000000000000000000000 P ea845a3e3060484f9ece4069e440908c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ea845a3e3060484f9ece4069e440908c" member_type: VOTER }
I20250112 15:08:46.449707 24433 raft_consensus.cc:397] T 00000000000000000000000000000000 P ea845a3e3060484f9ece4069e440908c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250112 15:08:46.449728 24433 raft_consensus.cc:491] T 00000000000000000000000000000000 P ea845a3e3060484f9ece4069e440908c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250112 15:08:46.449755 24433 raft_consensus.cc:3054] T 00000000000000000000000000000000 P ea845a3e3060484f9ece4069e440908c [term 0 FOLLOWER]: Advancing to term 1
I20250112 15:08:46.450238 24433 raft_consensus.cc:513] T 00000000000000000000000000000000 P ea845a3e3060484f9ece4069e440908c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ea845a3e3060484f9ece4069e440908c" member_type: VOTER }
I20250112 15:08:46.450292 24433 leader_election.cc:304] T 00000000000000000000000000000000 P ea845a3e3060484f9ece4069e440908c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ea845a3e3060484f9ece4069e440908c; no voters:
I20250112 15:08:46.450393 24433 leader_election.cc:290] T 00000000000000000000000000000000 P ea845a3e3060484f9ece4069e440908c [CANDIDATE]: Term 1 election: Requested vote from peers
I20250112 15:08:46.450450 24436 raft_consensus.cc:2798] T 00000000000000000000000000000000 P ea845a3e3060484f9ece4069e440908c [term 1 FOLLOWER]: Leader election won for term 1
I20250112 15:08:46.450538 24433 sys_catalog.cc:564] T 00000000000000000000000000000000 P ea845a3e3060484f9ece4069e440908c [sys.catalog]: configured and running, proceeding with master startup.
I20250112 15:08:46.450588 24436 raft_consensus.cc:695] T 00000000000000000000000000000000 P ea845a3e3060484f9ece4069e440908c [term 1 LEADER]: Becoming Leader. State: Replica: ea845a3e3060484f9ece4069e440908c, State: Running, Role: LEADER
I20250112 15:08:46.450644 24436 consensus_queue.cc:237] T 00000000000000000000000000000000 P ea845a3e3060484f9ece4069e440908c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ea845a3e3060484f9ece4069e440908c" member_type: VOTER }
I20250112 15:08:46.450884 24437 sys_catalog.cc:455] T 00000000000000000000000000000000 P ea845a3e3060484f9ece4069e440908c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ea845a3e3060484f9ece4069e440908c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ea845a3e3060484f9ece4069e440908c" member_type: VOTER } }
I20250112 15:08:46.450903 24438 sys_catalog.cc:455] T 00000000000000000000000000000000 P ea845a3e3060484f9ece4069e440908c [sys.catalog]: SysCatalogTable state changed. Reason: New leader ea845a3e3060484f9ece4069e440908c. Latest consensus state: current_term: 1 leader_uuid: "ea845a3e3060484f9ece4069e440908c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ea845a3e3060484f9ece4069e440908c" member_type: VOTER } }
I20250112 15:08:46.451031 24438 sys_catalog.cc:458] T 00000000000000000000000000000000 P ea845a3e3060484f9ece4069e440908c [sys.catalog]: This master's current role is: LEADER
I20250112 15:08:46.451226 24437 sys_catalog.cc:458] T 00000000000000000000000000000000 P ea845a3e3060484f9ece4069e440908c [sys.catalog]: This master's current role is: LEADER
I20250112 15:08:46.451218 24444 catalog_manager.cc:1476] Loading table and tablet metadata into memory...
I20250112 15:08:46.451399 24444 catalog_manager.cc:1485] Initializing Kudu cluster ID...
I20250112 15:08:46.451768 17651 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20250112 15:08:46.451972 24444 catalog_manager.cc:1348] Generated new cluster ID: 5d8a5ce08ab540769db1e1a94b098029
I20250112 15:08:46.452011 24444 catalog_manager.cc:1496] Initializing Kudu internal certificate authority...
I20250112 15:08:46.459873 24444 catalog_manager.cc:1371] Generated new certificate authority record
I20250112 15:08:46.460275 24444 catalog_manager.cc:1505] Loading token signing keys...
I20250112 15:08:46.466969 24444 catalog_manager.cc:5899] T 00000000000000000000000000000000 P ea845a3e3060484f9ece4069e440908c: Generated new TSK 0
I20250112 15:08:46.467056 24444 catalog_manager.cc:1515] Initializing in-progress tserver states...
I20250112 15:08:46.467392 17651 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250112 15:08:46.469259 24455 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250112 15:08:46.469866 24457 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250112 15:08:46.470132 24454 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250112 15:08:46.471318 17651 server_base.cc:1034] running on GCE node
I20250112 15:08:46.471398 17651 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250112 15:08:46.471422 17651 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250112 15:08:46.471434 17651 hybrid_clock.cc:648] HybridClock initialized: now 1736694526471435 us; error 0 us; skew 500 ppm
I20250112 15:08:46.472187 17651 webserver.cc:458] Webserver started at http://127.17.60.193:43709/ using document root <none> and password file <none>
I20250112 15:08:46.472288 17651 fs_manager.cc:362] Metadata directory not provided
I20250112 15:08:46.472337 17651 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250112 15:08:46.472391 17651 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250112 15:08:46.472721 17651 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task95dLSD/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestDeletedTables.1736694351070047-17651-0/minicluster-data/ts-0-root/instance:
uuid: "52bd510945874fa781ec841d40e251ab"
format_stamp: "Formatted at 2025-01-12 15:08:46 on dist-test-slave-b4nz"
I20250112 15:08:46.473620 17651 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20250112 15:08:46.474066 24462 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250112 15:08:46.474185 17651 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20250112 15:08:46.474238 17651 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task95dLSD/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestDeletedTables.1736694351070047-17651-0/minicluster-data/ts-0-root
uuid: "52bd510945874fa781ec841d40e251ab"
format_stamp: "Formatted at 2025-01-12 15:08:46 on dist-test-slave-b4nz"
I20250112 15:08:46.474287 17651 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task95dLSD/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestDeletedTables.1736694351070047-17651-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-task95dLSD/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestDeletedTables.1736694351070047-17651-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-task95dLSD/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestDeletedTables.1736694351070047-17651-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250112 15:08:46.484447 17651 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250112 15:08:46.484686 17651 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250112 15:08:46.484828 17651 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250112 15:08:46.485018 17651 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250112 15:08:46.485045 17651 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250112 15:08:46.485081 17651 ts_tablet_manager.cc:610] Registered 0 tablets
I20250112 15:08:46.485107 17651 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250112 15:08:46.487794 17651 rpc_server.cc:307] RPC server started. Bound to: 127.17.60.193:34851
I20250112 15:08:46.487856 24525 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.17.60.193:34851 every 8 connection(s)
I20250112 15:08:46.488169 17651 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250112 15:08:46.489560 24530 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250112 15:08:46.489785 24531 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250112 15:08:46.490679 24533 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250112 15:08:46.491506 17651 server_base.cc:1034] running on GCE node
I20250112 15:08:46.491514 24526 heartbeater.cc:346] Connected to a master server at 127.17.60.254:37937
I20250112 15:08:46.491603 24526 heartbeater.cc:463] Registering TS with master...
I20250112 15:08:46.491642 17651 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250112 15:08:46.491678 17651 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250112 15:08:46.491695 17651 hybrid_clock.cc:648] HybridClock initialized: now 1736694526491696 us; error 0 us; skew 500 ppm
I20250112 15:08:46.491726 24526 heartbeater.cc:510] Master 127.17.60.254:37937 requested a full tablet report, sending...
I20250112 15:08:46.491969 24397 ts_manager.cc:194] Registered new tserver with Master: 52bd510945874fa781ec841d40e251ab (127.17.60.193:34851)
I20250112 15:08:46.492353 17651 webserver.cc:458] Webserver started at http://127.17.60.194:40489/ using document root <none> and password file <none>
I20250112 15:08:46.492421 17651 fs_manager.cc:362] Metadata directory not provided
I20250112 15:08:46.492468 17651 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250112 15:08:46.492517 17651 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250112 15:08:46.492725 24397 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:60578
I20250112 15:08:46.492808 17651 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task95dLSD/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestDeletedTables.1736694351070047-17651-0/minicluster-data/ts-1-root/instance:
uuid: "56dd875b6ba54b40877146f15292bf2f"
format_stamp: "Formatted at 2025-01-12 15:08:46 on dist-test-slave-b4nz"
I20250112 15:08:46.493713 17651 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20250112 15:08:46.494117 24538 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250112 15:08:46.494243 17651 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20250112 15:08:46.494285 17651 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task95dLSD/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestDeletedTables.1736694351070047-17651-0/minicluster-data/ts-1-root
uuid: "56dd875b6ba54b40877146f15292bf2f"
format_stamp: "Formatted at 2025-01-12 15:08:46 on dist-test-slave-b4nz"
I20250112 15:08:46.494325 17651 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task95dLSD/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestDeletedTables.1736694351070047-17651-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-task95dLSD/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestDeletedTables.1736694351070047-17651-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-task95dLSD/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestDeletedTables.1736694351070047-17651-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250112 15:08:46.511047 17651 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250112 15:08:46.511278 17651 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250112 15:08:46.511435 17651 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250112 15:08:46.511652 17651 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250112 15:08:46.511683 17651 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250112 15:08:46.511718 17651 ts_tablet_manager.cc:610] Registered 0 tablets
I20250112 15:08:46.511744 17651 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250112 15:08:46.514537 17651 rpc_server.cc:307] RPC server started. Bound to: 127.17.60.194:32969
I20250112 15:08:46.514601 24601 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.17.60.194:32969 every 8 connection(s)
I20250112 15:08:46.514914 17651 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250112 15:08:46.516319 24605 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250112 15:08:46.516889 24606 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250112 15:08:46.517601 24608 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250112 15:08:46.518250 24602 heartbeater.cc:346] Connected to a master server at 127.17.60.254:37937
I20250112 15:08:46.518282 17651 server_base.cc:1034] running on GCE node
I20250112 15:08:46.518321 24602 heartbeater.cc:463] Registering TS with master...
I20250112 15:08:46.518433 17651 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
I20250112 15:08:46.518465 24602 heartbeater.cc:510] Master 127.17.60.254:37937 requested a full tablet report, sending...
W20250112 15:08:46.518471 17651 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250112 15:08:46.518520 17651 hybrid_clock.cc:648] HybridClock initialized: now 1736694526518521 us; error 0 us; skew 500 ppm
I20250112 15:08:46.518724 24397 ts_manager.cc:194] Registered new tserver with Master: 56dd875b6ba54b40877146f15292bf2f (127.17.60.194:32969)
I20250112 15:08:46.519090 24397 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:60590
I20250112 15:08:46.519264 17651 webserver.cc:458] Webserver started at http://127.17.60.195:37543/ using document root <none> and password file <none>
I20250112 15:08:46.519349 17651 fs_manager.cc:362] Metadata directory not provided
I20250112 15:08:46.519397 17651 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250112 15:08:46.519444 17651 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250112 15:08:46.519713 17651 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task95dLSD/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestDeletedTables.1736694351070047-17651-0/minicluster-data/ts-2-root/instance:
uuid: "f5b45ccea9c7438f8fba1b79073969e5"
format_stamp: "Formatted at 2025-01-12 15:08:46 on dist-test-slave-b4nz"
I20250112 15:08:46.520608 17651 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20250112 15:08:46.521072 24613 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250112 15:08:46.521201 17651 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20250112 15:08:46.521243 17651 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task95dLSD/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestDeletedTables.1736694351070047-17651-0/minicluster-data/ts-2-root
uuid: "f5b45ccea9c7438f8fba1b79073969e5"
format_stamp: "Formatted at 2025-01-12 15:08:46 on dist-test-slave-b4nz"
I20250112 15:08:46.521283 17651 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task95dLSD/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestDeletedTables.1736694351070047-17651-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-task95dLSD/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestDeletedTables.1736694351070047-17651-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-task95dLSD/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestDeletedTables.1736694351070047-17651-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250112 15:08:46.529064 17651 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250112 15:08:46.529290 17651 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250112 15:08:46.529435 17651 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250112 15:08:46.529610 17651 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250112 15:08:46.529637 17651 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250112 15:08:46.529661 17651 ts_tablet_manager.cc:610] Registered 0 tablets
I20250112 15:08:46.529688 17651 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250112 15:08:46.532474 17651 rpc_server.cc:307] RPC server started. Bound to: 127.17.60.195:40795
I20250112 15:08:46.532538 24676 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.17.60.195:40795 every 8 connection(s)
I20250112 15:08:46.535365 24677 heartbeater.cc:346] Connected to a master server at 127.17.60.254:37937
I20250112 15:08:46.535426 24677 heartbeater.cc:463] Registering TS with master...
I20250112 15:08:46.535526 24677 heartbeater.cc:510] Master 127.17.60.254:37937 requested a full tablet report, sending...
I20250112 15:08:46.535756 24397 ts_manager.cc:194] Registered new tserver with Master: f5b45ccea9c7438f8fba1b79073969e5 (127.17.60.195:40795)
I20250112 15:08:46.535987 17651 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.003323729s
I20250112 15:08:46.536137 24397 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:60594
I20250112 15:08:47.493201 24526 heartbeater.cc:502] Master 127.17.60.254:37937 was elected leader, sending a full tablet report...
I20250112 15:08:47.519651 24602 heartbeater.cc:502] Master 127.17.60.254:37937 was elected leader, sending a full tablet report...
I20250112 15:08:47.536662 24677 heartbeater.cc:502] Master 127.17.60.254:37937 was elected leader, sending a full tablet report...
I20250112 15:08:47.561048 17651 test_util.cc:274] Using random seed: -1268380332
I20250112 15:08:47.570669 24397 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:60596:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "\004\001\000\377\377\377\037\004\001\000\376\377\377?\004\001\000\375\377\377_""\004\001\000\377\377\377\037\004\001\000\376\377\377?\004\001\000\375\377\377_"
indirect_data: """"
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 15:08:47.570838 24397 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-workload in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 15:08:47.575760 24566 tablet_service.cc:1467] Processing CreateTablet for tablet c73b4e40028649e5bb9ad937f307e7bf (DEFAULT_TABLE table=test-workload [id=a7cea0e15506428cabd1af7ad2081b4c]), partition=RANGE (key) PARTITION VALUES < 536870911
I20250112 15:08:47.575902 24566 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c73b4e40028649e5bb9ad937f307e7bf. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 15:08:47.576171 24641 tablet_service.cc:1467] Processing CreateTablet for tablet c73b4e40028649e5bb9ad937f307e7bf (DEFAULT_TABLE table=test-workload [id=a7cea0e15506428cabd1af7ad2081b4c]), partition=RANGE (key) PARTITION VALUES < 536870911
I20250112 15:08:47.576257 24641 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c73b4e40028649e5bb9ad937f307e7bf. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 15:08:47.576476 24490 tablet_service.cc:1467] Processing CreateTablet for tablet c73b4e40028649e5bb9ad937f307e7bf (DEFAULT_TABLE table=test-workload [id=a7cea0e15506428cabd1af7ad2081b4c]), partition=RANGE (key) PARTITION VALUES < 536870911
I20250112 15:08:47.576602 24490 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c73b4e40028649e5bb9ad937f307e7bf. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 15:08:47.576678 24487 tablet_service.cc:1467] Processing CreateTablet for tablet 94172f983ecd4a37956654f2f86dba00 (DEFAULT_TABLE table=test-workload [id=a7cea0e15506428cabd1af7ad2081b4c]), partition=RANGE (key) PARTITION 1610612733 <= VALUES
I20250112 15:08:47.576753 24487 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 94172f983ecd4a37956654f2f86dba00. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 15:08:47.577538 24697 tablet_bootstrap.cc:492] T c73b4e40028649e5bb9ad937f307e7bf P 56dd875b6ba54b40877146f15292bf2f: Bootstrap starting.
I20250112 15:08:47.577888 24697 tablet_bootstrap.cc:654] T c73b4e40028649e5bb9ad937f307e7bf P 56dd875b6ba54b40877146f15292bf2f: Neither blocks nor log segments found. Creating new log.
I20250112 15:08:47.577986 24699 tablet_bootstrap.cc:492] T 94172f983ecd4a37956654f2f86dba00 P 52bd510945874fa781ec841d40e251ab: Bootstrap starting.
I20250112 15:08:47.578015 24488 tablet_service.cc:1467] Processing CreateTablet for tablet 45d29678d6704a888f2c2514af2f36a4 (DEFAULT_TABLE table=test-workload [id=a7cea0e15506428cabd1af7ad2081b4c]), partition=RANGE (key) PARTITION 1073741822 <= VALUES < 1610612733
I20250112 15:08:47.578091 24488 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 45d29678d6704a888f2c2514af2f36a4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 15:08:47.578194 24698 tablet_bootstrap.cc:492] T c73b4e40028649e5bb9ad937f307e7bf P f5b45ccea9c7438f8fba1b79073969e5: Bootstrap starting.
I20250112 15:08:47.578488 24638 tablet_service.cc:1467] Processing CreateTablet for tablet 94172f983ecd4a37956654f2f86dba00 (DEFAULT_TABLE table=test-workload [id=a7cea0e15506428cabd1af7ad2081b4c]), partition=RANGE (key) PARTITION 1610612733 <= VALUES
I20250112 15:08:47.578557 24697 tablet_bootstrap.cc:492] T c73b4e40028649e5bb9ad937f307e7bf P 56dd875b6ba54b40877146f15292bf2f: No bootstrap required, opened a new log
I20250112 15:08:47.578577 24638 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 94172f983ecd4a37956654f2f86dba00. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 15:08:47.578605 24697 ts_tablet_manager.cc:1397] T c73b4e40028649e5bb9ad937f307e7bf P 56dd875b6ba54b40877146f15292bf2f: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20250112 15:08:47.578652 24699 tablet_bootstrap.cc:654] T 94172f983ecd4a37956654f2f86dba00 P 52bd510945874fa781ec841d40e251ab: Neither blocks nor log segments found. Creating new log.
I20250112 15:08:47.578769 24697 raft_consensus.cc:357] T c73b4e40028649e5bb9ad937f307e7bf P 56dd875b6ba54b40877146f15292bf2f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56dd875b6ba54b40877146f15292bf2f" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 32969 } } peers { permanent_uuid: "52bd510945874fa781ec841d40e251ab" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 34851 } } peers { permanent_uuid: "f5b45ccea9c7438f8fba1b79073969e5" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 40795 } }
I20250112 15:08:47.578840 24697 raft_consensus.cc:383] T c73b4e40028649e5bb9ad937f307e7bf P 56dd875b6ba54b40877146f15292bf2f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 15:08:47.578859 24697 raft_consensus.cc:738] T c73b4e40028649e5bb9ad937f307e7bf P 56dd875b6ba54b40877146f15292bf2f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 56dd875b6ba54b40877146f15292bf2f, State: Initialized, Role: FOLLOWER
I20250112 15:08:47.578930 24697 consensus_queue.cc:260] T c73b4e40028649e5bb9ad937f307e7bf P 56dd875b6ba54b40877146f15292bf2f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56dd875b6ba54b40877146f15292bf2f" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 32969 } } peers { permanent_uuid: "52bd510945874fa781ec841d40e251ab" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 34851 } } peers { permanent_uuid: "f5b45ccea9c7438f8fba1b79073969e5" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 40795 } }
I20250112 15:08:47.579092 24697 ts_tablet_manager.cc:1428] T c73b4e40028649e5bb9ad937f307e7bf P 56dd875b6ba54b40877146f15292bf2f: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20250112 15:08:47.579406 24698 tablet_bootstrap.cc:654] T c73b4e40028649e5bb9ad937f307e7bf P f5b45ccea9c7438f8fba1b79073969e5: Neither blocks nor log segments found. Creating new log.
I20250112 15:08:47.579710 24639 tablet_service.cc:1467] Processing CreateTablet for tablet 45d29678d6704a888f2c2514af2f36a4 (DEFAULT_TABLE table=test-workload [id=a7cea0e15506428cabd1af7ad2081b4c]), partition=RANGE (key) PARTITION 1073741822 <= VALUES < 1610612733
I20250112 15:08:47.579775 24639 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 45d29678d6704a888f2c2514af2f36a4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 15:08:47.579855 24698 tablet_bootstrap.cc:492] T c73b4e40028649e5bb9ad937f307e7bf P f5b45ccea9c7438f8fba1b79073969e5: No bootstrap required, opened a new log
I20250112 15:08:47.579887 24698 ts_tablet_manager.cc:1397] T c73b4e40028649e5bb9ad937f307e7bf P f5b45ccea9c7438f8fba1b79073969e5: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20250112 15:08:47.579991 24698 raft_consensus.cc:357] T c73b4e40028649e5bb9ad937f307e7bf P f5b45ccea9c7438f8fba1b79073969e5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56dd875b6ba54b40877146f15292bf2f" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 32969 } } peers { permanent_uuid: "52bd510945874fa781ec841d40e251ab" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 34851 } } peers { permanent_uuid: "f5b45ccea9c7438f8fba1b79073969e5" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 40795 } }
I20250112 15:08:47.580032 24698 raft_consensus.cc:383] T c73b4e40028649e5bb9ad937f307e7bf P f5b45ccea9c7438f8fba1b79073969e5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 15:08:47.580045 24698 raft_consensus.cc:738] T c73b4e40028649e5bb9ad937f307e7bf P f5b45ccea9c7438f8fba1b79073969e5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f5b45ccea9c7438f8fba1b79073969e5, State: Initialized, Role: FOLLOWER
I20250112 15:08:47.580070 24698 consensus_queue.cc:260] T c73b4e40028649e5bb9ad937f307e7bf P f5b45ccea9c7438f8fba1b79073969e5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56dd875b6ba54b40877146f15292bf2f" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 32969 } } peers { permanent_uuid: "52bd510945874fa781ec841d40e251ab" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 34851 } } peers { permanent_uuid: "f5b45ccea9c7438f8fba1b79073969e5" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 40795 } }
I20250112 15:08:47.580191 24698 ts_tablet_manager.cc:1428] T c73b4e40028649e5bb9ad937f307e7bf P f5b45ccea9c7438f8fba1b79073969e5: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20250112 15:08:47.580242 24698 tablet_bootstrap.cc:492] T 94172f983ecd4a37956654f2f86dba00 P f5b45ccea9c7438f8fba1b79073969e5: Bootstrap starting.
I20250112 15:08:47.580634 24698 tablet_bootstrap.cc:654] T 94172f983ecd4a37956654f2f86dba00 P f5b45ccea9c7438f8fba1b79073969e5: Neither blocks nor log segments found. Creating new log.
I20250112 15:08:47.580691 24640 tablet_service.cc:1467] Processing CreateTablet for tablet cc5c837c3de946a5952f656946d883c8 (DEFAULT_TABLE table=test-workload [id=a7cea0e15506428cabd1af7ad2081b4c]), partition=RANGE (key) PARTITION 536870911 <= VALUES < 1073741822
I20250112 15:08:47.580775 24640 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cc5c837c3de946a5952f656946d883c8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 15:08:47.581135 24698 tablet_bootstrap.cc:492] T 94172f983ecd4a37956654f2f86dba00 P f5b45ccea9c7438f8fba1b79073969e5: No bootstrap required, opened a new log
I20250112 15:08:47.581164 24698 ts_tablet_manager.cc:1397] T 94172f983ecd4a37956654f2f86dba00 P f5b45ccea9c7438f8fba1b79073969e5: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20250112 15:08:47.581275 24698 raft_consensus.cc:357] T 94172f983ecd4a37956654f2f86dba00 P f5b45ccea9c7438f8fba1b79073969e5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52bd510945874fa781ec841d40e251ab" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 34851 } } peers { permanent_uuid: "56dd875b6ba54b40877146f15292bf2f" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 32969 } } peers { permanent_uuid: "f5b45ccea9c7438f8fba1b79073969e5" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 40795 } }
I20250112 15:08:47.581310 24698 raft_consensus.cc:383] T 94172f983ecd4a37956654f2f86dba00 P f5b45ccea9c7438f8fba1b79073969e5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 15:08:47.581321 24698 raft_consensus.cc:738] T 94172f983ecd4a37956654f2f86dba00 P f5b45ccea9c7438f8fba1b79073969e5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f5b45ccea9c7438f8fba1b79073969e5, State: Initialized, Role: FOLLOWER
I20250112 15:08:47.581341 24698 consensus_queue.cc:260] T 94172f983ecd4a37956654f2f86dba00 P f5b45ccea9c7438f8fba1b79073969e5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52bd510945874fa781ec841d40e251ab" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 34851 } } peers { permanent_uuid: "56dd875b6ba54b40877146f15292bf2f" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 32969 } } peers { permanent_uuid: "f5b45ccea9c7438f8fba1b79073969e5" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 40795 } }
I20250112 15:08:47.581405 24698 ts_tablet_manager.cc:1428] T 94172f983ecd4a37956654f2f86dba00 P f5b45ccea9c7438f8fba1b79073969e5: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20250112 15:08:47.581452 24698 tablet_bootstrap.cc:492] T 45d29678d6704a888f2c2514af2f36a4 P f5b45ccea9c7438f8fba1b79073969e5: Bootstrap starting.
I20250112 15:08:47.581777 24563 tablet_service.cc:1467] Processing CreateTablet for tablet 94172f983ecd4a37956654f2f86dba00 (DEFAULT_TABLE table=test-workload [id=a7cea0e15506428cabd1af7ad2081b4c]), partition=RANGE (key) PARTITION 1610612733 <= VALUES
I20250112 15:08:47.581835 24698 tablet_bootstrap.cc:654] T 45d29678d6704a888f2c2514af2f36a4 P f5b45ccea9c7438f8fba1b79073969e5: Neither blocks nor log segments found. Creating new log.
I20250112 15:08:47.581869 24563 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 94172f983ecd4a37956654f2f86dba00. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 15:08:47.582331 24698 tablet_bootstrap.cc:492] T 45d29678d6704a888f2c2514af2f36a4 P f5b45ccea9c7438f8fba1b79073969e5: No bootstrap required, opened a new log
I20250112 15:08:47.582362 24698 ts_tablet_manager.cc:1397] T 45d29678d6704a888f2c2514af2f36a4 P f5b45ccea9c7438f8fba1b79073969e5: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20250112 15:08:47.582463 24698 raft_consensus.cc:357] T 45d29678d6704a888f2c2514af2f36a4 P f5b45ccea9c7438f8fba1b79073969e5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52bd510945874fa781ec841d40e251ab" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 34851 } } peers { permanent_uuid: "56dd875b6ba54b40877146f15292bf2f" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 32969 } } peers { permanent_uuid: "f5b45ccea9c7438f8fba1b79073969e5" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 40795 } }
I20250112 15:08:47.582496 24698 raft_consensus.cc:383] T 45d29678d6704a888f2c2514af2f36a4 P f5b45ccea9c7438f8fba1b79073969e5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 15:08:47.582509 24698 raft_consensus.cc:738] T 45d29678d6704a888f2c2514af2f36a4 P f5b45ccea9c7438f8fba1b79073969e5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f5b45ccea9c7438f8fba1b79073969e5, State: Initialized, Role: FOLLOWER
I20250112 15:08:47.582548 24698 consensus_queue.cc:260] T 45d29678d6704a888f2c2514af2f36a4 P f5b45ccea9c7438f8fba1b79073969e5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52bd510945874fa781ec841d40e251ab" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 34851 } } peers { permanent_uuid: "56dd875b6ba54b40877146f15292bf2f" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 32969 } } peers { permanent_uuid: "f5b45ccea9c7438f8fba1b79073969e5" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 40795 } }
I20250112 15:08:47.582681 24697 tablet_bootstrap.cc:492] T 94172f983ecd4a37956654f2f86dba00 P 56dd875b6ba54b40877146f15292bf2f: Bootstrap starting.
I20250112 15:08:47.582707 24564 tablet_service.cc:1467] Processing CreateTablet for tablet 45d29678d6704a888f2c2514af2f36a4 (DEFAULT_TABLE table=test-workload [id=a7cea0e15506428cabd1af7ad2081b4c]), partition=RANGE (key) PARTITION 1073741822 <= VALUES < 1610612733
I20250112 15:08:47.582633 24698 ts_tablet_manager.cc:1428] T 45d29678d6704a888f2c2514af2f36a4 P f5b45ccea9c7438f8fba1b79073969e5: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20250112 15:08:47.582787 24564 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 45d29678d6704a888f2c2514af2f36a4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 15:08:47.582809 24698 tablet_bootstrap.cc:492] T cc5c837c3de946a5952f656946d883c8 P f5b45ccea9c7438f8fba1b79073969e5: Bootstrap starting.
I20250112 15:08:47.583078 24697 tablet_bootstrap.cc:654] T 94172f983ecd4a37956654f2f86dba00 P 56dd875b6ba54b40877146f15292bf2f: Neither blocks nor log segments found. Creating new log.
I20250112 15:08:47.583266 24698 tablet_bootstrap.cc:654] T cc5c837c3de946a5952f656946d883c8 P f5b45ccea9c7438f8fba1b79073969e5: Neither blocks nor log segments found. Creating new log.
I20250112 15:08:47.583669 24697 tablet_bootstrap.cc:492] T 94172f983ecd4a37956654f2f86dba00 P 56dd875b6ba54b40877146f15292bf2f: No bootstrap required, opened a new log
I20250112 15:08:47.583725 24697 ts_tablet_manager.cc:1397] T 94172f983ecd4a37956654f2f86dba00 P 56dd875b6ba54b40877146f15292bf2f: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20250112 15:08:47.583774 24698 tablet_bootstrap.cc:492] T cc5c837c3de946a5952f656946d883c8 P f5b45ccea9c7438f8fba1b79073969e5: No bootstrap required, opened a new log
I20250112 15:08:47.583808 24698 ts_tablet_manager.cc:1397] T cc5c837c3de946a5952f656946d883c8 P f5b45ccea9c7438f8fba1b79073969e5: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20250112 15:08:47.583870 24697 raft_consensus.cc:357] T 94172f983ecd4a37956654f2f86dba00 P 56dd875b6ba54b40877146f15292bf2f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52bd510945874fa781ec841d40e251ab" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 34851 } } peers { permanent_uuid: "56dd875b6ba54b40877146f15292bf2f" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 32969 } } peers { permanent_uuid: "f5b45ccea9c7438f8fba1b79073969e5" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 40795 } }
I20250112 15:08:47.583926 24698 raft_consensus.cc:357] T cc5c837c3de946a5952f656946d883c8 P f5b45ccea9c7438f8fba1b79073969e5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52bd510945874fa781ec841d40e251ab" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 34851 } } peers { permanent_uuid: "56dd875b6ba54b40877146f15292bf2f" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 32969 } } peers { permanent_uuid: "f5b45ccea9c7438f8fba1b79073969e5" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 40795 } }
I20250112 15:08:47.583928 24697 raft_consensus.cc:383] T 94172f983ecd4a37956654f2f86dba00 P 56dd875b6ba54b40877146f15292bf2f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 15:08:47.583977 24698 raft_consensus.cc:383] T cc5c837c3de946a5952f656946d883c8 P f5b45ccea9c7438f8fba1b79073969e5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 15:08:47.583985 24697 raft_consensus.cc:738] T 94172f983ecd4a37956654f2f86dba00 P 56dd875b6ba54b40877146f15292bf2f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 56dd875b6ba54b40877146f15292bf2f, State: Initialized, Role: FOLLOWER
I20250112 15:08:47.583992 24698 raft_consensus.cc:738] T cc5c837c3de946a5952f656946d883c8 P f5b45ccea9c7438f8fba1b79073969e5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f5b45ccea9c7438f8fba1b79073969e5, State: Initialized, Role: FOLLOWER
I20250112 15:08:47.584043 24697 consensus_queue.cc:260] T 94172f983ecd4a37956654f2f86dba00 P 56dd875b6ba54b40877146f15292bf2f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52bd510945874fa781ec841d40e251ab" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 34851 } } peers { permanent_uuid: "56dd875b6ba54b40877146f15292bf2f" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 32969 } } peers { permanent_uuid: "f5b45ccea9c7438f8fba1b79073969e5" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 40795 } }
I20250112 15:08:47.584101 24565 tablet_service.cc:1467] Processing CreateTablet for tablet cc5c837c3de946a5952f656946d883c8 (DEFAULT_TABLE table=test-workload [id=a7cea0e15506428cabd1af7ad2081b4c]), partition=RANGE (key) PARTITION 536870911 <= VALUES < 1073741822
I20250112 15:08:47.584137 24697 ts_tablet_manager.cc:1428] T 94172f983ecd4a37956654f2f86dba00 P 56dd875b6ba54b40877146f15292bf2f: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20250112 15:08:47.584199 24697 tablet_bootstrap.cc:492] T 45d29678d6704a888f2c2514af2f36a4 P 56dd875b6ba54b40877146f15292bf2f: Bootstrap starting.
I20250112 15:08:47.584185 24565 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cc5c837c3de946a5952f656946d883c8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 15:08:47.584046 24698 consensus_queue.cc:260] T cc5c837c3de946a5952f656946d883c8 P f5b45ccea9c7438f8fba1b79073969e5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52bd510945874fa781ec841d40e251ab" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 34851 } } peers { permanent_uuid: "56dd875b6ba54b40877146f15292bf2f" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 32969 } } peers { permanent_uuid: "f5b45ccea9c7438f8fba1b79073969e5" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 40795 } }
I20250112 15:08:47.584345 24489 tablet_service.cc:1467] Processing CreateTablet for tablet cc5c837c3de946a5952f656946d883c8 (DEFAULT_TABLE table=test-workload [id=a7cea0e15506428cabd1af7ad2081b4c]), partition=RANGE (key) PARTITION 536870911 <= VALUES < 1073741822
I20250112 15:08:47.584403 24698 ts_tablet_manager.cc:1428] T cc5c837c3de946a5952f656946d883c8 P f5b45ccea9c7438f8fba1b79073969e5: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20250112 15:08:47.584434 24489 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cc5c837c3de946a5952f656946d883c8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 15:08:47.584856 24697 tablet_bootstrap.cc:654] T 45d29678d6704a888f2c2514af2f36a4 P 56dd875b6ba54b40877146f15292bf2f: Neither blocks nor log segments found. Creating new log.
I20250112 15:08:47.585436 24697 tablet_bootstrap.cc:492] T 45d29678d6704a888f2c2514af2f36a4 P 56dd875b6ba54b40877146f15292bf2f: No bootstrap required, opened a new log
I20250112 15:08:47.585474 24697 ts_tablet_manager.cc:1397] T 45d29678d6704a888f2c2514af2f36a4 P 56dd875b6ba54b40877146f15292bf2f: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20250112 15:08:47.585599 24697 raft_consensus.cc:357] T 45d29678d6704a888f2c2514af2f36a4 P 56dd875b6ba54b40877146f15292bf2f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52bd510945874fa781ec841d40e251ab" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 34851 } } peers { permanent_uuid: "56dd875b6ba54b40877146f15292bf2f" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 32969 } } peers { permanent_uuid: "f5b45ccea9c7438f8fba1b79073969e5" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 40795 } }
I20250112 15:08:47.585649 24697 raft_consensus.cc:383] T 45d29678d6704a888f2c2514af2f36a4 P 56dd875b6ba54b40877146f15292bf2f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 15:08:47.585667 24697 raft_consensus.cc:738] T 45d29678d6704a888f2c2514af2f36a4 P 56dd875b6ba54b40877146f15292bf2f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 56dd875b6ba54b40877146f15292bf2f, State: Initialized, Role: FOLLOWER
I20250112 15:08:47.585719 24697 consensus_queue.cc:260] T 45d29678d6704a888f2c2514af2f36a4 P 56dd875b6ba54b40877146f15292bf2f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52bd510945874fa781ec841d40e251ab" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 34851 } } peers { permanent_uuid: "56dd875b6ba54b40877146f15292bf2f" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 32969 } } peers { permanent_uuid: "f5b45ccea9c7438f8fba1b79073969e5" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 40795 } }
I20250112 15:08:47.585800 24697 ts_tablet_manager.cc:1428] T 45d29678d6704a888f2c2514af2f36a4 P 56dd875b6ba54b40877146f15292bf2f: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20250112 15:08:47.585861 24697 tablet_bootstrap.cc:492] T cc5c837c3de946a5952f656946d883c8 P 56dd875b6ba54b40877146f15292bf2f: Bootstrap starting.
I20250112 15:08:47.586284 24697 tablet_bootstrap.cc:654] T cc5c837c3de946a5952f656946d883c8 P 56dd875b6ba54b40877146f15292bf2f: Neither blocks nor log segments found. Creating new log.
I20250112 15:08:47.586529 24699 tablet_bootstrap.cc:492] T 94172f983ecd4a37956654f2f86dba00 P 52bd510945874fa781ec841d40e251ab: No bootstrap required, opened a new log
I20250112 15:08:47.586584 24699 ts_tablet_manager.cc:1397] T 94172f983ecd4a37956654f2f86dba00 P 52bd510945874fa781ec841d40e251ab: Time spent bootstrapping tablet: real 0.009s user 0.001s sys 0.000s
I20250112 15:08:47.586709 24699 raft_consensus.cc:357] T 94172f983ecd4a37956654f2f86dba00 P 52bd510945874fa781ec841d40e251ab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52bd510945874fa781ec841d40e251ab" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 34851 } } peers { permanent_uuid: "56dd875b6ba54b40877146f15292bf2f" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 32969 } } peers { permanent_uuid: "f5b45ccea9c7438f8fba1b79073969e5" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 40795 } }
I20250112 15:08:47.586763 24699 raft_consensus.cc:383] T 94172f983ecd4a37956654f2f86dba00 P 52bd510945874fa781ec841d40e251ab [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 15:08:47.586786 24699 raft_consensus.cc:738] T 94172f983ecd4a37956654f2f86dba00 P 52bd510945874fa781ec841d40e251ab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 52bd510945874fa781ec841d40e251ab, State: Initialized, Role: FOLLOWER
I20250112 15:08:47.586860 24699 consensus_queue.cc:260] T 94172f983ecd4a37956654f2f86dba00 P 52bd510945874fa781ec841d40e251ab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52bd510945874fa781ec841d40e251ab" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 34851 } } peers { permanent_uuid: "56dd875b6ba54b40877146f15292bf2f" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 32969 } } peers { permanent_uuid: "f5b45ccea9c7438f8fba1b79073969e5" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 40795 } }
I20250112 15:08:47.586882 24697 tablet_bootstrap.cc:492] T cc5c837c3de946a5952f656946d883c8 P 56dd875b6ba54b40877146f15292bf2f: No bootstrap required, opened a new log
I20250112 15:08:47.586939 24697 ts_tablet_manager.cc:1397] T cc5c837c3de946a5952f656946d883c8 P 56dd875b6ba54b40877146f15292bf2f: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20250112 15:08:47.587039 24699 ts_tablet_manager.cc:1428] T 94172f983ecd4a37956654f2f86dba00 P 52bd510945874fa781ec841d40e251ab: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20250112 15:08:47.587087 24697 raft_consensus.cc:357] T cc5c837c3de946a5952f656946d883c8 P 56dd875b6ba54b40877146f15292bf2f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52bd510945874fa781ec841d40e251ab" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 34851 } } peers { permanent_uuid: "56dd875b6ba54b40877146f15292bf2f" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 32969 } } peers { permanent_uuid: "f5b45ccea9c7438f8fba1b79073969e5" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 40795 } }
I20250112 15:08:47.587131 24699 tablet_bootstrap.cc:492] T c73b4e40028649e5bb9ad937f307e7bf P 52bd510945874fa781ec841d40e251ab: Bootstrap starting.
I20250112 15:08:47.587147 24697 raft_consensus.cc:383] T cc5c837c3de946a5952f656946d883c8 P 56dd875b6ba54b40877146f15292bf2f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 15:08:47.587167 24697 raft_consensus.cc:738] T cc5c837c3de946a5952f656946d883c8 P 56dd875b6ba54b40877146f15292bf2f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 56dd875b6ba54b40877146f15292bf2f, State: Initialized, Role: FOLLOWER
I20250112 15:08:47.587213 24697 consensus_queue.cc:260] T cc5c837c3de946a5952f656946d883c8 P 56dd875b6ba54b40877146f15292bf2f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52bd510945874fa781ec841d40e251ab" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 34851 } } peers { permanent_uuid: "56dd875b6ba54b40877146f15292bf2f" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 32969 } } peers { permanent_uuid: "f5b45ccea9c7438f8fba1b79073969e5" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 40795 } }
I20250112 15:08:47.587352 24697 ts_tablet_manager.cc:1428] T cc5c837c3de946a5952f656946d883c8 P 56dd875b6ba54b40877146f15292bf2f: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20250112 15:08:47.587522 24699 tablet_bootstrap.cc:654] T c73b4e40028649e5bb9ad937f307e7bf P 52bd510945874fa781ec841d40e251ab: Neither blocks nor log segments found. Creating new log.
I20250112 15:08:47.588068 24699 tablet_bootstrap.cc:492] T c73b4e40028649e5bb9ad937f307e7bf P 52bd510945874fa781ec841d40e251ab: No bootstrap required, opened a new log
I20250112 15:08:47.588117 24699 ts_tablet_manager.cc:1397] T c73b4e40028649e5bb9ad937f307e7bf P 52bd510945874fa781ec841d40e251ab: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20250112 15:08:47.588266 24699 raft_consensus.cc:357] T c73b4e40028649e5bb9ad937f307e7bf P 52bd510945874fa781ec841d40e251ab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56dd875b6ba54b40877146f15292bf2f" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 32969 } } peers { permanent_uuid: "52bd510945874fa781ec841d40e251ab" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 34851 } } peers { permanent_uuid: "f5b45ccea9c7438f8fba1b79073969e5" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 40795 } }
I20250112 15:08:47.588310 24699 raft_consensus.cc:383] T c73b4e40028649e5bb9ad937f307e7bf P 52bd510945874fa781ec841d40e251ab [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 15:08:47.588335 24699 raft_consensus.cc:738] T c73b4e40028649e5bb9ad937f307e7bf P 52bd510945874fa781ec841d40e251ab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 52bd510945874fa781ec841d40e251ab, State: Initialized, Role: FOLLOWER
I20250112 15:08:47.588385 24699 consensus_queue.cc:260] T c73b4e40028649e5bb9ad937f307e7bf P 52bd510945874fa781ec841d40e251ab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56dd875b6ba54b40877146f15292bf2f" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 32969 } } peers { permanent_uuid: "52bd510945874fa781ec841d40e251ab" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 34851 } } peers { permanent_uuid: "f5b45ccea9c7438f8fba1b79073969e5" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 40795 } }
I20250112 15:08:47.588467 24699 ts_tablet_manager.cc:1428] T c73b4e40028649e5bb9ad937f307e7bf P 52bd510945874fa781ec841d40e251ab: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20250112 15:08:47.588526 24699 tablet_bootstrap.cc:492] T 45d29678d6704a888f2c2514af2f36a4 P 52bd510945874fa781ec841d40e251ab: Bootstrap starting.
I20250112 15:08:47.588964 24699 tablet_bootstrap.cc:654] T 45d29678d6704a888f2c2514af2f36a4 P 52bd510945874fa781ec841d40e251ab: Neither blocks nor log segments found. Creating new log.
I20250112 15:08:47.589419 24699 tablet_bootstrap.cc:492] T 45d29678d6704a888f2c2514af2f36a4 P 52bd510945874fa781ec841d40e251ab: No bootstrap required, opened a new log
I20250112 15:08:47.589458 24699 ts_tablet_manager.cc:1397] T 45d29678d6704a888f2c2514af2f36a4 P 52bd510945874fa781ec841d40e251ab: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20250112 15:08:47.589570 24699 raft_consensus.cc:357] T 45d29678d6704a888f2c2514af2f36a4 P 52bd510945874fa781ec841d40e251ab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52bd510945874fa781ec841d40e251ab" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 34851 } } peers { permanent_uuid: "56dd875b6ba54b40877146f15292bf2f" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 32969 } } peers { permanent_uuid: "f5b45ccea9c7438f8fba1b79073969e5" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 40795 } }
I20250112 15:08:47.589612 24699 raft_consensus.cc:383] T 45d29678d6704a888f2c2514af2f36a4 P 52bd510945874fa781ec841d40e251ab [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 15:08:47.589635 24699 raft_consensus.cc:738] T 45d29678d6704a888f2c2514af2f36a4 P 52bd510945874fa781ec841d40e251ab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 52bd510945874fa781ec841d40e251ab, State: Initialized, Role: FOLLOWER
I20250112 15:08:47.589690 24699 consensus_queue.cc:260] T 45d29678d6704a888f2c2514af2f36a4 P 52bd510945874fa781ec841d40e251ab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52bd510945874fa781ec841d40e251ab" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 34851 } } peers { permanent_uuid: "56dd875b6ba54b40877146f15292bf2f" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 32969 } } peers { permanent_uuid: "f5b45ccea9c7438f8fba1b79073969e5" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 40795 } }
I20250112 15:08:47.589767 24699 ts_tablet_manager.cc:1428] T 45d29678d6704a888f2c2514af2f36a4 P 52bd510945874fa781ec841d40e251ab: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20250112 15:08:47.589823 24699 tablet_bootstrap.cc:492] T cc5c837c3de946a5952f656946d883c8 P 52bd510945874fa781ec841d40e251ab: Bootstrap starting.
I20250112 15:08:47.589918 24711 raft_consensus.cc:491] T 94172f983ecd4a37956654f2f86dba00 P 52bd510945874fa781ec841d40e251ab [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 15:08:47.589978 24711 raft_consensus.cc:513] T 94172f983ecd4a37956654f2f86dba00 P 52bd510945874fa781ec841d40e251ab [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52bd510945874fa781ec841d40e251ab" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 34851 } } peers { permanent_uuid: "56dd875b6ba54b40877146f15292bf2f" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 32969 } } peers { permanent_uuid: "f5b45ccea9c7438f8fba1b79073969e5" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 40795 } }
I20250112 15:08:47.590143 24711 leader_election.cc:290] T 94172f983ecd4a37956654f2f86dba00 P 52bd510945874fa781ec841d40e251ab [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 56dd875b6ba54b40877146f15292bf2f (127.17.60.194:32969), f5b45ccea9c7438f8fba1b79073969e5 (127.17.60.195:40795)
I20250112 15:08:47.590219 24699 tablet_bootstrap.cc:654] T cc5c837c3de946a5952f656946d883c8 P 52bd510945874fa781ec841d40e251ab: Neither blocks nor log segments found. Creating new log.
I20250112 15:08:47.590878 24699 tablet_bootstrap.cc:492] T cc5c837c3de946a5952f656946d883c8 P 52bd510945874fa781ec841d40e251ab: No bootstrap required, opened a new log
I20250112 15:08:47.590926 24699 ts_tablet_manager.cc:1397] T cc5c837c3de946a5952f656946d883c8 P 52bd510945874fa781ec841d40e251ab: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20250112 15:08:47.591084 24699 raft_consensus.cc:357] T cc5c837c3de946a5952f656946d883c8 P 52bd510945874fa781ec841d40e251ab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52bd510945874fa781ec841d40e251ab" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 34851 } } peers { permanent_uuid: "56dd875b6ba54b40877146f15292bf2f" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 32969 } } peers { permanent_uuid: "f5b45ccea9c7438f8fba1b79073969e5" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 40795 } }
I20250112 15:08:47.591159 24699 raft_consensus.cc:383] T cc5c837c3de946a5952f656946d883c8 P 52bd510945874fa781ec841d40e251ab [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 15:08:47.591192 24699 raft_consensus.cc:738] T cc5c837c3de946a5952f656946d883c8 P 52bd510945874fa781ec841d40e251ab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 52bd510945874fa781ec841d40e251ab, State: Initialized, Role: FOLLOWER
I20250112 15:08:47.591238 24699 consensus_queue.cc:260] T cc5c837c3de946a5952f656946d883c8 P 52bd510945874fa781ec841d40e251ab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52bd510945874fa781ec841d40e251ab" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 34851 } } peers { permanent_uuid: "56dd875b6ba54b40877146f15292bf2f" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 32969 } } peers { permanent_uuid: "f5b45ccea9c7438f8fba1b79073969e5" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 40795 } }
I20250112 15:08:47.591356 24699 ts_tablet_manager.cc:1428] T cc5c837c3de946a5952f656946d883c8 P 52bd510945874fa781ec841d40e251ab: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20250112 15:08:47.593190 24576 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "94172f983ecd4a37956654f2f86dba00" candidate_uuid: "52bd510945874fa781ec841d40e251ab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "56dd875b6ba54b40877146f15292bf2f" is_pre_election: true
I20250112 15:08:47.593256 24576 raft_consensus.cc:2463] T 94172f983ecd4a37956654f2f86dba00 P 56dd875b6ba54b40877146f15292bf2f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52bd510945874fa781ec841d40e251ab in term 0.
I20250112 15:08:47.593393 24463 leader_election.cc:304] T 94172f983ecd4a37956654f2f86dba00 P 52bd510945874fa781ec841d40e251ab [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52bd510945874fa781ec841d40e251ab, 56dd875b6ba54b40877146f15292bf2f; no voters:
I20250112 15:08:47.593468 24711 raft_consensus.cc:2798] T 94172f983ecd4a37956654f2f86dba00 P 52bd510945874fa781ec841d40e251ab [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250112 15:08:47.593498 24651 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "94172f983ecd4a37956654f2f86dba00" candidate_uuid: "52bd510945874fa781ec841d40e251ab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f5b45ccea9c7438f8fba1b79073969e5" is_pre_election: true
I20250112 15:08:47.593549 24711 raft_consensus.cc:491] T 94172f983ecd4a37956654f2f86dba00 P 52bd510945874fa781ec841d40e251ab [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250112 15:08:47.593569 24651 raft_consensus.cc:2463] T 94172f983ecd4a37956654f2f86dba00 P f5b45ccea9c7438f8fba1b79073969e5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52bd510945874fa781ec841d40e251ab in term 0.
I20250112 15:08:47.593576 24711 raft_consensus.cc:3054] T 94172f983ecd4a37956654f2f86dba00 P 52bd510945874fa781ec841d40e251ab [term 0 FOLLOWER]: Advancing to term 1
I20250112 15:08:47.594200 24711 raft_consensus.cc:513] T 94172f983ecd4a37956654f2f86dba00 P 52bd510945874fa781ec841d40e251ab [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52bd510945874fa781ec841d40e251ab" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 34851 } } peers { permanent_uuid: "56dd875b6ba54b40877146f15292bf2f" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 32969 } } peers { permanent_uuid: "f5b45ccea9c7438f8fba1b79073969e5" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 40795 } }
I20250112 15:08:47.594295 24711 leader_election.cc:290] T 94172f983ecd4a37956654f2f86dba00 P 52bd510945874fa781ec841d40e251ab [CANDIDATE]: Term 1 election: Requested vote from peers 56dd875b6ba54b40877146f15292bf2f (127.17.60.194:32969), f5b45ccea9c7438f8fba1b79073969e5 (127.17.60.195:40795)
I20250112 15:08:47.594477 24651 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "94172f983ecd4a37956654f2f86dba00" candidate_uuid: "52bd510945874fa781ec841d40e251ab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f5b45ccea9c7438f8fba1b79073969e5"
I20250112 15:08:47.594537 24651 raft_consensus.cc:3054] T 94172f983ecd4a37956654f2f86dba00 P f5b45ccea9c7438f8fba1b79073969e5 [term 0 FOLLOWER]: Advancing to term 1
I20250112 15:08:47.594475 24576 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "94172f983ecd4a37956654f2f86dba00" candidate_uuid: "52bd510945874fa781ec841d40e251ab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "56dd875b6ba54b40877146f15292bf2f"
I20250112 15:08:47.594566 24576 raft_consensus.cc:3054] T 94172f983ecd4a37956654f2f86dba00 P 56dd875b6ba54b40877146f15292bf2f [term 0 FOLLOWER]: Advancing to term 1
I20250112 15:08:47.595074 24651 raft_consensus.cc:2463] T 94172f983ecd4a37956654f2f86dba00 P f5b45ccea9c7438f8fba1b79073969e5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 52bd510945874fa781ec841d40e251ab in term 1.
I20250112 15:08:47.595218 24463 leader_election.cc:304] T 94172f983ecd4a37956654f2f86dba00 P 52bd510945874fa781ec841d40e251ab [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52bd510945874fa781ec841d40e251ab, f5b45ccea9c7438f8fba1b79073969e5; no voters:
I20250112 15:08:47.595238 24576 raft_consensus.cc:2463] T 94172f983ecd4a37956654f2f86dba00 P 56dd875b6ba54b40877146f15292bf2f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 52bd510945874fa781ec841d40e251ab in term 1.
I20250112 15:08:47.595309 24711 raft_consensus.cc:2798] T 94172f983ecd4a37956654f2f86dba00 P 52bd510945874fa781ec841d40e251ab [term 1 FOLLOWER]: Leader election won for term 1
I20250112 15:08:47.595440 24711 raft_consensus.cc:695] T 94172f983ecd4a37956654f2f86dba00 P 52bd510945874fa781ec841d40e251ab [term 1 LEADER]: Becoming Leader. State: Replica: 52bd510945874fa781ec841d40e251ab, State: Running, Role: LEADER
I20250112 15:08:47.595496 24711 consensus_queue.cc:237] T 94172f983ecd4a37956654f2f86dba00 P 52bd510945874fa781ec841d40e251ab [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52bd510945874fa781ec841d40e251ab" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 34851 } } peers { permanent_uuid: "56dd875b6ba54b40877146f15292bf2f" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 32969 } } peers { permanent_uuid: "f5b45ccea9c7438f8fba1b79073969e5" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 40795 } }
I20250112 15:08:47.595952 24397 catalog_manager.cc:5526] T 94172f983ecd4a37956654f2f86dba00 P 52bd510945874fa781ec841d40e251ab reported cstate change: term changed from 0 to 1, leader changed from <none> to 52bd510945874fa781ec841d40e251ab (127.17.60.193). New cstate: current_term: 1 leader_uuid: "52bd510945874fa781ec841d40e251ab" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52bd510945874fa781ec841d40e251ab" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 34851 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "56dd875b6ba54b40877146f15292bf2f" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 32969 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f5b45ccea9c7438f8fba1b79073969e5" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 40795 } health_report { overall_health: UNKNOWN } } }
I20250112 15:08:47.648370 24704 raft_consensus.cc:491] T c73b4e40028649e5bb9ad937f307e7bf P f5b45ccea9c7438f8fba1b79073969e5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 15:08:47.648475 24704 raft_consensus.cc:513] T c73b4e40028649e5bb9ad937f307e7bf P f5b45ccea9c7438f8fba1b79073969e5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56dd875b6ba54b40877146f15292bf2f" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 32969 } } peers { permanent_uuid: "52bd510945874fa781ec841d40e251ab" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 34851 } } peers { permanent_uuid: "f5b45ccea9c7438f8fba1b79073969e5" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 40795 } }
I20250112 15:08:47.648695 24704 leader_election.cc:290] T c73b4e40028649e5bb9ad937f307e7bf P f5b45ccea9c7438f8fba1b79073969e5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 56dd875b6ba54b40877146f15292bf2f (127.17.60.194:32969), 52bd510945874fa781ec841d40e251ab (127.17.60.193:34851)
I20250112 15:08:47.652274 24576 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c73b4e40028649e5bb9ad937f307e7bf" candidate_uuid: "f5b45ccea9c7438f8fba1b79073969e5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "56dd875b6ba54b40877146f15292bf2f" is_pre_election: true
I20250112 15:08:47.652302 24500 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c73b4e40028649e5bb9ad937f307e7bf" candidate_uuid: "f5b45ccea9c7438f8fba1b79073969e5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "52bd510945874fa781ec841d40e251ab" is_pre_election: true
I20250112 15:08:47.652369 24576 raft_consensus.cc:2463] T c73b4e40028649e5bb9ad937f307e7bf P 56dd875b6ba54b40877146f15292bf2f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f5b45ccea9c7438f8fba1b79073969e5 in term 0.
I20250112 15:08:47.652379 24500 raft_consensus.cc:2463] T c73b4e40028649e5bb9ad937f307e7bf P 52bd510945874fa781ec841d40e251ab [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f5b45ccea9c7438f8fba1b79073969e5 in term 0.
I20250112 15:08:47.652510 24614 leader_election.cc:304] T c73b4e40028649e5bb9ad937f307e7bf P f5b45ccea9c7438f8fba1b79073969e5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 56dd875b6ba54b40877146f15292bf2f, f5b45ccea9c7438f8fba1b79073969e5; no voters:
I20250112 15:08:47.652607 24704 raft_consensus.cc:2798] T c73b4e40028649e5bb9ad937f307e7bf P f5b45ccea9c7438f8fba1b79073969e5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250112 15:08:47.652654 24704 raft_consensus.cc:491] T c73b4e40028649e5bb9ad937f307e7bf P f5b45ccea9c7438f8fba1b79073969e5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250112 15:08:47.652679 24704 raft_consensus.cc:3054] T c73b4e40028649e5bb9ad937f307e7bf P f5b45ccea9c7438f8fba1b79073969e5 [term 0 FOLLOWER]: Advancing to term 1
I20250112 15:08:47.653345 24704 raft_consensus.cc:513] T c73b4e40028649e5bb9ad937f307e7bf P f5b45ccea9c7438f8fba1b79073969e5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56dd875b6ba54b40877146f15292bf2f" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 32969 } } peers { permanent_uuid: "52bd510945874fa781ec841d40e251ab" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 34851 } } peers { permanent_uuid: "f5b45ccea9c7438f8fba1b79073969e5" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 40795 } }
I20250112 15:08:47.653466 24704 leader_election.cc:290] T c73b4e40028649e5bb9ad937f307e7bf P f5b45ccea9c7438f8fba1b79073969e5 [CANDIDATE]: Term 1 election: Requested vote from peers 56dd875b6ba54b40877146f15292bf2f (127.17.60.194:32969), 52bd510945874fa781ec841d40e251ab (127.17.60.193:34851)
I20250112 15:08:47.653626 24576 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c73b4e40028649e5bb9ad937f307e7bf" candidate_uuid: "f5b45ccea9c7438f8fba1b79073969e5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "56dd875b6ba54b40877146f15292bf2f"
I20250112 15:08:47.653656 24500 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c73b4e40028649e5bb9ad937f307e7bf" candidate_uuid: "f5b45ccea9c7438f8fba1b79073969e5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "52bd510945874fa781ec841d40e251ab"
I20250112 15:08:47.653687 24576 raft_consensus.cc:3054] T c73b4e40028649e5bb9ad937f307e7bf P 56dd875b6ba54b40877146f15292bf2f [term 0 FOLLOWER]: Advancing to term 1
I20250112 15:08:47.653704 24500 raft_consensus.cc:3054] T c73b4e40028649e5bb9ad937f307e7bf P 52bd510945874fa781ec841d40e251ab [term 0 FOLLOWER]: Advancing to term 1
I20250112 15:08:47.654196 24576 raft_consensus.cc:2463] T c73b4e40028649e5bb9ad937f307e7bf P 56dd875b6ba54b40877146f15292bf2f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f5b45ccea9c7438f8fba1b79073969e5 in term 1.
I20250112 15:08:47.654354 24500 raft_consensus.cc:2463] T c73b4e40028649e5bb9ad937f307e7bf P 52bd510945874fa781ec841d40e251ab [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f5b45ccea9c7438f8fba1b79073969e5 in term 1.
I20250112 15:08:47.654455 24614 leader_election.cc:304] T c73b4e40028649e5bb9ad937f307e7bf P f5b45ccea9c7438f8fba1b79073969e5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 56dd875b6ba54b40877146f15292bf2f, f5b45ccea9c7438f8fba1b79073969e5; no voters:
I20250112 15:08:47.654531 24704 raft_consensus.cc:2798] T c73b4e40028649e5bb9ad937f307e7bf P f5b45ccea9c7438f8fba1b79073969e5 [term 1 FOLLOWER]: Leader election won for term 1
I20250112 15:08:47.654675 24704 raft_consensus.cc:695] T c73b4e40028649e5bb9ad937f307e7bf P f5b45ccea9c7438f8fba1b79073969e5 [term 1 LEADER]: Becoming Leader. State: Replica: f5b45ccea9c7438f8fba1b79073969e5, State: Running, Role: LEADER
I20250112 15:08:47.654731 24704 consensus_queue.cc:237] T c73b4e40028649e5bb9ad937f307e7bf P f5b45ccea9c7438f8fba1b79073969e5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56dd875b6ba54b40877146f15292bf2f" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 32969 } } peers { permanent_uuid: "52bd510945874fa781ec841d40e251ab" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 34851 } } peers { permanent_uuid: "f5b45ccea9c7438f8fba1b79073969e5" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 40795 } }
I20250112 15:08:47.655303 24397 catalog_manager.cc:5526] T c73b4e40028649e5bb9ad937f307e7bf P f5b45ccea9c7438f8fba1b79073969e5 reported cstate change: term changed from 0 to 1, leader changed from <none> to f5b45ccea9c7438f8fba1b79073969e5 (127.17.60.195). New cstate: current_term: 1 leader_uuid: "f5b45ccea9c7438f8fba1b79073969e5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56dd875b6ba54b40877146f15292bf2f" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 32969 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "52bd510945874fa781ec841d40e251ab" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 34851 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f5b45ccea9c7438f8fba1b79073969e5" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 40795 } health_report { overall_health: HEALTHY } } }
I20250112 15:08:47.690214 24704 raft_consensus.cc:491] T 45d29678d6704a888f2c2514af2f36a4 P f5b45ccea9c7438f8fba1b79073969e5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 15:08:47.690335 24704 raft_consensus.cc:513] T 45d29678d6704a888f2c2514af2f36a4 P f5b45ccea9c7438f8fba1b79073969e5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52bd510945874fa781ec841d40e251ab" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 34851 } } peers { permanent_uuid: "56dd875b6ba54b40877146f15292bf2f" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 32969 } } peers { permanent_uuid: "f5b45ccea9c7438f8fba1b79073969e5" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 40795 } }
I20250112 15:08:47.690513 24704 leader_election.cc:290] T 45d29678d6704a888f2c2514af2f36a4 P f5b45ccea9c7438f8fba1b79073969e5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 52bd510945874fa781ec841d40e251ab (127.17.60.193:34851), 56dd875b6ba54b40877146f15292bf2f (127.17.60.194:32969)
I20250112 15:08:47.690740 24576 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "45d29678d6704a888f2c2514af2f36a4" candidate_uuid: "f5b45ccea9c7438f8fba1b79073969e5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "56dd875b6ba54b40877146f15292bf2f" is_pre_election: true
I20250112 15:08:47.690806 24576 raft_consensus.cc:2463] T 45d29678d6704a888f2c2514af2f36a4 P 56dd875b6ba54b40877146f15292bf2f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f5b45ccea9c7438f8fba1b79073969e5 in term 0.
I20250112 15:08:47.690816 24500 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "45d29678d6704a888f2c2514af2f36a4" candidate_uuid: "f5b45ccea9c7438f8fba1b79073969e5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "52bd510945874fa781ec841d40e251ab" is_pre_election: true
I20250112 15:08:47.690932 24500 raft_consensus.cc:2463] T 45d29678d6704a888f2c2514af2f36a4 P 52bd510945874fa781ec841d40e251ab [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f5b45ccea9c7438f8fba1b79073969e5 in term 0.
I20250112 15:08:47.690958 24614 leader_election.cc:304] T 45d29678d6704a888f2c2514af2f36a4 P f5b45ccea9c7438f8fba1b79073969e5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 56dd875b6ba54b40877146f15292bf2f, f5b45ccea9c7438f8fba1b79073969e5; no voters:
I20250112 15:08:47.691094 24704 raft_consensus.cc:2798] T 45d29678d6704a888f2c2514af2f36a4 P f5b45ccea9c7438f8fba1b79073969e5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250112 15:08:47.691133 24704 raft_consensus.cc:491] T 45d29678d6704a888f2c2514af2f36a4 P f5b45ccea9c7438f8fba1b79073969e5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250112 15:08:47.691155 24704 raft_consensus.cc:3054] T 45d29678d6704a888f2c2514af2f36a4 P f5b45ccea9c7438f8fba1b79073969e5 [term 0 FOLLOWER]: Advancing to term 1
I20250112 15:08:47.691833 24704 raft_consensus.cc:513] T 45d29678d6704a888f2c2514af2f36a4 P f5b45ccea9c7438f8fba1b79073969e5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52bd510945874fa781ec841d40e251ab" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 34851 } } peers { permanent_uuid: "56dd875b6ba54b40877146f15292bf2f" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 32969 } } peers { permanent_uuid: "f5b45ccea9c7438f8fba1b79073969e5" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 40795 } }
I20250112 15:08:47.691947 24704 leader_election.cc:290] T 45d29678d6704a888f2c2514af2f36a4 P f5b45ccea9c7438f8fba1b79073969e5 [CANDIDATE]: Term 1 election: Requested vote from peers 52bd510945874fa781ec841d40e251ab (127.17.60.193:34851), 56dd875b6ba54b40877146f15292bf2f (127.17.60.194:32969)
I20250112 15:08:47.692116 24500 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "45d29678d6704a888f2c2514af2f36a4" candidate_uuid: "f5b45ccea9c7438f8fba1b79073969e5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "52bd510945874fa781ec841d40e251ab"
I20250112 15:08:47.692171 24500 raft_consensus.cc:3054] T 45d29678d6704a888f2c2514af2f36a4 P 52bd510945874fa781ec841d40e251ab [term 0 FOLLOWER]: Advancing to term 1
I20250112 15:08:47.692167 24576 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "45d29678d6704a888f2c2514af2f36a4" candidate_uuid: "f5b45ccea9c7438f8fba1b79073969e5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "56dd875b6ba54b40877146f15292bf2f"
I20250112 15:08:47.692265 24576 raft_consensus.cc:3054] T 45d29678d6704a888f2c2514af2f36a4 P 56dd875b6ba54b40877146f15292bf2f [term 0 FOLLOWER]: Advancing to term 1
I20250112 15:08:47.692816 24500 raft_consensus.cc:2463] T 45d29678d6704a888f2c2514af2f36a4 P 52bd510945874fa781ec841d40e251ab [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f5b45ccea9c7438f8fba1b79073969e5 in term 1.
I20250112 15:08:47.692821 24576 raft_consensus.cc:2463] T 45d29678d6704a888f2c2514af2f36a4 P 56dd875b6ba54b40877146f15292bf2f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f5b45ccea9c7438f8fba1b79073969e5 in term 1.
I20250112 15:08:47.692996 24614 leader_election.cc:304] T 45d29678d6704a888f2c2514af2f36a4 P f5b45ccea9c7438f8fba1b79073969e5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52bd510945874fa781ec841d40e251ab, f5b45ccea9c7438f8fba1b79073969e5; no voters:
I20250112 15:08:47.693084 24704 raft_consensus.cc:2798] T 45d29678d6704a888f2c2514af2f36a4 P f5b45ccea9c7438f8fba1b79073969e5 [term 1 FOLLOWER]: Leader election won for term 1
I20250112 15:08:47.693140 24704 raft_consensus.cc:695] T 45d29678d6704a888f2c2514af2f36a4 P f5b45ccea9c7438f8fba1b79073969e5 [term 1 LEADER]: Becoming Leader. State: Replica: f5b45ccea9c7438f8fba1b79073969e5, State: Running, Role: LEADER
I20250112 15:08:47.693199 24704 consensus_queue.cc:237] T 45d29678d6704a888f2c2514af2f36a4 P f5b45ccea9c7438f8fba1b79073969e5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52bd510945874fa781ec841d40e251ab" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 34851 } } peers { permanent_uuid: "56dd875b6ba54b40877146f15292bf2f" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 32969 } } peers { permanent_uuid: "f5b45ccea9c7438f8fba1b79073969e5" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 40795 } }
I20250112 15:08:47.693750 24397 catalog_manager.cc:5526] T 45d29678d6704a888f2c2514af2f36a4 P f5b45ccea9c7438f8fba1b79073969e5 reported cstate change: term changed from 0 to 1, leader changed from <none> to f5b45ccea9c7438f8fba1b79073969e5 (127.17.60.195). New cstate: current_term: 1 leader_uuid: "f5b45ccea9c7438f8fba1b79073969e5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52bd510945874fa781ec841d40e251ab" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 34851 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "56dd875b6ba54b40877146f15292bf2f" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 32969 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f5b45ccea9c7438f8fba1b79073969e5" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 40795 } health_report { overall_health: HEALTHY } } }
I20250112 15:08:47.890094 24704 raft_consensus.cc:491] T cc5c837c3de946a5952f656946d883c8 P f5b45ccea9c7438f8fba1b79073969e5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 15:08:47.890206 24704 raft_consensus.cc:513] T cc5c837c3de946a5952f656946d883c8 P f5b45ccea9c7438f8fba1b79073969e5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52bd510945874fa781ec841d40e251ab" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 34851 } } peers { permanent_uuid: "56dd875b6ba54b40877146f15292bf2f" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 32969 } } peers { permanent_uuid: "f5b45ccea9c7438f8fba1b79073969e5" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 40795 } }
I20250112 15:08:47.890376 24704 leader_election.cc:290] T cc5c837c3de946a5952f656946d883c8 P f5b45ccea9c7438f8fba1b79073969e5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 52bd510945874fa781ec841d40e251ab (127.17.60.193:34851), 56dd875b6ba54b40877146f15292bf2f (127.17.60.194:32969)
I20250112 15:08:47.890631 24576 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc5c837c3de946a5952f656946d883c8" candidate_uuid: "f5b45ccea9c7438f8fba1b79073969e5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "56dd875b6ba54b40877146f15292bf2f" is_pre_election: true
I20250112 15:08:47.890631 24500 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc5c837c3de946a5952f656946d883c8" candidate_uuid: "f5b45ccea9c7438f8fba1b79073969e5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "52bd510945874fa781ec841d40e251ab" is_pre_election: true
I20250112 15:08:47.890758 24576 raft_consensus.cc:2463] T cc5c837c3de946a5952f656946d883c8 P 56dd875b6ba54b40877146f15292bf2f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f5b45ccea9c7438f8fba1b79073969e5 in term 0.
I20250112 15:08:47.890762 24500 raft_consensus.cc:2463] T cc5c837c3de946a5952f656946d883c8 P 52bd510945874fa781ec841d40e251ab [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f5b45ccea9c7438f8fba1b79073969e5 in term 0.
I20250112 15:08:47.890913 24614 leader_election.cc:304] T cc5c837c3de946a5952f656946d883c8 P f5b45ccea9c7438f8fba1b79073969e5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 56dd875b6ba54b40877146f15292bf2f, f5b45ccea9c7438f8fba1b79073969e5; no voters:
I20250112 15:08:47.891016 24704 raft_consensus.cc:2798] T cc5c837c3de946a5952f656946d883c8 P f5b45ccea9c7438f8fba1b79073969e5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250112 15:08:47.891065 24704 raft_consensus.cc:491] T cc5c837c3de946a5952f656946d883c8 P f5b45ccea9c7438f8fba1b79073969e5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250112 15:08:47.891078 24704 raft_consensus.cc:3054] T cc5c837c3de946a5952f656946d883c8 P f5b45ccea9c7438f8fba1b79073969e5 [term 0 FOLLOWER]: Advancing to term 1
I20250112 15:08:47.891640 24704 raft_consensus.cc:513] T cc5c837c3de946a5952f656946d883c8 P f5b45ccea9c7438f8fba1b79073969e5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52bd510945874fa781ec841d40e251ab" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 34851 } } peers { permanent_uuid: "56dd875b6ba54b40877146f15292bf2f" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 32969 } } peers { permanent_uuid: "f5b45ccea9c7438f8fba1b79073969e5" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 40795 } }
I20250112 15:08:47.891763 24704 leader_election.cc:290] T cc5c837c3de946a5952f656946d883c8 P f5b45ccea9c7438f8fba1b79073969e5 [CANDIDATE]: Term 1 election: Requested vote from peers 52bd510945874fa781ec841d40e251ab (127.17.60.193:34851), 56dd875b6ba54b40877146f15292bf2f (127.17.60.194:32969)
I20250112 15:08:47.891924 24500 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc5c837c3de946a5952f656946d883c8" candidate_uuid: "f5b45ccea9c7438f8fba1b79073969e5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "52bd510945874fa781ec841d40e251ab"
I20250112 15:08:47.891978 24500 raft_consensus.cc:3054] T cc5c837c3de946a5952f656946d883c8 P 52bd510945874fa781ec841d40e251ab [term 0 FOLLOWER]: Advancing to term 1
I20250112 15:08:47.892004 24576 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc5c837c3de946a5952f656946d883c8" candidate_uuid: "f5b45ccea9c7438f8fba1b79073969e5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "56dd875b6ba54b40877146f15292bf2f"
I20250112 15:08:47.892058 24576 raft_consensus.cc:3054] T cc5c837c3de946a5952f656946d883c8 P 56dd875b6ba54b40877146f15292bf2f [term 0 FOLLOWER]: Advancing to term 1
I20250112 15:08:47.892671 24576 raft_consensus.cc:2463] T cc5c837c3de946a5952f656946d883c8 P 56dd875b6ba54b40877146f15292bf2f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f5b45ccea9c7438f8fba1b79073969e5 in term 1.
I20250112 15:08:47.892671 24500 raft_consensus.cc:2463] T cc5c837c3de946a5952f656946d883c8 P 52bd510945874fa781ec841d40e251ab [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f5b45ccea9c7438f8fba1b79073969e5 in term 1.
I20250112 15:08:47.892910 24614 leader_election.cc:304] T cc5c837c3de946a5952f656946d883c8 P f5b45ccea9c7438f8fba1b79073969e5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 56dd875b6ba54b40877146f15292bf2f, f5b45ccea9c7438f8fba1b79073969e5; no voters:
I20250112 15:08:47.892988 24704 raft_consensus.cc:2798] T cc5c837c3de946a5952f656946d883c8 P f5b45ccea9c7438f8fba1b79073969e5 [term 1 FOLLOWER]: Leader election won for term 1
I20250112 15:08:47.893040 24704 raft_consensus.cc:695] T cc5c837c3de946a5952f656946d883c8 P f5b45ccea9c7438f8fba1b79073969e5 [term 1 LEADER]: Becoming Leader. State: Replica: f5b45ccea9c7438f8fba1b79073969e5, State: Running, Role: LEADER
I20250112 15:08:47.893098 24704 consensus_queue.cc:237] T cc5c837c3de946a5952f656946d883c8 P f5b45ccea9c7438f8fba1b79073969e5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52bd510945874fa781ec841d40e251ab" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 34851 } } peers { permanent_uuid: "56dd875b6ba54b40877146f15292bf2f" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 32969 } } peers { permanent_uuid: "f5b45ccea9c7438f8fba1b79073969e5" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 40795 } }
I20250112 15:08:47.893657 24397 catalog_manager.cc:5526] T cc5c837c3de946a5952f656946d883c8 P f5b45ccea9c7438f8fba1b79073969e5 reported cstate change: term changed from 0 to 1, leader changed from <none> to f5b45ccea9c7438f8fba1b79073969e5 (127.17.60.195). New cstate: current_term: 1 leader_uuid: "f5b45ccea9c7438f8fba1b79073969e5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52bd510945874fa781ec841d40e251ab" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 34851 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "56dd875b6ba54b40877146f15292bf2f" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 32969 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f5b45ccea9c7438f8fba1b79073969e5" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 40795 } health_report { overall_health: HEALTHY } } }
I20250112 15:08:47.933286 17651 test_util.cc:274] Using random seed: -1268008098
I20250112 15:08:47.938239 24397 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:60604:
name: "dugtrio"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "\004\001\000\377\377\377\037\004\001\000\376\377\377?\004\001\000\375\377\377_""\004\001\000\377\377\377\037\004\001\000\376\377\377?\004\001\000\375\377\377_"
indirect_data: """"
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 15:08:47.938405 24397 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table dugtrio in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 15:08:47.939867 24489 tablet_service.cc:1467] Processing CreateTablet for tablet bf900db2a1344c7b8fd0b9ba5e225201 (DEFAULT_TABLE table=dugtrio [id=271cc77f05ad43eb89d1c5c6291536e4]), partition=RANGE (key) PARTITION VALUES < 536870911
I20250112 15:08:47.939970 24488 tablet_service.cc:1467] Processing CreateTablet for tablet a3cc29d2a75b4778b21400e0e0b64a9e (DEFAULT_TABLE table=dugtrio [id=271cc77f05ad43eb89d1c5c6291536e4]), partition=RANGE (key) PARTITION 536870911 <= VALUES < 1073741822
I20250112 15:08:47.940007 24489 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bf900db2a1344c7b8fd0b9ba5e225201. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 15:08:47.940053 24490 tablet_service.cc:1467] Processing CreateTablet for tablet 934800150b1945dcab01b4f7c4edcdc8 (DEFAULT_TABLE table=dugtrio [id=271cc77f05ad43eb89d1c5c6291536e4]), partition=RANGE (key) PARTITION 1610612733 <= VALUES
I20250112 15:08:47.940119 24488 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a3cc29d2a75b4778b21400e0e0b64a9e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 15:08:47.940229 24490 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 934800150b1945dcab01b4f7c4edcdc8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 15:08:47.940255 24566 tablet_service.cc:1467] Processing CreateTablet for tablet 934800150b1945dcab01b4f7c4edcdc8 (DEFAULT_TABLE table=dugtrio [id=271cc77f05ad43eb89d1c5c6291536e4]), partition=RANGE (key) PARTITION 1610612733 <= VALUES
I20250112 15:08:47.940331 24566 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 934800150b1945dcab01b4f7c4edcdc8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 15:08:47.940701 24564 tablet_service.cc:1467] Processing CreateTablet for tablet a3cc29d2a75b4778b21400e0e0b64a9e (DEFAULT_TABLE table=dugtrio [id=271cc77f05ad43eb89d1c5c6291536e4]), partition=RANGE (key) PARTITION 536870911 <= VALUES < 1073741822
I20250112 15:08:47.940791 24564 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a3cc29d2a75b4778b21400e0e0b64a9e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 15:08:47.941511 24699 tablet_bootstrap.cc:492] T bf900db2a1344c7b8fd0b9ba5e225201 P 52bd510945874fa781ec841d40e251ab: Bootstrap starting.
I20250112 15:08:47.941514 24697 tablet_bootstrap.cc:492] T 934800150b1945dcab01b4f7c4edcdc8 P 56dd875b6ba54b40877146f15292bf2f: Bootstrap starting.
I20250112 15:08:47.942003 24697 tablet_bootstrap.cc:654] T 934800150b1945dcab01b4f7c4edcdc8 P 56dd875b6ba54b40877146f15292bf2f: Neither blocks nor log segments found. Creating new log.
I20250112 15:08:47.942072 24563 tablet_service.cc:1467] Processing CreateTablet for tablet 6914949901e147f2970585c79458b9d2 (DEFAULT_TABLE table=dugtrio [id=271cc77f05ad43eb89d1c5c6291536e4]), partition=RANGE (key) PARTITION 1073741822 <= VALUES < 1610612733
I20250112 15:08:47.942147 24563 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6914949901e147f2970585c79458b9d2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 15:08:47.942636 24697 tablet_bootstrap.cc:492] T 934800150b1945dcab01b4f7c4edcdc8 P 56dd875b6ba54b40877146f15292bf2f: No bootstrap required, opened a new log
I20250112 15:08:47.942668 24697 ts_tablet_manager.cc:1397] T 934800150b1945dcab01b4f7c4edcdc8 P 56dd875b6ba54b40877146f15292bf2f: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20250112 15:08:47.942785 24697 raft_consensus.cc:357] T 934800150b1945dcab01b4f7c4edcdc8 P 56dd875b6ba54b40877146f15292bf2f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52bd510945874fa781ec841d40e251ab" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 34851 } } peers { permanent_uuid: "56dd875b6ba54b40877146f15292bf2f" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 32969 } } peers { permanent_uuid: "f5b45ccea9c7438f8fba1b79073969e5" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 40795 } }
I20250112 15:08:47.942827 24697 raft_consensus.cc:383] T 934800150b1945dcab01b4f7c4edcdc8 P 56dd875b6ba54b40877146f15292bf2f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 15:08:47.942842 24697 raft_consensus.cc:738] T 934800150b1945dcab01b4f7c4edcdc8 P 56dd875b6ba54b40877146f15292bf2f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 56dd875b6ba54b40877146f15292bf2f, State: Initialized, Role: FOLLOWER
I20250112 15:08:47.942891 24697 consensus_queue.cc:260] T 934800150b1945dcab01b4f7c4edcdc8 P 56dd875b6ba54b40877146f15292bf2f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52bd510945874fa781ec841d40e251ab" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 34851 } } peers { permanent_uuid: "56dd875b6ba54b40877146f15292bf2f" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 32969 } } peers { permanent_uuid: "f5b45ccea9c7438f8fba1b79073969e5" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 40795 } }
I20250112 15:08:47.939880 24565 tablet_service.cc:1467] Processing CreateTablet for tablet bf900db2a1344c7b8fd0b9ba5e225201 (DEFAULT_TABLE table=dugtrio [id=271cc77f05ad43eb89d1c5c6291536e4]), partition=RANGE (key) PARTITION VALUES < 536870911
I20250112 15:08:47.943022 24697 ts_tablet_manager.cc:1428] T 934800150b1945dcab01b4f7c4edcdc8 P 56dd875b6ba54b40877146f15292bf2f: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20250112 15:08:47.943045 24565 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bf900db2a1344c7b8fd0b9ba5e225201. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 15:08:47.943089 24697 tablet_bootstrap.cc:492] T a3cc29d2a75b4778b21400e0e0b64a9e P 56dd875b6ba54b40877146f15292bf2f: Bootstrap starting.
I20250112 15:08:47.943488 24697 tablet_bootstrap.cc:654] T a3cc29d2a75b4778b21400e0e0b64a9e P 56dd875b6ba54b40877146f15292bf2f: Neither blocks nor log segments found. Creating new log.
I20250112 15:08:47.944015 24697 tablet_bootstrap.cc:492] T a3cc29d2a75b4778b21400e0e0b64a9e P 56dd875b6ba54b40877146f15292bf2f: No bootstrap required, opened a new log
I20250112 15:08:47.944044 24697 ts_tablet_manager.cc:1397] T a3cc29d2a75b4778b21400e0e0b64a9e P 56dd875b6ba54b40877146f15292bf2f: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20250112 15:08:47.941982 24487 tablet_service.cc:1467] Processing CreateTablet for tablet 6914949901e147f2970585c79458b9d2 (DEFAULT_TABLE table=dugtrio [id=271cc77f05ad43eb89d1c5c6291536e4]), partition=RANGE (key) PARTITION 1073741822 <= VALUES < 1610612733
I20250112 15:08:47.944173 24487 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6914949901e147f2970585c79458b9d2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 15:08:47.944185 24697 raft_consensus.cc:357] T a3cc29d2a75b4778b21400e0e0b64a9e P 56dd875b6ba54b40877146f15292bf2f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52bd510945874fa781ec841d40e251ab" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 34851 } } peers { permanent_uuid: "56dd875b6ba54b40877146f15292bf2f" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 32969 } } peers { permanent_uuid: "f5b45ccea9c7438f8fba1b79073969e5" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 40795 } }
I20250112 15:08:47.944254 24697 raft_consensus.cc:383] T a3cc29d2a75b4778b21400e0e0b64a9e P 56dd875b6ba54b40877146f15292bf2f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 15:08:47.944268 24697 raft_consensus.cc:738] T a3cc29d2a75b4778b21400e0e0b64a9e P 56dd875b6ba54b40877146f15292bf2f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 56dd875b6ba54b40877146f15292bf2f, State: Initialized, Role: FOLLOWER
I20250112 15:08:47.944308 24697 consensus_queue.cc:260] T a3cc29d2a75b4778b21400e0e0b64a9e P 56dd875b6ba54b40877146f15292bf2f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52bd510945874fa781ec841d40e251ab" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 34851 } } peers { permanent_uuid: "56dd875b6ba54b40877146f15292bf2f" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 32969 } } peers { permanent_uuid: "f5b45ccea9c7438f8fba1b79073969e5" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 40795 } }
I20250112 15:08:47.944396 24697 ts_tablet_manager.cc:1428] T a3cc29d2a75b4778b21400e0e0b64a9e P 56dd875b6ba54b40877146f15292bf2f: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20250112 15:08:47.944440 24697 tablet_bootstrap.cc:492] T 6914949901e147f2970585c79458b9d2 P 56dd875b6ba54b40877146f15292bf2f: Bootstrap starting.
I20250112 15:08:47.944885 24640 tablet_service.cc:1467] Processing CreateTablet for tablet bf900db2a1344c7b8fd0b9ba5e225201 (DEFAULT_TABLE table=dugtrio [id=271cc77f05ad43eb89d1c5c6291536e4]), partition=RANGE (key) PARTITION VALUES < 536870911
I20250112 15:08:47.944902 24697 tablet_bootstrap.cc:654] T 6914949901e147f2970585c79458b9d2 P 56dd875b6ba54b40877146f15292bf2f: Neither blocks nor log segments found. Creating new log.
I20250112 15:08:47.944918 24641 tablet_service.cc:1467] Processing CreateTablet for tablet 934800150b1945dcab01b4f7c4edcdc8 (DEFAULT_TABLE table=dugtrio [id=271cc77f05ad43eb89d1c5c6291536e4]), partition=RANGE (key) PARTITION 1610612733 <= VALUES
I20250112 15:08:47.945004 24640 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bf900db2a1344c7b8fd0b9ba5e225201. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 15:08:47.945116 24639 tablet_service.cc:1467] Processing CreateTablet for tablet a3cc29d2a75b4778b21400e0e0b64a9e (DEFAULT_TABLE table=dugtrio [id=271cc77f05ad43eb89d1c5c6291536e4]), partition=RANGE (key) PARTITION 536870911 <= VALUES < 1073741822
I20250112 15:08:47.945214 24639 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a3cc29d2a75b4778b21400e0e0b64a9e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 15:08:47.945971 24641 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 934800150b1945dcab01b4f7c4edcdc8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 15:08:47.945979 24698 tablet_bootstrap.cc:492] T bf900db2a1344c7b8fd0b9ba5e225201 P f5b45ccea9c7438f8fba1b79073969e5: Bootstrap starting.
I20250112 15:08:47.946106 24638 tablet_service.cc:1467] Processing CreateTablet for tablet 6914949901e147f2970585c79458b9d2 (DEFAULT_TABLE table=dugtrio [id=271cc77f05ad43eb89d1c5c6291536e4]), partition=RANGE (key) PARTITION 1073741822 <= VALUES < 1610612733
I20250112 15:08:47.946180 24638 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6914949901e147f2970585c79458b9d2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 15:08:47.946516 24698 tablet_bootstrap.cc:654] T bf900db2a1344c7b8fd0b9ba5e225201 P f5b45ccea9c7438f8fba1b79073969e5: Neither blocks nor log segments found. Creating new log.
I20250112 15:08:47.947108 24698 tablet_bootstrap.cc:492] T bf900db2a1344c7b8fd0b9ba5e225201 P f5b45ccea9c7438f8fba1b79073969e5: No bootstrap required, opened a new log
I20250112 15:08:47.947168 24698 ts_tablet_manager.cc:1397] T bf900db2a1344c7b8fd0b9ba5e225201 P f5b45ccea9c7438f8fba1b79073969e5: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20250112 15:08:47.947324 24698 raft_consensus.cc:357] T bf900db2a1344c7b8fd0b9ba5e225201 P f5b45ccea9c7438f8fba1b79073969e5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52bd510945874fa781ec841d40e251ab" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 34851 } } peers { permanent_uuid: "56dd875b6ba54b40877146f15292bf2f" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 32969 } } peers { permanent_uuid: "f5b45ccea9c7438f8fba1b79073969e5" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 40795 } }
I20250112 15:08:47.947381 24698 raft_consensus.cc:383] T bf900db2a1344c7b8fd0b9ba5e225201 P f5b45ccea9c7438f8fba1b79073969e5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 15:08:47.947398 24698 raft_consensus.cc:738] T bf900db2a1344c7b8fd0b9ba5e225201 P f5b45ccea9c7438f8fba1b79073969e5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f5b45ccea9c7438f8fba1b79073969e5, State: Initialized, Role: FOLLOWER
I20250112 15:08:47.947439 24698 consensus_queue.cc:260] T bf900db2a1344c7b8fd0b9ba5e225201 P f5b45ccea9c7438f8fba1b79073969e5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52bd510945874fa781ec841d40e251ab" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 34851 } } peers { permanent_uuid: "56dd875b6ba54b40877146f15292bf2f" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 32969 } } peers { permanent_uuid: "f5b45ccea9c7438f8fba1b79073969e5" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 40795 } }
I20250112 15:08:47.947515 24698 ts_tablet_manager.cc:1428] T bf900db2a1344c7b8fd0b9ba5e225201 P f5b45ccea9c7438f8fba1b79073969e5: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20250112 15:08:47.947566 24698 tablet_bootstrap.cc:492] T a3cc29d2a75b4778b21400e0e0b64a9e P f5b45ccea9c7438f8fba1b79073969e5: Bootstrap starting.
I20250112 15:08:47.947589 24697 tablet_bootstrap.cc:492] T 6914949901e147f2970585c79458b9d2 P 56dd875b6ba54b40877146f15292bf2f: No bootstrap required, opened a new log
I20250112 15:08:47.947631 24697 ts_tablet_manager.cc:1397] T 6914949901e147f2970585c79458b9d2 P 56dd875b6ba54b40877146f15292bf2f: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.001s
I20250112 15:08:47.947793 24697 raft_consensus.cc:357] T 6914949901e147f2970585c79458b9d2 P 56dd875b6ba54b40877146f15292bf2f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52bd510945874fa781ec841d40e251ab" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 34851 } } peers { permanent_uuid: "56dd875b6ba54b40877146f15292bf2f" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 32969 } } peers { permanent_uuid: "f5b45ccea9c7438f8fba1b79073969e5" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 40795 } }
I20250112 15:08:47.947842 24697 raft_consensus.cc:383] T 6914949901e147f2970585c79458b9d2 P 56dd875b6ba54b40877146f15292bf2f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 15:08:47.947865 24697 raft_consensus.cc:738] T 6914949901e147f2970585c79458b9d2 P 56dd875b6ba54b40877146f15292bf2f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 56dd875b6ba54b40877146f15292bf2f, State: Initialized, Role: FOLLOWER
I20250112 15:08:47.947908 24697 consensus_queue.cc:260] T 6914949901e147f2970585c79458b9d2 P 56dd875b6ba54b40877146f15292bf2f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52bd510945874fa781ec841d40e251ab" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 34851 } } peers { permanent_uuid: "56dd875b6ba54b40877146f15292bf2f" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 32969 } } peers { permanent_uuid: "f5b45ccea9c7438f8fba1b79073969e5" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 40795 } }
I20250112 15:08:47.947971 24698 tablet_bootstrap.cc:654] T a3cc29d2a75b4778b21400e0e0b64a9e P f5b45ccea9c7438f8fba1b79073969e5: Neither blocks nor log segments found. Creating new log.
I20250112 15:08:47.947993 24697 ts_tablet_manager.cc:1428] T 6914949901e147f2970585c79458b9d2 P 56dd875b6ba54b40877146f15292bf2f: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20250112 15:08:47.948037 24697 tablet_bootstrap.cc:492] T bf900db2a1344c7b8fd0b9ba5e225201 P 56dd875b6ba54b40877146f15292bf2f: Bootstrap starting.
I20250112 15:08:47.941996 24699 tablet_bootstrap.cc:654] T bf900db2a1344c7b8fd0b9ba5e225201 P 52bd510945874fa781ec841d40e251ab: Neither blocks nor log segments found. Creating new log.
I20250112 15:08:47.948452 24697 tablet_bootstrap.cc:654] T bf900db2a1344c7b8fd0b9ba5e225201 P 56dd875b6ba54b40877146f15292bf2f: Neither blocks nor log segments found. Creating new log.
I20250112 15:08:47.949031 24697 tablet_bootstrap.cc:492] T bf900db2a1344c7b8fd0b9ba5e225201 P 56dd875b6ba54b40877146f15292bf2f: No bootstrap required, opened a new log
I20250112 15:08:47.949039 24699 tablet_bootstrap.cc:492] T bf900db2a1344c7b8fd0b9ba5e225201 P 52bd510945874fa781ec841d40e251ab: No bootstrap required, opened a new log
I20250112 15:08:47.949080 24697 ts_tablet_manager.cc:1397] T bf900db2a1344c7b8fd0b9ba5e225201 P 56dd875b6ba54b40877146f15292bf2f: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20250112 15:08:47.949039 24698 tablet_bootstrap.cc:492] T a3cc29d2a75b4778b21400e0e0b64a9e P f5b45ccea9c7438f8fba1b79073969e5: No bootstrap required, opened a new log
I20250112 15:08:47.949111 24699 ts_tablet_manager.cc:1397] T bf900db2a1344c7b8fd0b9ba5e225201 P 52bd510945874fa781ec841d40e251ab: Time spent bootstrapping tablet: real 0.008s user 0.001s sys 0.001s
I20250112 15:08:47.949172 24698 ts_tablet_manager.cc:1397] T a3cc29d2a75b4778b21400e0e0b64a9e P f5b45ccea9c7438f8fba1b79073969e5: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20250112 15:08:47.949210 24697 raft_consensus.cc:357] T bf900db2a1344c7b8fd0b9ba5e225201 P 56dd875b6ba54b40877146f15292bf2f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52bd510945874fa781ec841d40e251ab" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 34851 } } peers { permanent_uuid: "56dd875b6ba54b40877146f15292bf2f" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 32969 } } peers { permanent_uuid: "f5b45ccea9c7438f8fba1b79073969e5" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 40795 } }
I20250112 15:08:47.949247 24697 raft_consensus.cc:383] T bf900db2a1344c7b8fd0b9ba5e225201 P 56dd875b6ba54b40877146f15292bf2f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 15:08:47.949259 24697 raft_consensus.cc:738] T bf900db2a1344c7b8fd0b9ba5e225201 P 56dd875b6ba54b40877146f15292bf2f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 56dd875b6ba54b40877146f15292bf2f, State: Initialized, Role: FOLLOWER
I20250112 15:08:47.949293 24697 consensus_queue.cc:260] T bf900db2a1344c7b8fd0b9ba5e225201 P 56dd875b6ba54b40877146f15292bf2f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52bd510945874fa781ec841d40e251ab" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 34851 } } peers { permanent_uuid: "56dd875b6ba54b40877146f15292bf2f" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 32969 } } peers { permanent_uuid: "f5b45ccea9c7438f8fba1b79073969e5" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 40795 } }
I20250112 15:08:47.949313 24698 raft_consensus.cc:357] T a3cc29d2a75b4778b21400e0e0b64a9e P f5b45ccea9c7438f8fba1b79073969e5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52bd510945874fa781ec841d40e251ab" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 34851 } } peers { permanent_uuid: "56dd875b6ba54b40877146f15292bf2f" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 32969 } } peers { permanent_uuid: "f5b45ccea9c7438f8fba1b79073969e5" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 40795 } }
I20250112 15:08:47.949296 24699 raft_consensus.cc:357] T bf900db2a1344c7b8fd0b9ba5e225201 P 52bd510945874fa781ec841d40e251ab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52bd510945874fa781ec841d40e251ab" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 34851 } } peers { permanent_uuid: "56dd875b6ba54b40877146f15292bf2f" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 32969 } } peers { permanent_uuid: "f5b45ccea9c7438f8fba1b79073969e5" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 40795 } }
I20250112 15:08:47.949375 24698 raft_consensus.cc:383] T a3cc29d2a75b4778b21400e0e0b64a9e P f5b45ccea9c7438f8fba1b79073969e5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 15:08:47.949381 24697 ts_tablet_manager.cc:1428] T bf900db2a1344c7b8fd0b9ba5e225201 P 56dd875b6ba54b40877146f15292bf2f: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20250112 15:08:47.949410 24698 raft_consensus.cc:738] T a3cc29d2a75b4778b21400e0e0b64a9e P f5b45ccea9c7438f8fba1b79073969e5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f5b45ccea9c7438f8fba1b79073969e5, State: Initialized, Role: FOLLOWER
I20250112 15:08:47.949383 24699 raft_consensus.cc:383] T bf900db2a1344c7b8fd0b9ba5e225201 P 52bd510945874fa781ec841d40e251ab [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 15:08:47.949528 24699 raft_consensus.cc:738] T bf900db2a1344c7b8fd0b9ba5e225201 P 52bd510945874fa781ec841d40e251ab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 52bd510945874fa781ec841d40e251ab, State: Initialized, Role: FOLLOWER
I20250112 15:08:47.949542 24698 consensus_queue.cc:260] T a3cc29d2a75b4778b21400e0e0b64a9e P f5b45ccea9c7438f8fba1b79073969e5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52bd510945874fa781ec841d40e251ab" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 34851 } } peers { permanent_uuid: "56dd875b6ba54b40877146f15292bf2f" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 32969 } } peers { permanent_uuid: "f5b45ccea9c7438f8fba1b79073969e5" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 40795 } }
I20250112 15:08:47.949566 24699 consensus_queue.cc:260] T bf900db2a1344c7b8fd0b9ba5e225201 P 52bd510945874fa781ec841d40e251ab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52bd510945874fa781ec841d40e251ab" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 34851 } } peers { permanent_uuid: "56dd875b6ba54b40877146f15292bf2f" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 32969 } } peers { permanent_uuid: "f5b45ccea9c7438f8fba1b79073969e5" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 40795 } }
I20250112 15:08:47.949630 24698 ts_tablet_manager.cc:1428] T a3cc29d2a75b4778b21400e0e0b64a9e P f5b45ccea9c7438f8fba1b79073969e5: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20250112 15:08:47.949641 24699 ts_tablet_manager.cc:1428] T bf900db2a1344c7b8fd0b9ba5e225201 P 52bd510945874fa781ec841d40e251ab: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20250112 15:08:47.949693 24699 tablet_bootstrap.cc:492] T 934800150b1945dcab01b4f7c4edcdc8 P 52bd510945874fa781ec841d40e251ab: Bootstrap starting.
I20250112 15:08:47.949692 24698 tablet_bootstrap.cc:492] T 934800150b1945dcab01b4f7c4edcdc8 P f5b45ccea9c7438f8fba1b79073969e5: Bootstrap starting.
I20250112 15:08:47.950126 24699 tablet_bootstrap.cc:654] T 934800150b1945dcab01b4f7c4edcdc8 P 52bd510945874fa781ec841d40e251ab: Neither blocks nor log segments found. Creating new log.
I20250112 15:08:47.950229 24698 tablet_bootstrap.cc:654] T 934800150b1945dcab01b4f7c4edcdc8 P f5b45ccea9c7438f8fba1b79073969e5: Neither blocks nor log segments found. Creating new log.
I20250112 15:08:47.950889 24699 tablet_bootstrap.cc:492] T 934800150b1945dcab01b4f7c4edcdc8 P 52bd510945874fa781ec841d40e251ab: No bootstrap required, opened a new log
I20250112 15:08:47.950909 24698 tablet_bootstrap.cc:492] T 934800150b1945dcab01b4f7c4edcdc8 P f5b45ccea9c7438f8fba1b79073969e5: No bootstrap required, opened a new log
I20250112 15:08:47.950942 24699 ts_tablet_manager.cc:1397] T 934800150b1945dcab01b4f7c4edcdc8 P 52bd510945874fa781ec841d40e251ab: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20250112 15:08:47.950947 24698 ts_tablet_manager.cc:1397] T 934800150b1945dcab01b4f7c4edcdc8 P f5b45ccea9c7438f8fba1b79073969e5: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20250112 15:08:47.951085 24699 raft_consensus.cc:357] T 934800150b1945dcab01b4f7c4edcdc8 P 52bd510945874fa781ec841d40e251ab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52bd510945874fa781ec841d40e251ab" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 34851 } } peers { permanent_uuid: "56dd875b6ba54b40877146f15292bf2f" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 32969 } } peers { permanent_uuid: "f5b45ccea9c7438f8fba1b79073969e5" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 40795 } }
I20250112 15:08:47.951135 24699 raft_consensus.cc:383] T 934800150b1945dcab01b4f7c4edcdc8 P 52bd510945874fa781ec841d40e251ab [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 15:08:47.951149 24699 raft_consensus.cc:738] T 934800150b1945dcab01b4f7c4edcdc8 P 52bd510945874fa781ec841d40e251ab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 52bd510945874fa781ec841d40e251ab, State: Initialized, Role: FOLLOWER
I20250112 15:08:47.951122 24698 raft_consensus.cc:357] T 934800150b1945dcab01b4f7c4edcdc8 P f5b45ccea9c7438f8fba1b79073969e5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52bd510945874fa781ec841d40e251ab" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 34851 } } peers { permanent_uuid: "56dd875b6ba54b40877146f15292bf2f" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 32969 } } peers { permanent_uuid: "f5b45ccea9c7438f8fba1b79073969e5" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 40795 } }
I20250112 15:08:47.951177 24698 raft_consensus.cc:383] T 934800150b1945dcab01b4f7c4edcdc8 P f5b45ccea9c7438f8fba1b79073969e5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 15:08:47.951182 24699 consensus_queue.cc:260] T 934800150b1945dcab01b4f7c4edcdc8 P 52bd510945874fa781ec841d40e251ab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52bd510945874fa781ec841d40e251ab" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 34851 } } peers { permanent_uuid: "56dd875b6ba54b40877146f15292bf2f" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 32969 } } peers { permanent_uuid: "f5b45ccea9c7438f8fba1b79073969e5" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 40795 } }
I20250112 15:08:47.951241 24698 raft_consensus.cc:738] T 934800150b1945dcab01b4f7c4edcdc8 P f5b45ccea9c7438f8fba1b79073969e5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f5b45ccea9c7438f8fba1b79073969e5, State: Initialized, Role: FOLLOWER
I20250112 15:08:47.951330 24699 ts_tablet_manager.cc:1428] T 934800150b1945dcab01b4f7c4edcdc8 P 52bd510945874fa781ec841d40e251ab: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20250112 15:08:47.951392 24699 tablet_bootstrap.cc:492] T a3cc29d2a75b4778b21400e0e0b64a9e P 52bd510945874fa781ec841d40e251ab: Bootstrap starting.
I20250112 15:08:47.951364 24698 consensus_queue.cc:260] T 934800150b1945dcab01b4f7c4edcdc8 P f5b45ccea9c7438f8fba1b79073969e5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52bd510945874fa781ec841d40e251ab" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 34851 } } peers { permanent_uuid: "56dd875b6ba54b40877146f15292bf2f" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 32969 } } peers { permanent_uuid: "f5b45ccea9c7438f8fba1b79073969e5" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 40795 } }
I20250112 15:08:47.951516 24698 ts_tablet_manager.cc:1428] T 934800150b1945dcab01b4f7c4edcdc8 P f5b45ccea9c7438f8fba1b79073969e5: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20250112 15:08:47.951625 24698 tablet_bootstrap.cc:492] T 6914949901e147f2970585c79458b9d2 P f5b45ccea9c7438f8fba1b79073969e5: Bootstrap starting.
I20250112 15:08:47.951930 24699 tablet_bootstrap.cc:654] T a3cc29d2a75b4778b21400e0e0b64a9e P 52bd510945874fa781ec841d40e251ab: Neither blocks nor log segments found. Creating new log.
I20250112 15:08:47.952057 24698 tablet_bootstrap.cc:654] T 6914949901e147f2970585c79458b9d2 P f5b45ccea9c7438f8fba1b79073969e5: Neither blocks nor log segments found. Creating new log.
I20250112 15:08:47.952663 24698 tablet_bootstrap.cc:492] T 6914949901e147f2970585c79458b9d2 P f5b45ccea9c7438f8fba1b79073969e5: No bootstrap required, opened a new log
I20250112 15:08:47.952659 24699 tablet_bootstrap.cc:492] T a3cc29d2a75b4778b21400e0e0b64a9e P 52bd510945874fa781ec841d40e251ab: No bootstrap required, opened a new log
I20250112 15:08:47.952742 24698 ts_tablet_manager.cc:1397] T 6914949901e147f2970585c79458b9d2 P f5b45ccea9c7438f8fba1b79073969e5: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20250112 15:08:47.952749 24699 ts_tablet_manager.cc:1397] T a3cc29d2a75b4778b21400e0e0b64a9e P 52bd510945874fa781ec841d40e251ab: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20250112 15:08:47.952934 24698 raft_consensus.cc:357] T 6914949901e147f2970585c79458b9d2 P f5b45ccea9c7438f8fba1b79073969e5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52bd510945874fa781ec841d40e251ab" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 34851 } } peers { permanent_uuid: "56dd875b6ba54b40877146f15292bf2f" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 32969 } } peers { permanent_uuid: "f5b45ccea9c7438f8fba1b79073969e5" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 40795 } }
I20250112 15:08:47.952934 24699 raft_consensus.cc:357] T a3cc29d2a75b4778b21400e0e0b64a9e P 52bd510945874fa781ec841d40e251ab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52bd510945874fa781ec841d40e251ab" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 34851 } } peers { permanent_uuid: "56dd875b6ba54b40877146f15292bf2f" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 32969 } } peers { permanent_uuid: "f5b45ccea9c7438f8fba1b79073969e5" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 40795 } }
I20250112 15:08:47.953029 24698 raft_consensus.cc:383] T 6914949901e147f2970585c79458b9d2 P f5b45ccea9c7438f8fba1b79073969e5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 15:08:47.953037 24699 raft_consensus.cc:383] T a3cc29d2a75b4778b21400e0e0b64a9e P 52bd510945874fa781ec841d40e251ab [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 15:08:47.953068 24698 raft_consensus.cc:738] T 6914949901e147f2970585c79458b9d2 P f5b45ccea9c7438f8fba1b79073969e5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f5b45ccea9c7438f8fba1b79073969e5, State: Initialized, Role: FOLLOWER
I20250112 15:08:47.953074 24699 raft_consensus.cc:738] T a3cc29d2a75b4778b21400e0e0b64a9e P 52bd510945874fa781ec841d40e251ab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 52bd510945874fa781ec841d40e251ab, State: Initialized, Role: FOLLOWER
I20250112 15:08:47.953150 24699 consensus_queue.cc:260] T a3cc29d2a75b4778b21400e0e0b64a9e P 52bd510945874fa781ec841d40e251ab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52bd510945874fa781ec841d40e251ab" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 34851 } } peers { permanent_uuid: "56dd875b6ba54b40877146f15292bf2f" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 32969 } } peers { permanent_uuid: "f5b45ccea9c7438f8fba1b79073969e5" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 40795 } }
I20250112 15:08:47.953153 24698 consensus_queue.cc:260] T 6914949901e147f2970585c79458b9d2 P f5b45ccea9c7438f8fba1b79073969e5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52bd510945874fa781ec841d40e251ab" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 34851 } } peers { permanent_uuid: "56dd875b6ba54b40877146f15292bf2f" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 32969 } } peers { permanent_uuid: "f5b45ccea9c7438f8fba1b79073969e5" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 40795 } }
I20250112 15:08:47.953269 24699 ts_tablet_manager.cc:1428] T a3cc29d2a75b4778b21400e0e0b64a9e P 52bd510945874fa781ec841d40e251ab: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20250112 15:08:47.953325 24699 tablet_bootstrap.cc:492] T 6914949901e147f2970585c79458b9d2 P 52bd510945874fa781ec841d40e251ab: Bootstrap starting.
I20250112 15:08:47.953729 24699 tablet_bootstrap.cc:654] T 6914949901e147f2970585c79458b9d2 P 52bd510945874fa781ec841d40e251ab: Neither blocks nor log segments found. Creating new log.
I20250112 15:08:47.953313 24698 ts_tablet_manager.cc:1428] T 6914949901e147f2970585c79458b9d2 P f5b45ccea9c7438f8fba1b79073969e5: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20250112 15:08:47.954411 24699 tablet_bootstrap.cc:492] T 6914949901e147f2970585c79458b9d2 P 52bd510945874fa781ec841d40e251ab: No bootstrap required, opened a new log
I20250112 15:08:47.954465 24699 ts_tablet_manager.cc:1397] T 6914949901e147f2970585c79458b9d2 P 52bd510945874fa781ec841d40e251ab: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.000s
I20250112 15:08:47.954582 24699 raft_consensus.cc:357] T 6914949901e147f2970585c79458b9d2 P 52bd510945874fa781ec841d40e251ab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52bd510945874fa781ec841d40e251ab" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 34851 } } peers { permanent_uuid: "56dd875b6ba54b40877146f15292bf2f" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 32969 } } peers { permanent_uuid: "f5b45ccea9c7438f8fba1b79073969e5" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 40795 } }
I20250112 15:08:47.954628 24699 raft_consensus.cc:383] T 6914949901e147f2970585c79458b9d2 P 52bd510945874fa781ec841d40e251ab [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 15:08:47.954641 24699 raft_consensus.cc:738] T 6914949901e147f2970585c79458b9d2 P 52bd510945874fa781ec841d40e251ab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 52bd510945874fa781ec841d40e251ab, State: Initialized, Role: FOLLOWER
I20250112 15:08:47.954674 24699 consensus_queue.cc:260] T 6914949901e147f2970585c79458b9d2 P 52bd510945874fa781ec841d40e251ab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52bd510945874fa781ec841d40e251ab" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 34851 } } peers { permanent_uuid: "56dd875b6ba54b40877146f15292bf2f" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 32969 } } peers { permanent_uuid: "f5b45ccea9c7438f8fba1b79073969e5" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 40795 } }
I20250112 15:08:47.954777 24699 ts_tablet_manager.cc:1428] T 6914949901e147f2970585c79458b9d2 P 52bd510945874fa781ec841d40e251ab: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20250112 15:08:47.964826 24702 raft_consensus.cc:491] T 6914949901e147f2970585c79458b9d2 P 56dd875b6ba54b40877146f15292bf2f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 15:08:47.964912 24702 raft_consensus.cc:513] T 6914949901e147f2970585c79458b9d2 P 56dd875b6ba54b40877146f15292bf2f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52bd510945874fa781ec841d40e251ab" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 34851 } } peers { permanent_uuid: "56dd875b6ba54b40877146f15292bf2f" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 32969 } } peers { permanent_uuid: "f5b45ccea9c7438f8fba1b79073969e5" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 40795 } }
I20250112 15:08:47.965092 24702 leader_election.cc:290] T 6914949901e147f2970585c79458b9d2 P 56dd875b6ba54b40877146f15292bf2f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 52bd510945874fa781ec841d40e251ab (127.17.60.193:34851), f5b45ccea9c7438f8fba1b79073969e5 (127.17.60.195:40795)
I20250112 15:08:47.967871 24500 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6914949901e147f2970585c79458b9d2" candidate_uuid: "56dd875b6ba54b40877146f15292bf2f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "52bd510945874fa781ec841d40e251ab" is_pre_election: true
I20250112 15:08:47.967981 24500 raft_consensus.cc:2463] T 6914949901e147f2970585c79458b9d2 P 52bd510945874fa781ec841d40e251ab [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 56dd875b6ba54b40877146f15292bf2f in term 0.
I20250112 15:08:47.968103 24539 leader_election.cc:304] T 6914949901e147f2970585c79458b9d2 P 56dd875b6ba54b40877146f15292bf2f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52bd510945874fa781ec841d40e251ab, 56dd875b6ba54b40877146f15292bf2f; no voters:
I20250112 15:08:47.968212 24702 raft_consensus.cc:2798] T 6914949901e147f2970585c79458b9d2 P 56dd875b6ba54b40877146f15292bf2f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250112 15:08:47.968256 24702 raft_consensus.cc:491] T 6914949901e147f2970585c79458b9d2 P 56dd875b6ba54b40877146f15292bf2f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250112 15:08:47.968286 24702 raft_consensus.cc:3054] T 6914949901e147f2970585c79458b9d2 P 56dd875b6ba54b40877146f15292bf2f [term 0 FOLLOWER]: Advancing to term 1
I20250112 15:08:47.968439 24651 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6914949901e147f2970585c79458b9d2" candidate_uuid: "56dd875b6ba54b40877146f15292bf2f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f5b45ccea9c7438f8fba1b79073969e5" is_pre_election: true
I20250112 15:08:47.968514 24651 raft_consensus.cc:2463] T 6914949901e147f2970585c79458b9d2 P f5b45ccea9c7438f8fba1b79073969e5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 56dd875b6ba54b40877146f15292bf2f in term 0.
I20250112 15:08:47.969017 24702 raft_consensus.cc:513] T 6914949901e147f2970585c79458b9d2 P 56dd875b6ba54b40877146f15292bf2f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52bd510945874fa781ec841d40e251ab" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 34851 } } peers { permanent_uuid: "56dd875b6ba54b40877146f15292bf2f" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 32969 } } peers { permanent_uuid: "f5b45ccea9c7438f8fba1b79073969e5" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 40795 } }
I20250112 15:08:47.969126 24702 leader_election.cc:290] T 6914949901e147f2970585c79458b9d2 P 56dd875b6ba54b40877146f15292bf2f [CANDIDATE]: Term 1 election: Requested vote from peers 52bd510945874fa781ec841d40e251ab (127.17.60.193:34851), f5b45ccea9c7438f8fba1b79073969e5 (127.17.60.195:40795)
I20250112 15:08:47.969272 24500 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6914949901e147f2970585c79458b9d2" candidate_uuid: "56dd875b6ba54b40877146f15292bf2f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "52bd510945874fa781ec841d40e251ab"
I20250112 15:08:47.969281 24651 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6914949901e147f2970585c79458b9d2" candidate_uuid: "56dd875b6ba54b40877146f15292bf2f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f5b45ccea9c7438f8fba1b79073969e5"
I20250112 15:08:47.969334 24500 raft_consensus.cc:3054] T 6914949901e147f2970585c79458b9d2 P 52bd510945874fa781ec841d40e251ab [term 0 FOLLOWER]: Advancing to term 1
I20250112 15:08:47.969344 24651 raft_consensus.cc:3054] T 6914949901e147f2970585c79458b9d2 P f5b45ccea9c7438f8fba1b79073969e5 [term 0 FOLLOWER]: Advancing to term 1
I20250112 15:08:47.969810 24651 raft_consensus.cc:2463] T 6914949901e147f2970585c79458b9d2 P f5b45ccea9c7438f8fba1b79073969e5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 56dd875b6ba54b40877146f15292bf2f in term 1.
I20250112 15:08:47.969817 24500 raft_consensus.cc:2463] T 6914949901e147f2970585c79458b9d2 P 52bd510945874fa781ec841d40e251ab [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 56dd875b6ba54b40877146f15292bf2f in term 1.
I20250112 15:08:47.971489 24539 leader_election.cc:304] T 6914949901e147f2970585c79458b9d2 P 56dd875b6ba54b40877146f15292bf2f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 56dd875b6ba54b40877146f15292bf2f, f5b45ccea9c7438f8fba1b79073969e5; no voters:
I20250112 15:08:47.971591 24702 raft_consensus.cc:2798] T 6914949901e147f2970585c79458b9d2 P 56dd875b6ba54b40877146f15292bf2f [term 1 FOLLOWER]: Leader election won for term 1
I20250112 15:08:47.971765 24702 raft_consensus.cc:695] T 6914949901e147f2970585c79458b9d2 P 56dd875b6ba54b40877146f15292bf2f [term 1 LEADER]: Becoming Leader. State: Replica: 56dd875b6ba54b40877146f15292bf2f, State: Running, Role: LEADER
I20250112 15:08:47.971823 24702 consensus_queue.cc:237] T 6914949901e147f2970585c79458b9d2 P 56dd875b6ba54b40877146f15292bf2f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52bd510945874fa781ec841d40e251ab" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 34851 } } peers { permanent_uuid: "56dd875b6ba54b40877146f15292bf2f" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 32969 } } peers { permanent_uuid: "f5b45ccea9c7438f8fba1b79073969e5" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 40795 } }
I20250112 15:08:47.972349 24397 catalog_manager.cc:5526] T 6914949901e147f2970585c79458b9d2 P 56dd875b6ba54b40877146f15292bf2f reported cstate change: term changed from 0 to 1, leader changed from <none> to 56dd875b6ba54b40877146f15292bf2f (127.17.60.194). New cstate: current_term: 1 leader_uuid: "56dd875b6ba54b40877146f15292bf2f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52bd510945874fa781ec841d40e251ab" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 34851 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "56dd875b6ba54b40877146f15292bf2f" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 32969 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f5b45ccea9c7438f8fba1b79073969e5" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 40795 } health_report { overall_health: UNKNOWN } } }
I20250112 15:08:47.996814 24711 consensus_queue.cc:1035] T 94172f983ecd4a37956654f2f86dba00 P 52bd510945874fa781ec841d40e251ab [LEADER]: Connected to new peer: Peer: permanent_uuid: "f5b45ccea9c7438f8fba1b79073969e5" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 40795 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250112 15:08:47.998337 24711 consensus_queue.cc:1035] T 94172f983ecd4a37956654f2f86dba00 P 52bd510945874fa781ec841d40e251ab [LEADER]: Connected to new peer: Peer: permanent_uuid: "56dd875b6ba54b40877146f15292bf2f" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 32969 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250112 15:08:48.032706 24744 raft_consensus.cc:491] T bf900db2a1344c7b8fd0b9ba5e225201 P 56dd875b6ba54b40877146f15292bf2f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 15:08:48.032816 24744 raft_consensus.cc:513] T bf900db2a1344c7b8fd0b9ba5e225201 P 56dd875b6ba54b40877146f15292bf2f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52bd510945874fa781ec841d40e251ab" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 34851 } } peers { permanent_uuid: "56dd875b6ba54b40877146f15292bf2f" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 32969 } } peers { permanent_uuid: "f5b45ccea9c7438f8fba1b79073969e5" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 40795 } }
I20250112 15:08:48.032997 24744 leader_election.cc:290] T bf900db2a1344c7b8fd0b9ba5e225201 P 56dd875b6ba54b40877146f15292bf2f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 52bd510945874fa781ec841d40e251ab (127.17.60.193:34851), f5b45ccea9c7438f8fba1b79073969e5 (127.17.60.195:40795)
I20250112 15:08:48.033212 24500 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bf900db2a1344c7b8fd0b9ba5e225201" candidate_uuid: "56dd875b6ba54b40877146f15292bf2f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "52bd510945874fa781ec841d40e251ab" is_pre_election: true
I20250112 15:08:48.033282 24500 raft_consensus.cc:2463] T bf900db2a1344c7b8fd0b9ba5e225201 P 52bd510945874fa781ec841d40e251ab [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 56dd875b6ba54b40877146f15292bf2f in term 0.
I20250112 15:08:48.033267 24651 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bf900db2a1344c7b8fd0b9ba5e225201" candidate_uuid: "56dd875b6ba54b40877146f15292bf2f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f5b45ccea9c7438f8fba1b79073969e5" is_pre_election: true
I20250112 15:08:48.033394 24651 raft_consensus.cc:2463] T bf900db2a1344c7b8fd0b9ba5e225201 P f5b45ccea9c7438f8fba1b79073969e5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 56dd875b6ba54b40877146f15292bf2f in term 0.
I20250112 15:08:48.033488 24539 leader_election.cc:304] T bf900db2a1344c7b8fd0b9ba5e225201 P 56dd875b6ba54b40877146f15292bf2f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52bd510945874fa781ec841d40e251ab, 56dd875b6ba54b40877146f15292bf2f; no voters:
I20250112 15:08:48.033583 24744 raft_consensus.cc:2798] T bf900db2a1344c7b8fd0b9ba5e225201 P 56dd875b6ba54b40877146f15292bf2f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250112 15:08:48.033627 24744 raft_consensus.cc:491] T bf900db2a1344c7b8fd0b9ba5e225201 P 56dd875b6ba54b40877146f15292bf2f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250112 15:08:48.033645 24744 raft_consensus.cc:3054] T bf900db2a1344c7b8fd0b9ba5e225201 P 56dd875b6ba54b40877146f15292bf2f [term 0 FOLLOWER]: Advancing to term 1
I20250112 15:08:48.034425 24744 raft_consensus.cc:513] T bf900db2a1344c7b8fd0b9ba5e225201 P 56dd875b6ba54b40877146f15292bf2f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52bd510945874fa781ec841d40e251ab" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 34851 } } peers { permanent_uuid: "56dd875b6ba54b40877146f15292bf2f" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 32969 } } peers { permanent_uuid: "f5b45ccea9c7438f8fba1b79073969e5" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 40795 } }
I20250112 15:08:48.034523 24744 leader_election.cc:290] T bf900db2a1344c7b8fd0b9ba5e225201 P 56dd875b6ba54b40877146f15292bf2f [CANDIDATE]: Term 1 election: Requested vote from peers 52bd510945874fa781ec841d40e251ab (127.17.60.193:34851), f5b45ccea9c7438f8fba1b79073969e5 (127.17.60.195:40795)
I20250112 15:08:48.034685 24500 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bf900db2a1344c7b8fd0b9ba5e225201" candidate_uuid: "56dd875b6ba54b40877146f15292bf2f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "52bd510945874fa781ec841d40e251ab"
I20250112 15:08:48.034705 24651 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bf900db2a1344c7b8fd0b9ba5e225201" candidate_uuid: "56dd875b6ba54b40877146f15292bf2f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f5b45ccea9c7438f8fba1b79073969e5"
I20250112 15:08:48.034773 24500 raft_consensus.cc:3054] T bf900db2a1344c7b8fd0b9ba5e225201 P 52bd510945874fa781ec841d40e251ab [term 0 FOLLOWER]: Advancing to term 1
I20250112 15:08:48.034785 24651 raft_consensus.cc:3054] T bf900db2a1344c7b8fd0b9ba5e225201 P f5b45ccea9c7438f8fba1b79073969e5 [term 0 FOLLOWER]: Advancing to term 1
I20250112 15:08:48.035388 24500 raft_consensus.cc:2463] T bf900db2a1344c7b8fd0b9ba5e225201 P 52bd510945874fa781ec841d40e251ab [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 56dd875b6ba54b40877146f15292bf2f in term 1.
I20250112 15:08:48.035388 24651 raft_consensus.cc:2463] T bf900db2a1344c7b8fd0b9ba5e225201 P f5b45ccea9c7438f8fba1b79073969e5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 56dd875b6ba54b40877146f15292bf2f in term 1.
I20250112 15:08:48.035568 24539 leader_election.cc:304] T bf900db2a1344c7b8fd0b9ba5e225201 P 56dd875b6ba54b40877146f15292bf2f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52bd510945874fa781ec841d40e251ab, 56dd875b6ba54b40877146f15292bf2f; no voters:
I20250112 15:08:48.035688 24744 raft_consensus.cc:2798] T bf900db2a1344c7b8fd0b9ba5e225201 P 56dd875b6ba54b40877146f15292bf2f [term 1 FOLLOWER]: Leader election won for term 1
I20250112 15:08:48.035745 24744 raft_consensus.cc:695] T bf900db2a1344c7b8fd0b9ba5e225201 P 56dd875b6ba54b40877146f15292bf2f [term 1 LEADER]: Becoming Leader. State: Replica: 56dd875b6ba54b40877146f15292bf2f, State: Running, Role: LEADER
I20250112 15:08:48.035799 24744 consensus_queue.cc:237] T bf900db2a1344c7b8fd0b9ba5e225201 P 56dd875b6ba54b40877146f15292bf2f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52bd510945874fa781ec841d40e251ab" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 34851 } } peers { permanent_uuid: "56dd875b6ba54b40877146f15292bf2f" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 32969 } } peers { permanent_uuid: "f5b45ccea9c7438f8fba1b79073969e5" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 40795 } }
I20250112 15:08:48.036434 24397 catalog_manager.cc:5526] T bf900db2a1344c7b8fd0b9ba5e225201 P 56dd875b6ba54b40877146f15292bf2f reported cstate change: term changed from 0 to 1, leader changed from <none> to 56dd875b6ba54b40877146f15292bf2f (127.17.60.194). New cstate: current_term: 1 leader_uuid: "56dd875b6ba54b40877146f15292bf2f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52bd510945874fa781ec841d40e251ab" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 34851 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "56dd875b6ba54b40877146f15292bf2f" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 32969 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f5b45ccea9c7438f8fba1b79073969e5" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 40795 } health_report { overall_health: UNKNOWN } } }
I20250112 15:08:48.051265 24744 raft_consensus.cc:491] T a3cc29d2a75b4778b21400e0e0b64a9e P 56dd875b6ba54b40877146f15292bf2f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 15:08:48.051364 24744 raft_consensus.cc:513] T a3cc29d2a75b4778b21400e0e0b64a9e P 56dd875b6ba54b40877146f15292bf2f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52bd510945874fa781ec841d40e251ab" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 34851 } } peers { permanent_uuid: "56dd875b6ba54b40877146f15292bf2f" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 32969 } } peers { permanent_uuid: "f5b45ccea9c7438f8fba1b79073969e5" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 40795 } }
I20250112 15:08:48.051497 24744 leader_election.cc:290] T a3cc29d2a75b4778b21400e0e0b64a9e P 56dd875b6ba54b40877146f15292bf2f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 52bd510945874fa781ec841d40e251ab (127.17.60.193:34851), f5b45ccea9c7438f8fba1b79073969e5 (127.17.60.195:40795)
I20250112 15:08:48.051680 24651 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3cc29d2a75b4778b21400e0e0b64a9e" candidate_uuid: "56dd875b6ba54b40877146f15292bf2f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f5b45ccea9c7438f8fba1b79073969e5" is_pre_election: true
I20250112 15:08:48.051771 24651 raft_consensus.cc:2463] T a3cc29d2a75b4778b21400e0e0b64a9e P f5b45ccea9c7438f8fba1b79073969e5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 56dd875b6ba54b40877146f15292bf2f in term 0.
I20250112 15:08:48.051656 24500 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3cc29d2a75b4778b21400e0e0b64a9e" candidate_uuid: "56dd875b6ba54b40877146f15292bf2f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "52bd510945874fa781ec841d40e251ab" is_pre_election: true
I20250112 15:08:48.051853 24500 raft_consensus.cc:2463] T a3cc29d2a75b4778b21400e0e0b64a9e P 52bd510945874fa781ec841d40e251ab [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 56dd875b6ba54b40877146f15292bf2f in term 0.
I20250112 15:08:48.051909 24539 leader_election.cc:304] T a3cc29d2a75b4778b21400e0e0b64a9e P 56dd875b6ba54b40877146f15292bf2f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 56dd875b6ba54b40877146f15292bf2f, f5b45ccea9c7438f8fba1b79073969e5; no voters:
I20250112 15:08:48.051987 24744 raft_consensus.cc:2798] T a3cc29d2a75b4778b21400e0e0b64a9e P 56dd875b6ba54b40877146f15292bf2f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250112 15:08:48.052027 24744 raft_consensus.cc:491] T a3cc29d2a75b4778b21400e0e0b64a9e P 56dd875b6ba54b40877146f15292bf2f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250112 15:08:48.052047 24744 raft_consensus.cc:3054] T a3cc29d2a75b4778b21400e0e0b64a9e P 56dd875b6ba54b40877146f15292bf2f [term 0 FOLLOWER]: Advancing to term 1
I20250112 15:08:48.052639 24744 raft_consensus.cc:513] T a3cc29d2a75b4778b21400e0e0b64a9e P 56dd875b6ba54b40877146f15292bf2f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52bd510945874fa781ec841d40e251ab" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 34851 } } peers { permanent_uuid: "56dd875b6ba54b40877146f15292bf2f" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 32969 } } peers { permanent_uuid: "f5b45ccea9c7438f8fba1b79073969e5" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 40795 } }
I20250112 15:08:48.052728 24744 leader_election.cc:290] T a3cc29d2a75b4778b21400e0e0b64a9e P 56dd875b6ba54b40877146f15292bf2f [CANDIDATE]: Term 1 election: Requested vote from peers 52bd510945874fa781ec841d40e251ab (127.17.60.193:34851), f5b45ccea9c7438f8fba1b79073969e5 (127.17.60.195:40795)
I20250112 15:08:48.052889 24500 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3cc29d2a75b4778b21400e0e0b64a9e" candidate_uuid: "56dd875b6ba54b40877146f15292bf2f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "52bd510945874fa781ec841d40e251ab"
I20250112 15:08:48.052955 24651 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3cc29d2a75b4778b21400e0e0b64a9e" candidate_uuid: "56dd875b6ba54b40877146f15292bf2f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f5b45ccea9c7438f8fba1b79073969e5"
I20250112 15:08:48.052975 24500 raft_consensus.cc:3054] T a3cc29d2a75b4778b21400e0e0b64a9e P 52bd510945874fa781ec841d40e251ab [term 0 FOLLOWER]: Advancing to term 1
I20250112 15:08:48.052999 24651 raft_consensus.cc:3054] T a3cc29d2a75b4778b21400e0e0b64a9e P f5b45ccea9c7438f8fba1b79073969e5 [term 0 FOLLOWER]: Advancing to term 1
I20250112 15:08:48.053464 24500 raft_consensus.cc:2463] T a3cc29d2a75b4778b21400e0e0b64a9e P 52bd510945874fa781ec841d40e251ab [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 56dd875b6ba54b40877146f15292bf2f in term 1.
I20250112 15:08:48.053484 24651 raft_consensus.cc:2463] T a3cc29d2a75b4778b21400e0e0b64a9e P f5b45ccea9c7438f8fba1b79073969e5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 56dd875b6ba54b40877146f15292bf2f in term 1.
I20250112 15:08:48.053611 24539 leader_election.cc:304] T a3cc29d2a75b4778b21400e0e0b64a9e P 56dd875b6ba54b40877146f15292bf2f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52bd510945874fa781ec841d40e251ab, 56dd875b6ba54b40877146f15292bf2f; no voters:
I20250112 15:08:48.053705 24744 raft_consensus.cc:2798] T a3cc29d2a75b4778b21400e0e0b64a9e P 56dd875b6ba54b40877146f15292bf2f [term 1 FOLLOWER]: Leader election won for term 1
I20250112 15:08:48.053772 24744 raft_consensus.cc:695] T a3cc29d2a75b4778b21400e0e0b64a9e P 56dd875b6ba54b40877146f15292bf2f [term 1 LEADER]: Becoming Leader. State: Replica: 56dd875b6ba54b40877146f15292bf2f, State: Running, Role: LEADER
I20250112 15:08:48.053826 24744 consensus_queue.cc:237] T a3cc29d2a75b4778b21400e0e0b64a9e P 56dd875b6ba54b40877146f15292bf2f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52bd510945874fa781ec841d40e251ab" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 34851 } } peers { permanent_uuid: "56dd875b6ba54b40877146f15292bf2f" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 32969 } } peers { permanent_uuid: "f5b45ccea9c7438f8fba1b79073969e5" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 40795 } }
I20250112 15:08:48.054399 24397 catalog_manager.cc:5526] T a3cc29d2a75b4778b21400e0e0b64a9e P 56dd875b6ba54b40877146f15292bf2f reported cstate change: term changed from 0 to 1, leader changed from <none> to 56dd875b6ba54b40877146f15292bf2f (127.17.60.194). New cstate: current_term: 1 leader_uuid: "56dd875b6ba54b40877146f15292bf2f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52bd510945874fa781ec841d40e251ab" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 34851 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "56dd875b6ba54b40877146f15292bf2f" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 32969 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f5b45ccea9c7438f8fba1b79073969e5" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 40795 } health_report { overall_health: UNKNOWN } } }
I20250112 15:08:48.069976 24704 consensus_queue.cc:1035] T 45d29678d6704a888f2c2514af2f36a4 P f5b45ccea9c7438f8fba1b79073969e5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "52bd510945874fa781ec841d40e251ab" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 34851 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250112 15:08:48.071213 24721 consensus_queue.cc:1035] T 45d29678d6704a888f2c2514af2f36a4 P f5b45ccea9c7438f8fba1b79073969e5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "56dd875b6ba54b40877146f15292bf2f" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 32969 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250112 15:08:48.087141 24721 consensus_queue.cc:1035] T c73b4e40028649e5bb9ad937f307e7bf P f5b45ccea9c7438f8fba1b79073969e5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "52bd510945874fa781ec841d40e251ab" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 34851 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250112 15:08:48.088428 24721 consensus_queue.cc:1035] T c73b4e40028649e5bb9ad937f307e7bf P f5b45ccea9c7438f8fba1b79073969e5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "56dd875b6ba54b40877146f15292bf2f" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 32969 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250112 15:08:48.133900 24721 raft_consensus.cc:491] T 934800150b1945dcab01b4f7c4edcdc8 P f5b45ccea9c7438f8fba1b79073969e5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 15:08:48.134014 24721 raft_consensus.cc:513] T 934800150b1945dcab01b4f7c4edcdc8 P f5b45ccea9c7438f8fba1b79073969e5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52bd510945874fa781ec841d40e251ab" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 34851 } } peers { permanent_uuid: "56dd875b6ba54b40877146f15292bf2f" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 32969 } } peers { permanent_uuid: "f5b45ccea9c7438f8fba1b79073969e5" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 40795 } }
I20250112 15:08:48.134212 24721 leader_election.cc:290] T 934800150b1945dcab01b4f7c4edcdc8 P f5b45ccea9c7438f8fba1b79073969e5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 52bd510945874fa781ec841d40e251ab (127.17.60.193:34851), 56dd875b6ba54b40877146f15292bf2f (127.17.60.194:32969)
I20250112 15:08:48.134452 24500 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "934800150b1945dcab01b4f7c4edcdc8" candidate_uuid: "f5b45ccea9c7438f8fba1b79073969e5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "52bd510945874fa781ec841d40e251ab" is_pre_election: true
I20250112 15:08:48.134466 24576 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "934800150b1945dcab01b4f7c4edcdc8" candidate_uuid: "f5b45ccea9c7438f8fba1b79073969e5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "56dd875b6ba54b40877146f15292bf2f" is_pre_election: true
I20250112 15:08:48.134548 24500 raft_consensus.cc:2463] T 934800150b1945dcab01b4f7c4edcdc8 P 52bd510945874fa781ec841d40e251ab [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f5b45ccea9c7438f8fba1b79073969e5 in term 0.
I20250112 15:08:48.134558 24576 raft_consensus.cc:2463] T 934800150b1945dcab01b4f7c4edcdc8 P 56dd875b6ba54b40877146f15292bf2f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f5b45ccea9c7438f8fba1b79073969e5 in term 0.
I20250112 15:08:48.134727 24614 leader_election.cc:304] T 934800150b1945dcab01b4f7c4edcdc8 P f5b45ccea9c7438f8fba1b79073969e5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52bd510945874fa781ec841d40e251ab, f5b45ccea9c7438f8fba1b79073969e5; no voters:
I20250112 15:08:48.134830 24721 raft_consensus.cc:2798] T 934800150b1945dcab01b4f7c4edcdc8 P f5b45ccea9c7438f8fba1b79073969e5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250112 15:08:48.134871 24721 raft_consensus.cc:491] T 934800150b1945dcab01b4f7c4edcdc8 P f5b45ccea9c7438f8fba1b79073969e5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250112 15:08:48.134894 24721 raft_consensus.cc:3054] T 934800150b1945dcab01b4f7c4edcdc8 P f5b45ccea9c7438f8fba1b79073969e5 [term 0 FOLLOWER]: Advancing to term 1
I20250112 15:08:48.135583 24721 raft_consensus.cc:513] T 934800150b1945dcab01b4f7c4edcdc8 P f5b45ccea9c7438f8fba1b79073969e5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52bd510945874fa781ec841d40e251ab" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 34851 } } peers { permanent_uuid: "56dd875b6ba54b40877146f15292bf2f" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 32969 } } peers { permanent_uuid: "f5b45ccea9c7438f8fba1b79073969e5" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 40795 } }
I20250112 15:08:48.135706 24721 leader_election.cc:290] T 934800150b1945dcab01b4f7c4edcdc8 P f5b45ccea9c7438f8fba1b79073969e5 [CANDIDATE]: Term 1 election: Requested vote from peers 52bd510945874fa781ec841d40e251ab (127.17.60.193:34851), 56dd875b6ba54b40877146f15292bf2f (127.17.60.194:32969)
I20250112 15:08:48.135843 24500 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "934800150b1945dcab01b4f7c4edcdc8" candidate_uuid: "f5b45ccea9c7438f8fba1b79073969e5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "52bd510945874fa781ec841d40e251ab"
I20250112 15:08:48.135859 24576 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "934800150b1945dcab01b4f7c4edcdc8" candidate_uuid: "f5b45ccea9c7438f8fba1b79073969e5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "56dd875b6ba54b40877146f15292bf2f"
I20250112 15:08:48.135921 24576 raft_consensus.cc:3054] T 934800150b1945dcab01b4f7c4edcdc8 P 56dd875b6ba54b40877146f15292bf2f [term 0 FOLLOWER]: Advancing to term 1
I20250112 15:08:48.135936 24500 raft_consensus.cc:3054] T 934800150b1945dcab01b4f7c4edcdc8 P 52bd510945874fa781ec841d40e251ab [term 0 FOLLOWER]: Advancing to term 1
I20250112 15:08:48.136545 24576 raft_consensus.cc:2463] T 934800150b1945dcab01b4f7c4edcdc8 P 56dd875b6ba54b40877146f15292bf2f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f5b45ccea9c7438f8fba1b79073969e5 in term 1.
I20250112 15:08:48.136545 24500 raft_consensus.cc:2463] T 934800150b1945dcab01b4f7c4edcdc8 P 52bd510945874fa781ec841d40e251ab [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f5b45ccea9c7438f8fba1b79073969e5 in term 1.
I20250112 15:08:48.136785 24614 leader_election.cc:304] T 934800150b1945dcab01b4f7c4edcdc8 P f5b45ccea9c7438f8fba1b79073969e5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 56dd875b6ba54b40877146f15292bf2f, f5b45ccea9c7438f8fba1b79073969e5; no voters:
I20250112 15:08:48.136870 24721 raft_consensus.cc:2798] T 934800150b1945dcab01b4f7c4edcdc8 P f5b45ccea9c7438f8fba1b79073969e5 [term 1 FOLLOWER]: Leader election won for term 1
I20250112 15:08:48.136924 24721 raft_consensus.cc:695] T 934800150b1945dcab01b4f7c4edcdc8 P f5b45ccea9c7438f8fba1b79073969e5 [term 1 LEADER]: Becoming Leader. State: Replica: f5b45ccea9c7438f8fba1b79073969e5, State: Running, Role: LEADER
I20250112 15:08:48.136976 24721 consensus_queue.cc:237] T 934800150b1945dcab01b4f7c4edcdc8 P f5b45ccea9c7438f8fba1b79073969e5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52bd510945874fa781ec841d40e251ab" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 34851 } } peers { permanent_uuid: "56dd875b6ba54b40877146f15292bf2f" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 32969 } } peers { permanent_uuid: "f5b45ccea9c7438f8fba1b79073969e5" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 40795 } }
I20250112 15:08:48.137522 24397 catalog_manager.cc:5526] T 934800150b1945dcab01b4f7c4edcdc8 P f5b45ccea9c7438f8fba1b79073969e5 reported cstate change: term changed from 0 to 1, leader changed from <none> to f5b45ccea9c7438f8fba1b79073969e5 (127.17.60.195). New cstate: current_term: 1 leader_uuid: "f5b45ccea9c7438f8fba1b79073969e5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52bd510945874fa781ec841d40e251ab" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 34851 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "56dd875b6ba54b40877146f15292bf2f" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 32969 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f5b45ccea9c7438f8fba1b79073969e5" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 40795 } health_report { overall_health: HEALTHY } } }
I20250112 15:08:48.171204 17651 tablet_server.cc:178] TabletServer@127.17.60.193:0 shutting down...
I20250112 15:08:48.173897 17651 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250112 15:08:48.174074 17651 tablet_replica.cc:331] T 6914949901e147f2970585c79458b9d2 P 52bd510945874fa781ec841d40e251ab: stopping tablet replica
I20250112 15:08:48.174160 17651 raft_consensus.cc:2238] T 6914949901e147f2970585c79458b9d2 P 52bd510945874fa781ec841d40e251ab [term 1 FOLLOWER]: Raft consensus shutting down.
I20250112 15:08:48.174206 17651 raft_consensus.cc:2267] T 6914949901e147f2970585c79458b9d2 P 52bd510945874fa781ec841d40e251ab [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 15:08:48.174379 17651 tablet_replica.cc:331] T a3cc29d2a75b4778b21400e0e0b64a9e P 52bd510945874fa781ec841d40e251ab: stopping tablet replica
I20250112 15:08:48.174414 17651 raft_consensus.cc:2238] T a3cc29d2a75b4778b21400e0e0b64a9e P 52bd510945874fa781ec841d40e251ab [term 1 FOLLOWER]: Raft consensus shutting down.
I20250112 15:08:48.174443 17651 raft_consensus.cc:2267] T a3cc29d2a75b4778b21400e0e0b64a9e P 52bd510945874fa781ec841d40e251ab [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 15:08:48.174597 17651 tablet_replica.cc:331] T c73b4e40028649e5bb9ad937f307e7bf P 52bd510945874fa781ec841d40e251ab: stopping tablet replica
I20250112 15:08:48.174633 17651 raft_consensus.cc:2238] T c73b4e40028649e5bb9ad937f307e7bf P 52bd510945874fa781ec841d40e251ab [term 1 FOLLOWER]: Raft consensus shutting down.
I20250112 15:08:48.174661 17651 raft_consensus.cc:2267] T c73b4e40028649e5bb9ad937f307e7bf P 52bd510945874fa781ec841d40e251ab [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 15:08:48.174860 17651 tablet_replica.cc:331] T 934800150b1945dcab01b4f7c4edcdc8 P 52bd510945874fa781ec841d40e251ab: stopping tablet replica
I20250112 15:08:48.174902 17651 raft_consensus.cc:2238] T 934800150b1945dcab01b4f7c4edcdc8 P 52bd510945874fa781ec841d40e251ab [term 1 FOLLOWER]: Raft consensus shutting down.
I20250112 15:08:48.174934 17651 raft_consensus.cc:2267] T 934800150b1945dcab01b4f7c4edcdc8 P 52bd510945874fa781ec841d40e251ab [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 15:08:48.175081 17651 tablet_replica.cc:331] T 45d29678d6704a888f2c2514af2f36a4 P 52bd510945874fa781ec841d40e251ab: stopping tablet replica
I20250112 15:08:48.175122 17651 raft_consensus.cc:2238] T 45d29678d6704a888f2c2514af2f36a4 P 52bd510945874fa781ec841d40e251ab [term 1 FOLLOWER]: Raft consensus shutting down.
I20250112 15:08:48.175168 17651 raft_consensus.cc:2267] T 45d29678d6704a888f2c2514af2f36a4 P 52bd510945874fa781ec841d40e251ab [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 15:08:48.175349 17651 tablet_replica.cc:331] T 94172f983ecd4a37956654f2f86dba00 P 52bd510945874fa781ec841d40e251ab: stopping tablet replica
I20250112 15:08:48.175391 17651 raft_consensus.cc:2238] T 94172f983ecd4a37956654f2f86dba00 P 52bd510945874fa781ec841d40e251ab [term 1 LEADER]: Raft consensus shutting down.
I20250112 15:08:48.175437 17651 raft_consensus.cc:2267] T 94172f983ecd4a37956654f2f86dba00 P 52bd510945874fa781ec841d40e251ab [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 15:08:48.175609 17651 tablet_replica.cc:331] T cc5c837c3de946a5952f656946d883c8 P 52bd510945874fa781ec841d40e251ab: stopping tablet replica
I20250112 15:08:48.175647 17651 raft_consensus.cc:2238] T cc5c837c3de946a5952f656946d883c8 P 52bd510945874fa781ec841d40e251ab [term 1 FOLLOWER]: Raft consensus shutting down.
I20250112 15:08:48.175671 17651 raft_consensus.cc:2267] T cc5c837c3de946a5952f656946d883c8 P 52bd510945874fa781ec841d40e251ab [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 15:08:48.175741 17651 tablet_replica.cc:331] T bf900db2a1344c7b8fd0b9ba5e225201 P 52bd510945874fa781ec841d40e251ab: stopping tablet replica
I20250112 15:08:48.175765 17651 raft_consensus.cc:2238] T bf900db2a1344c7b8fd0b9ba5e225201 P 52bd510945874fa781ec841d40e251ab [term 1 FOLLOWER]: Raft consensus shutting down.
I20250112 15:08:48.175786 17651 raft_consensus.cc:2267] T bf900db2a1344c7b8fd0b9ba5e225201 P 52bd510945874fa781ec841d40e251ab [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 15:08:48.177552 17651 tablet_server.cc:195] TabletServer@127.17.60.193:0 shutdown complete.
I20250112 15:08:48.349967 24721 consensus_queue.cc:1035] T cc5c837c3de946a5952f656946d883c8 P f5b45ccea9c7438f8fba1b79073969e5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "56dd875b6ba54b40877146f15292bf2f" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 32969 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20250112 15:08:48.351738 24614 consensus_peers.cc:487] T cc5c837c3de946a5952f656946d883c8 P f5b45ccea9c7438f8fba1b79073969e5 -> Peer 52bd510945874fa781ec841d40e251ab (127.17.60.193:34851): Couldn't send request to peer 52bd510945874fa781ec841d40e251ab. Status: Network error: Client connection negotiation failed: client connection to 127.17.60.193:34851: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20250112 15:08:48.473609 24614 consensus_peers.cc:487] T 45d29678d6704a888f2c2514af2f36a4 P f5b45ccea9c7438f8fba1b79073969e5 -> Peer 52bd510945874fa781ec841d40e251ab (127.17.60.193:34851): Couldn't send request to peer 52bd510945874fa781ec841d40e251ab. Status: Network error: Client connection negotiation failed: client connection to 127.17.60.193:34851: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20250112 15:08:48.498400 24539 consensus_peers.cc:487] T bf900db2a1344c7b8fd0b9ba5e225201 P 56dd875b6ba54b40877146f15292bf2f -> Peer 52bd510945874fa781ec841d40e251ab (127.17.60.193:34851): Couldn't send request to peer 52bd510945874fa781ec841d40e251ab. Status: Network error: Client connection negotiation failed: client connection to 127.17.60.193:34851: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20250112 15:08:48.509310 24744 consensus_queue.cc:1035] T bf900db2a1344c7b8fd0b9ba5e225201 P 56dd875b6ba54b40877146f15292bf2f [LEADER]: Connected to new peer: Peer: permanent_uuid: "f5b45ccea9c7438f8fba1b79073969e5" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 40795 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20250112 15:08:48.538676 24539 consensus_peers.cc:487] T 6914949901e147f2970585c79458b9d2 P 56dd875b6ba54b40877146f15292bf2f -> Peer 52bd510945874fa781ec841d40e251ab (127.17.60.193:34851): Couldn't send request to peer 52bd510945874fa781ec841d40e251ab. Status: Network error: Client connection negotiation failed: client connection to 127.17.60.193:34851: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20250112 15:08:48.552358 24744 consensus_queue.cc:1035] T 6914949901e147f2970585c79458b9d2 P 56dd875b6ba54b40877146f15292bf2f [LEADER]: Connected to new peer: Peer: permanent_uuid: "f5b45ccea9c7438f8fba1b79073969e5" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 40795 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20250112 15:08:48.553932 24614 consensus_peers.cc:487] T c73b4e40028649e5bb9ad937f307e7bf P f5b45ccea9c7438f8fba1b79073969e5 -> Peer 52bd510945874fa781ec841d40e251ab (127.17.60.193:34851): Couldn't send request to peer 52bd510945874fa781ec841d40e251ab. Status: Network error: Client connection negotiation failed: client connection to 127.17.60.193:34851: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20250112 15:08:48.607981 24539 consensus_peers.cc:487] T a3cc29d2a75b4778b21400e0e0b64a9e P 56dd875b6ba54b40877146f15292bf2f -> Peer 52bd510945874fa781ec841d40e251ab (127.17.60.193:34851): Couldn't send request to peer 52bd510945874fa781ec841d40e251ab. Status: Network error: Client connection negotiation failed: client connection to 127.17.60.193:34851: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20250112 15:08:48.679162 24744 consensus_queue.cc:1035] T a3cc29d2a75b4778b21400e0e0b64a9e P 56dd875b6ba54b40877146f15292bf2f [LEADER]: Connected to new peer: Peer: permanent_uuid: "f5b45ccea9c7438f8fba1b79073969e5" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 40795 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250112 15:08:48.715342 24721 consensus_queue.cc:1035] T 934800150b1945dcab01b4f7c4edcdc8 P f5b45ccea9c7438f8fba1b79073969e5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "56dd875b6ba54b40877146f15292bf2f" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 32969 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20250112 15:08:48.716908 24614 consensus_peers.cc:487] T 934800150b1945dcab01b4f7c4edcdc8 P f5b45ccea9c7438f8fba1b79073969e5 -> Peer 52bd510945874fa781ec841d40e251ab (127.17.60.193:34851): Couldn't send request to peer 52bd510945874fa781ec841d40e251ab. Status: Network error: Client connection negotiation failed: client connection to 127.17.60.193:34851: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20250112 15:08:49.126437 24756 consensus_queue.cc:579] T c73b4e40028649e5bb9ad937f307e7bf P f5b45ccea9c7438f8fba1b79073969e5 [LEADER]: Leader has been unable to successfully communicate with peer 52bd510945874fa781ec841d40e251ab for more than 1 seconds (1.038s)
I20250112 15:08:49.147322 24744 consensus_queue.cc:579] T 6914949901e147f2970585c79458b9d2 P 56dd875b6ba54b40877146f15292bf2f [LEADER]: Leader has been unable to successfully communicate with peer 52bd510945874fa781ec841d40e251ab for more than 1 seconds (1.175s)
I20250112 15:08:49.182262 24397 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:60596:
table { table_name: "dugtrio" } modify_external_catalogs: true
I20250112 15:08:49.182361 24397 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:60596:
table { table_name: "dugtrio" } modify_external_catalogs: true
I20250112 15:08:49.183434 24397 catalog_manager.cc:5813] T 00000000000000000000000000000000 P ea845a3e3060484f9ece4069e440908c: Sending DeleteTablet for 3 replicas of tablet bf900db2a1344c7b8fd0b9ba5e225201
I20250112 15:08:49.183530 24397 catalog_manager.cc:5813] T 00000000000000000000000000000000 P ea845a3e3060484f9ece4069e440908c: Sending DeleteTablet for 3 replicas of tablet a3cc29d2a75b4778b21400e0e0b64a9e
I20250112 15:08:49.183597 24397 catalog_manager.cc:5813] T 00000000000000000000000000000000 P ea845a3e3060484f9ece4069e440908c: Sending DeleteTablet for 3 replicas of tablet 6914949901e147f2970585c79458b9d2
I20250112 15:08:49.183643 24397 catalog_manager.cc:5813] T 00000000000000000000000000000000 P ea845a3e3060484f9ece4069e440908c: Sending DeleteTablet for 3 replicas of tablet 934800150b1945dcab01b4f7c4edcdc8
I20250112 15:08:49.183863 24565 tablet_service.cc:1514] Processing DeleteTablet for tablet bf900db2a1344c7b8fd0b9ba5e225201 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 15:08:49 UTC) from {username='slave'} at 127.0.0.1:49784
I20250112 15:08:49.183921 24638 tablet_service.cc:1514] Processing DeleteTablet for tablet bf900db2a1344c7b8fd0b9ba5e225201 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 15:08:49 UTC) from {username='slave'} at 127.0.0.1:35848
I20250112 15:08:49.184073 24639 tablet_service.cc:1514] Processing DeleteTablet for tablet 6914949901e147f2970585c79458b9d2 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 15:08:49 UTC) from {username='slave'} at 127.0.0.1:35848
I20250112 15:08:49.184098 24564 tablet_service.cc:1514] Processing DeleteTablet for tablet 934800150b1945dcab01b4f7c4edcdc8 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 15:08:49 UTC) from {username='slave'} at 127.0.0.1:49784
I20250112 15:08:49.184149 24641 tablet_service.cc:1514] Processing DeleteTablet for tablet a3cc29d2a75b4778b21400e0e0b64a9e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 15:08:49 UTC) from {username='slave'} at 127.0.0.1:35848
I20250112 15:08:49.184170 24776 tablet_replica.cc:331] T bf900db2a1344c7b8fd0b9ba5e225201 P 56dd875b6ba54b40877146f15292bf2f: stopping tablet replica
I20250112 15:08:49.184202 24640 tablet_service.cc:1514] Processing DeleteTablet for tablet 934800150b1945dcab01b4f7c4edcdc8 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 15:08:49 UTC) from {username='slave'} at 127.0.0.1:35848
I20250112 15:08:49.184263 24565 tablet_service.cc:1514] Processing DeleteTablet for tablet a3cc29d2a75b4778b21400e0e0b64a9e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 15:08:49 UTC) from {username='slave'} at 127.0.0.1:49784
I20250112 15:08:49.184217 24776 raft_consensus.cc:2238] T bf900db2a1344c7b8fd0b9ba5e225201 P 56dd875b6ba54b40877146f15292bf2f [term 1 LEADER]: Raft consensus shutting down.
I20250112 15:08:49.184307 24777 tablet_replica.cc:331] T bf900db2a1344c7b8fd0b9ba5e225201 P f5b45ccea9c7438f8fba1b79073969e5: stopping tablet replica
I20250112 15:08:49.184092 24563 tablet_service.cc:1514] Processing DeleteTablet for tablet 6914949901e147f2970585c79458b9d2 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 15:08:49 UTC) from {username='slave'} at 127.0.0.1:49784
I20250112 15:08:49.184116 17651 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20250112 15:08:49.184358 24777 raft_consensus.cc:2238] T bf900db2a1344c7b8fd0b9ba5e225201 P f5b45ccea9c7438f8fba1b79073969e5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250112 15:08:49.184408 24777 raft_consensus.cc:2267] T bf900db2a1344c7b8fd0b9ba5e225201 P f5b45ccea9c7438f8fba1b79073969e5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 15:08:49.184365 24776 raft_consensus.cc:2267] T bf900db2a1344c7b8fd0b9ba5e225201 P 56dd875b6ba54b40877146f15292bf2f [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 15:08:49.184695 24776 ts_tablet_manager.cc:1905] T bf900db2a1344c7b8fd0b9ba5e225201 P 56dd875b6ba54b40877146f15292bf2f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 15:08:49.184901 24777 ts_tablet_manager.cc:1905] T bf900db2a1344c7b8fd0b9ba5e225201 P f5b45ccea9c7438f8fba1b79073969e5: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250112 15:08:49.185858 24782 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250112 15:08:49.186300 24779 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250112 15:08:49.186679 24381 catalog_manager.cc:4615] TS 52bd510945874fa781ec841d40e251ab (127.17.60.193:34851): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet a3cc29d2a75b4778b21400e0e0b64a9e: Network error: Client connection negotiation failed: client connection to 127.17.60.193:34851: connect: Connection refused (error 111)
I20250112 15:08:49.186753 24777 ts_tablet_manager.cc:1918] T bf900db2a1344c7b8fd0b9ba5e225201 P f5b45ccea9c7438f8fba1b79073969e5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250112 15:08:49.186761 24776 ts_tablet_manager.cc:1918] T bf900db2a1344c7b8fd0b9ba5e225201 P 56dd875b6ba54b40877146f15292bf2f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250112 15:08:49.186822 24777 log.cc:1198] T bf900db2a1344c7b8fd0b9ba5e225201 P f5b45ccea9c7438f8fba1b79073969e5: Deleting WAL directory at /tmp/dist-test-task95dLSD/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestDeletedTables.1736694351070047-17651-0/minicluster-data/ts-2-root/wals/bf900db2a1344c7b8fd0b9ba5e225201
I20250112 15:08:49.186878 24776 log.cc:1198] T bf900db2a1344c7b8fd0b9ba5e225201 P 56dd875b6ba54b40877146f15292bf2f: Deleting WAL directory at /tmp/dist-test-task95dLSD/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestDeletedTables.1736694351070047-17651-0/minicluster-data/ts-1-root/wals/bf900db2a1344c7b8fd0b9ba5e225201
W20250112 15:08:49.187088 24778 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250112 15:08:49.187112 24777 ts_tablet_manager.cc:1939] T bf900db2a1344c7b8fd0b9ba5e225201 P f5b45ccea9c7438f8fba1b79073969e5: Deleting consensus metadata
I20250112 15:08:49.187176 24776 ts_tablet_manager.cc:1939] T bf900db2a1344c7b8fd0b9ba5e225201 P 56dd875b6ba54b40877146f15292bf2f: Deleting consensus metadata
I20250112 15:08:49.187386 24777 tablet_replica.cc:331] T 6914949901e147f2970585c79458b9d2 P f5b45ccea9c7438f8fba1b79073969e5: stopping tablet replica
I20250112 15:08:49.187402 24776 tablet_replica.cc:331] T 934800150b1945dcab01b4f7c4edcdc8 P 56dd875b6ba54b40877146f15292bf2f: stopping tablet replica
I20250112 15:08:49.187472 24381 catalog_manager.cc:4872] TS 56dd875b6ba54b40877146f15292bf2f (127.17.60.194:32969): tablet bf900db2a1344c7b8fd0b9ba5e225201 (table dugtrio [id=271cc77f05ad43eb89d1c5c6291536e4]) successfully deleted
I20250112 15:08:49.187500 24777 raft_consensus.cc:2238] T 6914949901e147f2970585c79458b9d2 P f5b45ccea9c7438f8fba1b79073969e5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250112 15:08:49.187575 24776 raft_consensus.cc:2238] T 934800150b1945dcab01b4f7c4edcdc8 P 56dd875b6ba54b40877146f15292bf2f [term 1 FOLLOWER]: Raft consensus shutting down.
I20250112 15:08:49.187577 24381 catalog_manager.cc:4872] TS f5b45ccea9c7438f8fba1b79073969e5 (127.17.60.195:40795): tablet bf900db2a1344c7b8fd0b9ba5e225201 (table dugtrio [id=271cc77f05ad43eb89d1c5c6291536e4]) successfully deleted
I20250112 15:08:49.187578 24777 raft_consensus.cc:2267] T 6914949901e147f2970585c79458b9d2 P f5b45ccea9c7438f8fba1b79073969e5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 15:08:49.187629 24776 raft_consensus.cc:2267] T 934800150b1945dcab01b4f7c4edcdc8 P 56dd875b6ba54b40877146f15292bf2f [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 15:08:49.187870 17651 server_base.cc:1034] running on GCE node
I20250112 15:08:49.187878 24777 ts_tablet_manager.cc:1905] T 6914949901e147f2970585c79458b9d2 P f5b45ccea9c7438f8fba1b79073969e5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 15:08:49.188067 17651 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
I20250112 15:08:49.188079 24776 ts_tablet_manager.cc:1905] T 934800150b1945dcab01b4f7c4edcdc8 P 56dd875b6ba54b40877146f15292bf2f: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250112 15:08:49.188112 17651 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250112 15:08:49.188140 17651 hybrid_clock.cc:648] HybridClock initialized: now 1736694529188140 us; error 0 us; skew 500 ppm
I20250112 15:08:49.189018 17651 webserver.cc:458] Webserver started at http://127.17.60.196:35661/ using document root <none> and password file <none>
I20250112 15:08:49.189100 17651 fs_manager.cc:362] Metadata directory not provided
I20250112 15:08:49.189101 24777 ts_tablet_manager.cc:1918] T 6914949901e147f2970585c79458b9d2 P f5b45ccea9c7438f8fba1b79073969e5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250112 15:08:49.189145 17651 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250112 15:08:49.189167 24777 log.cc:1198] T 6914949901e147f2970585c79458b9d2 P f5b45ccea9c7438f8fba1b79073969e5: Deleting WAL directory at /tmp/dist-test-task95dLSD/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestDeletedTables.1736694351070047-17651-0/minicluster-data/ts-2-root/wals/6914949901e147f2970585c79458b9d2
I20250112 15:08:49.189184 17651 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250112 15:08:49.189313 24777 ts_tablet_manager.cc:1939] T 6914949901e147f2970585c79458b9d2 P f5b45ccea9c7438f8fba1b79073969e5: Deleting consensus metadata
I20250112 15:08:49.189435 17651 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task95dLSD/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestDeletedTables.1736694351070047-17651-0/minicluster-data/ts-3-root/instance:
uuid: "a422b4873cf04ae9b9bc6530f2f077a2"
format_stamp: "Formatted at 2025-01-12 15:08:49 on dist-test-slave-b4nz"
I20250112 15:08:49.189502 24776 ts_tablet_manager.cc:1918] T 934800150b1945dcab01b4f7c4edcdc8 P 56dd875b6ba54b40877146f15292bf2f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250112 15:08:49.189544 24776 log.cc:1198] T 934800150b1945dcab01b4f7c4edcdc8 P 56dd875b6ba54b40877146f15292bf2f: Deleting WAL directory at /tmp/dist-test-task95dLSD/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestDeletedTables.1736694351070047-17651-0/minicluster-data/ts-1-root/wals/934800150b1945dcab01b4f7c4edcdc8
I20250112 15:08:49.189749 24777 tablet_replica.cc:331] T a3cc29d2a75b4778b21400e0e0b64a9e P f5b45ccea9c7438f8fba1b79073969e5: stopping tablet replica
I20250112 15:08:49.189772 24776 ts_tablet_manager.cc:1939] T 934800150b1945dcab01b4f7c4edcdc8 P 56dd875b6ba54b40877146f15292bf2f: Deleting consensus metadata
I20250112 15:08:49.189807 24777 raft_consensus.cc:2238] T a3cc29d2a75b4778b21400e0e0b64a9e P f5b45ccea9c7438f8fba1b79073969e5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250112 15:08:49.189837 24381 catalog_manager.cc:4872] TS f5b45ccea9c7438f8fba1b79073969e5 (127.17.60.195:40795): tablet 6914949901e147f2970585c79458b9d2 (table dugtrio [id=271cc77f05ad43eb89d1c5c6291536e4]) successfully deleted
I20250112 15:08:49.189846 24777 raft_consensus.cc:2267] T a3cc29d2a75b4778b21400e0e0b64a9e P f5b45ccea9c7438f8fba1b79073969e5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 15:08:49.190119 24776 tablet_replica.cc:331] T a3cc29d2a75b4778b21400e0e0b64a9e P 56dd875b6ba54b40877146f15292bf2f: stopping tablet replica
I20250112 15:08:49.190116 24777 ts_tablet_manager.cc:1905] T a3cc29d2a75b4778b21400e0e0b64a9e P f5b45ccea9c7438f8fba1b79073969e5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 15:08:49.190189 24776 raft_consensus.cc:2238] T a3cc29d2a75b4778b21400e0e0b64a9e P 56dd875b6ba54b40877146f15292bf2f [term 1 LEADER]: Raft consensus shutting down.
I20250112 15:08:49.190191 24381 catalog_manager.cc:4872] TS 56dd875b6ba54b40877146f15292bf2f (127.17.60.194:32969): tablet 934800150b1945dcab01b4f7c4edcdc8 (table dugtrio [id=271cc77f05ad43eb89d1c5c6291536e4]) successfully deleted
I20250112 15:08:49.190253 24776 raft_consensus.cc:2267] T a3cc29d2a75b4778b21400e0e0b64a9e P 56dd875b6ba54b40877146f15292bf2f [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 15:08:49.190443 24776 ts_tablet_manager.cc:1905] T a3cc29d2a75b4778b21400e0e0b64a9e P 56dd875b6ba54b40877146f15292bf2f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 15:08:49.190529 17651 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20250112 15:08:49.191001 24787 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250112 15:08:49.191102 17651 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20250112 15:08:49.191140 17651 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task95dLSD/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestDeletedTables.1736694351070047-17651-0/minicluster-data/ts-3-root
uuid: "a422b4873cf04ae9b9bc6530f2f077a2"
format_stamp: "Formatted at 2025-01-12 15:08:49 on dist-test-slave-b4nz"
I20250112 15:08:49.191198 17651 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task95dLSD/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestDeletedTables.1736694351070047-17651-0/minicluster-data/ts-3-root
metadata directory: /tmp/dist-test-task95dLSD/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestDeletedTables.1736694351070047-17651-0/minicluster-data/ts-3-root
1 data directories: /tmp/dist-test-task95dLSD/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestDeletedTables.1736694351070047-17651-0/minicluster-data/ts-3-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250112 15:08:49.191310 24777 ts_tablet_manager.cc:1918] T a3cc29d2a75b4778b21400e0e0b64a9e P f5b45ccea9c7438f8fba1b79073969e5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250112 15:08:49.191354 24777 log.cc:1198] T a3cc29d2a75b4778b21400e0e0b64a9e P f5b45ccea9c7438f8fba1b79073969e5: Deleting WAL directory at /tmp/dist-test-task95dLSD/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestDeletedTables.1736694351070047-17651-0/minicluster-data/ts-2-root/wals/a3cc29d2a75b4778b21400e0e0b64a9e
I20250112 15:08:49.191545 24777 ts_tablet_manager.cc:1939] T a3cc29d2a75b4778b21400e0e0b64a9e P f5b45ccea9c7438f8fba1b79073969e5: Deleting consensus metadata
I20250112 15:08:49.191684 24776 ts_tablet_manager.cc:1918] T a3cc29d2a75b4778b21400e0e0b64a9e P 56dd875b6ba54b40877146f15292bf2f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250112 15:08:49.191722 24776 log.cc:1198] T a3cc29d2a75b4778b21400e0e0b64a9e P 56dd875b6ba54b40877146f15292bf2f: Deleting WAL directory at /tmp/dist-test-task95dLSD/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestDeletedTables.1736694351070047-17651-0/minicluster-data/ts-1-root/wals/a3cc29d2a75b4778b21400e0e0b64a9e
I20250112 15:08:49.191871 24381 catalog_manager.cc:4872] TS f5b45ccea9c7438f8fba1b79073969e5 (127.17.60.195:40795): tablet a3cc29d2a75b4778b21400e0e0b64a9e (table dugtrio [id=271cc77f05ad43eb89d1c5c6291536e4]) successfully deleted
I20250112 15:08:49.191938 24777 tablet_replica.cc:331] T 934800150b1945dcab01b4f7c4edcdc8 P f5b45ccea9c7438f8fba1b79073969e5: stopping tablet replica
I20250112 15:08:49.191941 24776 ts_tablet_manager.cc:1939] T a3cc29d2a75b4778b21400e0e0b64a9e P 56dd875b6ba54b40877146f15292bf2f: Deleting consensus metadata
I20250112 15:08:49.191994 24777 raft_consensus.cc:2238] T 934800150b1945dcab01b4f7c4edcdc8 P f5b45ccea9c7438f8fba1b79073969e5 [term 1 LEADER]: Raft consensus shutting down.
I20250112 15:08:49.192059 24777 raft_consensus.cc:2267] T 934800150b1945dcab01b4f7c4edcdc8 P f5b45ccea9c7438f8fba1b79073969e5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 15:08:49.192271 24777 ts_tablet_manager.cc:1905] T 934800150b1945dcab01b4f7c4edcdc8 P f5b45ccea9c7438f8fba1b79073969e5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 15:08:49.192349 24776 tablet_replica.cc:331] T 6914949901e147f2970585c79458b9d2 P 56dd875b6ba54b40877146f15292bf2f: stopping tablet replica
I20250112 15:08:49.192392 24776 raft_consensus.cc:2238] T 6914949901e147f2970585c79458b9d2 P 56dd875b6ba54b40877146f15292bf2f [term 1 LEADER]: Raft consensus shutting down.
I20250112 15:08:49.192426 24381 catalog_manager.cc:4872] TS 56dd875b6ba54b40877146f15292bf2f (127.17.60.194:32969): tablet a3cc29d2a75b4778b21400e0e0b64a9e (table dugtrio [id=271cc77f05ad43eb89d1c5c6291536e4]) successfully deleted
I20250112 15:08:49.192445 24776 raft_consensus.cc:2267] T 6914949901e147f2970585c79458b9d2 P 56dd875b6ba54b40877146f15292bf2f [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 15:08:49.192708 24776 ts_tablet_manager.cc:1905] T 6914949901e147f2970585c79458b9d2 P 56dd875b6ba54b40877146f15292bf2f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 15:08:49.193789 24777 ts_tablet_manager.cc:1918] T 934800150b1945dcab01b4f7c4edcdc8 P f5b45ccea9c7438f8fba1b79073969e5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250112 15:08:49.193897 24776 ts_tablet_manager.cc:1918] T 6914949901e147f2970585c79458b9d2 P 56dd875b6ba54b40877146f15292bf2f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250112 15:08:49.193915 24777 log.cc:1198] T 934800150b1945dcab01b4f7c4edcdc8 P f5b45ccea9c7438f8fba1b79073969e5: Deleting WAL directory at /tmp/dist-test-task95dLSD/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestDeletedTables.1736694351070047-17651-0/minicluster-data/ts-2-root/wals/934800150b1945dcab01b4f7c4edcdc8
I20250112 15:08:49.193977 24776 log.cc:1198] T 6914949901e147f2970585c79458b9d2 P 56dd875b6ba54b40877146f15292bf2f: Deleting WAL directory at /tmp/dist-test-task95dLSD/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestDeletedTables.1736694351070047-17651-0/minicluster-data/ts-1-root/wals/6914949901e147f2970585c79458b9d2
I20250112 15:08:49.194149 24777 ts_tablet_manager.cc:1939] T 934800150b1945dcab01b4f7c4edcdc8 P f5b45ccea9c7438f8fba1b79073969e5: Deleting consensus metadata
I20250112 15:08:49.194175 24776 ts_tablet_manager.cc:1939] T 6914949901e147f2970585c79458b9d2 P 56dd875b6ba54b40877146f15292bf2f: Deleting consensus metadata
I20250112 15:08:49.194463 24381 catalog_manager.cc:4872] TS 56dd875b6ba54b40877146f15292bf2f (127.17.60.194:32969): tablet 6914949901e147f2970585c79458b9d2 (table dugtrio [id=271cc77f05ad43eb89d1c5c6291536e4]) successfully deleted
I20250112 15:08:49.194561 24381 catalog_manager.cc:4872] TS f5b45ccea9c7438f8fba1b79073969e5 (127.17.60.195:40795): tablet 934800150b1945dcab01b4f7c4edcdc8 (table dugtrio [id=271cc77f05ad43eb89d1c5c6291536e4]) successfully deleted
I20250112 15:08:49.202792 17651 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250112 15:08:49.202986 17651 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250112 15:08:49.203125 17651 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250112 15:08:49.203277 17651 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250112 15:08:49.203302 17651 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250112 15:08:49.203334 17651 ts_tablet_manager.cc:610] Registered 0 tablets
I20250112 15:08:49.203358 17651 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250112 15:08:49.205817 17651 rpc_server.cc:307] RPC server started. Bound to: 127.17.60.196:33207
I20250112 15:08:49.206377 17651 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20250112 15:08:49.206391 24850 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.17.60.196:33207 every 8 connection(s)
W20250112 15:08:49.207679 24854 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250112 15:08:49.208439 24857 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250112 15:08:49.209192 24855 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250112 15:08:49.209462 24851 heartbeater.cc:346] Connected to a master server at 127.17.60.254:37937
I20250112 15:08:49.209507 24851 heartbeater.cc:463] Registering TS with master...
I20250112 15:08:49.209597 24851 heartbeater.cc:510] Master 127.17.60.254:37937 requested a full tablet report, sending...
I20250112 15:08:49.209844 24397 ts_manager.cc:194] Registered new tserver with Master: a422b4873cf04ae9b9bc6530f2f077a2 (127.17.60.196:33207)
I20250112 15:08:49.209849 17651 server_base.cc:1034] running on GCE node
I20250112 15:08:49.209964 17651 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250112 15:08:49.209995 17651 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250112 15:08:49.210017 17651 hybrid_clock.cc:648] HybridClock initialized: now 1736694529210017 us; error 0 us; skew 500 ppm
I20250112 15:08:49.210283 24397 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:60620
I20250112 15:08:49.210742 17651 webserver.cc:458] Webserver started at http://127.17.60.193:43709/ using document root <none> and password file <none>
I20250112 15:08:49.210805 17651 fs_manager.cc:362] Metadata directory not provided
I20250112 15:08:49.210830 17651 fs_manager.cc:365] Using existing metadata directory in first data directory
I20250112 15:08:49.211335 17651 fs_manager.cc:714] Time spent opening directory manager: real 0.000s user 0.000s sys 0.000s
I20250112 15:08:49.212050 24862 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250112 15:08:49.212172 17651 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20250112 15:08:49.212217 17651 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task95dLSD/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestDeletedTables.1736694351070047-17651-0/minicluster-data/ts-0-root
uuid: "52bd510945874fa781ec841d40e251ab"
format_stamp: "Formatted at 2025-01-12 15:08:46 on dist-test-slave-b4nz"
I20250112 15:08:49.212289 17651 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task95dLSD/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestDeletedTables.1736694351070047-17651-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-task95dLSD/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestDeletedTables.1736694351070047-17651-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-task95dLSD/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestDeletedTables.1736694351070047-17651-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250112 15:08:49.227833 17651 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250112 15:08:49.228044 17651 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250112 15:08:49.228168 17651 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250112 15:08:49.228443 24869 ts_tablet_manager.cc:536] Loading tablet metadata (0/8 complete)
I20250112 15:08:49.230422 17651 ts_tablet_manager.cc:579] Loaded tablet metadata (8 total tablets, 8 live tablets)
I20250112 15:08:49.230477 17651 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20250112 15:08:49.230518 17651 ts_tablet_manager.cc:594] Registering tablets (0/8 complete)
I20250112 15:08:49.230960 24869 tablet_bootstrap.cc:492] T a3cc29d2a75b4778b21400e0e0b64a9e P 52bd510945874fa781ec841d40e251ab: Bootstrap starting.
I20250112 15:08:49.231876 24869 tablet_bootstrap.cc:492] T a3cc29d2a75b4778b21400e0e0b64a9e P 52bd510945874fa781ec841d40e251ab: Bootstrap replayed 1/1 log segments. Stats: ops{read=0 overwritten=0 applied=0 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250112 15:08:49.232036 24869 tablet_bootstrap.cc:492] T a3cc29d2a75b4778b21400e0e0b64a9e P 52bd510945874fa781ec841d40e251ab: Bootstrap complete.
I20250112 15:08:49.232146 24869 ts_tablet_manager.cc:1397] T a3cc29d2a75b4778b21400e0e0b64a9e P 52bd510945874fa781ec841d40e251ab: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20250112 15:08:49.232266 24869 raft_consensus.cc:357] T a3cc29d2a75b4778b21400e0e0b64a9e P 52bd510945874fa781ec841d40e251ab [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52bd510945874fa781ec841d40e251ab" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 34851 } } peers { permanent_uuid: "56dd875b6ba54b40877146f15292bf2f" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 32969 } } peers { permanent_uuid: "f5b45ccea9c7438f8fba1b79073969e5" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 40795 } }
I20250112 15:08:49.232339 24869 raft_consensus.cc:738] T a3cc29d2a75b4778b21400e0e0b64a9e P 52bd510945874fa781ec841d40e251ab [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 52bd510945874fa781ec841d40e251ab, State: Initialized, Role: FOLLOWER
I20250112 15:08:49.232383 24869 consensus_queue.cc:260] T a3cc29d2a75b4778b21400e0e0b64a9e P 52bd510945874fa781ec841d40e251ab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52bd510945874fa781ec841d40e251ab" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 34851 } } peers { permanent_uuid: "56dd875b6ba54b40877146f15292bf2f" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 32969 } } peers { permanent_uuid: "f5b45ccea9c7438f8fba1b79073969e5" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 40795 } }
I20250112 15:08:49.232386 17651 ts_tablet_manager.cc:610] Registered 8 tablets
I20250112 15:08:49.232466 17651 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.002s sys 0.000s
I20250112 15:08:49.232583 24869 ts_tablet_manager.cc:1428] T a3cc29d2a75b4778b21400e0e0b64a9e P 52bd510945874fa781ec841d40e251ab: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20250112 15:08:49.232650 24869 tablet_bootstrap.cc:492] T cc5c837c3de946a5952f656946d883c8 P 52bd510945874fa781ec841d40e251ab: Bootstrap starting.
I20250112 15:08:49.233418 24869 tablet_bootstrap.cc:492] T cc5c837c3de946a5952f656946d883c8 P 52bd510945874fa781ec841d40e251ab: Bootstrap replayed 1/1 log segments. Stats: ops{read=0 overwritten=0 applied=0 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250112 15:08:49.233582 24869 tablet_bootstrap.cc:492] T cc5c837c3de946a5952f656946d883c8 P 52bd510945874fa781ec841d40e251ab: Bootstrap complete.
I20250112 15:08:49.233686 24869 ts_tablet_manager.cc:1397] T cc5c837c3de946a5952f656946d883c8 P 52bd510945874fa781ec841d40e251ab: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20250112 15:08:49.233803 24869 raft_consensus.cc:357] T cc5c837c3de946a5952f656946d883c8 P 52bd510945874fa781ec841d40e251ab [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52bd510945874fa781ec841d40e251ab" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 34851 } } peers { permanent_uuid: "56dd875b6ba54b40877146f15292bf2f" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 32969 } } peers { permanent_uuid: "f5b45ccea9c7438f8fba1b79073969e5" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 40795 } }
I20250112 15:08:49.233868 24869 raft_consensus.cc:738] T cc5c837c3de946a5952f656946d883c8 P 52bd510945874fa781ec841d40e251ab [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 52bd510945874fa781ec841d40e251ab, State: Initialized, Role: FOLLOWER
I20250112 15:08:49.233920 24869 consensus_queue.cc:260] T cc5c837c3de946a5952f656946d883c8 P 52bd510945874fa781ec841d40e251ab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52bd510945874fa781ec841d40e251ab" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 34851 } } peers { permanent_uuid: "56dd875b6ba54b40877146f15292bf2f" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 32969 } } peers { permanent_uuid: "f5b45ccea9c7438f8fba1b79073969e5" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 40795 } }
I20250112 15:08:49.234009 24869 ts_tablet_manager.cc:1428] T cc5c837c3de946a5952f656946d883c8 P 52bd510945874fa781ec841d40e251ab: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20250112 15:08:49.234054 24869 tablet_bootstrap.cc:492] T 6914949901e147f2970585c79458b9d2 P 52bd510945874fa781ec841d40e251ab: Bootstrap starting.
I20250112 15:08:49.234972 24869 tablet_bootstrap.cc:492] T 6914949901e147f2970585c79458b9d2 P 52bd510945874fa781ec841d40e251ab: Bootstrap replayed 1/1 log segments. Stats: ops{read=0 overwritten=0 applied=0 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250112 15:08:49.235144 24869 tablet_bootstrap.cc:492] T 6914949901e147f2970585c79458b9d2 P 52bd510945874fa781ec841d40e251ab: Bootstrap complete.
I20250112 15:08:49.235249 24869 ts_tablet_manager.cc:1397] T 6914949901e147f2970585c79458b9d2 P 52bd510945874fa781ec841d40e251ab: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20250112 15:08:49.235363 24869 raft_consensus.cc:357] T 6914949901e147f2970585c79458b9d2 P 52bd510945874fa781ec841d40e251ab [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52bd510945874fa781ec841d40e251ab" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 34851 } } peers { permanent_uuid: "56dd875b6ba54b40877146f15292bf2f" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 32969 } } peers { permanent_uuid: "f5b45ccea9c7438f8fba1b79073969e5" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 40795 } }
I20250112 15:08:49.235420 24869 raft_consensus.cc:738] T 6914949901e147f2970585c79458b9d2 P 52bd510945874fa781ec841d40e251ab [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 52bd510945874fa781ec841d40e251ab, State: Initialized, Role: FOLLOWER
I20250112 15:08:49.235464 24869 consensus_queue.cc:260] T 6914949901e147f2970585c79458b9d2 P 52bd510945874fa781ec841d40e251ab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52bd510945874fa781ec841d40e251ab" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 34851 } } peers { permanent_uuid: "56dd875b6ba54b40877146f15292bf2f" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 32969 } } peers { permanent_uuid: "f5b45ccea9c7438f8fba1b79073969e5" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 40795 } }
I20250112 15:08:49.235549 24869 ts_tablet_manager.cc:1428] T 6914949901e147f2970585c79458b9d2 P 52bd510945874fa781ec841d40e251ab: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20250112 15:08:49.235591 24869 tablet_bootstrap.cc:492] T 45d29678d6704a888f2c2514af2f36a4 P 52bd510945874fa781ec841d40e251ab: Bootstrap starting.
I20250112 15:08:49.235778 17651 rpc_server.cc:307] RPC server started. Bound to: 127.17.60.193:34851
I20250112 15:08:49.235839 24930 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.17.60.193:34851 every 8 connection(s)
I20250112 15:08:49.236688 24869 tablet_bootstrap.cc:492] T 45d29678d6704a888f2c2514af2f36a4 P 52bd510945874fa781ec841d40e251ab: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250112 15:08:49.236918 24869 tablet_bootstrap.cc:492] T 45d29678d6704a888f2c2514af2f36a4 P 52bd510945874fa781ec841d40e251ab: Bootstrap complete.
I20250112 15:08:49.236954 24651 consensus_queue.cc:237] T c73b4e40028649e5bb9ad937f307e7bf P f5b45ccea9c7438f8fba1b79073969e5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "56dd875b6ba54b40877146f15292bf2f" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 32969 } } peers { permanent_uuid: "52bd510945874fa781ec841d40e251ab" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 34851 } } peers { permanent_uuid: "f5b45ccea9c7438f8fba1b79073969e5" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 40795 } } peers { permanent_uuid: "a422b4873cf04ae9b9bc6530f2f077a2" member_type: NON_VOTER last_known_addr { host: "127.17.60.196" port: 33207 } attrs { promote: true } }
I20250112 15:08:49.237017 24869 ts_tablet_manager.cc:1397] T 45d29678d6704a888f2c2514af2f36a4 P 52bd510945874fa781ec841d40e251ab: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20250112 15:08:49.237187 24869 raft_consensus.cc:357] T 45d29678d6704a888f2c2514af2f36a4 P 52bd510945874fa781ec841d40e251ab [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52bd510945874fa781ec841d40e251ab" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 34851 } } peers { permanent_uuid: "56dd875b6ba54b40877146f15292bf2f" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 32969 } } peers { permanent_uuid: "f5b45ccea9c7438f8fba1b79073969e5" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 40795 } }
I20250112 15:08:49.237252 24869 raft_consensus.cc:738] T 45d29678d6704a888f2c2514af2f36a4 P 52bd510945874fa781ec841d40e251ab [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 52bd510945874fa781ec841d40e251ab, State: Initialized, Role: FOLLOWER
I20250112 15:08:49.237306 24869 consensus_queue.cc:260] T 45d29678d6704a888f2c2514af2f36a4 P 52bd510945874fa781ec841d40e251ab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52bd510945874fa781ec841d40e251ab" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 34851 } } peers { permanent_uuid: "56dd875b6ba54b40877146f15292bf2f" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 32969 } } peers { permanent_uuid: "f5b45ccea9c7438f8fba1b79073969e5" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 40795 } }
I20250112 15:08:49.237387 24869 ts_tablet_manager.cc:1428] T 45d29678d6704a888f2c2514af2f36a4 P 52bd510945874fa781ec841d40e251ab: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20250112 15:08:49.237435 24869 tablet_bootstrap.cc:492] T c73b4e40028649e5bb9ad937f307e7bf P 52bd510945874fa781ec841d40e251ab: Bootstrap starting.
I20250112 15:08:49.237744 24576 raft_consensus.cc:1270] T c73b4e40028649e5bb9ad937f307e7bf P 56dd875b6ba54b40877146f15292bf2f [term 1 FOLLOWER]: Refusing update from remote peer f5b45ccea9c7438f8fba1b79073969e5: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250112 15:08:49.238215 24756 consensus_queue.cc:1035] T c73b4e40028649e5bb9ad937f307e7bf P f5b45ccea9c7438f8fba1b79073969e5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "56dd875b6ba54b40877146f15292bf2f" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 32969 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250112 15:08:49.239282 24756 raft_consensus.cc:2949] T c73b4e40028649e5bb9ad937f307e7bf P f5b45ccea9c7438f8fba1b79073969e5 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER a422b4873cf04ae9b9bc6530f2f077a2 (127.17.60.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "56dd875b6ba54b40877146f15292bf2f" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 32969 } } peers { permanent_uuid: "52bd510945874fa781ec841d40e251ab" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 34851 } } peers { permanent_uuid: "f5b45ccea9c7438f8fba1b79073969e5" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 40795 } } peers { permanent_uuid: "a422b4873cf04ae9b9bc6530f2f077a2" member_type: NON_VOTER last_known_addr { host: "127.17.60.196" port: 33207 } attrs { promote: true } } }
I20250112 15:08:49.239627 24869 tablet_bootstrap.cc:492] T c73b4e40028649e5bb9ad937f307e7bf P 52bd510945874fa781ec841d40e251ab: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250112 15:08:49.239833 24869 tablet_bootstrap.cc:492] T c73b4e40028649e5bb9ad937f307e7bf P 52bd510945874fa781ec841d40e251ab: Bootstrap complete.
I20250112 15:08:49.239919 24869 ts_tablet_manager.cc:1397] T c73b4e40028649e5bb9ad937f307e7bf P 52bd510945874fa781ec841d40e251ab: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20250112 15:08:49.240017 24869 raft_consensus.cc:357] T c73b4e40028649e5bb9ad937f307e7bf P 52bd510945874fa781ec841d40e251ab [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56dd875b6ba54b40877146f15292bf2f" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 32969 } } peers { permanent_uuid: "52bd510945874fa781ec841d40e251ab" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 34851 } } peers { permanent_uuid: "f5b45ccea9c7438f8fba1b79073969e5" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 40795 } }
I20250112 15:08:49.240065 24869 raft_consensus.cc:738] T c73b4e40028649e5bb9ad937f307e7bf P 52bd510945874fa781ec841d40e251ab [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 52bd510945874fa781ec841d40e251ab, State: Initialized, Role: FOLLOWER
I20250112 15:08:49.240105 24869 consensus_queue.cc:260] T c73b4e40028649e5bb9ad937f307e7bf P 52bd510945874fa781ec841d40e251ab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56dd875b6ba54b40877146f15292bf2f" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 32969 } } peers { permanent_uuid: "52bd510945874fa781ec841d40e251ab" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 34851 } } peers { permanent_uuid: "f5b45ccea9c7438f8fba1b79073969e5" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 40795 } }
I20250112 15:08:49.240186 24869 ts_tablet_manager.cc:1428] T c73b4e40028649e5bb9ad937f307e7bf P 52bd510945874fa781ec841d40e251ab: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20250112 15:08:49.240226 24869 tablet_bootstrap.cc:492] T 934800150b1945dcab01b4f7c4edcdc8 P 52bd510945874fa781ec841d40e251ab: Bootstrap starting.
I20250112 15:08:49.240478 24381 catalog_manager.cc:5039] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet c73b4e40028649e5bb9ad937f307e7bf with cas_config_opid_index -1: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 3)
I20250112 15:08:49.241097 24397 catalog_manager.cc:5526] T c73b4e40028649e5bb9ad937f307e7bf P f5b45ccea9c7438f8fba1b79073969e5 reported cstate change: config changed from index -1 to 2, NON_VOTER a422b4873cf04ae9b9bc6530f2f077a2 (127.17.60.196) added. New cstate: current_term: 1 leader_uuid: "f5b45ccea9c7438f8fba1b79073969e5" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "56dd875b6ba54b40877146f15292bf2f" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 32969 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "52bd510945874fa781ec841d40e251ab" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 34851 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f5b45ccea9c7438f8fba1b79073969e5" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 40795 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a422b4873cf04ae9b9bc6530f2f077a2" member_type: NON_VOTER last_known_addr { host: "127.17.60.196" port: 33207 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20250112 15:08:49.241463 24904 raft_consensus.cc:1270] T c73b4e40028649e5bb9ad937f307e7bf P 52bd510945874fa781ec841d40e251ab [term 1 FOLLOWER]: Refusing update from remote peer f5b45ccea9c7438f8fba1b79073969e5: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250112 15:08:49.241638 24721 consensus_queue.cc:1035] T c73b4e40028649e5bb9ad937f307e7bf P f5b45ccea9c7438f8fba1b79073969e5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "52bd510945874fa781ec841d40e251ab" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 34851 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20250112 15:08:49.241888 24617 consensus_peers.cc:487] T c73b4e40028649e5bb9ad937f307e7bf P f5b45ccea9c7438f8fba1b79073969e5 -> Peer a422b4873cf04ae9b9bc6530f2f077a2 (127.17.60.196:33207): Couldn't send request to peer a422b4873cf04ae9b9bc6530f2f077a2. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: c73b4e40028649e5bb9ad937f307e7bf. This is attempt 1: this message will repeat every 5th retry.
I20250112 15:08:49.241858 24576 raft_consensus.cc:2949] T c73b4e40028649e5bb9ad937f307e7bf P 56dd875b6ba54b40877146f15292bf2f [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER a422b4873cf04ae9b9bc6530f2f077a2 (127.17.60.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "56dd875b6ba54b40877146f15292bf2f" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 32969 } } peers { permanent_uuid: "52bd510945874fa781ec841d40e251ab" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 34851 } } peers { permanent_uuid: "f5b45ccea9c7438f8fba1b79073969e5" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 40795 } } peers { permanent_uuid: "a422b4873cf04ae9b9bc6530f2f077a2" member_type: NON_VOTER last_known_addr { host: "127.17.60.196" port: 33207 } attrs { promote: true } } }
I20250112 15:08:49.242071 24931 heartbeater.cc:346] Connected to a master server at 127.17.60.254:37937
I20250112 15:08:49.242033 24904 raft_consensus.cc:2949] T c73b4e40028649e5bb9ad937f307e7bf P 52bd510945874fa781ec841d40e251ab [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER a422b4873cf04ae9b9bc6530f2f077a2 (127.17.60.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "56dd875b6ba54b40877146f15292bf2f" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 32969 } } peers { permanent_uuid: "52bd510945874fa781ec841d40e251ab" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 34851 } } peers { permanent_uuid: "f5b45ccea9c7438f8fba1b79073969e5" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 40795 } } peers { permanent_uuid: "a422b4873cf04ae9b9bc6530f2f077a2" member_type: NON_VOTER last_known_addr { host: "127.17.60.196" port: 33207 } attrs { promote: true } } }
I20250112 15:08:49.242161 24931 heartbeater.cc:463] Registering TS with master...
I20250112 15:08:49.242260 24931 heartbeater.cc:510] Master 127.17.60.254:37937 requested a full tablet report, sending...
I20250112 15:08:49.242467 24869 tablet_bootstrap.cc:492] T 934800150b1945dcab01b4f7c4edcdc8 P 52bd510945874fa781ec841d40e251ab: Bootstrap replayed 1/1 log segments. Stats: ops{read=0 overwritten=0 applied=0 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250112 15:08:49.242631 24869 tablet_bootstrap.cc:492] T 934800150b1945dcab01b4f7c4edcdc8 P 52bd510945874fa781ec841d40e251ab: Bootstrap complete.
I20250112 15:08:49.242846 24869 ts_tablet_manager.cc:1397] T 934800150b1945dcab01b4f7c4edcdc8 P 52bd510945874fa781ec841d40e251ab: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.001s
I20250112 15:08:49.242993 24869 raft_consensus.cc:357] T 934800150b1945dcab01b4f7c4edcdc8 P 52bd510945874fa781ec841d40e251ab [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52bd510945874fa781ec841d40e251ab" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 34851 } } peers { permanent_uuid: "56dd875b6ba54b40877146f15292bf2f" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 32969 } } peers { permanent_uuid: "f5b45ccea9c7438f8fba1b79073969e5" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 40795 } }
I20250112 15:08:49.243085 24869 raft_consensus.cc:738] T 934800150b1945dcab01b4f7c4edcdc8 P 52bd510945874fa781ec841d40e251ab [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 52bd510945874fa781ec841d40e251ab, State: Initialized, Role: FOLLOWER
I20250112 15:08:49.243125 24869 consensus_queue.cc:260] T 934800150b1945dcab01b4f7c4edcdc8 P 52bd510945874fa781ec841d40e251ab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52bd510945874fa781ec841d40e251ab" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 34851 } } peers { permanent_uuid: "56dd875b6ba54b40877146f15292bf2f" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 32969 } } peers { permanent_uuid: "f5b45ccea9c7438f8fba1b79073969e5" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 40795 } }
I20250112 15:08:49.243218 24869 ts_tablet_manager.cc:1428] T 934800150b1945dcab01b4f7c4edcdc8 P 52bd510945874fa781ec841d40e251ab: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20250112 15:08:49.243266 24869 tablet_bootstrap.cc:492] T 94172f983ecd4a37956654f2f86dba00 P 52bd510945874fa781ec841d40e251ab: Bootstrap starting.
I20250112 15:08:49.243438 24397 ts_manager.cc:194] Re-registered known tserver with Master: 52bd510945874fa781ec841d40e251ab (127.17.60.193:34851)
I20250112 15:08:49.243777 24397 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:60626
I20250112 15:08:49.244158 24931 heartbeater.cc:502] Master 127.17.60.254:37937 was elected leader, sending a full tablet report...
I20250112 15:08:49.244504 24869 tablet_bootstrap.cc:492] T 94172f983ecd4a37956654f2f86dba00 P 52bd510945874fa781ec841d40e251ab: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250112 15:08:49.244717 24869 tablet_bootstrap.cc:492] T 94172f983ecd4a37956654f2f86dba00 P 52bd510945874fa781ec841d40e251ab: Bootstrap complete.
I20250112 15:08:49.244804 24869 ts_tablet_manager.cc:1397] T 94172f983ecd4a37956654f2f86dba00 P 52bd510945874fa781ec841d40e251ab: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20250112 15:08:49.244932 24869 raft_consensus.cc:357] T 94172f983ecd4a37956654f2f86dba00 P 52bd510945874fa781ec841d40e251ab [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52bd510945874fa781ec841d40e251ab" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 34851 } } peers { permanent_uuid: "56dd875b6ba54b40877146f15292bf2f" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 32969 } } peers { permanent_uuid: "f5b45ccea9c7438f8fba1b79073969e5" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 40795 } }
I20250112 15:08:49.244973 24869 raft_consensus.cc:738] T 94172f983ecd4a37956654f2f86dba00 P 52bd510945874fa781ec841d40e251ab [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 52bd510945874fa781ec841d40e251ab, State: Initialized, Role: FOLLOWER
I20250112 15:08:49.245019 24869 consensus_queue.cc:260] T 94172f983ecd4a37956654f2f86dba00 P 52bd510945874fa781ec841d40e251ab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52bd510945874fa781ec841d40e251ab" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 34851 } } peers { permanent_uuid: "56dd875b6ba54b40877146f15292bf2f" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 32969 } } peers { permanent_uuid: "f5b45ccea9c7438f8fba1b79073969e5" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 40795 } }
I20250112 15:08:49.245103 24869 ts_tablet_manager.cc:1428] T 94172f983ecd4a37956654f2f86dba00 P 52bd510945874fa781ec841d40e251ab: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20250112 15:08:49.245152 24869 tablet_bootstrap.cc:492] T bf900db2a1344c7b8fd0b9ba5e225201 P 52bd510945874fa781ec841d40e251ab: Bootstrap starting.
I20250112 15:08:49.245980 24869 tablet_bootstrap.cc:492] T bf900db2a1344c7b8fd0b9ba5e225201 P 52bd510945874fa781ec841d40e251ab: Bootstrap replayed 1/1 log segments. Stats: ops{read=0 overwritten=0 applied=0 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250112 15:08:49.246125 24869 tablet_bootstrap.cc:492] T bf900db2a1344c7b8fd0b9ba5e225201 P 52bd510945874fa781ec841d40e251ab: Bootstrap complete.
I20250112 15:08:49.246215 24869 ts_tablet_manager.cc:1397] T bf900db2a1344c7b8fd0b9ba5e225201 P 52bd510945874fa781ec841d40e251ab: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20250112 15:08:49.246345 24869 raft_consensus.cc:357] T bf900db2a1344c7b8fd0b9ba5e225201 P 52bd510945874fa781ec841d40e251ab [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52bd510945874fa781ec841d40e251ab" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 34851 } } peers { permanent_uuid: "56dd875b6ba54b40877146f15292bf2f" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 32969 } } peers { permanent_uuid: "f5b45ccea9c7438f8fba1b79073969e5" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 40795 } }
I20250112 15:08:49.246389 24869 raft_consensus.cc:738] T bf900db2a1344c7b8fd0b9ba5e225201 P 52bd510945874fa781ec841d40e251ab [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 52bd510945874fa781ec841d40e251ab, State: Initialized, Role: FOLLOWER
I20250112 15:08:49.246438 24869 consensus_queue.cc:260] T bf900db2a1344c7b8fd0b9ba5e225201 P 52bd510945874fa781ec841d40e251ab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52bd510945874fa781ec841d40e251ab" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 34851 } } peers { permanent_uuid: "56dd875b6ba54b40877146f15292bf2f" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 32969 } } peers { permanent_uuid: "f5b45ccea9c7438f8fba1b79073969e5" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 40795 } }
I20250112 15:08:49.246526 24869 ts_tablet_manager.cc:1428] T bf900db2a1344c7b8fd0b9ba5e225201 P 52bd510945874fa781ec841d40e251ab: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20250112 15:08:49.253371 24894 tablet_service.cc:1514] Processing DeleteTablet for tablet bf900db2a1344c7b8fd0b9ba5e225201 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 15:08:49 UTC) from {username='slave'} at 127.0.0.1:56452
I20250112 15:08:49.253574 24945 tablet_replica.cc:331] stopping tablet replica
I20250112 15:08:49.253635 24945 raft_consensus.cc:2238] T bf900db2a1344c7b8fd0b9ba5e225201 P 52bd510945874fa781ec841d40e251ab [term 1 FOLLOWER]: Raft consensus shutting down.
I20250112 15:08:49.253677 24945 raft_consensus.cc:2267] T bf900db2a1344c7b8fd0b9ba5e225201 P 52bd510945874fa781ec841d40e251ab [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 15:08:49.253890 24945 ts_tablet_manager.cc:1905] T bf900db2a1344c7b8fd0b9ba5e225201 P 52bd510945874fa781ec841d40e251ab: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 15:08:49.255288 24945 ts_tablet_manager.cc:1918] T bf900db2a1344c7b8fd0b9ba5e225201 P 52bd510945874fa781ec841d40e251ab: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250112 15:08:49.255333 24945 log.cc:1198] T bf900db2a1344c7b8fd0b9ba5e225201 P 52bd510945874fa781ec841d40e251ab: Deleting WAL directory at /tmp/dist-test-task95dLSD/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestDeletedTables.1736694351070047-17651-0/minicluster-data/ts-0-root/wals/bf900db2a1344c7b8fd0b9ba5e225201
I20250112 15:08:49.255482 24945 ts_tablet_manager.cc:1939] T bf900db2a1344c7b8fd0b9ba5e225201 P 52bd510945874fa781ec841d40e251ab: Deleting consensus metadata
I20250112 15:08:49.255744 24381 catalog_manager.cc:4872] TS 52bd510945874fa781ec841d40e251ab (127.17.60.193:34851): tablet bf900db2a1344c7b8fd0b9ba5e225201 (table dugtrio [id=271cc77f05ad43eb89d1c5c6291536e4]) successfully deleted
I20250112 15:08:49.260036 24894 tablet_service.cc:1514] Processing DeleteTablet for tablet a3cc29d2a75b4778b21400e0e0b64a9e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 15:08:49 UTC) from {username='slave'} at 127.0.0.1:56452
I20250112 15:08:49.260129 24945 tablet_replica.cc:331] stopping tablet replica
I20250112 15:08:49.260208 24945 raft_consensus.cc:2238] T a3cc29d2a75b4778b21400e0e0b64a9e P 52bd510945874fa781ec841d40e251ab [term 1 FOLLOWER]: Raft consensus shutting down.
I20250112 15:08:49.260250 24945 raft_consensus.cc:2267] T a3cc29d2a75b4778b21400e0e0b64a9e P 52bd510945874fa781ec841d40e251ab [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 15:08:49.260433 24945 ts_tablet_manager.cc:1905] T a3cc29d2a75b4778b21400e0e0b64a9e P 52bd510945874fa781ec841d40e251ab: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 15:08:49.261579 24945 ts_tablet_manager.cc:1918] T a3cc29d2a75b4778b21400e0e0b64a9e P 52bd510945874fa781ec841d40e251ab: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250112 15:08:49.261627 24945 log.cc:1198] T a3cc29d2a75b4778b21400e0e0b64a9e P 52bd510945874fa781ec841d40e251ab: Deleting WAL directory at /tmp/dist-test-task95dLSD/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestDeletedTables.1736694351070047-17651-0/minicluster-data/ts-0-root/wals/a3cc29d2a75b4778b21400e0e0b64a9e
I20250112 15:08:49.261763 24945 ts_tablet_manager.cc:1939] T a3cc29d2a75b4778b21400e0e0b64a9e P 52bd510945874fa781ec841d40e251ab: Deleting consensus metadata
I20250112 15:08:49.262068 24381 catalog_manager.cc:4872] TS 52bd510945874fa781ec841d40e251ab (127.17.60.193:34851): tablet a3cc29d2a75b4778b21400e0e0b64a9e (table dugtrio [id=271cc77f05ad43eb89d1c5c6291536e4]) successfully deleted
I20250112 15:08:49.263317 24894 tablet_service.cc:1514] Processing DeleteTablet for tablet 6914949901e147f2970585c79458b9d2 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 15:08:49 UTC) from {username='slave'} at 127.0.0.1:56452
I20250112 15:08:49.263398 24945 tablet_replica.cc:331] stopping tablet replica
I20250112 15:08:49.263465 24945 raft_consensus.cc:2238] T 6914949901e147f2970585c79458b9d2 P 52bd510945874fa781ec841d40e251ab [term 1 FOLLOWER]: Raft consensus shutting down.
I20250112 15:08:49.263509 24945 raft_consensus.cc:2267] T 6914949901e147f2970585c79458b9d2 P 52bd510945874fa781ec841d40e251ab [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 15:08:49.263674 24945 ts_tablet_manager.cc:1905] T 6914949901e147f2970585c79458b9d2 P 52bd510945874fa781ec841d40e251ab: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 15:08:49.264797 24945 ts_tablet_manager.cc:1918] T 6914949901e147f2970585c79458b9d2 P 52bd510945874fa781ec841d40e251ab: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250112 15:08:49.264856 24945 log.cc:1198] T 6914949901e147f2970585c79458b9d2 P 52bd510945874fa781ec841d40e251ab: Deleting WAL directory at /tmp/dist-test-task95dLSD/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestDeletedTables.1736694351070047-17651-0/minicluster-data/ts-0-root/wals/6914949901e147f2970585c79458b9d2
I20250112 15:08:49.265027 24945 ts_tablet_manager.cc:1939] T 6914949901e147f2970585c79458b9d2 P 52bd510945874fa781ec841d40e251ab: Deleting consensus metadata
I20250112 15:08:49.265287 24381 catalog_manager.cc:4872] TS 52bd510945874fa781ec841d40e251ab (127.17.60.193:34851): tablet 6914949901e147f2970585c79458b9d2 (table dugtrio [id=271cc77f05ad43eb89d1c5c6291536e4]) successfully deleted
I20250112 15:08:49.285130 24721 consensus_queue.cc:579] T cc5c837c3de946a5952f656946d883c8 P f5b45ccea9c7438f8fba1b79073969e5 [LEADER]: Leader has been unable to successfully communicate with peer 52bd510945874fa781ec841d40e251ab for more than 1 seconds (1.392s)
I20250112 15:08:49.286008 24651 consensus_queue.cc:237] T cc5c837c3de946a5952f656946d883c8 P f5b45ccea9c7438f8fba1b79073969e5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "52bd510945874fa781ec841d40e251ab" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 34851 } } peers { permanent_uuid: "56dd875b6ba54b40877146f15292bf2f" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 32969 } } peers { permanent_uuid: "f5b45ccea9c7438f8fba1b79073969e5" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 40795 } } peers { permanent_uuid: "a422b4873cf04ae9b9bc6530f2f077a2" member_type: NON_VOTER last_known_addr { host: "127.17.60.196" port: 33207 } attrs { promote: true } }
I20250112 15:08:49.286485 24576 raft_consensus.cc:1270] T cc5c837c3de946a5952f656946d883c8 P 56dd875b6ba54b40877146f15292bf2f [term 1 FOLLOWER]: Refusing update from remote peer f5b45ccea9c7438f8fba1b79073969e5: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250112 15:08:49.286530 24904 raft_consensus.cc:1270] T cc5c837c3de946a5952f656946d883c8 P 52bd510945874fa781ec841d40e251ab [term 1 FOLLOWER]: Refusing update from remote peer f5b45ccea9c7438f8fba1b79073969e5: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20250112 15:08:49.286585 24617 consensus_peers.cc:487] T cc5c837c3de946a5952f656946d883c8 P f5b45ccea9c7438f8fba1b79073969e5 -> Peer a422b4873cf04ae9b9bc6530f2f077a2 (127.17.60.196:33207): Couldn't send request to peer a422b4873cf04ae9b9bc6530f2f077a2. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: cc5c837c3de946a5952f656946d883c8. This is attempt 1: this message will repeat every 5th retry.
I20250112 15:08:49.286691 24756 consensus_queue.cc:1035] T cc5c837c3de946a5952f656946d883c8 P f5b45ccea9c7438f8fba1b79073969e5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "52bd510945874fa781ec841d40e251ab" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 34851 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250112 15:08:49.286772 24704 consensus_queue.cc:1035] T cc5c837c3de946a5952f656946d883c8 P f5b45ccea9c7438f8fba1b79073969e5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "56dd875b6ba54b40877146f15292bf2f" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 32969 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250112 15:08:49.287307 24721 raft_consensus.cc:2949] T cc5c837c3de946a5952f656946d883c8 P f5b45ccea9c7438f8fba1b79073969e5 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER a422b4873cf04ae9b9bc6530f2f077a2 (127.17.60.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "52bd510945874fa781ec841d40e251ab" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 34851 } } peers { permanent_uuid: "56dd875b6ba54b40877146f15292bf2f" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 32969 } } peers { permanent_uuid: "f5b45ccea9c7438f8fba1b79073969e5" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 40795 } } peers { permanent_uuid: "a422b4873cf04ae9b9bc6530f2f077a2" member_type: NON_VOTER last_known_addr { host: "127.17.60.196" port: 33207 } attrs { promote: true } } }
I20250112 15:08:49.287422 24904 raft_consensus.cc:2949] T cc5c837c3de946a5952f656946d883c8 P 52bd510945874fa781ec841d40e251ab [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER a422b4873cf04ae9b9bc6530f2f077a2 (127.17.60.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "52bd510945874fa781ec841d40e251ab" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 34851 } } peers { permanent_uuid: "56dd875b6ba54b40877146f15292bf2f" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 32969 } } peers { permanent_uuid: "f5b45ccea9c7438f8fba1b79073969e5" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 40795 } } peers { permanent_uuid: "a422b4873cf04ae9b9bc6530f2f077a2" member_type: NON_VOTER last_known_addr { host: "127.17.60.196" port: 33207 } attrs { promote: true } } }
I20250112 15:08:49.287470 24576 raft_consensus.cc:2949] T cc5c837c3de946a5952f656946d883c8 P 56dd875b6ba54b40877146f15292bf2f [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER a422b4873cf04ae9b9bc6530f2f077a2 (127.17.60.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "52bd510945874fa781ec841d40e251ab" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 34851 } } peers { permanent_uuid: "56dd875b6ba54b40877146f15292bf2f" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 32969 } } peers { permanent_uuid: "f5b45ccea9c7438f8fba1b79073969e5" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 40795 } } peers { permanent_uuid: "a422b4873cf04ae9b9bc6530f2f077a2" member_type: NON_VOTER last_known_addr { host: "127.17.60.196" port: 33207 } attrs { promote: true } } }
I20250112 15:08:49.288288 24381 catalog_manager.cc:5039] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet cc5c837c3de946a5952f656946d883c8 with cas_config_opid_index -1: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20250112 15:08:49.288443 24397 catalog_manager.cc:5526] T cc5c837c3de946a5952f656946d883c8 P 56dd875b6ba54b40877146f15292bf2f reported cstate change: config changed from index -1 to 2, NON_VOTER a422b4873cf04ae9b9bc6530f2f077a2 (127.17.60.196) added. New cstate: current_term: 1 leader_uuid: "f5b45ccea9c7438f8fba1b79073969e5" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "52bd510945874fa781ec841d40e251ab" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 34851 } } peers { permanent_uuid: "56dd875b6ba54b40877146f15292bf2f" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 32969 } } peers { permanent_uuid: "f5b45ccea9c7438f8fba1b79073969e5" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 40795 } } peers { permanent_uuid: "a422b4873cf04ae9b9bc6530f2f077a2" member_type: NON_VOTER last_known_addr { host: "127.17.60.196" port: 33207 } attrs { promote: true } } }
I20250112 15:08:49.289320 24651 consensus_queue.cc:237] T cc5c837c3de946a5952f656946d883c8 P f5b45ccea9c7438f8fba1b79073969e5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "52bd510945874fa781ec841d40e251ab" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 34851 } } peers { permanent_uuid: "56dd875b6ba54b40877146f15292bf2f" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 32969 } } peers { permanent_uuid: "f5b45ccea9c7438f8fba1b79073969e5" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 40795 } }
I20250112 15:08:49.289690 24904 raft_consensus.cc:1270] T cc5c837c3de946a5952f656946d883c8 P 52bd510945874fa781ec841d40e251ab [term 1 FOLLOWER]: Refusing update from remote peer f5b45ccea9c7438f8fba1b79073969e5: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250112 15:08:49.289690 24576 raft_consensus.cc:1270] T cc5c837c3de946a5952f656946d883c8 P 56dd875b6ba54b40877146f15292bf2f [term 1 FOLLOWER]: Refusing update from remote peer f5b45ccea9c7438f8fba1b79073969e5: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250112 15:08:49.289870 24704 consensus_queue.cc:1035] T cc5c837c3de946a5952f656946d883c8 P f5b45ccea9c7438f8fba1b79073969e5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "56dd875b6ba54b40877146f15292bf2f" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 32969 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250112 15:08:49.289964 24721 consensus_queue.cc:1035] T cc5c837c3de946a5952f656946d883c8 P f5b45ccea9c7438f8fba1b79073969e5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "52bd510945874fa781ec841d40e251ab" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 34851 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250112 15:08:49.290436 24704 raft_consensus.cc:2949] T cc5c837c3de946a5952f656946d883c8 P f5b45ccea9c7438f8fba1b79073969e5 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, NON_VOTER a422b4873cf04ae9b9bc6530f2f077a2 (127.17.60.196) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "52bd510945874fa781ec841d40e251ab" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 34851 } } peers { permanent_uuid: "56dd875b6ba54b40877146f15292bf2f" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 32969 } } peers { permanent_uuid: "f5b45ccea9c7438f8fba1b79073969e5" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 40795 } } }
I20250112 15:08:49.290562 24904 raft_consensus.cc:2949] T cc5c837c3de946a5952f656946d883c8 P 52bd510945874fa781ec841d40e251ab [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, NON_VOTER a422b4873cf04ae9b9bc6530f2f077a2 (127.17.60.196) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "52bd510945874fa781ec841d40e251ab" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 34851 } } peers { permanent_uuid: "56dd875b6ba54b40877146f15292bf2f" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 32969 } } peers { permanent_uuid: "f5b45ccea9c7438f8fba1b79073969e5" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 40795 } } }
I20250112 15:08:49.290587 24576 raft_consensus.cc:2949] T cc5c837c3de946a5952f656946d883c8 P 56dd875b6ba54b40877146f15292bf2f [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, NON_VOTER a422b4873cf04ae9b9bc6530f2f077a2 (127.17.60.196) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "52bd510945874fa781ec841d40e251ab" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 34851 } } peers { permanent_uuid: "56dd875b6ba54b40877146f15292bf2f" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 32969 } } peers { permanent_uuid: "f5b45ccea9c7438f8fba1b79073969e5" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 40795 } } }
I20250112 15:08:49.291458 24381 catalog_manager.cc:5039] ChangeConfig:REMOVE_PEER RPC for tablet cc5c837c3de946a5952f656946d883c8 with cas_config_opid_index 2: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250112 15:08:49.291479 24396 catalog_manager.cc:5526] T cc5c837c3de946a5952f656946d883c8 P 56dd875b6ba54b40877146f15292bf2f reported cstate change: config changed from index 2 to 3, NON_VOTER a422b4873cf04ae9b9bc6530f2f077a2 (127.17.60.196) evicted. New cstate: current_term: 1 leader_uuid: "f5b45ccea9c7438f8fba1b79073969e5" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "52bd510945874fa781ec841d40e251ab" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 34851 } } peers { permanent_uuid: "56dd875b6ba54b40877146f15292bf2f" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 32969 } } peers { permanent_uuid: "f5b45ccea9c7438f8fba1b79073969e5" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 40795 } } }
I20250112 15:08:49.294509 24815 tablet_service.cc:1514] Processing DeleteTablet for tablet cc5c837c3de946a5952f656946d883c8 with delete_type TABLET_DATA_TOMBSTONED (TS a422b4873cf04ae9b9bc6530f2f077a2 not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:46800
I20250112 15:08:49.294623 24894 tablet_service.cc:1514] Processing DeleteTablet for tablet 934800150b1945dcab01b4f7c4edcdc8 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 15:08:49 UTC) from {username='slave'} at 127.0.0.1:56452
I20250112 15:08:49.294729 24945 tablet_replica.cc:331] stopping tablet replica
I20250112 15:08:49.294798 24945 raft_consensus.cc:2238] T 934800150b1945dcab01b4f7c4edcdc8 P 52bd510945874fa781ec841d40e251ab [term 1 FOLLOWER]: Raft consensus shutting down.
I20250112 15:08:49.294837 24945 raft_consensus.cc:2267] T 934800150b1945dcab01b4f7c4edcdc8 P 52bd510945874fa781ec841d40e251ab [term 1 FOLLOWER]: Raft consensus is shut down!
W20250112 15:08:49.294847 24384 catalog_manager.cc:4839] TS a422b4873cf04ae9b9bc6530f2f077a2 (127.17.60.196:33207): delete failed for tablet cc5c837c3de946a5952f656946d883c8 because the tablet was not found. No further retry: Not found: Tablet not found: cc5c837c3de946a5952f656946d883c8
I20250112 15:08:49.295015 24945 ts_tablet_manager.cc:1905] T 934800150b1945dcab01b4f7c4edcdc8 P 52bd510945874fa781ec841d40e251ab: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 15:08:49.296383 24945 ts_tablet_manager.cc:1918] T 934800150b1945dcab01b4f7c4edcdc8 P 52bd510945874fa781ec841d40e251ab: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250112 15:08:49.296430 24945 log.cc:1198] T 934800150b1945dcab01b4f7c4edcdc8 P 52bd510945874fa781ec841d40e251ab: Deleting WAL directory at /tmp/dist-test-task95dLSD/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestDeletedTables.1736694351070047-17651-0/minicluster-data/ts-0-root/wals/934800150b1945dcab01b4f7c4edcdc8
I20250112 15:08:49.296636 24945 ts_tablet_manager.cc:1939] T 934800150b1945dcab01b4f7c4edcdc8 P 52bd510945874fa781ec841d40e251ab: Deleting consensus metadata
I20250112 15:08:49.296917 24381 catalog_manager.cc:4872] TS 52bd510945874fa781ec841d40e251ab (127.17.60.193:34851): tablet 934800150b1945dcab01b4f7c4edcdc8 (table dugtrio [id=271cc77f05ad43eb89d1c5c6291536e4]) successfully deleted
W20250112 15:08:49.455125 24446 auto_rebalancer.cc:253] failed to send replica move request: Invalid argument: Server with UUID a422b4873cf04ae9b9bc6530f2f077a2 is already a member of the config. RaftConfig: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "56dd875b6ba54b40877146f15292bf2f" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 32969 } } peers { permanent_uuid: "52bd510945874fa781ec841d40e251ab" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 34851 } } peers { permanent_uuid: "f5b45ccea9c7438f8fba1b79073969e5" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 40795 } } peers { permanent_uuid: "a422b4873cf04ae9b9bc6530f2f077a2" member_type: NON_VOTER last_known_addr { host: "127.17.60.196" port: 33207 } attrs { promote: true } }
W20250112 15:08:49.455535 24446 auto_rebalancer.cc:663] Could not move replica: Illegal state: c73b4e40028649e5bb9ad937f307e7bf: source replica 52bd510945874fa781ec841d40e251ab does not have REPLACE attribute set
W20250112 15:08:49.455576 24446 auto_rebalancer.cc:263] scheduled replica move failed to complete: Illegal state: c73b4e40028649e5bb9ad937f307e7bf: source replica 52bd510945874fa781ec841d40e251ab does not have REPLACE attribute set
W20250112 15:08:49.455930 24446 auto_rebalancer.cc:663] Could not move replica: Incomplete: tablet 94172f983ecd4a37956654f2f86dba00, TS f5b45ccea9c7438f8fba1b79073969e5 -> TS a422b4873cf04ae9b9bc6530f2f077a2 move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52bd510945874fa781ec841d40e251ab" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 34851 } } peers { permanent_uuid: "56dd875b6ba54b40877146f15292bf2f" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 32969 } } peers { permanent_uuid: "f5b45ccea9c7438f8fba1b79073969e5" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 40795 } }
W20250112 15:08:49.455969 24446 auto_rebalancer.cc:263] scheduled replica move failed to complete: Incomplete: tablet 94172f983ecd4a37956654f2f86dba00, TS f5b45ccea9c7438f8fba1b79073969e5 -> TS a422b4873cf04ae9b9bc6530f2f077a2 move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52bd510945874fa781ec841d40e251ab" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 34851 } } peers { permanent_uuid: "56dd875b6ba54b40877146f15292bf2f" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 32969 } } peers { permanent_uuid: "f5b45ccea9c7438f8fba1b79073969e5" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 40795 } }
W20250112 15:08:49.456252 24446 auto_rebalancer.cc:663] Could not move replica: Incomplete: tablet 45d29678d6704a888f2c2514af2f36a4, TS 56dd875b6ba54b40877146f15292bf2f -> TS a422b4873cf04ae9b9bc6530f2f077a2 move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52bd510945874fa781ec841d40e251ab" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 34851 } } peers { permanent_uuid: "56dd875b6ba54b40877146f15292bf2f" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 32969 } } peers { permanent_uuid: "f5b45ccea9c7438f8fba1b79073969e5" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 40795 } }
W20250112 15:08:49.456287 24446 auto_rebalancer.cc:263] scheduled replica move failed to complete: Incomplete: tablet 45d29678d6704a888f2c2514af2f36a4, TS 56dd875b6ba54b40877146f15292bf2f -> TS a422b4873cf04ae9b9bc6530f2f077a2 move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52bd510945874fa781ec841d40e251ab" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 34851 } } peers { permanent_uuid: "56dd875b6ba54b40877146f15292bf2f" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 32969 } } peers { permanent_uuid: "f5b45ccea9c7438f8fba1b79073969e5" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 40795 } }
I20250112 15:08:49.489300 24721 consensus_queue.cc:579] T 45d29678d6704a888f2c2514af2f36a4 P f5b45ccea9c7438f8fba1b79073969e5 [LEADER]: Leader has been unable to successfully communicate with peer 52bd510945874fa781ec841d40e251ab for more than 1 seconds (1.418s)
I20250112 15:08:49.490187 24651 consensus_queue.cc:237] T 45d29678d6704a888f2c2514af2f36a4 P f5b45ccea9c7438f8fba1b79073969e5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "52bd510945874fa781ec841d40e251ab" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 34851 } } peers { permanent_uuid: "56dd875b6ba54b40877146f15292bf2f" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 32969 } } peers { permanent_uuid: "f5b45ccea9c7438f8fba1b79073969e5" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 40795 } } peers { permanent_uuid: "a422b4873cf04ae9b9bc6530f2f077a2" member_type: NON_VOTER last_known_addr { host: "127.17.60.196" port: 33207 } attrs { promote: true } }
I20250112 15:08:49.490746 24576 raft_consensus.cc:1270] T 45d29678d6704a888f2c2514af2f36a4 P 56dd875b6ba54b40877146f15292bf2f [term 1 FOLLOWER]: Refusing update from remote peer f5b45ccea9c7438f8fba1b79073969e5: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250112 15:08:49.490765 24904 raft_consensus.cc:1270] T 45d29678d6704a888f2c2514af2f36a4 P 52bd510945874fa781ec841d40e251ab [term 1 FOLLOWER]: Refusing update from remote peer f5b45ccea9c7438f8fba1b79073969e5: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20250112 15:08:49.490875 24617 consensus_peers.cc:487] T 45d29678d6704a888f2c2514af2f36a4 P f5b45ccea9c7438f8fba1b79073969e5 -> Peer a422b4873cf04ae9b9bc6530f2f077a2 (127.17.60.196:33207): Couldn't send request to peer a422b4873cf04ae9b9bc6530f2f077a2. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 45d29678d6704a888f2c2514af2f36a4. This is attempt 1: this message will repeat every 5th retry.
I20250112 15:08:49.490962 24721 consensus_queue.cc:1035] T 45d29678d6704a888f2c2514af2f36a4 P f5b45ccea9c7438f8fba1b79073969e5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "56dd875b6ba54b40877146f15292bf2f" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 32969 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250112 15:08:49.491041 24704 consensus_queue.cc:1035] T 45d29678d6704a888f2c2514af2f36a4 P f5b45ccea9c7438f8fba1b79073969e5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "52bd510945874fa781ec841d40e251ab" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 34851 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250112 15:08:49.491734 24704 raft_consensus.cc:2949] T 45d29678d6704a888f2c2514af2f36a4 P f5b45ccea9c7438f8fba1b79073969e5 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER a422b4873cf04ae9b9bc6530f2f077a2 (127.17.60.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "52bd510945874fa781ec841d40e251ab" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 34851 } } peers { permanent_uuid: "56dd875b6ba54b40877146f15292bf2f" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 32969 } } peers { permanent_uuid: "f5b45ccea9c7438f8fba1b79073969e5" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 40795 } } peers { permanent_uuid: "a422b4873cf04ae9b9bc6530f2f077a2" member_type: NON_VOTER last_known_addr { host: "127.17.60.196" port: 33207 } attrs { promote: true } } }
I20250112 15:08:49.491918 24904 raft_consensus.cc:2949] T 45d29678d6704a888f2c2514af2f36a4 P 52bd510945874fa781ec841d40e251ab [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER a422b4873cf04ae9b9bc6530f2f077a2 (127.17.60.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "52bd510945874fa781ec841d40e251ab" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 34851 } } peers { permanent_uuid: "56dd875b6ba54b40877146f15292bf2f" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 32969 } } peers { permanent_uuid: "f5b45ccea9c7438f8fba1b79073969e5" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 40795 } } peers { permanent_uuid: "a422b4873cf04ae9b9bc6530f2f077a2" member_type: NON_VOTER last_known_addr { host: "127.17.60.196" port: 33207 } attrs { promote: true } } }
I20250112 15:08:49.491941 24576 raft_consensus.cc:2949] T 45d29678d6704a888f2c2514af2f36a4 P 56dd875b6ba54b40877146f15292bf2f [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER a422b4873cf04ae9b9bc6530f2f077a2 (127.17.60.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "52bd510945874fa781ec841d40e251ab" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 34851 } } peers { permanent_uuid: "56dd875b6ba54b40877146f15292bf2f" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 32969 } } peers { permanent_uuid: "f5b45ccea9c7438f8fba1b79073969e5" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 40795 } } peers { permanent_uuid: "a422b4873cf04ae9b9bc6530f2f077a2" member_type: NON_VOTER last_known_addr { host: "127.17.60.196" port: 33207 } attrs { promote: true } } }
I20250112 15:08:49.492972 24381 catalog_manager.cc:5039] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 45d29678d6704a888f2c2514af2f36a4 with cas_config_opid_index -1: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20250112 15:08:49.492996 24396 catalog_manager.cc:5526] T 45d29678d6704a888f2c2514af2f36a4 P f5b45ccea9c7438f8fba1b79073969e5 reported cstate change: config changed from index -1 to 2, NON_VOTER a422b4873cf04ae9b9bc6530f2f077a2 (127.17.60.196) added. New cstate: current_term: 1 leader_uuid: "f5b45ccea9c7438f8fba1b79073969e5" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "52bd510945874fa781ec841d40e251ab" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 34851 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "56dd875b6ba54b40877146f15292bf2f" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 32969 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f5b45ccea9c7438f8fba1b79073969e5" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 40795 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a422b4873cf04ae9b9bc6530f2f077a2" member_type: NON_VOTER last_known_addr { host: "127.17.60.196" port: 33207 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20250112 15:08:49.494100 24651 consensus_queue.cc:237] T 45d29678d6704a888f2c2514af2f36a4 P f5b45ccea9c7438f8fba1b79073969e5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "52bd510945874fa781ec841d40e251ab" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 34851 } } peers { permanent_uuid: "56dd875b6ba54b40877146f15292bf2f" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 32969 } } peers { permanent_uuid: "f5b45ccea9c7438f8fba1b79073969e5" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 40795 } }
I20250112 15:08:49.494441 24576 raft_consensus.cc:1270] T 45d29678d6704a888f2c2514af2f36a4 P 56dd875b6ba54b40877146f15292bf2f [term 1 FOLLOWER]: Refusing update from remote peer f5b45ccea9c7438f8fba1b79073969e5: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250112 15:08:49.494454 24904 raft_consensus.cc:1270] T 45d29678d6704a888f2c2514af2f36a4 P 52bd510945874fa781ec841d40e251ab [term 1 FOLLOWER]: Refusing update from remote peer f5b45ccea9c7438f8fba1b79073969e5: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250112 15:08:49.494603 24756 consensus_queue.cc:1035] T 45d29678d6704a888f2c2514af2f36a4 P f5b45ccea9c7438f8fba1b79073969e5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "56dd875b6ba54b40877146f15292bf2f" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 32969 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250112 15:08:49.494666 24721 consensus_queue.cc:1035] T 45d29678d6704a888f2c2514af2f36a4 P f5b45ccea9c7438f8fba1b79073969e5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "52bd510945874fa781ec841d40e251ab" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 34851 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250112 15:08:49.495234 24756 raft_consensus.cc:2949] T 45d29678d6704a888f2c2514af2f36a4 P f5b45ccea9c7438f8fba1b79073969e5 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, NON_VOTER a422b4873cf04ae9b9bc6530f2f077a2 (127.17.60.196) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "52bd510945874fa781ec841d40e251ab" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 34851 } } peers { permanent_uuid: "56dd875b6ba54b40877146f15292bf2f" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 32969 } } peers { permanent_uuid: "f5b45ccea9c7438f8fba1b79073969e5" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 40795 } } }
I20250112 15:08:49.496043 24381 catalog_manager.cc:5039] ChangeConfig:REMOVE_PEER RPC for tablet 45d29678d6704a888f2c2514af2f36a4 with cas_config_opid_index 2: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250112 15:08:49.496058 24576 raft_consensus.cc:2949] T 45d29678d6704a888f2c2514af2f36a4 P 56dd875b6ba54b40877146f15292bf2f [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, NON_VOTER a422b4873cf04ae9b9bc6530f2f077a2 (127.17.60.196) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "52bd510945874fa781ec841d40e251ab" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 34851 } } peers { permanent_uuid: "56dd875b6ba54b40877146f15292bf2f" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 32969 } } peers { permanent_uuid: "f5b45ccea9c7438f8fba1b79073969e5" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 40795 } } }
I20250112 15:08:49.496146 24397 catalog_manager.cc:5526] T 45d29678d6704a888f2c2514af2f36a4 P f5b45ccea9c7438f8fba1b79073969e5 reported cstate change: config changed from index 2 to 3, NON_VOTER a422b4873cf04ae9b9bc6530f2f077a2 (127.17.60.196) evicted. New cstate: current_term: 1 leader_uuid: "f5b45ccea9c7438f8fba1b79073969e5" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "52bd510945874fa781ec841d40e251ab" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 34851 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "56dd875b6ba54b40877146f15292bf2f" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 32969 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f5b45ccea9c7438f8fba1b79073969e5" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 40795 } health_report { overall_health: HEALTHY } } }
I20250112 15:08:49.496891 24815 tablet_service.cc:1514] Processing DeleteTablet for tablet 45d29678d6704a888f2c2514af2f36a4 with delete_type TABLET_DATA_TOMBSTONED (TS a422b4873cf04ae9b9bc6530f2f077a2 not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:46800
I20250112 15:08:49.496598 24904 raft_consensus.cc:2949] T 45d29678d6704a888f2c2514af2f36a4 P 52bd510945874fa781ec841d40e251ab [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, NON_VOTER a422b4873cf04ae9b9bc6530f2f077a2 (127.17.60.196) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "52bd510945874fa781ec841d40e251ab" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 34851 } } peers { permanent_uuid: "56dd875b6ba54b40877146f15292bf2f" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 32969 } } peers { permanent_uuid: "f5b45ccea9c7438f8fba1b79073969e5" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 40795 } } }
W20250112 15:08:49.497112 24384 catalog_manager.cc:4839] TS a422b4873cf04ae9b9bc6530f2f077a2 (127.17.60.196:33207): delete failed for tablet 45d29678d6704a888f2c2514af2f36a4 because the tablet was not found. No further retry: Not found: Tablet not found: 45d29678d6704a888f2c2514af2f36a4
I20250112 15:08:49.499126 24768 raft_consensus.cc:491] T 94172f983ecd4a37956654f2f86dba00 P 56dd875b6ba54b40877146f15292bf2f [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 52bd510945874fa781ec841d40e251ab)
I20250112 15:08:49.499181 24768 raft_consensus.cc:513] T 94172f983ecd4a37956654f2f86dba00 P 56dd875b6ba54b40877146f15292bf2f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52bd510945874fa781ec841d40e251ab" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 34851 } } peers { permanent_uuid: "56dd875b6ba54b40877146f15292bf2f" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 32969 } } peers { permanent_uuid: "f5b45ccea9c7438f8fba1b79073969e5" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 40795 } }
I20250112 15:08:49.499305 24768 leader_election.cc:290] T 94172f983ecd4a37956654f2f86dba00 P 56dd875b6ba54b40877146f15292bf2f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 52bd510945874fa781ec841d40e251ab (127.17.60.193:34851), f5b45ccea9c7438f8fba1b79073969e5 (127.17.60.195:40795)
I20250112 15:08:49.499647 24721 raft_consensus.cc:491] T 94172f983ecd4a37956654f2f86dba00 P f5b45ccea9c7438f8fba1b79073969e5 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 52bd510945874fa781ec841d40e251ab)
I20250112 15:08:49.499699 24721 raft_consensus.cc:513] T 94172f983ecd4a37956654f2f86dba00 P f5b45ccea9c7438f8fba1b79073969e5 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52bd510945874fa781ec841d40e251ab" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 34851 } } peers { permanent_uuid: "56dd875b6ba54b40877146f15292bf2f" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 32969 } } peers { permanent_uuid: "f5b45ccea9c7438f8fba1b79073969e5" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 40795 } }
I20250112 15:08:49.499724 24651 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "94172f983ecd4a37956654f2f86dba00" candidate_uuid: "56dd875b6ba54b40877146f15292bf2f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f5b45ccea9c7438f8fba1b79073969e5" is_pre_election: true
I20250112 15:08:49.499770 24651 raft_consensus.cc:2463] T 94172f983ecd4a37956654f2f86dba00 P f5b45ccea9c7438f8fba1b79073969e5 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 56dd875b6ba54b40877146f15292bf2f in term 1.
I20250112 15:08:49.499804 24721 leader_election.cc:290] T 94172f983ecd4a37956654f2f86dba00 P f5b45ccea9c7438f8fba1b79073969e5 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 52bd510945874fa781ec841d40e251ab (127.17.60.193:34851), 56dd875b6ba54b40877146f15292bf2f (127.17.60.194:32969)
I20250112 15:08:49.499922 24539 leader_election.cc:304] T 94172f983ecd4a37956654f2f86dba00 P 56dd875b6ba54b40877146f15292bf2f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 56dd875b6ba54b40877146f15292bf2f, f5b45ccea9c7438f8fba1b79073969e5; no voters:
I20250112 15:08:49.500001 24768 raft_consensus.cc:2798] T 94172f983ecd4a37956654f2f86dba00 P 56dd875b6ba54b40877146f15292bf2f [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250112 15:08:49.500041 24768 raft_consensus.cc:491] T 94172f983ecd4a37956654f2f86dba00 P 56dd875b6ba54b40877146f15292bf2f [term 1 FOLLOWER]: Starting leader election (detected failure of leader 52bd510945874fa781ec841d40e251ab)
I20250112 15:08:49.500057 24768 raft_consensus.cc:3054] T 94172f983ecd4a37956654f2f86dba00 P 56dd875b6ba54b40877146f15292bf2f [term 1 FOLLOWER]: Advancing to term 2
I20250112 15:08:49.500048 24576 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "94172f983ecd4a37956654f2f86dba00" candidate_uuid: "f5b45ccea9c7438f8fba1b79073969e5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "56dd875b6ba54b40877146f15292bf2f" is_pre_election: true
I20250112 15:08:49.500753 24768 raft_consensus.cc:513] T 94172f983ecd4a37956654f2f86dba00 P 56dd875b6ba54b40877146f15292bf2f [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52bd510945874fa781ec841d40e251ab" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 34851 } } peers { permanent_uuid: "56dd875b6ba54b40877146f15292bf2f" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 32969 } } peers { permanent_uuid: "f5b45ccea9c7438f8fba1b79073969e5" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 40795 } }
I20250112 15:08:49.500850 24576 raft_consensus.cc:2388] T 94172f983ecd4a37956654f2f86dba00 P 56dd875b6ba54b40877146f15292bf2f [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f5b45ccea9c7438f8fba1b79073969e5 in current term 2: Already voted for candidate 56dd875b6ba54b40877146f15292bf2f in this term.
I20250112 15:08:49.500862 24768 leader_election.cc:290] T 94172f983ecd4a37956654f2f86dba00 P 56dd875b6ba54b40877146f15292bf2f [CANDIDATE]: Term 2 election: Requested vote from peers 52bd510945874fa781ec841d40e251ab (127.17.60.193:34851), f5b45ccea9c7438f8fba1b79073969e5 (127.17.60.195:40795)
I20250112 15:08:49.501016 24651 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "94172f983ecd4a37956654f2f86dba00" candidate_uuid: "56dd875b6ba54b40877146f15292bf2f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f5b45ccea9c7438f8fba1b79073969e5"
I20250112 15:08:49.501086 24651 raft_consensus.cc:3054] T 94172f983ecd4a37956654f2f86dba00 P f5b45ccea9c7438f8fba1b79073969e5 [term 1 FOLLOWER]: Advancing to term 2
I20250112 15:08:49.501479 24651 raft_consensus.cc:2463] T 94172f983ecd4a37956654f2f86dba00 P f5b45ccea9c7438f8fba1b79073969e5 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 56dd875b6ba54b40877146f15292bf2f in term 2.
I20250112 15:08:49.501637 24539 leader_election.cc:304] T 94172f983ecd4a37956654f2f86dba00 P 56dd875b6ba54b40877146f15292bf2f [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 56dd875b6ba54b40877146f15292bf2f, f5b45ccea9c7438f8fba1b79073969e5; no voters:
I20250112 15:08:49.501737 24904 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "94172f983ecd4a37956654f2f86dba00" candidate_uuid: "f5b45ccea9c7438f8fba1b79073969e5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "52bd510945874fa781ec841d40e251ab" is_pre_election: true
I20250112 15:08:49.501741 24768 raft_consensus.cc:2798] T 94172f983ecd4a37956654f2f86dba00 P 56dd875b6ba54b40877146f15292bf2f [term 2 FOLLOWER]: Leader election won for term 2
I20250112 15:08:49.501808 24904 raft_consensus.cc:2463] T 94172f983ecd4a37956654f2f86dba00 P 52bd510945874fa781ec841d40e251ab [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f5b45ccea9c7438f8fba1b79073969e5 in term 1.
I20250112 15:08:49.501827 24768 raft_consensus.cc:695] T 94172f983ecd4a37956654f2f86dba00 P 56dd875b6ba54b40877146f15292bf2f [term 2 LEADER]: Becoming Leader. State: Replica: 56dd875b6ba54b40877146f15292bf2f, State: Running, Role: LEADER
I20250112 15:08:49.502048 24768 consensus_queue.cc:237] T 94172f983ecd4a37956654f2f86dba00 P 56dd875b6ba54b40877146f15292bf2f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52bd510945874fa781ec841d40e251ab" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 34851 } } peers { permanent_uuid: "56dd875b6ba54b40877146f15292bf2f" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 32969 } } peers { permanent_uuid: "f5b45ccea9c7438f8fba1b79073969e5" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 40795 } }
I20250112 15:08:49.502118 24903 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "94172f983ecd4a37956654f2f86dba00" candidate_uuid: "56dd875b6ba54b40877146f15292bf2f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "52bd510945874fa781ec841d40e251ab" is_pre_election: true
I20250112 15:08:49.502122 24902 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "94172f983ecd4a37956654f2f86dba00" candidate_uuid: "56dd875b6ba54b40877146f15292bf2f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "52bd510945874fa781ec841d40e251ab"
I20250112 15:08:49.502559 24614 leader_election.cc:304] T 94172f983ecd4a37956654f2f86dba00 P f5b45ccea9c7438f8fba1b79073969e5 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 52bd510945874fa781ec841d40e251ab, f5b45ccea9c7438f8fba1b79073969e5; no voters: 56dd875b6ba54b40877146f15292bf2f
I20250112 15:08:49.502640 24721 raft_consensus.cc:2758] T 94172f983ecd4a37956654f2f86dba00 P f5b45ccea9c7438f8fba1b79073969e5 [term 2 FOLLOWER]: Leader pre-election decision vote started in defunct term 1: won
I20250112 15:08:49.502672 24397 catalog_manager.cc:5526] T 94172f983ecd4a37956654f2f86dba00 P 56dd875b6ba54b40877146f15292bf2f reported cstate change: term changed from 1 to 2, leader changed from 52bd510945874fa781ec841d40e251ab (127.17.60.193) to 56dd875b6ba54b40877146f15292bf2f (127.17.60.194). New cstate: current_term: 2 leader_uuid: "56dd875b6ba54b40877146f15292bf2f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52bd510945874fa781ec841d40e251ab" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 34851 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "56dd875b6ba54b40877146f15292bf2f" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 32969 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f5b45ccea9c7438f8fba1b79073969e5" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 40795 } health_report { overall_health: UNKNOWN } } }
I20250112 15:08:49.739773 24955 ts_tablet_manager.cc:927] T c73b4e40028649e5bb9ad937f307e7bf P a422b4873cf04ae9b9bc6530f2f077a2: Initiating tablet copy from peer f5b45ccea9c7438f8fba1b79073969e5 (127.17.60.195:40795)
I20250112 15:08:49.739919 24955 tablet_copy_client.cc:323] T c73b4e40028649e5bb9ad937f307e7bf P a422b4873cf04ae9b9bc6530f2f077a2: tablet copy: Beginning tablet copy session from remote peer at address 127.17.60.195:40795
I20250112 15:08:49.742367 24661 tablet_copy_service.cc:140] P f5b45ccea9c7438f8fba1b79073969e5: Received BeginTabletCopySession request for tablet c73b4e40028649e5bb9ad937f307e7bf from peer a422b4873cf04ae9b9bc6530f2f077a2 ({username='slave'} at 127.0.0.1:38216)
I20250112 15:08:49.742435 24661 tablet_copy_service.cc:161] P f5b45ccea9c7438f8fba1b79073969e5: Beginning new tablet copy session on tablet c73b4e40028649e5bb9ad937f307e7bf from peer a422b4873cf04ae9b9bc6530f2f077a2 at {username='slave'} at 127.0.0.1:38216: session id = a422b4873cf04ae9b9bc6530f2f077a2-c73b4e40028649e5bb9ad937f307e7bf
I20250112 15:08:49.742935 24661 tablet_copy_source_session.cc:215] T c73b4e40028649e5bb9ad937f307e7bf P f5b45ccea9c7438f8fba1b79073969e5: Tablet Copy: opened 0 blocks and 1 log segments
I20250112 15:08:49.743294 24955 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c73b4e40028649e5bb9ad937f307e7bf. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 15:08:49.744279 24955 tablet_copy_client.cc:806] T c73b4e40028649e5bb9ad937f307e7bf P a422b4873cf04ae9b9bc6530f2f077a2: tablet copy: Starting download of 0 data blocks...
I20250112 15:08:49.744423 24955 tablet_copy_client.cc:670] T c73b4e40028649e5bb9ad937f307e7bf P a422b4873cf04ae9b9bc6530f2f077a2: tablet copy: Starting download of 1 WAL segments...
I20250112 15:08:49.744990 24955 tablet_copy_client.cc:538] T c73b4e40028649e5bb9ad937f307e7bf P a422b4873cf04ae9b9bc6530f2f077a2: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250112 15:08:49.745507 24955 tablet_bootstrap.cc:492] T c73b4e40028649e5bb9ad937f307e7bf P a422b4873cf04ae9b9bc6530f2f077a2: Bootstrap starting.
I20250112 15:08:49.746479 24955 tablet_bootstrap.cc:492] T c73b4e40028649e5bb9ad937f307e7bf P a422b4873cf04ae9b9bc6530f2f077a2: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250112 15:08:49.746601 24955 tablet_bootstrap.cc:492] T c73b4e40028649e5bb9ad937f307e7bf P a422b4873cf04ae9b9bc6530f2f077a2: Bootstrap complete.
I20250112 15:08:49.746675 24955 ts_tablet_manager.cc:1397] T c73b4e40028649e5bb9ad937f307e7bf P a422b4873cf04ae9b9bc6530f2f077a2: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20250112 15:08:49.746796 24955 raft_consensus.cc:357] T c73b4e40028649e5bb9ad937f307e7bf P a422b4873cf04ae9b9bc6530f2f077a2 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "56dd875b6ba54b40877146f15292bf2f" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 32969 } } peers { permanent_uuid: "52bd510945874fa781ec841d40e251ab" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 34851 } } peers { permanent_uuid: "f5b45ccea9c7438f8fba1b79073969e5" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 40795 } } peers { permanent_uuid: "a422b4873cf04ae9b9bc6530f2f077a2" member_type: NON_VOTER last_known_addr { host: "127.17.60.196" port: 33207 } attrs { promote: true } }
I20250112 15:08:49.746865 24955 raft_consensus.cc:738] T c73b4e40028649e5bb9ad937f307e7bf P a422b4873cf04ae9b9bc6530f2f077a2 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: a422b4873cf04ae9b9bc6530f2f077a2, State: Initialized, Role: LEARNER
I20250112 15:08:49.746886 24955 consensus_queue.cc:260] T c73b4e40028649e5bb9ad937f307e7bf P a422b4873cf04ae9b9bc6530f2f077a2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "56dd875b6ba54b40877146f15292bf2f" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 32969 } } peers { permanent_uuid: "52bd510945874fa781ec841d40e251ab" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 34851 } } peers { permanent_uuid: "f5b45ccea9c7438f8fba1b79073969e5" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 40795 } } peers { permanent_uuid: "a422b4873cf04ae9b9bc6530f2f077a2" member_type: NON_VOTER last_known_addr { host: "127.17.60.196" port: 33207 } attrs { promote: true } }
I20250112 15:08:49.746984 24955 ts_tablet_manager.cc:1428] T c73b4e40028649e5bb9ad937f307e7bf P a422b4873cf04ae9b9bc6530f2f077a2: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20250112 15:08:49.747040 24851 heartbeater.cc:502] Master 127.17.60.254:37937 was elected leader, sending a full tablet report...
I20250112 15:08:49.747150 24661 tablet_copy_service.cc:342] P f5b45ccea9c7438f8fba1b79073969e5: Request end of tablet copy session a422b4873cf04ae9b9bc6530f2f077a2-c73b4e40028649e5bb9ad937f307e7bf received from {username='slave'} at 127.0.0.1:38216
I20250112 15:08:49.747208 24661 tablet_copy_service.cc:434] P f5b45ccea9c7438f8fba1b79073969e5: ending tablet copy session a422b4873cf04ae9b9bc6530f2f077a2-c73b4e40028649e5bb9ad937f307e7bf on tablet c73b4e40028649e5bb9ad937f307e7bf with peer a422b4873cf04ae9b9bc6530f2f077a2
I20250112 15:08:49.864526 24651 consensus_queue.cc:237] T c73b4e40028649e5bb9ad937f307e7bf P f5b45ccea9c7438f8fba1b79073969e5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "56dd875b6ba54b40877146f15292bf2f" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 32969 } } peers { permanent_uuid: "52bd510945874fa781ec841d40e251ab" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 34851 } } peers { permanent_uuid: "f5b45ccea9c7438f8fba1b79073969e5" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 40795 } }
I20250112 15:08:49.865012 24904 raft_consensus.cc:1270] T c73b4e40028649e5bb9ad937f307e7bf P 52bd510945874fa781ec841d40e251ab [term 1 FOLLOWER]: Refusing update from remote peer f5b45ccea9c7438f8fba1b79073969e5: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250112 15:08:49.865012 24576 raft_consensus.cc:1270] T c73b4e40028649e5bb9ad937f307e7bf P 56dd875b6ba54b40877146f15292bf2f [term 1 FOLLOWER]: Refusing update from remote peer f5b45ccea9c7438f8fba1b79073969e5: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250112 15:08:49.865252 24704 consensus_queue.cc:1035] T c73b4e40028649e5bb9ad937f307e7bf P f5b45ccea9c7438f8fba1b79073969e5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "56dd875b6ba54b40877146f15292bf2f" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 32969 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250112 15:08:49.865319 24756 consensus_queue.cc:1035] T c73b4e40028649e5bb9ad937f307e7bf P f5b45ccea9c7438f8fba1b79073969e5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "52bd510945874fa781ec841d40e251ab" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 34851 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250112 15:08:49.865901 24704 raft_consensus.cc:2949] T c73b4e40028649e5bb9ad937f307e7bf P f5b45ccea9c7438f8fba1b79073969e5 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, NON_VOTER a422b4873cf04ae9b9bc6530f2f077a2 (127.17.60.196) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "56dd875b6ba54b40877146f15292bf2f" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 32969 } } peers { permanent_uuid: "52bd510945874fa781ec841d40e251ab" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 34851 } } peers { permanent_uuid: "f5b45ccea9c7438f8fba1b79073969e5" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 40795 } } }
I20250112 15:08:49.866035 24576 raft_consensus.cc:2949] T c73b4e40028649e5bb9ad937f307e7bf P 56dd875b6ba54b40877146f15292bf2f [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, NON_VOTER a422b4873cf04ae9b9bc6530f2f077a2 (127.17.60.196) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "56dd875b6ba54b40877146f15292bf2f" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 32969 } } peers { permanent_uuid: "52bd510945874fa781ec841d40e251ab" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 34851 } } peers { permanent_uuid: "f5b45ccea9c7438f8fba1b79073969e5" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 40795 } } }
I20250112 15:08:49.866909 24904 raft_consensus.cc:2949] T c73b4e40028649e5bb9ad937f307e7bf P 52bd510945874fa781ec841d40e251ab [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, NON_VOTER a422b4873cf04ae9b9bc6530f2f077a2 (127.17.60.196) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "56dd875b6ba54b40877146f15292bf2f" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 32969 } } peers { permanent_uuid: "52bd510945874fa781ec841d40e251ab" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 34851 } } peers { permanent_uuid: "f5b45ccea9c7438f8fba1b79073969e5" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 40795 } } }
I20250112 15:08:49.867067 24397 catalog_manager.cc:5526] T c73b4e40028649e5bb9ad937f307e7bf P 56dd875b6ba54b40877146f15292bf2f reported cstate change: config changed from index 2 to 3, NON_VOTER a422b4873cf04ae9b9bc6530f2f077a2 (127.17.60.196) evicted. New cstate: current_term: 1 leader_uuid: "f5b45ccea9c7438f8fba1b79073969e5" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "56dd875b6ba54b40877146f15292bf2f" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 32969 } } peers { permanent_uuid: "52bd510945874fa781ec841d40e251ab" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 34851 } } peers { permanent_uuid: "f5b45ccea9c7438f8fba1b79073969e5" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 40795 } } }
I20250112 15:08:49.867857 24815 tablet_service.cc:1514] Processing DeleteTablet for tablet c73b4e40028649e5bb9ad937f307e7bf with delete_type TABLET_DATA_TOMBSTONED (TS a422b4873cf04ae9b9bc6530f2f077a2 not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:46800
I20250112 15:08:49.867972 24948 tablet_replica.cc:331] T c73b4e40028649e5bb9ad937f307e7bf P a422b4873cf04ae9b9bc6530f2f077a2: stopping tablet replica
I20250112 15:08:49.868026 24948 raft_consensus.cc:2238] T c73b4e40028649e5bb9ad937f307e7bf P a422b4873cf04ae9b9bc6530f2f077a2 [term 1 LEARNER]: Raft consensus shutting down.
I20250112 15:08:49.868062 24948 raft_consensus.cc:2267] T c73b4e40028649e5bb9ad937f307e7bf P a422b4873cf04ae9b9bc6530f2f077a2 [term 1 LEARNER]: Raft consensus is shut down!
I20250112 15:08:49.867880 24381 catalog_manager.cc:5039] ChangeConfig:REMOVE_PEER RPC for tablet c73b4e40028649e5bb9ad937f307e7bf with cas_config_opid_index 2: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250112 15:08:49.868268 24948 ts_tablet_manager.cc:1905] T c73b4e40028649e5bb9ad937f307e7bf P a422b4873cf04ae9b9bc6530f2f077a2: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250112 15:08:49.869465 24948 ts_tablet_manager.cc:1918] T c73b4e40028649e5bb9ad937f307e7bf P a422b4873cf04ae9b9bc6530f2f077a2: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20250112 15:08:49.869519 24948 log.cc:1198] T c73b4e40028649e5bb9ad937f307e7bf P a422b4873cf04ae9b9bc6530f2f077a2: Deleting WAL directory at /tmp/dist-test-task95dLSD/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestDeletedTables.1736694351070047-17651-0/minicluster-data/ts-3-root/wals/c73b4e40028649e5bb9ad937f307e7bf
I20250112 15:08:49.869836 24384 catalog_manager.cc:4872] TS a422b4873cf04ae9b9bc6530f2f077a2 (127.17.60.196:33207): tablet c73b4e40028649e5bb9ad937f307e7bf (table test-workload [id=a7cea0e15506428cabd1af7ad2081b4c]) successfully deleted
I20250112 15:08:49.912659 24651 raft_consensus.cc:1270] T 94172f983ecd4a37956654f2f86dba00 P f5b45ccea9c7438f8fba1b79073969e5 [term 2 FOLLOWER]: Refusing update from remote peer 56dd875b6ba54b40877146f15292bf2f: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250112 15:08:49.912953 24768 consensus_queue.cc:1035] T 94172f983ecd4a37956654f2f86dba00 P 56dd875b6ba54b40877146f15292bf2f [LEADER]: Connected to new peer: Peer: permanent_uuid: "f5b45ccea9c7438f8fba1b79073969e5" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 40795 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250112 15:08:49.913964 24904 raft_consensus.cc:3054] T 94172f983ecd4a37956654f2f86dba00 P 52bd510945874fa781ec841d40e251ab [term 1 FOLLOWER]: Advancing to term 2
I20250112 15:08:49.914703 24904 raft_consensus.cc:1270] T 94172f983ecd4a37956654f2f86dba00 P 52bd510945874fa781ec841d40e251ab [term 2 FOLLOWER]: Refusing update from remote peer 56dd875b6ba54b40877146f15292bf2f: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250112 15:08:49.914881 24702 consensus_queue.cc:1035] T 94172f983ecd4a37956654f2f86dba00 P 56dd875b6ba54b40877146f15292bf2f [LEADER]: Connected to new peer: Peer: permanent_uuid: "52bd510945874fa781ec841d40e251ab" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 34851 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250112 15:08:50.459597 24576 consensus_queue.cc:237] T 94172f983ecd4a37956654f2f86dba00 P 56dd875b6ba54b40877146f15292bf2f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "52bd510945874fa781ec841d40e251ab" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 34851 } } peers { permanent_uuid: "56dd875b6ba54b40877146f15292bf2f" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 32969 } } peers { permanent_uuid: "f5b45ccea9c7438f8fba1b79073969e5" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 40795 } attrs { replace: true } } peers { permanent_uuid: "a422b4873cf04ae9b9bc6530f2f077a2" member_type: NON_VOTER last_known_addr { host: "127.17.60.196" port: 33207 } attrs { promote: true } }
I20250112 15:08:50.460256 24904 raft_consensus.cc:1270] T 94172f983ecd4a37956654f2f86dba00 P 52bd510945874fa781ec841d40e251ab [term 2 FOLLOWER]: Refusing update from remote peer 56dd875b6ba54b40877146f15292bf2f: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20250112 15:08:50.460291 24651 raft_consensus.cc:1270] T 94172f983ecd4a37956654f2f86dba00 P f5b45ccea9c7438f8fba1b79073969e5 [term 2 FOLLOWER]: Refusing update from remote peer 56dd875b6ba54b40877146f15292bf2f: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20250112 15:08:50.460464 24967 consensus_queue.cc:1035] T 94172f983ecd4a37956654f2f86dba00 P 56dd875b6ba54b40877146f15292bf2f [LEADER]: Connected to new peer: Peer: permanent_uuid: "f5b45ccea9c7438f8fba1b79073969e5" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 40795 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250112 15:08:50.460606 24968 consensus_queue.cc:1035] T 94172f983ecd4a37956654f2f86dba00 P 56dd875b6ba54b40877146f15292bf2f [LEADER]: Connected to new peer: Peer: permanent_uuid: "52bd510945874fa781ec841d40e251ab" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 34851 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250112 15:08:50.460994 24967 raft_consensus.cc:2949] T 94172f983ecd4a37956654f2f86dba00 P 56dd875b6ba54b40877146f15292bf2f [term 2 LEADER]: Committing config change with OpId 2.3: config changed from index -1 to 3, NON_VOTER a422b4873cf04ae9b9bc6530f2f077a2 (127.17.60.196) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "52bd510945874fa781ec841d40e251ab" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 34851 } } peers { permanent_uuid: "56dd875b6ba54b40877146f15292bf2f" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 32969 } } peers { permanent_uuid: "f5b45ccea9c7438f8fba1b79073969e5" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 40795 } attrs { replace: true } } peers { permanent_uuid: "a422b4873cf04ae9b9bc6530f2f077a2" member_type: NON_VOTER last_known_addr { host: "127.17.60.196" port: 33207 } attrs { promote: true } } }
I20250112 15:08:50.461280 24904 raft_consensus.cc:2949] T 94172f983ecd4a37956654f2f86dba00 P 52bd510945874fa781ec841d40e251ab [term 2 FOLLOWER]: Committing config change with OpId 2.3: config changed from index -1 to 3, NON_VOTER a422b4873cf04ae9b9bc6530f2f077a2 (127.17.60.196) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "52bd510945874fa781ec841d40e251ab" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 34851 } } peers { permanent_uuid: "56dd875b6ba54b40877146f15292bf2f" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 32969 } } peers { permanent_uuid: "f5b45ccea9c7438f8fba1b79073969e5" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 40795 } attrs { replace: true } } peers { permanent_uuid: "a422b4873cf04ae9b9bc6530f2f077a2" member_type: NON_VOTER last_known_addr { host: "127.17.60.196" port: 33207 } attrs { promote: true } } }
I20250112 15:08:50.462263 24397 catalog_manager.cc:5526] T 94172f983ecd4a37956654f2f86dba00 P 56dd875b6ba54b40877146f15292bf2f reported cstate change: config changed from index -1 to 3, NON_VOTER a422b4873cf04ae9b9bc6530f2f077a2 (127.17.60.196) added. New cstate: current_term: 2 leader_uuid: "56dd875b6ba54b40877146f15292bf2f" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "52bd510945874fa781ec841d40e251ab" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 34851 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "56dd875b6ba54b40877146f15292bf2f" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 32969 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f5b45ccea9c7438f8fba1b79073969e5" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 40795 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a422b4873cf04ae9b9bc6530f2f077a2" member_type: NON_VOTER last_known_addr { host: "127.17.60.196" port: 33207 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20250112 15:08:50.462394 24651 consensus_queue.cc:237] T cc5c837c3de946a5952f656946d883c8 P f5b45ccea9c7438f8fba1b79073969e5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52bd510945874fa781ec841d40e251ab" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 34851 } attrs { replace: true } } peers { permanent_uuid: "56dd875b6ba54b40877146f15292bf2f" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 32969 } } peers { permanent_uuid: "f5b45ccea9c7438f8fba1b79073969e5" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 40795 } } peers { permanent_uuid: "a422b4873cf04ae9b9bc6530f2f077a2" member_type: NON_VOTER last_known_addr { host: "127.17.60.196" port: 33207 } attrs { promote: true } }
I20250112 15:08:50.462796 24650 raft_consensus.cc:2949] T 94172f983ecd4a37956654f2f86dba00 P f5b45ccea9c7438f8fba1b79073969e5 [term 2 FOLLOWER]: Committing config change with OpId 2.3: config changed from index -1 to 3, NON_VOTER a422b4873cf04ae9b9bc6530f2f077a2 (127.17.60.196) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "52bd510945874fa781ec841d40e251ab" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 34851 } } peers { permanent_uuid: "56dd875b6ba54b40877146f15292bf2f" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 32969 } } peers { permanent_uuid: "f5b45ccea9c7438f8fba1b79073969e5" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 40795 } attrs { replace: true } } peers { permanent_uuid: "a422b4873cf04ae9b9bc6530f2f077a2" member_type: NON_VOTER last_known_addr { host: "127.17.60.196" port: 33207 } attrs { promote: true } } }
I20250112 15:08:50.463143 24576 raft_consensus.cc:1270] T cc5c837c3de946a5952f656946d883c8 P 56dd875b6ba54b40877146f15292bf2f [term 1 FOLLOWER]: Refusing update from remote peer f5b45ccea9c7438f8fba1b79073969e5: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20250112 15:08:50.463150 24904 raft_consensus.cc:1270] T cc5c837c3de946a5952f656946d883c8 P 52bd510945874fa781ec841d40e251ab [term 1 FOLLOWER]: Refusing update from remote peer f5b45ccea9c7438f8fba1b79073969e5: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20250112 15:08:50.463366 24975 consensus_queue.cc:1035] T cc5c837c3de946a5952f656946d883c8 P f5b45ccea9c7438f8fba1b79073969e5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "52bd510945874fa781ec841d40e251ab" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 34851 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20250112 15:08:50.463462 24617 consensus_peers.cc:487] T cc5c837c3de946a5952f656946d883c8 P f5b45ccea9c7438f8fba1b79073969e5 -> Peer a422b4873cf04ae9b9bc6530f2f077a2 (127.17.60.196:33207): Couldn't send request to peer a422b4873cf04ae9b9bc6530f2f077a2. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: cc5c837c3de946a5952f656946d883c8. This is attempt 1: this message will repeat every 5th retry.
I20250112 15:08:50.463502 24976 consensus_queue.cc:1035] T cc5c837c3de946a5952f656946d883c8 P f5b45ccea9c7438f8fba1b79073969e5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "56dd875b6ba54b40877146f15292bf2f" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 32969 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250112 15:08:50.464529 24721 raft_consensus.cc:2949] T cc5c837c3de946a5952f656946d883c8 P f5b45ccea9c7438f8fba1b79073969e5 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, NON_VOTER a422b4873cf04ae9b9bc6530f2f077a2 (127.17.60.196) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52bd510945874fa781ec841d40e251ab" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 34851 } attrs { replace: true } } peers { permanent_uuid: "56dd875b6ba54b40877146f15292bf2f" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 32969 } } peers { permanent_uuid: "f5b45ccea9c7438f8fba1b79073969e5" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 40795 } } peers { permanent_uuid: "a422b4873cf04ae9b9bc6530f2f077a2" member_type: NON_VOTER last_known_addr { host: "127.17.60.196" port: 33207 } attrs { promote: true } } }
I20250112 15:08:50.464764 24576 raft_consensus.cc:2949] T cc5c837c3de946a5952f656946d883c8 P 56dd875b6ba54b40877146f15292bf2f [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, NON_VOTER a422b4873cf04ae9b9bc6530f2f077a2 (127.17.60.196) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52bd510945874fa781ec841d40e251ab" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 34851 } attrs { replace: true } } peers { permanent_uuid: "56dd875b6ba54b40877146f15292bf2f" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 32969 } } peers { permanent_uuid: "f5b45ccea9c7438f8fba1b79073969e5" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 40795 } } peers { permanent_uuid: "a422b4873cf04ae9b9bc6530f2f077a2" member_type: NON_VOTER last_known_addr { host: "127.17.60.196" port: 33207 } attrs { promote: true } } }
I20250112 15:08:50.464829 24904 raft_consensus.cc:2949] T cc5c837c3de946a5952f656946d883c8 P 52bd510945874fa781ec841d40e251ab [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, NON_VOTER a422b4873cf04ae9b9bc6530f2f077a2 (127.17.60.196) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52bd510945874fa781ec841d40e251ab" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 34851 } attrs { replace: true } } peers { permanent_uuid: "56dd875b6ba54b40877146f15292bf2f" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 32969 } } peers { permanent_uuid: "f5b45ccea9c7438f8fba1b79073969e5" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 40795 } } peers { permanent_uuid: "a422b4873cf04ae9b9bc6530f2f077a2" member_type: NON_VOTER last_known_addr { host: "127.17.60.196" port: 33207 } attrs { promote: true } } }
W20250112 15:08:50.465512 24542 consensus_peers.cc:487] T 94172f983ecd4a37956654f2f86dba00 P 56dd875b6ba54b40877146f15292bf2f -> Peer a422b4873cf04ae9b9bc6530f2f077a2 (127.17.60.196:33207): Couldn't send request to peer a422b4873cf04ae9b9bc6530f2f077a2. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 94172f983ecd4a37956654f2f86dba00. This is attempt 1: this message will repeat every 5th retry.
I20250112 15:08:50.465873 24395 catalog_manager.cc:5526] T cc5c837c3de946a5952f656946d883c8 P 52bd510945874fa781ec841d40e251ab reported cstate change: config changed from index 3 to 4, NON_VOTER a422b4873cf04ae9b9bc6530f2f077a2 (127.17.60.196) added. New cstate: current_term: 1 leader_uuid: "f5b45ccea9c7438f8fba1b79073969e5" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52bd510945874fa781ec841d40e251ab" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 34851 } attrs { replace: true } } peers { permanent_uuid: "56dd875b6ba54b40877146f15292bf2f" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 32969 } } peers { permanent_uuid: "f5b45ccea9c7438f8fba1b79073969e5" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 40795 } } peers { permanent_uuid: "a422b4873cf04ae9b9bc6530f2f077a2" member_type: NON_VOTER last_known_addr { host: "127.17.60.196" port: 33207 } attrs { promote: true } } }
I20250112 15:08:50.466109 24650 consensus_queue.cc:237] T c73b4e40028649e5bb9ad937f307e7bf P f5b45ccea9c7438f8fba1b79073969e5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "56dd875b6ba54b40877146f15292bf2f" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 32969 } attrs { replace: true } } peers { permanent_uuid: "52bd510945874fa781ec841d40e251ab" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 34851 } } peers { permanent_uuid: "f5b45ccea9c7438f8fba1b79073969e5" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 40795 } } peers { permanent_uuid: "a422b4873cf04ae9b9bc6530f2f077a2" member_type: NON_VOTER last_known_addr { host: "127.17.60.196" port: 33207 } attrs { promote: true } }
W20250112 15:08:50.466554 24617 consensus_peers.cc:487] T c73b4e40028649e5bb9ad937f307e7bf P f5b45ccea9c7438f8fba1b79073969e5 -> Peer a422b4873cf04ae9b9bc6530f2f077a2 (127.17.60.196:33207): Couldn't send request to peer a422b4873cf04ae9b9bc6530f2f077a2. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20250112 15:08:50.466784 24576 raft_consensus.cc:1270] T c73b4e40028649e5bb9ad937f307e7bf P 56dd875b6ba54b40877146f15292bf2f [term 1 FOLLOWER]: Refusing update from remote peer f5b45ccea9c7438f8fba1b79073969e5: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20250112 15:08:50.466868 24904 raft_consensus.cc:1270] T c73b4e40028649e5bb9ad937f307e7bf P 52bd510945874fa781ec841d40e251ab [term 1 FOLLOWER]: Refusing update from remote peer f5b45ccea9c7438f8fba1b79073969e5: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20250112 15:08:50.466984 24976 consensus_queue.cc:1035] T c73b4e40028649e5bb9ad937f307e7bf P f5b45ccea9c7438f8fba1b79073969e5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "56dd875b6ba54b40877146f15292bf2f" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 32969 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250112 15:08:50.467087 24975 consensus_queue.cc:1035] T c73b4e40028649e5bb9ad937f307e7bf P f5b45ccea9c7438f8fba1b79073969e5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "52bd510945874fa781ec841d40e251ab" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 34851 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250112 15:08:50.467661 24721 raft_consensus.cc:2949] T c73b4e40028649e5bb9ad937f307e7bf P f5b45ccea9c7438f8fba1b79073969e5 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, NON_VOTER a422b4873cf04ae9b9bc6530f2f077a2 (127.17.60.196) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "56dd875b6ba54b40877146f15292bf2f" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 32969 } attrs { replace: true } } peers { permanent_uuid: "52bd510945874fa781ec841d40e251ab" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 34851 } } peers { permanent_uuid: "f5b45ccea9c7438f8fba1b79073969e5" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 40795 } } peers { permanent_uuid: "a422b4873cf04ae9b9bc6530f2f077a2" member_type: NON_VOTER last_known_addr { host: "127.17.60.196" port: 33207 } attrs { promote: true } } }
I20250112 15:08:50.467762 24576 raft_consensus.cc:2949] T c73b4e40028649e5bb9ad937f307e7bf P 56dd875b6ba54b40877146f15292bf2f [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, NON_VOTER a422b4873cf04ae9b9bc6530f2f077a2 (127.17.60.196) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "56dd875b6ba54b40877146f15292bf2f" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 32969 } attrs { replace: true } } peers { permanent_uuid: "52bd510945874fa781ec841d40e251ab" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 34851 } } peers { permanent_uuid: "f5b45ccea9c7438f8fba1b79073969e5" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 40795 } } peers { permanent_uuid: "a422b4873cf04ae9b9bc6530f2f077a2" member_type: NON_VOTER last_known_addr { host: "127.17.60.196" port: 33207 } attrs { promote: true } } }
I20250112 15:08:50.467809 24904 raft_consensus.cc:2949] T c73b4e40028649e5bb9ad937f307e7bf P 52bd510945874fa781ec841d40e251ab [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, NON_VOTER a422b4873cf04ae9b9bc6530f2f077a2 (127.17.60.196) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "56dd875b6ba54b40877146f15292bf2f" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 32969 } attrs { replace: true } } peers { permanent_uuid: "52bd510945874fa781ec841d40e251ab" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 34851 } } peers { permanent_uuid: "f5b45ccea9c7438f8fba1b79073969e5" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 40795 } } peers { permanent_uuid: "a422b4873cf04ae9b9bc6530f2f077a2" member_type: NON_VOTER last_known_addr { host: "127.17.60.196" port: 33207 } attrs { promote: true } } }
I20250112 15:08:50.468756 24397 catalog_manager.cc:5526] T c73b4e40028649e5bb9ad937f307e7bf P f5b45ccea9c7438f8fba1b79073969e5 reported cstate change: config changed from index 3 to 4, NON_VOTER a422b4873cf04ae9b9bc6530f2f077a2 (127.17.60.196) added. New cstate: current_term: 1 leader_uuid: "f5b45ccea9c7438f8fba1b79073969e5" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "56dd875b6ba54b40877146f15292bf2f" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 32969 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "52bd510945874fa781ec841d40e251ab" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 34851 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f5b45ccea9c7438f8fba1b79073969e5" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 40795 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a422b4873cf04ae9b9bc6530f2f077a2" member_type: NON_VOTER last_known_addr { host: "127.17.60.196" port: 33207 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20250112 15:08:50.954036 24981 ts_tablet_manager.cc:927] T c73b4e40028649e5bb9ad937f307e7bf P a422b4873cf04ae9b9bc6530f2f077a2: Initiating tablet copy from peer f5b45ccea9c7438f8fba1b79073969e5 (127.17.60.195:40795)
I20250112 15:08:50.954277 24981 tablet_copy_client.cc:323] T c73b4e40028649e5bb9ad937f307e7bf P a422b4873cf04ae9b9bc6530f2f077a2: tablet copy: Beginning tablet copy session from remote peer at address 127.17.60.195:40795
I20250112 15:08:50.962198 24661 tablet_copy_service.cc:140] P f5b45ccea9c7438f8fba1b79073969e5: Received BeginTabletCopySession request for tablet c73b4e40028649e5bb9ad937f307e7bf from peer a422b4873cf04ae9b9bc6530f2f077a2 ({username='slave'} at 127.0.0.1:38216)
I20250112 15:08:50.962302 24661 tablet_copy_service.cc:161] P f5b45ccea9c7438f8fba1b79073969e5: Beginning new tablet copy session on tablet c73b4e40028649e5bb9ad937f307e7bf from peer a422b4873cf04ae9b9bc6530f2f077a2 at {username='slave'} at 127.0.0.1:38216: session id = a422b4873cf04ae9b9bc6530f2f077a2-c73b4e40028649e5bb9ad937f307e7bf
I20250112 15:08:50.962951 24661 tablet_copy_source_session.cc:215] T c73b4e40028649e5bb9ad937f307e7bf P f5b45ccea9c7438f8fba1b79073969e5: Tablet Copy: opened 0 blocks and 1 log segments
I20250112 15:08:50.964267 24981 ts_tablet_manager.cc:1905] T c73b4e40028649e5bb9ad937f307e7bf P a422b4873cf04ae9b9bc6530f2f077a2: Deleting tablet data with delete state TABLET_DATA_COPYING
I20250112 15:08:50.966127 24981 ts_tablet_manager.cc:1918] T c73b4e40028649e5bb9ad937f307e7bf P a422b4873cf04ae9b9bc6530f2f077a2: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 1.2
I20250112 15:08:50.966248 24981 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c73b4e40028649e5bb9ad937f307e7bf. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 15:08:50.966945 24981 tablet_copy_client.cc:806] T c73b4e40028649e5bb9ad937f307e7bf P a422b4873cf04ae9b9bc6530f2f077a2: tablet copy: Starting download of 0 data blocks...
I20250112 15:08:50.967084 24981 tablet_copy_client.cc:670] T c73b4e40028649e5bb9ad937f307e7bf P a422b4873cf04ae9b9bc6530f2f077a2: tablet copy: Starting download of 1 WAL segments...
I20250112 15:08:50.967633 24981 tablet_copy_client.cc:538] T c73b4e40028649e5bb9ad937f307e7bf P a422b4873cf04ae9b9bc6530f2f077a2: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250112 15:08:50.968180 24981 tablet_bootstrap.cc:492] T c73b4e40028649e5bb9ad937f307e7bf P a422b4873cf04ae9b9bc6530f2f077a2: Bootstrap starting.
I20250112 15:08:50.969760 24981 tablet_bootstrap.cc:492] T c73b4e40028649e5bb9ad937f307e7bf P a422b4873cf04ae9b9bc6530f2f077a2: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250112 15:08:50.969955 24981 tablet_bootstrap.cc:492] T c73b4e40028649e5bb9ad937f307e7bf P a422b4873cf04ae9b9bc6530f2f077a2: Bootstrap complete.
I20250112 15:08:50.970041 24981 ts_tablet_manager.cc:1397] T c73b4e40028649e5bb9ad937f307e7bf P a422b4873cf04ae9b9bc6530f2f077a2: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20250112 15:08:50.970188 24981 raft_consensus.cc:357] T c73b4e40028649e5bb9ad937f307e7bf P a422b4873cf04ae9b9bc6530f2f077a2 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "56dd875b6ba54b40877146f15292bf2f" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 32969 } attrs { replace: true } } peers { permanent_uuid: "52bd510945874fa781ec841d40e251ab" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 34851 } } peers { permanent_uuid: "f5b45ccea9c7438f8fba1b79073969e5" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 40795 } } peers { permanent_uuid: "a422b4873cf04ae9b9bc6530f2f077a2" member_type: NON_VOTER last_known_addr { host: "127.17.60.196" port: 33207 } attrs { promote: true } }
I20250112 15:08:50.970279 24981 raft_consensus.cc:738] T c73b4e40028649e5bb9ad937f307e7bf P a422b4873cf04ae9b9bc6530f2f077a2 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: a422b4873cf04ae9b9bc6530f2f077a2, State: Initialized, Role: LEARNER
I20250112 15:08:50.970312 24981 consensus_queue.cc:260] T c73b4e40028649e5bb9ad937f307e7bf P a422b4873cf04ae9b9bc6530f2f077a2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "56dd875b6ba54b40877146f15292bf2f" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 32969 } attrs { replace: true } } peers { permanent_uuid: "52bd510945874fa781ec841d40e251ab" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 34851 } } peers { permanent_uuid: "f5b45ccea9c7438f8fba1b79073969e5" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 40795 } } peers { permanent_uuid: "a422b4873cf04ae9b9bc6530f2f077a2" member_type: NON_VOTER last_known_addr { host: "127.17.60.196" port: 33207 } attrs { promote: true } }
I20250112 15:08:50.970449 24981 ts_tablet_manager.cc:1428] T c73b4e40028649e5bb9ad937f307e7bf P a422b4873cf04ae9b9bc6530f2f077a2: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20250112 15:08:50.970656 24661 tablet_copy_service.cc:342] P f5b45ccea9c7438f8fba1b79073969e5: Request end of tablet copy session a422b4873cf04ae9b9bc6530f2f077a2-c73b4e40028649e5bb9ad937f307e7bf received from {username='slave'} at 127.0.0.1:38216
I20250112 15:08:50.970711 24661 tablet_copy_service.cc:434] P f5b45ccea9c7438f8fba1b79073969e5: ending tablet copy session a422b4873cf04ae9b9bc6530f2f077a2-c73b4e40028649e5bb9ad937f307e7bf on tablet c73b4e40028649e5bb9ad937f307e7bf with peer a422b4873cf04ae9b9bc6530f2f077a2
I20250112 15:08:51.009240 24981 ts_tablet_manager.cc:927] T 94172f983ecd4a37956654f2f86dba00 P a422b4873cf04ae9b9bc6530f2f077a2: Initiating tablet copy from peer 56dd875b6ba54b40877146f15292bf2f (127.17.60.194:32969)
I20250112 15:08:51.009442 24981 tablet_copy_client.cc:323] T 94172f983ecd4a37956654f2f86dba00 P a422b4873cf04ae9b9bc6530f2f077a2: tablet copy: Beginning tablet copy session from remote peer at address 127.17.60.194:32969
I20250112 15:08:51.012518 24586 tablet_copy_service.cc:140] P 56dd875b6ba54b40877146f15292bf2f: Received BeginTabletCopySession request for tablet 94172f983ecd4a37956654f2f86dba00 from peer a422b4873cf04ae9b9bc6530f2f077a2 ({username='slave'} at 127.0.0.1:44578)
I20250112 15:08:51.012619 24586 tablet_copy_service.cc:161] P 56dd875b6ba54b40877146f15292bf2f: Beginning new tablet copy session on tablet 94172f983ecd4a37956654f2f86dba00 from peer a422b4873cf04ae9b9bc6530f2f077a2 at {username='slave'} at 127.0.0.1:44578: session id = a422b4873cf04ae9b9bc6530f2f077a2-94172f983ecd4a37956654f2f86dba00
I20250112 15:08:51.013144 24586 tablet_copy_source_session.cc:215] T 94172f983ecd4a37956654f2f86dba00 P 56dd875b6ba54b40877146f15292bf2f: Tablet Copy: opened 0 blocks and 1 log segments
I20250112 15:08:51.013466 24981 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 94172f983ecd4a37956654f2f86dba00. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 15:08:51.014698 24981 tablet_copy_client.cc:806] T 94172f983ecd4a37956654f2f86dba00 P a422b4873cf04ae9b9bc6530f2f077a2: tablet copy: Starting download of 0 data blocks...
I20250112 15:08:51.014811 24981 tablet_copy_client.cc:670] T 94172f983ecd4a37956654f2f86dba00 P a422b4873cf04ae9b9bc6530f2f077a2: tablet copy: Starting download of 1 WAL segments...
I20250112 15:08:51.015312 24981 tablet_copy_client.cc:538] T 94172f983ecd4a37956654f2f86dba00 P a422b4873cf04ae9b9bc6530f2f077a2: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250112 15:08:51.015895 24981 tablet_bootstrap.cc:492] T 94172f983ecd4a37956654f2f86dba00 P a422b4873cf04ae9b9bc6530f2f077a2: Bootstrap starting.
I20250112 15:08:51.016894 24981 tablet_bootstrap.cc:492] T 94172f983ecd4a37956654f2f86dba00 P a422b4873cf04ae9b9bc6530f2f077a2: Bootstrap replayed 1/1 log segments. Stats: ops{read=3 overwritten=0 applied=3 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250112 15:08:51.017078 24981 tablet_bootstrap.cc:492] T 94172f983ecd4a37956654f2f86dba00 P a422b4873cf04ae9b9bc6530f2f077a2: Bootstrap complete.
I20250112 15:08:51.017164 24981 ts_tablet_manager.cc:1397] T 94172f983ecd4a37956654f2f86dba00 P a422b4873cf04ae9b9bc6530f2f077a2: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.001s
I20250112 15:08:51.017321 24981 raft_consensus.cc:357] T 94172f983ecd4a37956654f2f86dba00 P a422b4873cf04ae9b9bc6530f2f077a2 [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "52bd510945874fa781ec841d40e251ab" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 34851 } } peers { permanent_uuid: "56dd875b6ba54b40877146f15292bf2f" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 32969 } } peers { permanent_uuid: "f5b45ccea9c7438f8fba1b79073969e5" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 40795 } attrs { replace: true } } peers { permanent_uuid: "a422b4873cf04ae9b9bc6530f2f077a2" member_type: NON_VOTER last_known_addr { host: "127.17.60.196" port: 33207 } attrs { promote: true } }
I20250112 15:08:51.017417 24981 raft_consensus.cc:738] T 94172f983ecd4a37956654f2f86dba00 P a422b4873cf04ae9b9bc6530f2f077a2 [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: a422b4873cf04ae9b9bc6530f2f077a2, State: Initialized, Role: LEARNER
I20250112 15:08:51.017477 24981 consensus_queue.cc:260] T 94172f983ecd4a37956654f2f86dba00 P a422b4873cf04ae9b9bc6530f2f077a2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 2.3, Last appended by leader: 3, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "52bd510945874fa781ec841d40e251ab" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 34851 } } peers { permanent_uuid: "56dd875b6ba54b40877146f15292bf2f" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 32969 } } peers { permanent_uuid: "f5b45ccea9c7438f8fba1b79073969e5" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 40795 } attrs { replace: true } } peers { permanent_uuid: "a422b4873cf04ae9b9bc6530f2f077a2" member_type: NON_VOTER last_known_addr { host: "127.17.60.196" port: 33207 } attrs { promote: true } }
I20250112 15:08:51.017572 24981 ts_tablet_manager.cc:1428] T 94172f983ecd4a37956654f2f86dba00 P a422b4873cf04ae9b9bc6530f2f077a2: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20250112 15:08:51.017782 24586 tablet_copy_service.cc:342] P 56dd875b6ba54b40877146f15292bf2f: Request end of tablet copy session a422b4873cf04ae9b9bc6530f2f077a2-94172f983ecd4a37956654f2f86dba00 received from {username='slave'} at 127.0.0.1:44578
I20250112 15:08:51.017850 24586 tablet_copy_service.cc:434] P 56dd875b6ba54b40877146f15292bf2f: ending tablet copy session a422b4873cf04ae9b9bc6530f2f077a2-94172f983ecd4a37956654f2f86dba00 on tablet 94172f983ecd4a37956654f2f86dba00 with peer a422b4873cf04ae9b9bc6530f2f077a2
I20250112 15:08:51.021220 24981 ts_tablet_manager.cc:927] T cc5c837c3de946a5952f656946d883c8 P a422b4873cf04ae9b9bc6530f2f077a2: Initiating tablet copy from peer f5b45ccea9c7438f8fba1b79073969e5 (127.17.60.195:40795)
I20250112 15:08:51.021360 24981 tablet_copy_client.cc:323] T cc5c837c3de946a5952f656946d883c8 P a422b4873cf04ae9b9bc6530f2f077a2: tablet copy: Beginning tablet copy session from remote peer at address 127.17.60.195:40795
I20250112 15:08:51.021507 24661 tablet_copy_service.cc:140] P f5b45ccea9c7438f8fba1b79073969e5: Received BeginTabletCopySession request for tablet cc5c837c3de946a5952f656946d883c8 from peer a422b4873cf04ae9b9bc6530f2f077a2 ({username='slave'} at 127.0.0.1:38216)
I20250112 15:08:51.021569 24661 tablet_copy_service.cc:161] P f5b45ccea9c7438f8fba1b79073969e5: Beginning new tablet copy session on tablet cc5c837c3de946a5952f656946d883c8 from peer a422b4873cf04ae9b9bc6530f2f077a2 at {username='slave'} at 127.0.0.1:38216: session id = a422b4873cf04ae9b9bc6530f2f077a2-cc5c837c3de946a5952f656946d883c8
I20250112 15:08:51.022089 24661 tablet_copy_source_session.cc:215] T cc5c837c3de946a5952f656946d883c8 P f5b45ccea9c7438f8fba1b79073969e5: Tablet Copy: opened 0 blocks and 1 log segments
I20250112 15:08:51.022405 24981 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cc5c837c3de946a5952f656946d883c8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 15:08:51.023635 24981 tablet_copy_client.cc:806] T cc5c837c3de946a5952f656946d883c8 P a422b4873cf04ae9b9bc6530f2f077a2: tablet copy: Starting download of 0 data blocks...
I20250112 15:08:51.023741 24981 tablet_copy_client.cc:670] T cc5c837c3de946a5952f656946d883c8 P a422b4873cf04ae9b9bc6530f2f077a2: tablet copy: Starting download of 1 WAL segments...
I20250112 15:08:51.024140 24981 tablet_copy_client.cc:538] T cc5c837c3de946a5952f656946d883c8 P a422b4873cf04ae9b9bc6530f2f077a2: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250112 15:08:51.024856 24981 tablet_bootstrap.cc:492] T cc5c837c3de946a5952f656946d883c8 P a422b4873cf04ae9b9bc6530f2f077a2: Bootstrap starting.
I20250112 15:08:51.026257 24981 tablet_bootstrap.cc:492] T cc5c837c3de946a5952f656946d883c8 P a422b4873cf04ae9b9bc6530f2f077a2: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250112 15:08:51.026425 24981 tablet_bootstrap.cc:492] T cc5c837c3de946a5952f656946d883c8 P a422b4873cf04ae9b9bc6530f2f077a2: Bootstrap complete.
I20250112 15:08:51.026531 24981 ts_tablet_manager.cc:1397] T cc5c837c3de946a5952f656946d883c8 P a422b4873cf04ae9b9bc6530f2f077a2: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20250112 15:08:51.026669 24981 raft_consensus.cc:357] T cc5c837c3de946a5952f656946d883c8 P a422b4873cf04ae9b9bc6530f2f077a2 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52bd510945874fa781ec841d40e251ab" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 34851 } attrs { replace: true } } peers { permanent_uuid: "56dd875b6ba54b40877146f15292bf2f" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 32969 } } peers { permanent_uuid: "f5b45ccea9c7438f8fba1b79073969e5" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 40795 } } peers { permanent_uuid: "a422b4873cf04ae9b9bc6530f2f077a2" member_type: NON_VOTER last_known_addr { host: "127.17.60.196" port: 33207 } attrs { promote: true } }
I20250112 15:08:51.026731 24981 raft_consensus.cc:738] T cc5c837c3de946a5952f656946d883c8 P a422b4873cf04ae9b9bc6530f2f077a2 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: a422b4873cf04ae9b9bc6530f2f077a2, State: Initialized, Role: LEARNER
I20250112 15:08:51.026765 24981 consensus_queue.cc:260] T cc5c837c3de946a5952f656946d883c8 P a422b4873cf04ae9b9bc6530f2f077a2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52bd510945874fa781ec841d40e251ab" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 34851 } attrs { replace: true } } peers { permanent_uuid: "56dd875b6ba54b40877146f15292bf2f" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 32969 } } peers { permanent_uuid: "f5b45ccea9c7438f8fba1b79073969e5" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 40795 } } peers { permanent_uuid: "a422b4873cf04ae9b9bc6530f2f077a2" member_type: NON_VOTER last_known_addr { host: "127.17.60.196" port: 33207 } attrs { promote: true } }
I20250112 15:08:51.026858 24981 ts_tablet_manager.cc:1428] T cc5c837c3de946a5952f656946d883c8 P a422b4873cf04ae9b9bc6530f2f077a2: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20250112 15:08:51.027020 24661 tablet_copy_service.cc:342] P f5b45ccea9c7438f8fba1b79073969e5: Request end of tablet copy session a422b4873cf04ae9b9bc6530f2f077a2-cc5c837c3de946a5952f656946d883c8 received from {username='slave'} at 127.0.0.1:38216
I20250112 15:08:51.027074 24661 tablet_copy_service.cc:434] P f5b45ccea9c7438f8fba1b79073969e5: ending tablet copy session a422b4873cf04ae9b9bc6530f2f077a2-cc5c837c3de946a5952f656946d883c8 on tablet cc5c837c3de946a5952f656946d883c8 with peer a422b4873cf04ae9b9bc6530f2f077a2
I20250112 15:08:51.430063 24825 raft_consensus.cc:1212] T 94172f983ecd4a37956654f2f86dba00 P a422b4873cf04ae9b9bc6530f2f077a2 [term 2 LEARNER]: Deduplicated request from leader. Original: 2.2->[2.3-2.3] Dedup: 2.3->[]
I20250112 15:08:51.479938 24825 raft_consensus.cc:1212] T c73b4e40028649e5bb9ad937f307e7bf P a422b4873cf04ae9b9bc6530f2f077a2 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20250112 15:08:51.530570 24825 raft_consensus.cc:1212] T cc5c837c3de946a5952f656946d883c8 P a422b4873cf04ae9b9bc6530f2f077a2 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20250112 15:08:51.941982 24994 raft_consensus.cc:1059] T 94172f983ecd4a37956654f2f86dba00 P 56dd875b6ba54b40877146f15292bf2f: attempting to promote NON_VOTER a422b4873cf04ae9b9bc6530f2f077a2 to VOTER
I20250112 15:08:51.942239 24994 consensus_queue.cc:237] T 94172f983ecd4a37956654f2f86dba00 P 56dd875b6ba54b40877146f15292bf2f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.3, Last appended by leader: 1, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52bd510945874fa781ec841d40e251ab" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 34851 } } peers { permanent_uuid: "56dd875b6ba54b40877146f15292bf2f" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 32969 } } peers { permanent_uuid: "f5b45ccea9c7438f8fba1b79073969e5" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 40795 } attrs { replace: true } } peers { permanent_uuid: "a422b4873cf04ae9b9bc6530f2f077a2" member_type: VOTER last_known_addr { host: "127.17.60.196" port: 33207 } attrs { promote: false } }
I20250112 15:08:51.942798 24825 raft_consensus.cc:1270] T 94172f983ecd4a37956654f2f86dba00 P a422b4873cf04ae9b9bc6530f2f077a2 [term 2 LEARNER]: Refusing update from remote peer 56dd875b6ba54b40877146f15292bf2f: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20250112 15:08:51.942895 24650 raft_consensus.cc:1270] T 94172f983ecd4a37956654f2f86dba00 P f5b45ccea9c7438f8fba1b79073969e5 [term 2 FOLLOWER]: Refusing update from remote peer 56dd875b6ba54b40877146f15292bf2f: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20250112 15:08:51.943054 24997 consensus_queue.cc:1035] T 94172f983ecd4a37956654f2f86dba00 P 56dd875b6ba54b40877146f15292bf2f [LEADER]: Connected to new peer: Peer: permanent_uuid: "a422b4873cf04ae9b9bc6530f2f077a2" member_type: VOTER last_known_addr { host: "127.17.60.196" port: 33207 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250112 15:08:51.943171 24994 consensus_queue.cc:1035] T 94172f983ecd4a37956654f2f86dba00 P 56dd875b6ba54b40877146f15292bf2f [LEADER]: Connected to new peer: Peer: permanent_uuid: "f5b45ccea9c7438f8fba1b79073969e5" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 40795 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250112 15:08:51.943532 24904 raft_consensus.cc:1270] T 94172f983ecd4a37956654f2f86dba00 P 52bd510945874fa781ec841d40e251ab [term 2 FOLLOWER]: Refusing update from remote peer 56dd875b6ba54b40877146f15292bf2f: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20250112 15:08:51.943704 24991 consensus_queue.cc:1035] T 94172f983ecd4a37956654f2f86dba00 P 56dd875b6ba54b40877146f15292bf2f [LEADER]: Connected to new peer: Peer: permanent_uuid: "52bd510945874fa781ec841d40e251ab" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 34851 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250112 15:08:51.944065 24994 raft_consensus.cc:2949] T 94172f983ecd4a37956654f2f86dba00 P 56dd875b6ba54b40877146f15292bf2f [term 2 LEADER]: Committing config change with OpId 2.4: config changed from index 3 to 4, a422b4873cf04ae9b9bc6530f2f077a2 (127.17.60.196) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52bd510945874fa781ec841d40e251ab" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 34851 } } peers { permanent_uuid: "56dd875b6ba54b40877146f15292bf2f" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 32969 } } peers { permanent_uuid: "f5b45ccea9c7438f8fba1b79073969e5" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 40795 } attrs { replace: true } } peers { permanent_uuid: "a422b4873cf04ae9b9bc6530f2f077a2" member_type: VOTER last_known_addr { host: "127.17.60.196" port: 33207 } attrs { promote: false } } }
I20250112 15:08:51.944466 24904 raft_consensus.cc:2949] T 94172f983ecd4a37956654f2f86dba00 P 52bd510945874fa781ec841d40e251ab [term 2 FOLLOWER]: Committing config change with OpId 2.4: config changed from index 3 to 4, a422b4873cf04ae9b9bc6530f2f077a2 (127.17.60.196) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52bd510945874fa781ec841d40e251ab" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 34851 } } peers { permanent_uuid: "56dd875b6ba54b40877146f15292bf2f" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 32969 } } peers { permanent_uuid: "f5b45ccea9c7438f8fba1b79073969e5" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 40795 } attrs { replace: true } } peers { permanent_uuid: "a422b4873cf04ae9b9bc6530f2f077a2" member_type: VOTER last_known_addr { host: "127.17.60.196" port: 33207 } attrs { promote: false } } }
I20250112 15:08:51.945210 24825 raft_consensus.cc:2949] T 94172f983ecd4a37956654f2f86dba00 P a422b4873cf04ae9b9bc6530f2f077a2 [term 2 FOLLOWER]: Committing config change with OpId 2.4: config changed from index 3 to 4, a422b4873cf04ae9b9bc6530f2f077a2 (127.17.60.196) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52bd510945874fa781ec841d40e251ab" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 34851 } } peers { permanent_uuid: "56dd875b6ba54b40877146f15292bf2f" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 32969 } } peers { permanent_uuid: "f5b45ccea9c7438f8fba1b79073969e5" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 40795 } attrs { replace: true } } peers { permanent_uuid: "a422b4873cf04ae9b9bc6530f2f077a2" member_type: VOTER last_known_addr { host: "127.17.60.196" port: 33207 } attrs { promote: false } } }
I20250112 15:08:51.945523 24395 catalog_manager.cc:5526] T 94172f983ecd4a37956654f2f86dba00 P 52bd510945874fa781ec841d40e251ab reported cstate change: config changed from index 3 to 4, a422b4873cf04ae9b9bc6530f2f077a2 (127.17.60.196) changed from NON_VOTER to VOTER. New cstate: current_term: 2 leader_uuid: "56dd875b6ba54b40877146f15292bf2f" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52bd510945874fa781ec841d40e251ab" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 34851 } } peers { permanent_uuid: "56dd875b6ba54b40877146f15292bf2f" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 32969 } } peers { permanent_uuid: "f5b45ccea9c7438f8fba1b79073969e5" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 40795 } attrs { replace: true } } peers { permanent_uuid: "a422b4873cf04ae9b9bc6530f2f077a2" member_type: VOTER last_known_addr { host: "127.17.60.196" port: 33207 } attrs { promote: false } } }
I20250112 15:08:51.945941 24651 raft_consensus.cc:2949] T 94172f983ecd4a37956654f2f86dba00 P f5b45ccea9c7438f8fba1b79073969e5 [term 2 FOLLOWER]: Committing config change with OpId 2.4: config changed from index 3 to 4, a422b4873cf04ae9b9bc6530f2f077a2 (127.17.60.196) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52bd510945874fa781ec841d40e251ab" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 34851 } } peers { permanent_uuid: "56dd875b6ba54b40877146f15292bf2f" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 32969 } } peers { permanent_uuid: "f5b45ccea9c7438f8fba1b79073969e5" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 40795 } attrs { replace: true } } peers { permanent_uuid: "a422b4873cf04ae9b9bc6530f2f077a2" member_type: VOTER last_known_addr { host: "127.17.60.196" port: 33207 } attrs { promote: false } } }
I20250112 15:08:51.947741 24575 consensus_queue.cc:237] T 94172f983ecd4a37956654f2f86dba00 P 56dd875b6ba54b40877146f15292bf2f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.4, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "52bd510945874fa781ec841d40e251ab" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 34851 } } peers { permanent_uuid: "56dd875b6ba54b40877146f15292bf2f" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 32969 } } peers { permanent_uuid: "a422b4873cf04ae9b9bc6530f2f077a2" member_type: VOTER last_known_addr { host: "127.17.60.196" port: 33207 } attrs { promote: false } }
I20250112 15:08:51.948123 24904 raft_consensus.cc:1270] T 94172f983ecd4a37956654f2f86dba00 P 52bd510945874fa781ec841d40e251ab [term 2 FOLLOWER]: Refusing update from remote peer 56dd875b6ba54b40877146f15292bf2f: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20250112 15:08:51.948290 24825 raft_consensus.cc:1270] T 94172f983ecd4a37956654f2f86dba00 P a422b4873cf04ae9b9bc6530f2f077a2 [term 2 FOLLOWER]: Refusing update from remote peer 56dd875b6ba54b40877146f15292bf2f: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20250112 15:08:51.948318 24991 consensus_queue.cc:1035] T 94172f983ecd4a37956654f2f86dba00 P 56dd875b6ba54b40877146f15292bf2f [LEADER]: Connected to new peer: Peer: permanent_uuid: "52bd510945874fa781ec841d40e251ab" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 34851 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20250112 15:08:51.948530 24991 consensus_queue.cc:1035] T 94172f983ecd4a37956654f2f86dba00 P 56dd875b6ba54b40877146f15292bf2f [LEADER]: Connected to new peer: Peer: permanent_uuid: "a422b4873cf04ae9b9bc6530f2f077a2" member_type: VOTER last_known_addr { host: "127.17.60.196" port: 33207 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20250112 15:08:51.948900 24996 raft_consensus.cc:2949] T 94172f983ecd4a37956654f2f86dba00 P 56dd875b6ba54b40877146f15292bf2f [term 2 LEADER]: Committing config change with OpId 2.5: config changed from index 4 to 5, VOTER f5b45ccea9c7438f8fba1b79073969e5 (127.17.60.195) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "52bd510945874fa781ec841d40e251ab" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 34851 } } peers { permanent_uuid: "56dd875b6ba54b40877146f15292bf2f" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 32969 } } peers { permanent_uuid: "a422b4873cf04ae9b9bc6530f2f077a2" member_type: VOTER last_known_addr { host: "127.17.60.196" port: 33207 } attrs { promote: false } } }
I20250112 15:08:51.949110 24904 raft_consensus.cc:2949] T 94172f983ecd4a37956654f2f86dba00 P 52bd510945874fa781ec841d40e251ab [term 2 FOLLOWER]: Committing config change with OpId 2.5: config changed from index 4 to 5, VOTER f5b45ccea9c7438f8fba1b79073969e5 (127.17.60.195) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "52bd510945874fa781ec841d40e251ab" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 34851 } } peers { permanent_uuid: "56dd875b6ba54b40877146f15292bf2f" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 32969 } } peers { permanent_uuid: "a422b4873cf04ae9b9bc6530f2f077a2" member_type: VOTER last_known_addr { host: "127.17.60.196" port: 33207 } attrs { promote: false } } }
I20250112 15:08:51.949841 24825 raft_consensus.cc:2949] T 94172f983ecd4a37956654f2f86dba00 P a422b4873cf04ae9b9bc6530f2f077a2 [term 2 FOLLOWER]: Committing config change with OpId 2.5: config changed from index 4 to 5, VOTER f5b45ccea9c7438f8fba1b79073969e5 (127.17.60.195) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "52bd510945874fa781ec841d40e251ab" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 34851 } } peers { permanent_uuid: "56dd875b6ba54b40877146f15292bf2f" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 32969 } } peers { permanent_uuid: "a422b4873cf04ae9b9bc6530f2f077a2" member_type: VOTER last_known_addr { host: "127.17.60.196" port: 33207 } attrs { promote: false } } }
I20250112 15:08:51.949934 24381 catalog_manager.cc:5039] ChangeConfig:REMOVE_PEER RPC for tablet 94172f983ecd4a37956654f2f86dba00 with cas_config_opid_index 4: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250112 15:08:51.949973 24396 catalog_manager.cc:5526] T 94172f983ecd4a37956654f2f86dba00 P 52bd510945874fa781ec841d40e251ab reported cstate change: config changed from index 4 to 5, VOTER f5b45ccea9c7438f8fba1b79073969e5 (127.17.60.195) evicted. New cstate: current_term: 2 leader_uuid: "56dd875b6ba54b40877146f15292bf2f" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "52bd510945874fa781ec841d40e251ab" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 34851 } } peers { permanent_uuid: "56dd875b6ba54b40877146f15292bf2f" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 32969 } } peers { permanent_uuid: "a422b4873cf04ae9b9bc6530f2f077a2" member_type: VOTER last_known_addr { host: "127.17.60.196" port: 33207 } attrs { promote: false } } }
I20250112 15:08:51.950623 24640 tablet_service.cc:1514] Processing DeleteTablet for tablet 94172f983ecd4a37956654f2f86dba00 with delete_type TABLET_DATA_TOMBSTONED (TS f5b45ccea9c7438f8fba1b79073969e5 not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:35848
I20250112 15:08:51.950824 25006 tablet_replica.cc:331] T 94172f983ecd4a37956654f2f86dba00 P f5b45ccea9c7438f8fba1b79073969e5: stopping tablet replica
I20250112 15:08:51.950886 25006 raft_consensus.cc:2238] T 94172f983ecd4a37956654f2f86dba00 P f5b45ccea9c7438f8fba1b79073969e5 [term 2 FOLLOWER]: Raft consensus shutting down.
I20250112 15:08:51.950942 25006 raft_consensus.cc:2267] T 94172f983ecd4a37956654f2f86dba00 P f5b45ccea9c7438f8fba1b79073969e5 [term 2 FOLLOWER]: Raft consensus is shut down!
I20250112 15:08:51.951304 25006 ts_tablet_manager.cc:1905] T 94172f983ecd4a37956654f2f86dba00 P f5b45ccea9c7438f8fba1b79073969e5: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250112 15:08:51.952729 25006 ts_tablet_manager.cc:1918] T 94172f983ecd4a37956654f2f86dba00 P f5b45ccea9c7438f8fba1b79073969e5: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.4
I20250112 15:08:51.952790 25006 log.cc:1198] T 94172f983ecd4a37956654f2f86dba00 P f5b45ccea9c7438f8fba1b79073969e5: Deleting WAL directory at /tmp/dist-test-task95dLSD/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestDeletedTables.1736694351070047-17651-0/minicluster-data/ts-2-root/wals/94172f983ecd4a37956654f2f86dba00
I20250112 15:08:51.953109 24381 catalog_manager.cc:4872] TS f5b45ccea9c7438f8fba1b79073969e5 (127.17.60.195:40795): tablet 94172f983ecd4a37956654f2f86dba00 (table test-workload [id=a7cea0e15506428cabd1af7ad2081b4c]) successfully deleted
I20250112 15:08:52.049027 25007 raft_consensus.cc:1059] T cc5c837c3de946a5952f656946d883c8 P f5b45ccea9c7438f8fba1b79073969e5: attempting to promote NON_VOTER a422b4873cf04ae9b9bc6530f2f077a2 to VOTER
I20250112 15:08:52.049283 25007 consensus_queue.cc:237] T cc5c837c3de946a5952f656946d883c8 P f5b45ccea9c7438f8fba1b79073969e5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "52bd510945874fa781ec841d40e251ab" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 34851 } attrs { replace: true } } peers { permanent_uuid: "56dd875b6ba54b40877146f15292bf2f" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 32969 } } peers { permanent_uuid: "f5b45ccea9c7438f8fba1b79073969e5" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 40795 } } peers { permanent_uuid: "a422b4873cf04ae9b9bc6530f2f077a2" member_type: VOTER last_known_addr { host: "127.17.60.196" port: 33207 } attrs { promote: false } }
I20250112 15:08:52.049906 24576 raft_consensus.cc:1270] T cc5c837c3de946a5952f656946d883c8 P 56dd875b6ba54b40877146f15292bf2f [term 1 FOLLOWER]: Refusing update from remote peer f5b45ccea9c7438f8fba1b79073969e5: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20250112 15:08:52.049937 24825 raft_consensus.cc:1270] T cc5c837c3de946a5952f656946d883c8 P a422b4873cf04ae9b9bc6530f2f077a2 [term 1 LEARNER]: Refusing update from remote peer f5b45ccea9c7438f8fba1b79073969e5: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20250112 15:08:52.049906 24904 raft_consensus.cc:1270] T cc5c837c3de946a5952f656946d883c8 P 52bd510945874fa781ec841d40e251ab [term 1 FOLLOWER]: Refusing update from remote peer f5b45ccea9c7438f8fba1b79073969e5: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20250112 15:08:52.050114 25010 consensus_queue.cc:1035] T cc5c837c3de946a5952f656946d883c8 P f5b45ccea9c7438f8fba1b79073969e5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a422b4873cf04ae9b9bc6530f2f077a2" member_type: VOTER last_known_addr { host: "127.17.60.196" port: 33207 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20250112 15:08:52.050340 24993 consensus_queue.cc:1035] T cc5c837c3de946a5952f656946d883c8 P f5b45ccea9c7438f8fba1b79073969e5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "56dd875b6ba54b40877146f15292bf2f" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 32969 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20250112 15:08:52.050520 25009 consensus_queue.cc:1035] T cc5c837c3de946a5952f656946d883c8 P f5b45ccea9c7438f8fba1b79073969e5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "52bd510945874fa781ec841d40e251ab" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 34851 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20250112 15:08:52.051355 24993 raft_consensus.cc:2949] T cc5c837c3de946a5952f656946d883c8 P f5b45ccea9c7438f8fba1b79073969e5 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, a422b4873cf04ae9b9bc6530f2f077a2 (127.17.60.196) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "52bd510945874fa781ec841d40e251ab" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 34851 } attrs { replace: true } } peers { permanent_uuid: "56dd875b6ba54b40877146f15292bf2f" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 32969 } } peers { permanent_uuid: "f5b45ccea9c7438f8fba1b79073969e5" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 40795 } } peers { permanent_uuid: "a422b4873cf04ae9b9bc6530f2f077a2" member_type: VOTER last_known_addr { host: "127.17.60.196" port: 33207 } attrs { promote: false } } }
I20250112 15:08:52.051474 24576 raft_consensus.cc:2949] T cc5c837c3de946a5952f656946d883c8 P 56dd875b6ba54b40877146f15292bf2f [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, a422b4873cf04ae9b9bc6530f2f077a2 (127.17.60.196) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "52bd510945874fa781ec841d40e251ab" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 34851 } attrs { replace: true } } peers { permanent_uuid: "56dd875b6ba54b40877146f15292bf2f" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 32969 } } peers { permanent_uuid: "f5b45ccea9c7438f8fba1b79073969e5" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 40795 } } peers { permanent_uuid: "a422b4873cf04ae9b9bc6530f2f077a2" member_type: VOTER last_known_addr { host: "127.17.60.196" port: 33207 } attrs { promote: false } } }
I20250112 15:08:52.051640 24904 raft_consensus.cc:2949] T cc5c837c3de946a5952f656946d883c8 P 52bd510945874fa781ec841d40e251ab [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, a422b4873cf04ae9b9bc6530f2f077a2 (127.17.60.196) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "52bd510945874fa781ec841d40e251ab" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 34851 } attrs { replace: true } } peers { permanent_uuid: "56dd875b6ba54b40877146f15292bf2f" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 32969 } } peers { permanent_uuid: "f5b45ccea9c7438f8fba1b79073969e5" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 40795 } } peers { permanent_uuid: "a422b4873cf04ae9b9bc6530f2f077a2" member_type: VOTER last_known_addr { host: "127.17.60.196" port: 33207 } attrs { promote: false } } }
I20250112 15:08:52.052541 24396 catalog_manager.cc:5526] T cc5c837c3de946a5952f656946d883c8 P 56dd875b6ba54b40877146f15292bf2f reported cstate change: config changed from index 4 to 5, a422b4873cf04ae9b9bc6530f2f077a2 (127.17.60.196) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "f5b45ccea9c7438f8fba1b79073969e5" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "52bd510945874fa781ec841d40e251ab" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 34851 } attrs { replace: true } } peers { permanent_uuid: "56dd875b6ba54b40877146f15292bf2f" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 32969 } } peers { permanent_uuid: "f5b45ccea9c7438f8fba1b79073969e5" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 40795 } } peers { permanent_uuid: "a422b4873cf04ae9b9bc6530f2f077a2" member_type: VOTER last_known_addr { host: "127.17.60.196" port: 33207 } attrs { promote: false } } }
I20250112 15:08:52.053017 24825 raft_consensus.cc:2949] T cc5c837c3de946a5952f656946d883c8 P a422b4873cf04ae9b9bc6530f2f077a2 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, a422b4873cf04ae9b9bc6530f2f077a2 (127.17.60.196) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "52bd510945874fa781ec841d40e251ab" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 34851 } attrs { replace: true } } peers { permanent_uuid: "56dd875b6ba54b40877146f15292bf2f" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 32969 } } peers { permanent_uuid: "f5b45ccea9c7438f8fba1b79073969e5" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 40795 } } peers { permanent_uuid: "a422b4873cf04ae9b9bc6530f2f077a2" member_type: VOTER last_known_addr { host: "127.17.60.196" port: 33207 } attrs { promote: false } } }
I20250112 15:08:52.053453 24650 consensus_queue.cc:237] T cc5c837c3de946a5952f656946d883c8 P f5b45ccea9c7438f8fba1b79073969e5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "56dd875b6ba54b40877146f15292bf2f" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 32969 } } peers { permanent_uuid: "f5b45ccea9c7438f8fba1b79073969e5" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 40795 } } peers { permanent_uuid: "a422b4873cf04ae9b9bc6530f2f077a2" member_type: VOTER last_known_addr { host: "127.17.60.196" port: 33207 } attrs { promote: false } }
I20250112 15:08:52.054013 24825 raft_consensus.cc:1270] T cc5c837c3de946a5952f656946d883c8 P a422b4873cf04ae9b9bc6530f2f077a2 [term 1 FOLLOWER]: Refusing update from remote peer f5b45ccea9c7438f8fba1b79073969e5: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20250112 15:08:52.054158 24576 raft_consensus.cc:1270] T cc5c837c3de946a5952f656946d883c8 P 56dd875b6ba54b40877146f15292bf2f [term 1 FOLLOWER]: Refusing update from remote peer f5b45ccea9c7438f8fba1b79073969e5: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20250112 15:08:52.054226 25009 consensus_queue.cc:1035] T cc5c837c3de946a5952f656946d883c8 P f5b45ccea9c7438f8fba1b79073969e5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a422b4873cf04ae9b9bc6530f2f077a2" member_type: VOTER last_known_addr { host: "127.17.60.196" port: 33207 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20250112 15:08:52.054335 25007 consensus_queue.cc:1035] T cc5c837c3de946a5952f656946d883c8 P f5b45ccea9c7438f8fba1b79073969e5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "56dd875b6ba54b40877146f15292bf2f" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 32969 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20250112 15:08:52.054998 25009 raft_consensus.cc:2949] T cc5c837c3de946a5952f656946d883c8 P f5b45ccea9c7438f8fba1b79073969e5 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 52bd510945874fa781ec841d40e251ab (127.17.60.193) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "56dd875b6ba54b40877146f15292bf2f" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 32969 } } peers { permanent_uuid: "f5b45ccea9c7438f8fba1b79073969e5" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 40795 } } peers { permanent_uuid: "a422b4873cf04ae9b9bc6530f2f077a2" member_type: VOTER last_known_addr { host: "127.17.60.196" port: 33207 } attrs { promote: false } } }
I20250112 15:08:52.055097 24576 raft_consensus.cc:2949] T cc5c837c3de946a5952f656946d883c8 P 56dd875b6ba54b40877146f15292bf2f [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 52bd510945874fa781ec841d40e251ab (127.17.60.193) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "56dd875b6ba54b40877146f15292bf2f" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 32969 } } peers { permanent_uuid: "f5b45ccea9c7438f8fba1b79073969e5" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 40795 } } peers { permanent_uuid: "a422b4873cf04ae9b9bc6530f2f077a2" member_type: VOTER last_known_addr { host: "127.17.60.196" port: 33207 } attrs { promote: false } } }
I20250112 15:08:52.055898 24381 catalog_manager.cc:5039] ChangeConfig:REMOVE_PEER RPC for tablet cc5c837c3de946a5952f656946d883c8 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250112 15:08:52.055965 24397 catalog_manager.cc:5526] T cc5c837c3de946a5952f656946d883c8 P 56dd875b6ba54b40877146f15292bf2f reported cstate change: config changed from index 5 to 6, VOTER 52bd510945874fa781ec841d40e251ab (127.17.60.193) evicted. New cstate: current_term: 1 leader_uuid: "f5b45ccea9c7438f8fba1b79073969e5" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "56dd875b6ba54b40877146f15292bf2f" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 32969 } } peers { permanent_uuid: "f5b45ccea9c7438f8fba1b79073969e5" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 40795 } } peers { permanent_uuid: "a422b4873cf04ae9b9bc6530f2f077a2" member_type: VOTER last_known_addr { host: "127.17.60.196" port: 33207 } attrs { promote: false } } }
I20250112 15:08:52.056365 24825 raft_consensus.cc:2949] T cc5c837c3de946a5952f656946d883c8 P a422b4873cf04ae9b9bc6530f2f077a2 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 52bd510945874fa781ec841d40e251ab (127.17.60.193) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "56dd875b6ba54b40877146f15292bf2f" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 32969 } } peers { permanent_uuid: "f5b45ccea9c7438f8fba1b79073969e5" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 40795 } } peers { permanent_uuid: "a422b4873cf04ae9b9bc6530f2f077a2" member_type: VOTER last_known_addr { host: "127.17.60.196" port: 33207 } attrs { promote: false } } }
I20250112 15:08:52.056687 24894 tablet_service.cc:1514] Processing DeleteTablet for tablet cc5c837c3de946a5952f656946d883c8 with delete_type TABLET_DATA_TOMBSTONED (TS 52bd510945874fa781ec841d40e251ab not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:56452
I20250112 15:08:52.056893 25014 tablet_replica.cc:331] stopping tablet replica
I20250112 15:08:52.056979 25014 raft_consensus.cc:2238] T cc5c837c3de946a5952f656946d883c8 P 52bd510945874fa781ec841d40e251ab [term 1 FOLLOWER]: Raft consensus shutting down.
I20250112 15:08:52.057045 25014 raft_consensus.cc:2267] T cc5c837c3de946a5952f656946d883c8 P 52bd510945874fa781ec841d40e251ab [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 15:08:52.057250 25014 ts_tablet_manager.cc:1905] T cc5c837c3de946a5952f656946d883c8 P 52bd510945874fa781ec841d40e251ab: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250112 15:08:52.058774 25014 ts_tablet_manager.cc:1918] T cc5c837c3de946a5952f656946d883c8 P 52bd510945874fa781ec841d40e251ab: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20250112 15:08:52.058864 25014 log.cc:1198] T cc5c837c3de946a5952f656946d883c8 P 52bd510945874fa781ec841d40e251ab: Deleting WAL directory at /tmp/dist-test-task95dLSD/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestDeletedTables.1736694351070047-17651-0/minicluster-data/ts-0-root/wals/cc5c837c3de946a5952f656946d883c8
I20250112 15:08:52.059221 24381 catalog_manager.cc:4872] TS 52bd510945874fa781ec841d40e251ab (127.17.60.193:34851): tablet cc5c837c3de946a5952f656946d883c8 (table test-workload [id=a7cea0e15506428cabd1af7ad2081b4c]) successfully deleted
I20250112 15:08:52.079221 25009 raft_consensus.cc:1059] T c73b4e40028649e5bb9ad937f307e7bf P f5b45ccea9c7438f8fba1b79073969e5: attempting to promote NON_VOTER a422b4873cf04ae9b9bc6530f2f077a2 to VOTER
I20250112 15:08:52.079409 25009 consensus_queue.cc:237] T c73b4e40028649e5bb9ad937f307e7bf P f5b45ccea9c7438f8fba1b79073969e5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "56dd875b6ba54b40877146f15292bf2f" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 32969 } attrs { replace: true } } peers { permanent_uuid: "52bd510945874fa781ec841d40e251ab" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 34851 } } peers { permanent_uuid: "f5b45ccea9c7438f8fba1b79073969e5" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 40795 } } peers { permanent_uuid: "a422b4873cf04ae9b9bc6530f2f077a2" member_type: VOTER last_known_addr { host: "127.17.60.196" port: 33207 } attrs { promote: false } }
I20250112 15:08:52.079888 24825 raft_consensus.cc:1270] T c73b4e40028649e5bb9ad937f307e7bf P a422b4873cf04ae9b9bc6530f2f077a2 [term 1 LEARNER]: Refusing update from remote peer f5b45ccea9c7438f8fba1b79073969e5: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20250112 15:08:52.079898 24904 raft_consensus.cc:1270] T c73b4e40028649e5bb9ad937f307e7bf P 52bd510945874fa781ec841d40e251ab [term 1 FOLLOWER]: Refusing update from remote peer f5b45ccea9c7438f8fba1b79073969e5: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20250112 15:08:52.080005 24576 raft_consensus.cc:1270] T c73b4e40028649e5bb9ad937f307e7bf P 56dd875b6ba54b40877146f15292bf2f [term 1 FOLLOWER]: Refusing update from remote peer f5b45ccea9c7438f8fba1b79073969e5: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20250112 15:08:52.080101 25010 consensus_queue.cc:1035] T c73b4e40028649e5bb9ad937f307e7bf P f5b45ccea9c7438f8fba1b79073969e5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a422b4873cf04ae9b9bc6530f2f077a2" member_type: VOTER last_known_addr { host: "127.17.60.196" port: 33207 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20250112 15:08:52.080227 25007 consensus_queue.cc:1035] T c73b4e40028649e5bb9ad937f307e7bf P f5b45ccea9c7438f8fba1b79073969e5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "52bd510945874fa781ec841d40e251ab" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 34851 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20250112 15:08:52.080381 25007 consensus_queue.cc:1035] T c73b4e40028649e5bb9ad937f307e7bf P f5b45ccea9c7438f8fba1b79073969e5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "56dd875b6ba54b40877146f15292bf2f" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 32969 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20250112 15:08:52.081001 24993 raft_consensus.cc:2949] T c73b4e40028649e5bb9ad937f307e7bf P f5b45ccea9c7438f8fba1b79073969e5 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, a422b4873cf04ae9b9bc6530f2f077a2 (127.17.60.196) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "56dd875b6ba54b40877146f15292bf2f" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 32969 } attrs { replace: true } } peers { permanent_uuid: "52bd510945874fa781ec841d40e251ab" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 34851 } } peers { permanent_uuid: "f5b45ccea9c7438f8fba1b79073969e5" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 40795 } } peers { permanent_uuid: "a422b4873cf04ae9b9bc6530f2f077a2" member_type: VOTER last_known_addr { host: "127.17.60.196" port: 33207 } attrs { promote: false } } }
I20250112 15:08:52.081095 24825 raft_consensus.cc:2949] T c73b4e40028649e5bb9ad937f307e7bf P a422b4873cf04ae9b9bc6530f2f077a2 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, a422b4873cf04ae9b9bc6530f2f077a2 (127.17.60.196) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "56dd875b6ba54b40877146f15292bf2f" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 32969 } attrs { replace: true } } peers { permanent_uuid: "52bd510945874fa781ec841d40e251ab" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 34851 } } peers { permanent_uuid: "f5b45ccea9c7438f8fba1b79073969e5" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 40795 } } peers { permanent_uuid: "a422b4873cf04ae9b9bc6530f2f077a2" member_type: VOTER last_known_addr { host: "127.17.60.196" port: 33207 } attrs { promote: false } } }
I20250112 15:08:52.082015 24904 raft_consensus.cc:2949] T c73b4e40028649e5bb9ad937f307e7bf P 52bd510945874fa781ec841d40e251ab [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, a422b4873cf04ae9b9bc6530f2f077a2 (127.17.60.196) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "56dd875b6ba54b40877146f15292bf2f" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 32969 } attrs { replace: true } } peers { permanent_uuid: "52bd510945874fa781ec841d40e251ab" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 34851 } } peers { permanent_uuid: "f5b45ccea9c7438f8fba1b79073969e5" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 40795 } } peers { permanent_uuid: "a422b4873cf04ae9b9bc6530f2f077a2" member_type: VOTER last_known_addr { host: "127.17.60.196" port: 33207 } attrs { promote: false } } }
I20250112 15:08:52.082041 24394 catalog_manager.cc:5526] T c73b4e40028649e5bb9ad937f307e7bf P a422b4873cf04ae9b9bc6530f2f077a2 reported cstate change: config changed from index 4 to 5, a422b4873cf04ae9b9bc6530f2f077a2 (127.17.60.196) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "f5b45ccea9c7438f8fba1b79073969e5" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "56dd875b6ba54b40877146f15292bf2f" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 32969 } attrs { replace: true } } peers { permanent_uuid: "52bd510945874fa781ec841d40e251ab" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 34851 } } peers { permanent_uuid: "f5b45ccea9c7438f8fba1b79073969e5" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 40795 } } peers { permanent_uuid: "a422b4873cf04ae9b9bc6530f2f077a2" member_type: VOTER last_known_addr { host: "127.17.60.196" port: 33207 } attrs { promote: false } } }
I20250112 15:08:52.082206 24576 raft_consensus.cc:2949] T c73b4e40028649e5bb9ad937f307e7bf P 56dd875b6ba54b40877146f15292bf2f [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, a422b4873cf04ae9b9bc6530f2f077a2 (127.17.60.196) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "56dd875b6ba54b40877146f15292bf2f" member_type: VOTER last_known_addr { host: "127.17.60.194" port: 32969 } attrs { replace: true } } peers { permanent_uuid: "52bd510945874fa781ec841d40e251ab" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 34851 } } peers { permanent_uuid: "f5b45ccea9c7438f8fba1b79073969e5" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 40795 } } peers { permanent_uuid: "a422b4873cf04ae9b9bc6530f2f077a2" member_type: VOTER last_known_addr { host: "127.17.60.196" port: 33207 } attrs { promote: false } } }
I20250112 15:08:52.083761 24651 consensus_queue.cc:237] T c73b4e40028649e5bb9ad937f307e7bf P f5b45ccea9c7438f8fba1b79073969e5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "52bd510945874fa781ec841d40e251ab" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 34851 } } peers { permanent_uuid: "f5b45ccea9c7438f8fba1b79073969e5" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 40795 } } peers { permanent_uuid: "a422b4873cf04ae9b9bc6530f2f077a2" member_type: VOTER last_known_addr { host: "127.17.60.196" port: 33207 } attrs { promote: false } }
I20250112 15:08:52.084072 24904 raft_consensus.cc:1270] T c73b4e40028649e5bb9ad937f307e7bf P 52bd510945874fa781ec841d40e251ab [term 1 FOLLOWER]: Refusing update from remote peer f5b45ccea9c7438f8fba1b79073969e5: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20250112 15:08:52.084125 24825 raft_consensus.cc:1270] T c73b4e40028649e5bb9ad937f307e7bf P a422b4873cf04ae9b9bc6530f2f077a2 [term 1 FOLLOWER]: Refusing update from remote peer f5b45ccea9c7438f8fba1b79073969e5: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20250112 15:08:52.084301 24993 consensus_queue.cc:1035] T c73b4e40028649e5bb9ad937f307e7bf P f5b45ccea9c7438f8fba1b79073969e5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a422b4873cf04ae9b9bc6530f2f077a2" member_type: VOTER last_known_addr { host: "127.17.60.196" port: 33207 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20250112 15:08:52.084380 25010 consensus_queue.cc:1035] T c73b4e40028649e5bb9ad937f307e7bf P f5b45ccea9c7438f8fba1b79073969e5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "52bd510945874fa781ec841d40e251ab" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 34851 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20250112 15:08:52.085008 25010 raft_consensus.cc:2949] T c73b4e40028649e5bb9ad937f307e7bf P f5b45ccea9c7438f8fba1b79073969e5 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 56dd875b6ba54b40877146f15292bf2f (127.17.60.194) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "52bd510945874fa781ec841d40e251ab" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 34851 } } peers { permanent_uuid: "f5b45ccea9c7438f8fba1b79073969e5" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 40795 } } peers { permanent_uuid: "a422b4873cf04ae9b9bc6530f2f077a2" member_type: VOTER last_known_addr { host: "127.17.60.196" port: 33207 } attrs { promote: false } } }
I20250112 15:08:52.085112 24904 raft_consensus.cc:2949] T c73b4e40028649e5bb9ad937f307e7bf P 52bd510945874fa781ec841d40e251ab [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 56dd875b6ba54b40877146f15292bf2f (127.17.60.194) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "52bd510945874fa781ec841d40e251ab" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 34851 } } peers { permanent_uuid: "f5b45ccea9c7438f8fba1b79073969e5" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 40795 } } peers { permanent_uuid: "a422b4873cf04ae9b9bc6530f2f077a2" member_type: VOTER last_known_addr { host: "127.17.60.196" port: 33207 } attrs { promote: false } } }
I20250112 15:08:52.085927 24825 raft_consensus.cc:2949] T c73b4e40028649e5bb9ad937f307e7bf P a422b4873cf04ae9b9bc6530f2f077a2 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 56dd875b6ba54b40877146f15292bf2f (127.17.60.194) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "52bd510945874fa781ec841d40e251ab" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 34851 } } peers { permanent_uuid: "f5b45ccea9c7438f8fba1b79073969e5" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 40795 } } peers { permanent_uuid: "a422b4873cf04ae9b9bc6530f2f077a2" member_type: VOTER last_known_addr { host: "127.17.60.196" port: 33207 } attrs { promote: false } } }
I20250112 15:08:52.086091 24395 catalog_manager.cc:5526] T c73b4e40028649e5bb9ad937f307e7bf P f5b45ccea9c7438f8fba1b79073969e5 reported cstate change: config changed from index 5 to 6, VOTER 56dd875b6ba54b40877146f15292bf2f (127.17.60.194) evicted. New cstate: current_term: 1 leader_uuid: "f5b45ccea9c7438f8fba1b79073969e5" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "52bd510945874fa781ec841d40e251ab" member_type: VOTER last_known_addr { host: "127.17.60.193" port: 34851 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f5b45ccea9c7438f8fba1b79073969e5" member_type: VOTER last_known_addr { host: "127.17.60.195" port: 40795 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a422b4873cf04ae9b9bc6530f2f077a2" member_type: VOTER last_known_addr { host: "127.17.60.196" port: 33207 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20250112 15:08:52.086740 24563 tablet_service.cc:1514] Processing DeleteTablet for tablet c73b4e40028649e5bb9ad937f307e7bf with delete_type TABLET_DATA_TOMBSTONED (TS 56dd875b6ba54b40877146f15292bf2f not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:49784
I20250112 15:08:52.086740 24381 catalog_manager.cc:5039] ChangeConfig:REMOVE_PEER RPC for tablet c73b4e40028649e5bb9ad937f307e7bf with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250112 15:08:52.087088 25019 tablet_replica.cc:331] T c73b4e40028649e5bb9ad937f307e7bf P 56dd875b6ba54b40877146f15292bf2f: stopping tablet replica
I20250112 15:08:52.087191 25019 raft_consensus.cc:2238] T c73b4e40028649e5bb9ad937f307e7bf P 56dd875b6ba54b40877146f15292bf2f [term 1 FOLLOWER]: Raft consensus shutting down.
I20250112 15:08:52.087255 25019 raft_consensus.cc:2267] T c73b4e40028649e5bb9ad937f307e7bf P 56dd875b6ba54b40877146f15292bf2f [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 15:08:52.087440 25019 ts_tablet_manager.cc:1905] T c73b4e40028649e5bb9ad937f307e7bf P 56dd875b6ba54b40877146f15292bf2f: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250112 15:08:52.088615 25019 ts_tablet_manager.cc:1918] T c73b4e40028649e5bb9ad937f307e7bf P 56dd875b6ba54b40877146f15292bf2f: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20250112 15:08:52.088662 25019 log.cc:1198] T c73b4e40028649e5bb9ad937f307e7bf P 56dd875b6ba54b40877146f15292bf2f: Deleting WAL directory at /tmp/dist-test-task95dLSD/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestDeletedTables.1736694351070047-17651-0/minicluster-data/ts-1-root/wals/c73b4e40028649e5bb9ad937f307e7bf
I20250112 15:08:52.088970 24381 catalog_manager.cc:4872] TS 56dd875b6ba54b40877146f15292bf2f (127.17.60.194:32969): tablet c73b4e40028649e5bb9ad937f307e7bf (table test-workload [id=a7cea0e15506428cabd1af7ad2081b4c]) successfully deleted
W20250112 15:08:53.087174 24446 auto_rebalancer.cc:227] Could not retrieve cluster info: Not found: tserver 52bd510945874fa781ec841d40e251ab not available for placement
W20250112 15:08:54.087433 24446 auto_rebalancer.cc:227] Could not retrieve cluster info: Not found: tserver f5b45ccea9c7438f8fba1b79073969e5 not available for placement
I20250112 15:08:54.775151 17651 tablet_server.cc:178] TabletServer@127.17.60.193:34851 shutting down...
I20250112 15:08:54.777720 17651 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250112 15:08:54.777825 17651 tablet_replica.cc:331] stopping tablet replica
I20250112 15:08:54.777891 17651 raft_consensus.cc:2238] T 94172f983ecd4a37956654f2f86dba00 P 52bd510945874fa781ec841d40e251ab [term 2 FOLLOWER]: Raft consensus shutting down.
I20250112 15:08:54.777956 17651 raft_consensus.cc:2267] T 94172f983ecd4a37956654f2f86dba00 P 52bd510945874fa781ec841d40e251ab [term 2 FOLLOWER]: Raft consensus is shut down!
I20250112 15:08:54.778107 17651 tablet_replica.cc:331] stopping tablet replica
I20250112 15:08:54.778136 17651 raft_consensus.cc:2238] T 45d29678d6704a888f2c2514af2f36a4 P 52bd510945874fa781ec841d40e251ab [term 1 FOLLOWER]: Raft consensus shutting down.
I20250112 15:08:54.778167 17651 raft_consensus.cc:2267] T 45d29678d6704a888f2c2514af2f36a4 P 52bd510945874fa781ec841d40e251ab [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 15:08:54.778244 17651 tablet_replica.cc:331] stopping tablet replica
I20250112 15:08:54.778266 17651 raft_consensus.cc:2238] T c73b4e40028649e5bb9ad937f307e7bf P 52bd510945874fa781ec841d40e251ab [term 1 FOLLOWER]: Raft consensus shutting down.
I20250112 15:08:54.778316 17651 raft_consensus.cc:2267] T c73b4e40028649e5bb9ad937f307e7bf P 52bd510945874fa781ec841d40e251ab [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 15:08:54.789834 17651 tablet_server.cc:195] TabletServer@127.17.60.193:34851 shutdown complete.
I20250112 15:08:54.791254 17651 tablet_server.cc:178] TabletServer@127.17.60.194:0 shutting down...
I20250112 15:08:54.794359 17651 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250112 15:08:54.794471 17651 tablet_replica.cc:331] T 45d29678d6704a888f2c2514af2f36a4 P 56dd875b6ba54b40877146f15292bf2f: stopping tablet replica
I20250112 15:08:54.794554 17651 raft_consensus.cc:2238] T 45d29678d6704a888f2c2514af2f36a4 P 56dd875b6ba54b40877146f15292bf2f [term 1 FOLLOWER]: Raft consensus shutting down.
I20250112 15:08:54.794605 17651 raft_consensus.cc:2267] T 45d29678d6704a888f2c2514af2f36a4 P 56dd875b6ba54b40877146f15292bf2f [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 15:08:54.794741 17651 tablet_replica.cc:331] T 94172f983ecd4a37956654f2f86dba00 P 56dd875b6ba54b40877146f15292bf2f: stopping tablet replica
I20250112 15:08:54.794770 17651 raft_consensus.cc:2238] T 94172f983ecd4a37956654f2f86dba00 P 56dd875b6ba54b40877146f15292bf2f [term 2 LEADER]: Raft consensus shutting down.
I20250112 15:08:54.794806 17651 raft_consensus.cc:2267] T 94172f983ecd4a37956654f2f86dba00 P 56dd875b6ba54b40877146f15292bf2f [term 2 FOLLOWER]: Raft consensus is shut down!
I20250112 15:08:54.794912 17651 tablet_replica.cc:331] T cc5c837c3de946a5952f656946d883c8 P 56dd875b6ba54b40877146f15292bf2f: stopping tablet replica
I20250112 15:08:54.794948 17651 raft_consensus.cc:2238] T cc5c837c3de946a5952f656946d883c8 P 56dd875b6ba54b40877146f15292bf2f [term 1 FOLLOWER]: Raft consensus shutting down.
I20250112 15:08:54.794989 17651 raft_consensus.cc:2267] T cc5c837c3de946a5952f656946d883c8 P 56dd875b6ba54b40877146f15292bf2f [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 15:08:54.806705 17651 tablet_server.cc:195] TabletServer@127.17.60.194:0 shutdown complete.
I20250112 15:08:54.807832 17651 tablet_server.cc:178] TabletServer@127.17.60.195:0 shutting down...
I20250112 15:08:54.810696 17651 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250112 15:08:54.810794 17651 tablet_replica.cc:331] T c73b4e40028649e5bb9ad937f307e7bf P f5b45ccea9c7438f8fba1b79073969e5: stopping tablet replica
I20250112 15:08:54.810863 17651 raft_consensus.cc:2238] T c73b4e40028649e5bb9ad937f307e7bf P f5b45ccea9c7438f8fba1b79073969e5 [term 1 LEADER]: Raft consensus shutting down.
I20250112 15:08:54.810922 17651 raft_consensus.cc:2267] T c73b4e40028649e5bb9ad937f307e7bf P f5b45ccea9c7438f8fba1b79073969e5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 15:08:54.811043 17651 tablet_replica.cc:331] T 45d29678d6704a888f2c2514af2f36a4 P f5b45ccea9c7438f8fba1b79073969e5: stopping tablet replica
I20250112 15:08:54.811072 17651 raft_consensus.cc:2238] T 45d29678d6704a888f2c2514af2f36a4 P f5b45ccea9c7438f8fba1b79073969e5 [term 1 LEADER]: Raft consensus shutting down.
I20250112 15:08:54.811100 17651 raft_consensus.cc:2267] T 45d29678d6704a888f2c2514af2f36a4 P f5b45ccea9c7438f8fba1b79073969e5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 15:08:54.811169 17651 tablet_replica.cc:331] T cc5c837c3de946a5952f656946d883c8 P f5b45ccea9c7438f8fba1b79073969e5: stopping tablet replica
I20250112 15:08:54.811192 17651 raft_consensus.cc:2238] T cc5c837c3de946a5952f656946d883c8 P f5b45ccea9c7438f8fba1b79073969e5 [term 1 LEADER]: Raft consensus shutting down.
I20250112 15:08:54.811218 17651 raft_consensus.cc:2267] T cc5c837c3de946a5952f656946d883c8 P f5b45ccea9c7438f8fba1b79073969e5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 15:08:54.822654 17651 tablet_server.cc:195] TabletServer@127.17.60.195:0 shutdown complete.
I20250112 15:08:54.823766 17651 tablet_server.cc:178] TabletServer@127.17.60.196:0 shutting down...
I20250112 15:08:54.826114 17651 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250112 15:08:54.826234 17651 tablet_replica.cc:331] T cc5c837c3de946a5952f656946d883c8 P a422b4873cf04ae9b9bc6530f2f077a2: stopping tablet replica
I20250112 15:08:54.826287 17651 raft_consensus.cc:2238] T cc5c837c3de946a5952f656946d883c8 P a422b4873cf04ae9b9bc6530f2f077a2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250112 15:08:54.826347 17651 raft_consensus.cc:2267] T cc5c837c3de946a5952f656946d883c8 P a422b4873cf04ae9b9bc6530f2f077a2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 15:08:54.826493 17651 tablet_replica.cc:331] T c73b4e40028649e5bb9ad937f307e7bf P a422b4873cf04ae9b9bc6530f2f077a2: stopping tablet replica
I20250112 15:08:54.826531 17651 raft_consensus.cc:2238] T c73b4e40028649e5bb9ad937f307e7bf P a422b4873cf04ae9b9bc6530f2f077a2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250112 15:08:54.826576 17651 raft_consensus.cc:2267] T c73b4e40028649e5bb9ad937f307e7bf P a422b4873cf04ae9b9bc6530f2f077a2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 15:08:54.826683 17651 tablet_replica.cc:331] T 94172f983ecd4a37956654f2f86dba00 P a422b4873cf04ae9b9bc6530f2f077a2: stopping tablet replica
I20250112 15:08:54.826731 17651 raft_consensus.cc:2238] T 94172f983ecd4a37956654f2f86dba00 P a422b4873cf04ae9b9bc6530f2f077a2 [term 2 FOLLOWER]: Raft consensus shutting down.
I20250112 15:08:54.826774 17651 raft_consensus.cc:2267] T 94172f983ecd4a37956654f2f86dba00 P a422b4873cf04ae9b9bc6530f2f077a2 [term 2 FOLLOWER]: Raft consensus is shut down!
I20250112 15:08:54.838156 17651 tablet_server.cc:195] TabletServer@127.17.60.196:0 shutdown complete.
I20250112 15:08:54.839215 17651 master.cc:537] Master@127.17.60.254:37937 shutting down...
I20250112 15:08:54.841662 17651 raft_consensus.cc:2238] T 00000000000000000000000000000000 P ea845a3e3060484f9ece4069e440908c [term 1 LEADER]: Raft consensus shutting down.
I20250112 15:08:54.841720 17651 raft_consensus.cc:2267] T 00000000000000000000000000000000 P ea845a3e3060484f9ece4069e440908c [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 15:08:54.841743 17651 tablet_replica.cc:331] T 00000000000000000000000000000000 P ea845a3e3060484f9ece4069e440908c: stopping tablet replica
I20250112 15:08:54.853193 17651 master.cc:559] Master@127.17.60.254:37937 shutdown complete.
[ OK ] AutoRebalancerTest.TestDeletedTables (8440 ms)
[----------] 14 tests from AutoRebalancerTest (183784 ms total)
[----------] Global test environment tear-down
[==========] 14 tests from 1 test suite ran. (183785 ms total)
[ PASSED ] 13 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] AutoRebalancerTest.TestMaxMovesPerServer
1 FAILED TEST
I20250112 15:08:54.859402 17651 logging.cc:424] LogThrottler /home/jenkins-slave/workspace/build_and_test_flaky/src/kudu/tserver/ts_tablet_manager.cc:594: suppressed but not reported on 7 messages since previous log ~5 seconds ago
I20250112 15:08:54.859434 17651 logging.cc:424] LogThrottler /home/jenkins-slave/workspace/build_and_test_flaky/src/kudu/tserver/ts_tablet_manager.cc:536: suppressed but not reported on 7 messages since previous log ~5 seconds ago
I20250112 15:08:54.859448 17651 logging.cc:424] LogThrottler /home/jenkins-slave/workspace/build_and_test_flaky/src/kudu/master/catalog_manager.cc:4615: suppressed but not reported on 7 messages since previous log ~5 seconds ago
I20250112 15:08:54.859459 17651 logging.cc:424] LogThrottler /home/jenkins-slave/workspace/build_and_test_flaky/src/kudu/master/catalog_manager.cc:5204: suppressed but not reported on 6 messages since previous log ~30 seconds ago
I20250112 15:08:54.859479 17651 logging.cc:424] LogThrottler /home/jenkins-slave/workspace/build_and_test_flaky/src/kudu/tserver/heartbeater.cc:643: suppressed but not reported on 11 messages since previous log ~170 seconds ago
I20250112 15:08:54.859498 17651 logging.cc:424] LogThrottler /home/jenkins-slave/workspace/build_and_test_flaky/src/kudu/tablet/tablet.cc:2367: suppressed but not reported on 50 messages since previous log ~18 seconds ago
I20250112 15:08:54.859555 17651 logging.cc:424] LogThrottler /home/jenkins-slave/workspace/build_and_test_flaky/src/kudu/rpc/proxy.cc:239: suppressed but not reported on 24 messages since previous log ~8 seconds ago