[==========] Running 14 tests from 1 test suite.
[----------] Global test environment set-up.
[----------] 14 tests from AutoRebalancerTest
[ RUN ] AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing
WARNING: Logging before InitGoogleLogging() is written to STDERR
I20240503 23:24:48.543890 400 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.0.100.62:46349,127.0.100.61:33979,127.0.100.60:33981
I20240503 23:24:48.546258 400 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 23:24:48.547801 400 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 23:24:48.562479 406 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 23:24:48.563180 405 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 23:24:48.563941 408 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 23:24:48.564819 400 server_base.cc:1034] running on GCE node
I20240503 23:24:48.565973 400 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240503 23:24:48.566181 400 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240503 23:24:48.566295 400 hybrid_clock.cc:648] HybridClock initialized: now 1714778688566257 us; error 0 us; skew 500 ppm
I20240503 23:24:48.566725 400 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 23:24:48.570537 400 webserver.cc:460] Webserver started at http://127.0.100.62:40735/ using document root <none> and password file <none>
I20240503 23:24:48.571558 400 fs_manager.cc:359] Metadata directory not provided
I20240503 23:24:48.571683 400 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 23:24:48.572162 400 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 23:24:48.576697 400 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZhIiPz/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714778688451250-400-0/minicluster-data/master-0-root/instance:
uuid: "7a4cf86e12b74089b8fbea8b8a032a01"
format_stamp: "Formatted at 2024-05-03 23:24:48 on dist-test-slave-jvbp"
I20240503 23:24:48.585281 400 fs_manager.cc:691] Time spent creating directory manager: real 0.008s user 0.007s sys 0.001s
I20240503 23:24:48.589920 413 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 23:24:48.592087 400 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20240503 23:24:48.592346 400 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZhIiPz/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714778688451250-400-0/minicluster-data/master-0-root
uuid: "7a4cf86e12b74089b8fbea8b8a032a01"
format_stamp: "Formatted at 2024-05-03 23:24:48 on dist-test-slave-jvbp"
I20240503 23:24:48.592626 400 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZhIiPz/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714778688451250-400-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskZhIiPz/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714778688451250-400-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskZhIiPz/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714778688451250-400-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 23:24:48.614748 400 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 23:24:48.615924 400 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 23:24:48.616313 400 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 23:24:48.642668 400 rpc_server.cc:307] RPC server started. Bound to: 127.0.100.62:46349
I20240503 23:24:48.642668 464 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.100.62:46349 every 8 connection(s)
I20240503 23:24:48.645685 400 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 23:24:48.650609 467 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 23:24:48.651391 468 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 23:24:48.652045 465 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 23:24:48.652290 400 server_base.cc:1034] running on GCE node
W20240503 23:24:48.652897 470 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 23:24:48.653388 400 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240503 23:24:48.653492 400 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240503 23:24:48.653576 400 hybrid_clock.cc:648] HybridClock initialized: now 1714778688653534 us; error 0 us; skew 500 ppm
I20240503 23:24:48.653802 400 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 23:24:48.655546 400 webserver.cc:460] Webserver started at http://127.0.100.61:43961/ using document root <none> and password file <none>
I20240503 23:24:48.655934 400 fs_manager.cc:359] Metadata directory not provided
I20240503 23:24:48.656075 400 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 23:24:48.656252 400 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 23:24:48.657155 400 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZhIiPz/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714778688451250-400-0/minicluster-data/master-1-root/instance:
uuid: "01d3e7fb2d0344a885db17c0fe4b84a6"
format_stamp: "Formatted at 2024-05-03 23:24:48 on dist-test-slave-jvbp"
I20240503 23:24:48.660594 400 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.003s
I20240503 23:24:48.664147 475 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 23:24:48.664741 400 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20240503 23:24:48.664928 400 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZhIiPz/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714778688451250-400-0/minicluster-data/master-1-root
uuid: "01d3e7fb2d0344a885db17c0fe4b84a6"
format_stamp: "Formatted at 2024-05-03 23:24:48 on dist-test-slave-jvbp"
I20240503 23:24:48.665118 400 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZhIiPz/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714778688451250-400-0/minicluster-data/master-1-root
metadata directory: /tmp/dist-test-taskZhIiPz/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714778688451250-400-0/minicluster-data/master-1-root
1 data directories: /tmp/dist-test-taskZhIiPz/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714778688451250-400-0/minicluster-data/master-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 23:24:48.668674 465 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.100.62" port: 46349 } has no permanent_uuid. Determining permanent_uuid...
I20240503 23:24:48.676702 400 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 23:24:48.677561 400 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 23:24:48.700407 465 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.100.61" port: 33979 } has no permanent_uuid. Determining permanent_uuid...
W20240503 23:24:48.702520 416 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.100.61:33979: connect: Connection refused (error 111)
W20240503 23:24:48.705272 465 consensus_peers.cc:644] Error getting permanent uuid from config peer 127.0.100.61:33979: Network error: Client connection negotiation failed: client connection to 127.0.100.61:33979: connect: Connection refused (error 111)
I20240503 23:24:48.708221 400 rpc_server.cc:307] RPC server started. Bound to: 127.0.100.61:33979
I20240503 23:24:48.708254 529 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.100.61:33979 every 8 connection(s)
I20240503 23:24:48.710173 400 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20240503 23:24:48.710326 530 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
W20240503 23:24:48.715374 532 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 23:24:48.715539 533 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 23:24:48.715956 530 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.100.62" port: 46349 } has no permanent_uuid. Determining permanent_uuid...
W20240503 23:24:48.718230 535 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 23:24:48.724681 400 server_base.cc:1034] running on GCE node
I20240503 23:24:48.724985 530 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.100.61" port: 33979 } has no permanent_uuid. Determining permanent_uuid...
I20240503 23:24:48.725100 400 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240503 23:24:48.725181 400 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240503 23:24:48.725260 400 hybrid_clock.cc:648] HybridClock initialized: now 1714778688725218 us; error 0 us; skew 500 ppm
I20240503 23:24:48.725447 400 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 23:24:48.732254 400 webserver.cc:460] Webserver started at http://127.0.100.60:40815/ using document root <none> and password file <none>
I20240503 23:24:48.732620 400 fs_manager.cc:359] Metadata directory not provided
I20240503 23:24:48.732770 400 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 23:24:48.733592 530 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.100.60" port: 33981 } has no permanent_uuid. Determining permanent_uuid...
I20240503 23:24:48.733716 400 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 23:24:48.734789 400 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZhIiPz/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714778688451250-400-0/minicluster-data/master-2-root/instance:
uuid: "2bae372273f24a8e98f912be100fdc47"
format_stamp: "Formatted at 2024-05-03 23:24:48 on dist-test-slave-jvbp"
W20240503 23:24:48.736500 530 consensus_peers.cc:644] Error getting permanent uuid from config peer 127.0.100.60:33981: Network error: Client connection negotiation failed: client connection to 127.0.100.60:33981: connect: Connection refused (error 111)
I20240503 23:24:48.738284 400 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.004s
I20240503 23:24:48.740465 543 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 23:24:48.741086 400 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.001s
I20240503 23:24:48.741286 400 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZhIiPz/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714778688451250-400-0/minicluster-data/master-2-root
uuid: "2bae372273f24a8e98f912be100fdc47"
format_stamp: "Formatted at 2024-05-03 23:24:48 on dist-test-slave-jvbp"
I20240503 23:24:48.741518 400 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZhIiPz/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714778688451250-400-0/minicluster-data/master-2-root
metadata directory: /tmp/dist-test-taskZhIiPz/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714778688451250-400-0/minicluster-data/master-2-root
1 data directories: /tmp/dist-test-taskZhIiPz/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714778688451250-400-0/minicluster-data/master-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 23:24:48.754914 465 consensus_peers.cc:654] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.100.61" port: 33979 } attempt: 1
I20240503 23:24:48.756371 400 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 23:24:48.757483 400 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 23:24:48.764796 465 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.100.60" port: 33981 } has no permanent_uuid. Determining permanent_uuid...
W20240503 23:24:48.767279 465 consensus_peers.cc:644] Error getting permanent uuid from config peer 127.0.100.60:33981: Network error: Client connection negotiation failed: client connection to 127.0.100.60:33981: connect: Connection refused (error 111)
I20240503 23:24:48.781517 400 rpc_server.cc:307] RPC server started. Bound to: 127.0.100.60:33981
I20240503 23:24:48.781625 594 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.100.60:33981 every 8 connection(s)
I20240503 23:24:48.782876 400 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20240503 23:24:48.783499 595 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 23:24:48.786373 595 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.100.62" port: 46349 } has no permanent_uuid. Determining permanent_uuid...
I20240503 23:24:48.788789 530 consensus_peers.cc:654] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.100.60" port: 33981 } attempt: 1
I20240503 23:24:48.796411 595 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.100.61" port: 33979 } has no permanent_uuid. Determining permanent_uuid...
I20240503 23:24:48.803752 595 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.100.60" port: 33981 } has no permanent_uuid. Determining permanent_uuid...
I20240503 23:24:48.803953 530 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 01d3e7fb2d0344a885db17c0fe4b84a6: Bootstrap starting.
I20240503 23:24:48.810580 465 consensus_peers.cc:654] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.100.60" port: 33981 } attempt: 1
I20240503 23:24:48.811198 530 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 01d3e7fb2d0344a885db17c0fe4b84a6: Neither blocks nor log segments found. Creating new log.
I20240503 23:24:48.813668 530 log.cc:824] T 00000000000000000000000000000000 P 01d3e7fb2d0344a885db17c0fe4b84a6: Log is configured to *not* fsync() on all Append() calls
I20240503 23:24:48.815500 595 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2bae372273f24a8e98f912be100fdc47: Bootstrap starting.
I20240503 23:24:48.817775 595 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2bae372273f24a8e98f912be100fdc47: Neither blocks nor log segments found. Creating new log.
I20240503 23:24:48.819308 530 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 01d3e7fb2d0344a885db17c0fe4b84a6: No bootstrap required, opened a new log
I20240503 23:24:48.821065 595 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2bae372273f24a8e98f912be100fdc47: No bootstrap required, opened a new log
I20240503 23:24:48.822342 465 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7a4cf86e12b74089b8fbea8b8a032a01: Bootstrap starting.
I20240503 23:24:48.823904 465 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7a4cf86e12b74089b8fbea8b8a032a01: Neither blocks nor log segments found. Creating new log.
I20240503 23:24:48.826156 465 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7a4cf86e12b74089b8fbea8b8a032a01: No bootstrap required, opened a new log
I20240503 23:24:48.828686 465 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7a4cf86e12b74089b8fbea8b8a032a01 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a4cf86e12b74089b8fbea8b8a032a01" member_type: VOTER last_known_addr { host: "127.0.100.62" port: 46349 } } peers { permanent_uuid: "01d3e7fb2d0344a885db17c0fe4b84a6" member_type: VOTER last_known_addr { host: "127.0.100.61" port: 33979 } } peers { permanent_uuid: "2bae372273f24a8e98f912be100fdc47" member_type: VOTER last_known_addr { host: "127.0.100.60" port: 33981 } }
I20240503 23:24:48.828680 595 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2bae372273f24a8e98f912be100fdc47 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a4cf86e12b74089b8fbea8b8a032a01" member_type: VOTER last_known_addr { host: "127.0.100.62" port: 46349 } } peers { permanent_uuid: "01d3e7fb2d0344a885db17c0fe4b84a6" member_type: VOTER last_known_addr { host: "127.0.100.61" port: 33979 } } peers { permanent_uuid: "2bae372273f24a8e98f912be100fdc47" member_type: VOTER last_known_addr { host: "127.0.100.60" port: 33981 } }
I20240503 23:24:48.829788 530 raft_consensus.cc:348] T 00000000000000000000000000000000 P 01d3e7fb2d0344a885db17c0fe4b84a6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a4cf86e12b74089b8fbea8b8a032a01" member_type: VOTER last_known_addr { host: "127.0.100.62" port: 46349 } } peers { permanent_uuid: "01d3e7fb2d0344a885db17c0fe4b84a6" member_type: VOTER last_known_addr { host: "127.0.100.61" port: 33979 } } peers { permanent_uuid: "2bae372273f24a8e98f912be100fdc47" member_type: VOTER last_known_addr { host: "127.0.100.60" port: 33981 } }
I20240503 23:24:48.830024 530 raft_consensus.cc:374] T 00000000000000000000000000000000 P 01d3e7fb2d0344a885db17c0fe4b84a6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 23:24:48.830072 595 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2bae372273f24a8e98f912be100fdc47 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 23:24:48.830029 465 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7a4cf86e12b74089b8fbea8b8a032a01 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 23:24:48.830192 530 raft_consensus.cc:729] T 00000000000000000000000000000000 P 01d3e7fb2d0344a885db17c0fe4b84a6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 01d3e7fb2d0344a885db17c0fe4b84a6, State: Initialized, Role: FOLLOWER
I20240503 23:24:48.830214 595 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2bae372273f24a8e98f912be100fdc47 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2bae372273f24a8e98f912be100fdc47, State: Initialized, Role: FOLLOWER
I20240503 23:24:48.830345 465 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7a4cf86e12b74089b8fbea8b8a032a01 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a4cf86e12b74089b8fbea8b8a032a01, State: Initialized, Role: FOLLOWER
I20240503 23:24:48.831341 595 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2bae372273f24a8e98f912be100fdc47 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a4cf86e12b74089b8fbea8b8a032a01" member_type: VOTER last_known_addr { host: "127.0.100.62" port: 46349 } } peers { permanent_uuid: "01d3e7fb2d0344a885db17c0fe4b84a6" member_type: VOTER last_known_addr { host: "127.0.100.61" port: 33979 } } peers { permanent_uuid: "2bae372273f24a8e98f912be100fdc47" member_type: VOTER last_known_addr { host: "127.0.100.60" port: 33981 } }
I20240503 23:24:48.831375 465 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7a4cf86e12b74089b8fbea8b8a032a01 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a4cf86e12b74089b8fbea8b8a032a01" member_type: VOTER last_known_addr { host: "127.0.100.62" port: 46349 } } peers { permanent_uuid: "01d3e7fb2d0344a885db17c0fe4b84a6" member_type: VOTER last_known_addr { host: "127.0.100.61" port: 33979 } } peers { permanent_uuid: "2bae372273f24a8e98f912be100fdc47" member_type: VOTER last_known_addr { host: "127.0.100.60" port: 33981 } }
I20240503 23:24:48.831341 530 consensus_queue.cc:260] T 00000000000000000000000000000000 P 01d3e7fb2d0344a885db17c0fe4b84a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a4cf86e12b74089b8fbea8b8a032a01" member_type: VOTER last_known_addr { host: "127.0.100.62" port: 46349 } } peers { permanent_uuid: "01d3e7fb2d0344a885db17c0fe4b84a6" member_type: VOTER last_known_addr { host: "127.0.100.61" port: 33979 } } peers { permanent_uuid: "2bae372273f24a8e98f912be100fdc47" member_type: VOTER last_known_addr { host: "127.0.100.60" port: 33981 } }
I20240503 23:24:48.833057 603 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7a4cf86e12b74089b8fbea8b8a032a01 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a4cf86e12b74089b8fbea8b8a032a01" member_type: VOTER last_known_addr { host: "127.0.100.62" port: 46349 } } peers { permanent_uuid: "01d3e7fb2d0344a885db17c0fe4b84a6" member_type: VOTER last_known_addr { host: "127.0.100.61" port: 33979 } } peers { permanent_uuid: "2bae372273f24a8e98f912be100fdc47" member_type: VOTER last_known_addr { host: "127.0.100.60" port: 33981 } } }
I20240503 23:24:48.833926 603 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7a4cf86e12b74089b8fbea8b8a032a01 [sys.catalog]: This master's current role is: FOLLOWER
I20240503 23:24:48.833640 602 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2bae372273f24a8e98f912be100fdc47 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a4cf86e12b74089b8fbea8b8a032a01" member_type: VOTER last_known_addr { host: "127.0.100.62" port: 46349 } } peers { permanent_uuid: "01d3e7fb2d0344a885db17c0fe4b84a6" member_type: VOTER last_known_addr { host: "127.0.100.61" port: 33979 } } peers { permanent_uuid: "2bae372273f24a8e98f912be100fdc47" member_type: VOTER last_known_addr { host: "127.0.100.60" port: 33981 } } }
I20240503 23:24:48.834256 602 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2bae372273f24a8e98f912be100fdc47 [sys.catalog]: This master's current role is: FOLLOWER
I20240503 23:24:48.834813 604 sys_catalog.cc:455] T 00000000000000000000000000000000 P 01d3e7fb2d0344a885db17c0fe4b84a6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a4cf86e12b74089b8fbea8b8a032a01" member_type: VOTER last_known_addr { host: "127.0.100.62" port: 46349 } } peers { permanent_uuid: "01d3e7fb2d0344a885db17c0fe4b84a6" member_type: VOTER last_known_addr { host: "127.0.100.61" port: 33979 } } peers { permanent_uuid: "2bae372273f24a8e98f912be100fdc47" member_type: VOTER last_known_addr { host: "127.0.100.60" port: 33981 } } }
I20240503 23:24:48.835022 465 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7a4cf86e12b74089b8fbea8b8a032a01 [sys.catalog]: configured and running, proceeding with master startup.
I20240503 23:24:48.835135 604 sys_catalog.cc:458] T 00000000000000000000000000000000 P 01d3e7fb2d0344a885db17c0fe4b84a6 [sys.catalog]: This master's current role is: FOLLOWER
I20240503 23:24:48.835032 595 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2bae372273f24a8e98f912be100fdc47 [sys.catalog]: configured and running, proceeding with master startup.
I20240503 23:24:48.835078 530 sys_catalog.cc:564] T 00000000000000000000000000000000 P 01d3e7fb2d0344a885db17c0fe4b84a6 [sys.catalog]: configured and running, proceeding with master startup.
I20240503 23:24:48.845621 400 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 1
W20240503 23:24:48.854445 621 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 7a4cf86e12b74089b8fbea8b8a032a01: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20240503 23:24:48.854804 621 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
W20240503 23:24:48.867952 637 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 2bae372273f24a8e98f912be100fdc47: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20240503 23:24:48.868172 637 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20240503 23:24:48.868634 400 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 2
W20240503 23:24:48.869071 419 tablet.cc:2369] T 00000000000000000000000000000000 P 7a4cf86e12b74089b8fbea8b8a032a01: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240503 23:24:48.870131 400 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 23:24:48.870604 638 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 01d3e7fb2d0344a885db17c0fe4b84a6: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20240503 23:24:48.870715 638 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
W20240503 23:24:48.877094 639 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 23:24:48.877765 640 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 23:24:48.879024 400 server_base.cc:1034] running on GCE node
W20240503 23:24:48.879403 642 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 23:24:48.879922 400 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240503 23:24:48.880035 400 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240503 23:24:48.880129 400 hybrid_clock.cc:648] HybridClock initialized: now 1714778688880103 us; error 0 us; skew 500 ppm
I20240503 23:24:48.880337 400 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 23:24:48.882207 400 webserver.cc:460] Webserver started at http://127.0.100.1:34541/ using document root <none> and password file <none>
I20240503 23:24:48.882550 400 fs_manager.cc:359] Metadata directory not provided
I20240503 23:24:48.882660 400 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 23:24:48.882881 400 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 23:24:48.884832 400 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZhIiPz/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714778688451250-400-0/minicluster-data/ts-0-root/instance:
uuid: "8181f028a7dd43a1af4ff6ebd6011af8"
format_stamp: "Formatted at 2024-05-03 23:24:48 on dist-test-slave-jvbp"
I20240503 23:24:48.889097 400 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.003s
I20240503 23:24:48.891490 647 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 23:24:48.892004 400 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20240503 23:24:48.892138 400 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZhIiPz/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714778688451250-400-0/minicluster-data/ts-0-root
uuid: "8181f028a7dd43a1af4ff6ebd6011af8"
format_stamp: "Formatted at 2024-05-03 23:24:48 on dist-test-slave-jvbp"
I20240503 23:24:48.892297 400 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZhIiPz/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714778688451250-400-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskZhIiPz/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714778688451250-400-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskZhIiPz/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714778688451250-400-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 23:24:48.905596 400 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 23:24:48.906436 400 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 23:24:48.908854 400 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 23:24:48.910856 400 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 23:24:48.911023 400 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 23:24:48.911175 400 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 23:24:48.911267 400 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 23:24:48.941732 709 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.100.1:34815 every 8 connection(s)
I20240503 23:24:48.941721 400 rpc_server.cc:307] RPC server started. Bound to: 127.0.100.1:34815
I20240503 23:24:48.952836 603 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7a4cf86e12b74089b8fbea8b8a032a01 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 23:24:48.953264 603 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7a4cf86e12b74089b8fbea8b8a032a01 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a4cf86e12b74089b8fbea8b8a032a01" member_type: VOTER last_known_addr { host: "127.0.100.62" port: 46349 } } peers { permanent_uuid: "01d3e7fb2d0344a885db17c0fe4b84a6" member_type: VOTER last_known_addr { host: "127.0.100.61" port: 33979 } } peers { permanent_uuid: "2bae372273f24a8e98f912be100fdc47" member_type: VOTER last_known_addr { host: "127.0.100.60" port: 33981 } }
I20240503 23:24:48.959270 504 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "7a4cf86e12b74089b8fbea8b8a032a01" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "01d3e7fb2d0344a885db17c0fe4b84a6" is_pre_election: true
I20240503 23:24:48.959884 711 heartbeater.cc:347] Connected to a master server at 127.0.100.62:46349
I20240503 23:24:48.960086 504 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 01d3e7fb2d0344a885db17c0fe4b84a6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7a4cf86e12b74089b8fbea8b8a032a01 in term 0.
I20240503 23:24:48.960191 711 heartbeater.cc:464] Registering TS with master...
I20240503 23:24:48.961190 416 leader_election.cc:302] T 00000000000000000000000000000000 P 7a4cf86e12b74089b8fbea8b8a032a01 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 01d3e7fb2d0344a885db17c0fe4b84a6, 7a4cf86e12b74089b8fbea8b8a032a01; no voters:
I20240503 23:24:48.961926 711 heartbeater.cc:511] Master 127.0.100.62:46349 requested a full tablet report, sending...
I20240503 23:24:48.962931 717 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7a4cf86e12b74089b8fbea8b8a032a01 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 23:24:48.963222 717 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7a4cf86e12b74089b8fbea8b8a032a01 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 23:24:48.969439 570 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "7a4cf86e12b74089b8fbea8b8a032a01" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2bae372273f24a8e98f912be100fdc47" is_pre_election: true
I20240503 23:24:48.969911 570 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 2bae372273f24a8e98f912be100fdc47 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7a4cf86e12b74089b8fbea8b8a032a01 in term 0.
I20240503 23:24:48.969983 400 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20240503 23:24:48.970566 712 heartbeater.cc:347] Connected to a master server at 127.0.100.61:33979
I20240503 23:24:48.970769 712 heartbeater.cc:464] Registering TS with master...
I20240503 23:24:48.971417 712 heartbeater.cc:511] Master 127.0.100.61:33979 requested a full tablet report, sending...
I20240503 23:24:48.972991 710 heartbeater.cc:347] Connected to a master server at 127.0.100.60:33981
I20240503 23:24:48.973235 710 heartbeater.cc:464] Registering TS with master...
I20240503 23:24:48.973937 494 ts_manager.cc:194] Registered new tserver with Master: 8181f028a7dd43a1af4ff6ebd6011af8 (127.0.100.1:34815)
I20240503 23:24:48.973994 710 heartbeater.cc:511] Master 127.0.100.60:33981 requested a full tablet report, sending...
I20240503 23:24:48.975394 603 leader_election.cc:288] T 00000000000000000000000000000000 P 7a4cf86e12b74089b8fbea8b8a032a01 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 01d3e7fb2d0344a885db17c0fe4b84a6 (127.0.100.61:33979), 2bae372273f24a8e98f912be100fdc47 (127.0.100.60:33981)
I20240503 23:24:48.975515 560 ts_manager.cc:194] Registered new tserver with Master: 8181f028a7dd43a1af4ff6ebd6011af8 (127.0.100.1:34815)
I20240503 23:24:48.976227 717 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7a4cf86e12b74089b8fbea8b8a032a01 [term 0 FOLLOWER]: Advancing to term 1
I20240503 23:24:48.979367 717 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7a4cf86e12b74089b8fbea8b8a032a01 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a4cf86e12b74089b8fbea8b8a032a01" member_type: VOTER last_known_addr { host: "127.0.100.62" port: 46349 } } peers { permanent_uuid: "01d3e7fb2d0344a885db17c0fe4b84a6" member_type: VOTER last_known_addr { host: "127.0.100.61" port: 33979 } } peers { permanent_uuid: "2bae372273f24a8e98f912be100fdc47" member_type: VOTER last_known_addr { host: "127.0.100.60" port: 33981 } }
I20240503 23:24:48.982002 430 ts_manager.cc:194] Registered new tserver with Master: 8181f028a7dd43a1af4ff6ebd6011af8 (127.0.100.1:34815)
I20240503 23:24:48.983342 504 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "7a4cf86e12b74089b8fbea8b8a032a01" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "01d3e7fb2d0344a885db17c0fe4b84a6"
I20240503 23:24:48.983650 504 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 01d3e7fb2d0344a885db17c0fe4b84a6 [term 0 FOLLOWER]: Advancing to term 1
I20240503 23:24:48.984990 717 leader_election.cc:288] T 00000000000000000000000000000000 P 7a4cf86e12b74089b8fbea8b8a032a01 [CANDIDATE]: Term 1 election: Requested vote from peers 01d3e7fb2d0344a885db17c0fe4b84a6 (127.0.100.61:33979), 2bae372273f24a8e98f912be100fdc47 (127.0.100.60:33981)
I20240503 23:24:48.986002 570 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "7a4cf86e12b74089b8fbea8b8a032a01" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2bae372273f24a8e98f912be100fdc47"
I20240503 23:24:48.986724 570 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2bae372273f24a8e98f912be100fdc47 [term 0 FOLLOWER]: Advancing to term 1
W20240503 23:24:48.987869 718 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 23:24:48.990550 504 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 01d3e7fb2d0344a885db17c0fe4b84a6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7a4cf86e12b74089b8fbea8b8a032a01 in term 1.
W20240503 23:24:48.991444 719 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 23:24:48.991662 416 leader_election.cc:302] T 00000000000000000000000000000000 P 7a4cf86e12b74089b8fbea8b8a032a01 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 01d3e7fb2d0344a885db17c0fe4b84a6, 7a4cf86e12b74089b8fbea8b8a032a01; no voters:
I20240503 23:24:48.992043 717 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7a4cf86e12b74089b8fbea8b8a032a01 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 23:24:48.992282 717 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7a4cf86e12b74089b8fbea8b8a032a01 [term 1 LEADER]: Becoming Leader. State: Replica: 7a4cf86e12b74089b8fbea8b8a032a01, State: Running, Role: LEADER
I20240503 23:24:48.992646 570 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 2bae372273f24a8e98f912be100fdc47 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7a4cf86e12b74089b8fbea8b8a032a01 in term 1.
I20240503 23:24:48.993393 717 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7a4cf86e12b74089b8fbea8b8a032a01 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a4cf86e12b74089b8fbea8b8a032a01" member_type: VOTER last_known_addr { host: "127.0.100.62" port: 46349 } } peers { permanent_uuid: "01d3e7fb2d0344a885db17c0fe4b84a6" member_type: VOTER last_known_addr { host: "127.0.100.61" port: 33979 } } peers { permanent_uuid: "2bae372273f24a8e98f912be100fdc47" member_type: VOTER last_known_addr { host: "127.0.100.60" port: 33981 } }
W20240503 23:24:49.000309 721 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 23:24:49.000308 603 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7a4cf86e12b74089b8fbea8b8a032a01 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7a4cf86e12b74089b8fbea8b8a032a01. Latest consensus state: current_term: 1 leader_uuid: "7a4cf86e12b74089b8fbea8b8a032a01" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a4cf86e12b74089b8fbea8b8a032a01" member_type: VOTER last_known_addr { host: "127.0.100.62" port: 46349 } } peers { permanent_uuid: "01d3e7fb2d0344a885db17c0fe4b84a6" member_type: VOTER last_known_addr { host: "127.0.100.61" port: 33979 } } peers { permanent_uuid: "2bae372273f24a8e98f912be100fdc47" member_type: VOTER last_known_addr { host: "127.0.100.60" port: 33981 } } }
I20240503 23:24:49.001564 603 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7a4cf86e12b74089b8fbea8b8a032a01 [sys.catalog]: This master's current role is: LEADER
I20240503 23:24:49.004159 724 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240503 23:24:49.007227 724 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240503 23:24:49.007829 400 server_base.cc:1034] running on GCE node
I20240503 23:24:49.008213 400 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240503 23:24:49.008301 400 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240503 23:24:49.008386 400 hybrid_clock.cc:648] HybridClock initialized: now 1714778689008341 us; error 0 us; skew 500 ppm
I20240503 23:24:49.008602 400 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 23:24:49.010690 400 webserver.cc:460] Webserver started at http://127.0.100.2:33621/ using document root <none> and password file <none>
I20240503 23:24:49.011000 400 fs_manager.cc:359] Metadata directory not provided
I20240503 23:24:49.011128 400 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 23:24:49.011296 400 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 23:24:49.012182 400 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZhIiPz/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714778688451250-400-0/minicluster-data/ts-1-root/instance:
uuid: "06b178e84b2e43c68511466d60b5213f"
format_stamp: "Formatted at 2024-05-03 23:24:49 on dist-test-slave-jvbp"
I20240503 23:24:49.017346 400 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.002s
I20240503 23:24:49.020217 729 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 23:24:49.021682 570 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 2bae372273f24a8e98f912be100fdc47 [term 1 FOLLOWER]: Refusing update from remote peer 7a4cf86e12b74089b8fbea8b8a032a01: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 23:24:49.021683 504 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 01d3e7fb2d0344a885db17c0fe4b84a6 [term 1 FOLLOWER]: Refusing update from remote peer 7a4cf86e12b74089b8fbea8b8a032a01: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 23:24:49.022425 400 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20240503 23:24:49.022595 400 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZhIiPz/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714778688451250-400-0/minicluster-data/ts-1-root
uuid: "06b178e84b2e43c68511466d60b5213f"
format_stamp: "Formatted at 2024-05-03 23:24:49 on dist-test-slave-jvbp"
I20240503 23:24:49.022799 400 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZhIiPz/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714778688451250-400-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskZhIiPz/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714778688451250-400-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskZhIiPz/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714778688451250-400-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 23:24:49.022795 603 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 7a4cf86e12b74089b8fbea8b8a032a01 [LEADER]: Connected to new peer: Peer: permanent_uuid: "01d3e7fb2d0344a885db17c0fe4b84a6" member_type: VOTER last_known_addr { host: "127.0.100.61" port: 33979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 23:24:49.023183 717 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 7a4cf86e12b74089b8fbea8b8a032a01 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2bae372273f24a8e98f912be100fdc47" member_type: VOTER last_known_addr { host: "127.0.100.60" port: 33981 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 23:24:49.043722 604 sys_catalog.cc:455] T 00000000000000000000000000000000 P 01d3e7fb2d0344a885db17c0fe4b84a6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7a4cf86e12b74089b8fbea8b8a032a01. Latest consensus state: current_term: 1 leader_uuid: "7a4cf86e12b74089b8fbea8b8a032a01" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a4cf86e12b74089b8fbea8b8a032a01" member_type: VOTER last_known_addr { host: "127.0.100.62" port: 46349 } } peers { permanent_uuid: "01d3e7fb2d0344a885db17c0fe4b84a6" member_type: VOTER last_known_addr { host: "127.0.100.61" port: 33979 } } peers { permanent_uuid: "2bae372273f24a8e98f912be100fdc47" member_type: VOTER last_known_addr { host: "127.0.100.60" port: 33981 } } }
I20240503 23:24:49.044246 604 sys_catalog.cc:458] T 00000000000000000000000000000000 P 01d3e7fb2d0344a885db17c0fe4b84a6 [sys.catalog]: This master's current role is: FOLLOWER
I20240503 23:24:49.045472 602 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2bae372273f24a8e98f912be100fdc47 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7a4cf86e12b74089b8fbea8b8a032a01. Latest consensus state: current_term: 1 leader_uuid: "7a4cf86e12b74089b8fbea8b8a032a01" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a4cf86e12b74089b8fbea8b8a032a01" member_type: VOTER last_known_addr { host: "127.0.100.62" port: 46349 } } peers { permanent_uuid: "01d3e7fb2d0344a885db17c0fe4b84a6" member_type: VOTER last_known_addr { host: "127.0.100.61" port: 33979 } } peers { permanent_uuid: "2bae372273f24a8e98f912be100fdc47" member_type: VOTER last_known_addr { host: "127.0.100.60" port: 33981 } } }
I20240503 23:24:49.045797 602 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2bae372273f24a8e98f912be100fdc47 [sys.catalog]: This master's current role is: FOLLOWER
I20240503 23:24:49.051458 732 mvcc.cc:205] Tried to move back new op lower bound from 7023733510217351168 to 7023733510131027968. Current Snapshot: MvccSnapshot[applied={T|T < 7023733510217351168}]
I20240503 23:24:49.060830 724 catalog_manager.cc:1347] Generated new cluster ID: c3ed65145b3648dcac89dac2e94f3ddc
I20240503 23:24:49.061008 724 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240503 23:24:49.080618 603 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7a4cf86e12b74089b8fbea8b8a032a01 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "7a4cf86e12b74089b8fbea8b8a032a01" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a4cf86e12b74089b8fbea8b8a032a01" member_type: VOTER last_known_addr { host: "127.0.100.62" port: 46349 } } peers { permanent_uuid: "01d3e7fb2d0344a885db17c0fe4b84a6" member_type: VOTER last_known_addr { host: "127.0.100.61" port: 33979 } } peers { permanent_uuid: "2bae372273f24a8e98f912be100fdc47" member_type: VOTER last_known_addr { host: "127.0.100.60" port: 33981 } } }
I20240503 23:24:49.080811 604 sys_catalog.cc:455] T 00000000000000000000000000000000 P 01d3e7fb2d0344a885db17c0fe4b84a6 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "7a4cf86e12b74089b8fbea8b8a032a01" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a4cf86e12b74089b8fbea8b8a032a01" member_type: VOTER last_known_addr { host: "127.0.100.62" port: 46349 } } peers { permanent_uuid: "01d3e7fb2d0344a885db17c0fe4b84a6" member_type: VOTER last_known_addr { host: "127.0.100.61" port: 33979 } } peers { permanent_uuid: "2bae372273f24a8e98f912be100fdc47" member_type: VOTER last_known_addr { host: "127.0.100.60" port: 33981 } } }
I20240503 23:24:49.081024 603 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7a4cf86e12b74089b8fbea8b8a032a01 [sys.catalog]: This master's current role is: LEADER
I20240503 23:24:49.081121 604 sys_catalog.cc:458] T 00000000000000000000000000000000 P 01d3e7fb2d0344a885db17c0fe4b84a6 [sys.catalog]: This master's current role is: FOLLOWER
I20240503 23:24:49.082633 602 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2bae372273f24a8e98f912be100fdc47 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "7a4cf86e12b74089b8fbea8b8a032a01" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a4cf86e12b74089b8fbea8b8a032a01" member_type: VOTER last_known_addr { host: "127.0.100.62" port: 46349 } } peers { permanent_uuid: "01d3e7fb2d0344a885db17c0fe4b84a6" member_type: VOTER last_known_addr { host: "127.0.100.61" port: 33979 } } peers { permanent_uuid: "2bae372273f24a8e98f912be100fdc47" member_type: VOTER last_known_addr { host: "127.0.100.60" port: 33981 } } }
I20240503 23:24:49.082950 602 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2bae372273f24a8e98f912be100fdc47 [sys.catalog]: This master's current role is: FOLLOWER
I20240503 23:24:49.083082 717 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7a4cf86e12b74089b8fbea8b8a032a01 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "7a4cf86e12b74089b8fbea8b8a032a01" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a4cf86e12b74089b8fbea8b8a032a01" member_type: VOTER last_known_addr { host: "127.0.100.62" port: 46349 } } peers { permanent_uuid: "01d3e7fb2d0344a885db17c0fe4b84a6" member_type: VOTER last_known_addr { host: "127.0.100.61" port: 33979 } } peers { permanent_uuid: "2bae372273f24a8e98f912be100fdc47" member_type: VOTER last_known_addr { host: "127.0.100.60" port: 33981 } } }
I20240503 23:24:49.083357 717 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7a4cf86e12b74089b8fbea8b8a032a01 [sys.catalog]: This master's current role is: LEADER
I20240503 23:24:49.101943 400 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 23:24:49.103004 400 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 23:24:49.109221 724 catalog_manager.cc:1370] Generated new certificate authority record
I20240503 23:24:49.111609 724 catalog_manager.cc:1504] Loading token signing keys...
I20240503 23:24:49.112973 400 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 23:24:49.118757 400 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 23:24:49.118911 400 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 23:24:49.119057 400 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 23:24:49.119138 400 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 23:24:49.140614 724 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 7a4cf86e12b74089b8fbea8b8a032a01: Generated new TSK 0
I20240503 23:24:49.142881 724 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240503 23:24:49.170097 799 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.100.2:40145 every 8 connection(s)
I20240503 23:24:49.170097 400 rpc_server.cc:307] RPC server started. Bound to: 127.0.100.2:40145
I20240503 23:24:49.195511 400 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20240503 23:24:49.201438 801 heartbeater.cc:347] Connected to a master server at 127.0.100.62:46349
I20240503 23:24:49.201685 801 heartbeater.cc:464] Registering TS with master...
I20240503 23:24:49.202940 800 heartbeater.cc:347] Connected to a master server at 127.0.100.60:33981
I20240503 23:24:49.203172 800 heartbeater.cc:464] Registering TS with master...
I20240503 23:24:49.203325 802 heartbeater.cc:347] Connected to a master server at 127.0.100.61:33979
I20240503 23:24:49.203454 802 heartbeater.cc:464] Registering TS with master...
I20240503 23:24:49.205770 801 heartbeater.cc:511] Master 127.0.100.62:46349 requested a full tablet report, sending...
I20240503 23:24:49.205786 800 heartbeater.cc:511] Master 127.0.100.60:33981 requested a full tablet report, sending...
I20240503 23:24:49.207060 430 ts_manager.cc:194] Registered new tserver with Master: 06b178e84b2e43c68511466d60b5213f (127.0.100.2:40145)
I20240503 23:24:49.207443 560 ts_manager.cc:194] Registered new tserver with Master: 06b178e84b2e43c68511466d60b5213f (127.0.100.2:40145)
I20240503 23:24:49.208410 802 heartbeater.cc:511] Master 127.0.100.61:33979 requested a full tablet report, sending...
I20240503 23:24:49.209472 430 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:38528
I20240503 23:24:49.210053 494 ts_manager.cc:194] Registered new tserver with Master: 06b178e84b2e43c68511466d60b5213f (127.0.100.2:40145)
W20240503 23:24:49.210175 808 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 23:24:49.211081 807 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 23:24:49.219084 810 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 23:24:49.219884 400 server_base.cc:1034] running on GCE node
I20240503 23:24:49.220234 400 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240503 23:24:49.220331 400 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240503 23:24:49.220402 400 hybrid_clock.cc:648] HybridClock initialized: now 1714778689220374 us; error 0 us; skew 500 ppm
I20240503 23:24:49.220649 400 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 23:24:49.222609 400 webserver.cc:460] Webserver started at http://127.0.100.3:36093/ using document root <none> and password file <none>
I20240503 23:24:49.222919 400 fs_manager.cc:359] Metadata directory not provided
I20240503 23:24:49.223022 400 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 23:24:49.223203 400 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 23:24:49.225004 400 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZhIiPz/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714778688451250-400-0/minicluster-data/ts-2-root/instance:
uuid: "fa5c2ac5b68e47649e922a2176ccfd28"
format_stamp: "Formatted at 2024-05-03 23:24:49 on dist-test-slave-jvbp"
I20240503 23:24:49.228394 400 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20240503 23:24:49.230685 815 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 23:24:49.231155 400 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.002s sys 0.000s
I20240503 23:24:49.231281 400 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZhIiPz/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714778688451250-400-0/minicluster-data/ts-2-root
uuid: "fa5c2ac5b68e47649e922a2176ccfd28"
format_stamp: "Formatted at 2024-05-03 23:24:49 on dist-test-slave-jvbp"
I20240503 23:24:49.231428 400 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZhIiPz/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714778688451250-400-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-taskZhIiPz/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714778688451250-400-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-taskZhIiPz/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714778688451250-400-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 23:24:49.246511 400 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 23:24:49.247437 400 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 23:24:49.248221 400 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 23:24:49.249532 400 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 23:24:49.249651 400 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 23:24:49.249764 400 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 23:24:49.249845 400 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 23:24:49.286675 400 rpc_server.cc:307] RPC server started. Bound to: 127.0.100.3:36505
I20240503 23:24:49.286717 877 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.100.3:36505 every 8 connection(s)
I20240503 23:24:49.312719 878 heartbeater.cc:347] Connected to a master server at 127.0.100.60:33981
I20240503 23:24:49.312991 878 heartbeater.cc:464] Registering TS with master...
I20240503 23:24:49.314878 878 heartbeater.cc:511] Master 127.0.100.60:33981 requested a full tablet report, sending...
I20240503 23:24:49.316619 560 ts_manager.cc:194] Registered new tserver with Master: fa5c2ac5b68e47649e922a2176ccfd28 (127.0.100.3:36505)
I20240503 23:24:49.320951 879 heartbeater.cc:347] Connected to a master server at 127.0.100.62:46349
I20240503 23:24:49.321159 879 heartbeater.cc:464] Registering TS with master...
I20240503 23:24:49.321815 879 heartbeater.cc:511] Master 127.0.100.62:46349 requested a full tablet report, sending...
I20240503 23:24:49.322746 880 heartbeater.cc:347] Connected to a master server at 127.0.100.61:33979
I20240503 23:24:49.322914 880 heartbeater.cc:464] Registering TS with master...
I20240503 23:24:49.323158 430 ts_manager.cc:194] Registered new tserver with Master: fa5c2ac5b68e47649e922a2176ccfd28 (127.0.100.3:36505)
I20240503 23:24:49.323611 880 heartbeater.cc:511] Master 127.0.100.61:33979 requested a full tablet report, sending...
I20240503 23:24:49.324462 430 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:38536
I20240503 23:24:49.324832 494 ts_manager.cc:194] Registered new tserver with Master: fa5c2ac5b68e47649e922a2176ccfd28 (127.0.100.3:36505)
I20240503 23:24:49.325145 400 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.012886328s
I20240503 23:24:49.871094 637 catalog_manager.cc:1259] Loaded cluster ID: c3ed65145b3648dcac89dac2e94f3ddc
I20240503 23:24:49.871244 637 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 2bae372273f24a8e98f912be100fdc47: loading cluster ID for follower catalog manager: success
I20240503 23:24:49.872323 638 catalog_manager.cc:1259] Loaded cluster ID: c3ed65145b3648dcac89dac2e94f3ddc
I20240503 23:24:49.872439 638 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 01d3e7fb2d0344a885db17c0fe4b84a6: loading cluster ID for follower catalog manager: success
I20240503 23:24:49.874150 637 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 2bae372273f24a8e98f912be100fdc47: acquiring CA information for follower catalog manager: success
I20240503 23:24:49.874595 638 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 01d3e7fb2d0344a885db17c0fe4b84a6: acquiring CA information for follower catalog manager: success
I20240503 23:24:49.875811 638 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 01d3e7fb2d0344a885db17c0fe4b84a6: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20240503 23:24:49.875823 637 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 2bae372273f24a8e98f912be100fdc47: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20240503 23:24:49.985194 430 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:38512
I20240503 23:24:50.212654 801 heartbeater.cc:503] Master 127.0.100.62:46349 was elected leader, sending a full tablet report...
I20240503 23:24:50.326640 879 heartbeater.cc:503] Master 127.0.100.62:46349 was elected leader, sending a full tablet report...
I20240503 23:24:50.844843 620 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240503 23:24:50.872884 400 test_util.cc:268] Using random seed: 113042965
I20240503 23:24:50.913872 430 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33730:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 1
split_rows_range_bounds {
rows: "\004\001\000\377\377\377\037\004\001\000\376\377\377?\004\001\000\375\377\377_""\004\001\000\377\377\377\037\004\001\000\376\377\377?\004\001\000\375\377\377_"
indirect_data: """"
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240503 23:24:50.956532 843 tablet_service.cc:1467] Processing CreateTablet for tablet f16a349decb2402abf31109ee05bdb50 (DEFAULT_TABLE table=test-workload [id=ab9b58b93b2f4e6fa89ead6a368228b9]), partition=RANGE (key) PARTITION VALUES < 536870911
I20240503 23:24:50.958065 843 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f16a349decb2402abf31109ee05bdb50. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 23:24:50.966734 675 tablet_service.cc:1467] Processing CreateTablet for tablet 2d504dad95654ef9ae3f1543c2dfaf26 (DEFAULT_TABLE table=test-workload [id=ab9b58b93b2f4e6fa89ead6a368228b9]), partition=RANGE (key) PARTITION 536870911 <= VALUES < 1073741822
I20240503 23:24:50.966997 765 tablet_service.cc:1467] Processing CreateTablet for tablet b0e54b5c573b48c8bf2a6fa8841330eb (DEFAULT_TABLE table=test-workload [id=ab9b58b93b2f4e6fa89ead6a368228b9]), partition=RANGE (key) PARTITION 1073741822 <= VALUES < 1610612733
I20240503 23:24:50.967607 675 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2d504dad95654ef9ae3f1543c2dfaf26. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 23:24:50.968987 911 tablet_bootstrap.cc:492] T f16a349decb2402abf31109ee05bdb50 P fa5c2ac5b68e47649e922a2176ccfd28: Bootstrap starting.
I20240503 23:24:50.969125 765 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b0e54b5c573b48c8bf2a6fa8841330eb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 23:24:50.969350 674 tablet_service.cc:1467] Processing CreateTablet for tablet 8f7f130c19b149f5b4fb1350c2ff9e6e (DEFAULT_TABLE table=test-workload [id=ab9b58b93b2f4e6fa89ead6a368228b9]), partition=RANGE (key) PARTITION 1610612733 <= VALUES
I20240503 23:24:50.969998 674 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8f7f130c19b149f5b4fb1350c2ff9e6e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 23:24:50.971912 911 tablet_bootstrap.cc:654] T f16a349decb2402abf31109ee05bdb50 P fa5c2ac5b68e47649e922a2176ccfd28: Neither blocks nor log segments found. Creating new log.
I20240503 23:24:50.981324 913 tablet_bootstrap.cc:492] T b0e54b5c573b48c8bf2a6fa8841330eb P 06b178e84b2e43c68511466d60b5213f: Bootstrap starting.
I20240503 23:24:50.982363 914 tablet_bootstrap.cc:492] T 2d504dad95654ef9ae3f1543c2dfaf26 P 8181f028a7dd43a1af4ff6ebd6011af8: Bootstrap starting.
I20240503 23:24:50.983165 911 tablet_bootstrap.cc:492] T f16a349decb2402abf31109ee05bdb50 P fa5c2ac5b68e47649e922a2176ccfd28: No bootstrap required, opened a new log
I20240503 23:24:50.983371 911 ts_tablet_manager.cc:1367] T f16a349decb2402abf31109ee05bdb50 P fa5c2ac5b68e47649e922a2176ccfd28: Time spent bootstrapping tablet: real 0.015s user 0.008s sys 0.004s
I20240503 23:24:50.984529 911 raft_consensus.cc:348] T f16a349decb2402abf31109ee05bdb50 P fa5c2ac5b68e47649e922a2176ccfd28 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fa5c2ac5b68e47649e922a2176ccfd28" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 36505 } }
I20240503 23:24:50.984817 911 raft_consensus.cc:374] T f16a349decb2402abf31109ee05bdb50 P fa5c2ac5b68e47649e922a2176ccfd28 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 23:24:50.984990 911 raft_consensus.cc:729] T f16a349decb2402abf31109ee05bdb50 P fa5c2ac5b68e47649e922a2176ccfd28 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fa5c2ac5b68e47649e922a2176ccfd28, State: Initialized, Role: FOLLOWER
I20240503 23:24:50.985049 914 tablet_bootstrap.cc:654] T 2d504dad95654ef9ae3f1543c2dfaf26 P 8181f028a7dd43a1af4ff6ebd6011af8: Neither blocks nor log segments found. Creating new log.
I20240503 23:24:50.985402 911 consensus_queue.cc:260] T f16a349decb2402abf31109ee05bdb50 P fa5c2ac5b68e47649e922a2176ccfd28 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fa5c2ac5b68e47649e922a2176ccfd28" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 36505 } }
I20240503 23:24:50.985569 913 tablet_bootstrap.cc:654] T b0e54b5c573b48c8bf2a6fa8841330eb P 06b178e84b2e43c68511466d60b5213f: Neither blocks nor log segments found. Creating new log.
I20240503 23:24:50.985687 911 raft_consensus.cc:388] T f16a349decb2402abf31109ee05bdb50 P fa5c2ac5b68e47649e922a2176ccfd28 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 23:24:50.985829 911 raft_consensus.cc:482] T f16a349decb2402abf31109ee05bdb50 P fa5c2ac5b68e47649e922a2176ccfd28 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 23:24:50.985996 911 raft_consensus.cc:3037] T f16a349decb2402abf31109ee05bdb50 P fa5c2ac5b68e47649e922a2176ccfd28 [term 0 FOLLOWER]: Advancing to term 1
I20240503 23:24:50.987129 711 heartbeater.cc:503] Master 127.0.100.62:46349 was elected leader, sending a full tablet report...
I20240503 23:24:50.991151 911 raft_consensus.cc:504] T f16a349decb2402abf31109ee05bdb50 P fa5c2ac5b68e47649e922a2176ccfd28 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fa5c2ac5b68e47649e922a2176ccfd28" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 36505 } }
I20240503 23:24:50.991636 911 leader_election.cc:302] T f16a349decb2402abf31109ee05bdb50 P fa5c2ac5b68e47649e922a2176ccfd28 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fa5c2ac5b68e47649e922a2176ccfd28; no voters:
I20240503 23:24:50.992385 911 leader_election.cc:288] T f16a349decb2402abf31109ee05bdb50 P fa5c2ac5b68e47649e922a2176ccfd28 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240503 23:24:50.993219 917 raft_consensus.cc:2781] T f16a349decb2402abf31109ee05bdb50 P fa5c2ac5b68e47649e922a2176ccfd28 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 23:24:50.995018 914 tablet_bootstrap.cc:492] T 2d504dad95654ef9ae3f1543c2dfaf26 P 8181f028a7dd43a1af4ff6ebd6011af8: No bootstrap required, opened a new log
I20240503 23:24:50.995399 914 ts_tablet_manager.cc:1367] T 2d504dad95654ef9ae3f1543c2dfaf26 P 8181f028a7dd43a1af4ff6ebd6011af8: Time spent bootstrapping tablet: real 0.013s user 0.004s sys 0.008s
I20240503 23:24:50.995498 913 tablet_bootstrap.cc:492] T b0e54b5c573b48c8bf2a6fa8841330eb P 06b178e84b2e43c68511466d60b5213f: No bootstrap required, opened a new log
I20240503 23:24:50.995805 913 ts_tablet_manager.cc:1367] T b0e54b5c573b48c8bf2a6fa8841330eb P 06b178e84b2e43c68511466d60b5213f: Time spent bootstrapping tablet: real 0.015s user 0.008s sys 0.000s
I20240503 23:24:50.997197 913 raft_consensus.cc:348] T b0e54b5c573b48c8bf2a6fa8841330eb P 06b178e84b2e43c68511466d60b5213f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "06b178e84b2e43c68511466d60b5213f" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 40145 } }
I20240503 23:24:50.997211 914 raft_consensus.cc:348] T 2d504dad95654ef9ae3f1543c2dfaf26 P 8181f028a7dd43a1af4ff6ebd6011af8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8181f028a7dd43a1af4ff6ebd6011af8" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 34815 } }
I20240503 23:24:50.997586 913 raft_consensus.cc:374] T b0e54b5c573b48c8bf2a6fa8841330eb P 06b178e84b2e43c68511466d60b5213f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 23:24:50.997776 914 raft_consensus.cc:374] T 2d504dad95654ef9ae3f1543c2dfaf26 P 8181f028a7dd43a1af4ff6ebd6011af8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 23:24:50.997810 913 raft_consensus.cc:729] T b0e54b5c573b48c8bf2a6fa8841330eb P 06b178e84b2e43c68511466d60b5213f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 06b178e84b2e43c68511466d60b5213f, State: Initialized, Role: FOLLOWER
I20240503 23:24:50.998030 914 raft_consensus.cc:729] T 2d504dad95654ef9ae3f1543c2dfaf26 P 8181f028a7dd43a1af4ff6ebd6011af8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8181f028a7dd43a1af4ff6ebd6011af8, State: Initialized, Role: FOLLOWER
I20240503 23:24:50.998245 913 consensus_queue.cc:260] T b0e54b5c573b48c8bf2a6fa8841330eb P 06b178e84b2e43c68511466d60b5213f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "06b178e84b2e43c68511466d60b5213f" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 40145 } }
I20240503 23:24:50.998632 913 raft_consensus.cc:388] T b0e54b5c573b48c8bf2a6fa8841330eb P 06b178e84b2e43c68511466d60b5213f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 23:24:50.998512 914 consensus_queue.cc:260] T 2d504dad95654ef9ae3f1543c2dfaf26 P 8181f028a7dd43a1af4ff6ebd6011af8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8181f028a7dd43a1af4ff6ebd6011af8" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 34815 } }
I20240503 23:24:50.998843 913 raft_consensus.cc:482] T b0e54b5c573b48c8bf2a6fa8841330eb P 06b178e84b2e43c68511466d60b5213f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 23:24:50.999069 913 raft_consensus.cc:3037] T b0e54b5c573b48c8bf2a6fa8841330eb P 06b178e84b2e43c68511466d60b5213f [term 0 FOLLOWER]: Advancing to term 1
I20240503 23:24:50.999118 914 raft_consensus.cc:388] T 2d504dad95654ef9ae3f1543c2dfaf26 P 8181f028a7dd43a1af4ff6ebd6011af8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 23:24:50.999352 914 raft_consensus.cc:482] T 2d504dad95654ef9ae3f1543c2dfaf26 P 8181f028a7dd43a1af4ff6ebd6011af8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 23:24:50.999562 914 raft_consensus.cc:3037] T 2d504dad95654ef9ae3f1543c2dfaf26 P 8181f028a7dd43a1af4ff6ebd6011af8 [term 0 FOLLOWER]: Advancing to term 1
I20240503 23:24:51.000804 917 raft_consensus.cc:686] T f16a349decb2402abf31109ee05bdb50 P fa5c2ac5b68e47649e922a2176ccfd28 [term 1 LEADER]: Becoming Leader. State: Replica: fa5c2ac5b68e47649e922a2176ccfd28, State: Running, Role: LEADER
I20240503 23:24:51.002346 917 consensus_queue.cc:237] T f16a349decb2402abf31109ee05bdb50 P fa5c2ac5b68e47649e922a2176ccfd28 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fa5c2ac5b68e47649e922a2176ccfd28" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 36505 } }
I20240503 23:24:51.003962 911 ts_tablet_manager.cc:1398] T f16a349decb2402abf31109ee05bdb50 P fa5c2ac5b68e47649e922a2176ccfd28: Time spent starting tablet: real 0.020s user 0.015s sys 0.005s
I20240503 23:24:51.004069 913 raft_consensus.cc:504] T b0e54b5c573b48c8bf2a6fa8841330eb P 06b178e84b2e43c68511466d60b5213f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "06b178e84b2e43c68511466d60b5213f" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 40145 } }
I20240503 23:24:51.004317 913 leader_election.cc:302] T b0e54b5c573b48c8bf2a6fa8841330eb P 06b178e84b2e43c68511466d60b5213f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 06b178e84b2e43c68511466d60b5213f; no voters:
I20240503 23:24:51.004928 913 leader_election.cc:288] T b0e54b5c573b48c8bf2a6fa8841330eb P 06b178e84b2e43c68511466d60b5213f [CANDIDATE]: Term 1 election: Requested vote from peers
I20240503 23:24:51.005167 920 raft_consensus.cc:2781] T b0e54b5c573b48c8bf2a6fa8841330eb P 06b178e84b2e43c68511466d60b5213f [term 1 FOLLOWER]: Leader election won for term 1
I20240503 23:24:51.005336 914 raft_consensus.cc:504] T 2d504dad95654ef9ae3f1543c2dfaf26 P 8181f028a7dd43a1af4ff6ebd6011af8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8181f028a7dd43a1af4ff6ebd6011af8" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 34815 } }
I20240503 23:24:51.005614 914 leader_election.cc:302] T 2d504dad95654ef9ae3f1543c2dfaf26 P 8181f028a7dd43a1af4ff6ebd6011af8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8181f028a7dd43a1af4ff6ebd6011af8; no voters:
I20240503 23:24:51.007071 924 raft_consensus.cc:2781] T 2d504dad95654ef9ae3f1543c2dfaf26 P 8181f028a7dd43a1af4ff6ebd6011af8 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 23:24:51.007696 914 leader_election.cc:288] T 2d504dad95654ef9ae3f1543c2dfaf26 P 8181f028a7dd43a1af4ff6ebd6011af8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240503 23:24:51.009025 914 ts_tablet_manager.cc:1398] T 2d504dad95654ef9ae3f1543c2dfaf26 P 8181f028a7dd43a1af4ff6ebd6011af8: Time spent starting tablet: real 0.013s user 0.007s sys 0.003s
I20240503 23:24:51.010188 914 tablet_bootstrap.cc:492] T 8f7f130c19b149f5b4fb1350c2ff9e6e P 8181f028a7dd43a1af4ff6ebd6011af8: Bootstrap starting.
I20240503 23:24:51.013386 914 tablet_bootstrap.cc:654] T 8f7f130c19b149f5b4fb1350c2ff9e6e P 8181f028a7dd43a1af4ff6ebd6011af8: Neither blocks nor log segments found. Creating new log.
I20240503 23:24:51.015722 924 raft_consensus.cc:686] T 2d504dad95654ef9ae3f1543c2dfaf26 P 8181f028a7dd43a1af4ff6ebd6011af8 [term 1 LEADER]: Becoming Leader. State: Replica: 8181f028a7dd43a1af4ff6ebd6011af8, State: Running, Role: LEADER
I20240503 23:24:51.016364 924 consensus_queue.cc:237] T 2d504dad95654ef9ae3f1543c2dfaf26 P 8181f028a7dd43a1af4ff6ebd6011af8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8181f028a7dd43a1af4ff6ebd6011af8" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 34815 } }
I20240503 23:24:51.019474 914 tablet_bootstrap.cc:492] T 8f7f130c19b149f5b4fb1350c2ff9e6e P 8181f028a7dd43a1af4ff6ebd6011af8: No bootstrap required, opened a new log
I20240503 23:24:51.019753 914 ts_tablet_manager.cc:1367] T 8f7f130c19b149f5b4fb1350c2ff9e6e P 8181f028a7dd43a1af4ff6ebd6011af8: Time spent bootstrapping tablet: real 0.010s user 0.006s sys 0.000s
I20240503 23:24:51.024489 914 raft_consensus.cc:348] T 8f7f130c19b149f5b4fb1350c2ff9e6e P 8181f028a7dd43a1af4ff6ebd6011af8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8181f028a7dd43a1af4ff6ebd6011af8" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 34815 } }
I20240503 23:24:51.026355 913 ts_tablet_manager.cc:1398] T b0e54b5c573b48c8bf2a6fa8841330eb P 06b178e84b2e43c68511466d60b5213f: Time spent starting tablet: real 0.027s user 0.010s sys 0.015s
I20240503 23:24:51.028256 914 raft_consensus.cc:374] T 8f7f130c19b149f5b4fb1350c2ff9e6e P 8181f028a7dd43a1af4ff6ebd6011af8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 23:24:51.028621 914 raft_consensus.cc:729] T 8f7f130c19b149f5b4fb1350c2ff9e6e P 8181f028a7dd43a1af4ff6ebd6011af8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8181f028a7dd43a1af4ff6ebd6011af8, State: Initialized, Role: FOLLOWER
I20240503 23:24:51.027781 430 catalog_manager.cc:5484] T f16a349decb2402abf31109ee05bdb50 P fa5c2ac5b68e47649e922a2176ccfd28 reported cstate change: term changed from 0 to 1, leader changed from <none> to fa5c2ac5b68e47649e922a2176ccfd28 (127.0.100.3). New cstate: current_term: 1 leader_uuid: "fa5c2ac5b68e47649e922a2176ccfd28" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fa5c2ac5b68e47649e922a2176ccfd28" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 36505 } health_report { overall_health: HEALTHY } } }
I20240503 23:24:51.032944 920 raft_consensus.cc:686] T b0e54b5c573b48c8bf2a6fa8841330eb P 06b178e84b2e43c68511466d60b5213f [term 1 LEADER]: Becoming Leader. State: Replica: 06b178e84b2e43c68511466d60b5213f, State: Running, Role: LEADER
I20240503 23:24:51.032577 914 consensus_queue.cc:260] T 8f7f130c19b149f5b4fb1350c2ff9e6e P 8181f028a7dd43a1af4ff6ebd6011af8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8181f028a7dd43a1af4ff6ebd6011af8" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 34815 } }
I20240503 23:24:51.033464 914 raft_consensus.cc:388] T 8f7f130c19b149f5b4fb1350c2ff9e6e P 8181f028a7dd43a1af4ff6ebd6011af8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 23:24:51.033635 914 raft_consensus.cc:482] T 8f7f130c19b149f5b4fb1350c2ff9e6e P 8181f028a7dd43a1af4ff6ebd6011af8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 23:24:51.033792 914 raft_consensus.cc:3037] T 8f7f130c19b149f5b4fb1350c2ff9e6e P 8181f028a7dd43a1af4ff6ebd6011af8 [term 0 FOLLOWER]: Advancing to term 1
I20240503 23:24:51.036871 920 consensus_queue.cc:237] T b0e54b5c573b48c8bf2a6fa8841330eb P 06b178e84b2e43c68511466d60b5213f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "06b178e84b2e43c68511466d60b5213f" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 40145 } }
I20240503 23:24:51.038412 914 raft_consensus.cc:504] T 8f7f130c19b149f5b4fb1350c2ff9e6e P 8181f028a7dd43a1af4ff6ebd6011af8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8181f028a7dd43a1af4ff6ebd6011af8" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 34815 } }
I20240503 23:24:51.038844 914 leader_election.cc:302] T 8f7f130c19b149f5b4fb1350c2ff9e6e P 8181f028a7dd43a1af4ff6ebd6011af8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8181f028a7dd43a1af4ff6ebd6011af8; no voters:
I20240503 23:24:51.039147 914 leader_election.cc:288] T 8f7f130c19b149f5b4fb1350c2ff9e6e P 8181f028a7dd43a1af4ff6ebd6011af8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240503 23:24:51.039820 914 ts_tablet_manager.cc:1398] T 8f7f130c19b149f5b4fb1350c2ff9e6e P 8181f028a7dd43a1af4ff6ebd6011af8: Time spent starting tablet: real 0.020s user 0.002s sys 0.005s
I20240503 23:24:51.041350 925 raft_consensus.cc:2781] T 8f7f130c19b149f5b4fb1350c2ff9e6e P 8181f028a7dd43a1af4ff6ebd6011af8 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 23:24:51.045315 429 catalog_manager.cc:5484] T 2d504dad95654ef9ae3f1543c2dfaf26 P 8181f028a7dd43a1af4ff6ebd6011af8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8181f028a7dd43a1af4ff6ebd6011af8 (127.0.100.1). New cstate: current_term: 1 leader_uuid: "8181f028a7dd43a1af4ff6ebd6011af8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8181f028a7dd43a1af4ff6ebd6011af8" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 34815 } health_report { overall_health: HEALTHY } } }
I20240503 23:24:51.046963 925 raft_consensus.cc:686] T 8f7f130c19b149f5b4fb1350c2ff9e6e P 8181f028a7dd43a1af4ff6ebd6011af8 [term 1 LEADER]: Becoming Leader. State: Replica: 8181f028a7dd43a1af4ff6ebd6011af8, State: Running, Role: LEADER
I20240503 23:24:51.047551 925 consensus_queue.cc:237] T 8f7f130c19b149f5b4fb1350c2ff9e6e P 8181f028a7dd43a1af4ff6ebd6011af8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8181f028a7dd43a1af4ff6ebd6011af8" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 34815 } }
I20240503 23:24:51.057369 430 catalog_manager.cc:5484] T b0e54b5c573b48c8bf2a6fa8841330eb P 06b178e84b2e43c68511466d60b5213f reported cstate change: term changed from 0 to 1, leader changed from <none> to 06b178e84b2e43c68511466d60b5213f (127.0.100.2). New cstate: current_term: 1 leader_uuid: "06b178e84b2e43c68511466d60b5213f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "06b178e84b2e43c68511466d60b5213f" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 40145 } health_report { overall_health: HEALTHY } } }
I20240503 23:24:51.060750 429 catalog_manager.cc:5484] T 8f7f130c19b149f5b4fb1350c2ff9e6e P 8181f028a7dd43a1af4ff6ebd6011af8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8181f028a7dd43a1af4ff6ebd6011af8 (127.0.100.1). New cstate: current_term: 1 leader_uuid: "8181f028a7dd43a1af4ff6ebd6011af8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8181f028a7dd43a1af4ff6ebd6011af8" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 34815 } health_report { overall_health: HEALTHY } } }
I20240503 23:24:51.075150 400 tablet_server.cc:178] TabletServer@127.0.100.1:0 shutting down...
I20240503 23:24:51.108011 400 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240503 23:24:51.108580 400 tablet_replica.cc:330] T 8f7f130c19b149f5b4fb1350c2ff9e6e P 8181f028a7dd43a1af4ff6ebd6011af8: stopping tablet replica
I20240503 23:24:51.109246 400 raft_consensus.cc:2229] T 8f7f130c19b149f5b4fb1350c2ff9e6e P 8181f028a7dd43a1af4ff6ebd6011af8 [term 1 LEADER]: Raft consensus shutting down.
I20240503 23:24:51.109660 400 raft_consensus.cc:2258] T 8f7f130c19b149f5b4fb1350c2ff9e6e P 8181f028a7dd43a1af4ff6ebd6011af8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 23:24:51.112090 400 tablet_replica.cc:330] T 2d504dad95654ef9ae3f1543c2dfaf26 P 8181f028a7dd43a1af4ff6ebd6011af8: stopping tablet replica
I20240503 23:24:51.112303 400 raft_consensus.cc:2229] T 2d504dad95654ef9ae3f1543c2dfaf26 P 8181f028a7dd43a1af4ff6ebd6011af8 [term 1 LEADER]: Raft consensus shutting down.
I20240503 23:24:51.112504 400 raft_consensus.cc:2258] T 2d504dad95654ef9ae3f1543c2dfaf26 P 8181f028a7dd43a1af4ff6ebd6011af8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 23:24:51.129899 400 tablet_server.cc:194] TabletServer@127.0.100.1:0 shutdown complete.
I20240503 23:24:51.137075 400 tablet_server.cc:178] TabletServer@127.0.100.2:0 shutting down...
I20240503 23:24:51.146354 400 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240503 23:24:51.146857 400 tablet_replica.cc:330] T b0e54b5c573b48c8bf2a6fa8841330eb P 06b178e84b2e43c68511466d60b5213f: stopping tablet replica
I20240503 23:24:51.147081 400 raft_consensus.cc:2229] T b0e54b5c573b48c8bf2a6fa8841330eb P 06b178e84b2e43c68511466d60b5213f [term 1 LEADER]: Raft consensus shutting down.
I20240503 23:24:51.147322 400 raft_consensus.cc:2258] T b0e54b5c573b48c8bf2a6fa8841330eb P 06b178e84b2e43c68511466d60b5213f [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 23:24:51.162412 400 tablet_server.cc:194] TabletServer@127.0.100.2:0 shutdown complete.
I20240503 23:24:51.165935 400 tablet_server.cc:178] TabletServer@127.0.100.3:0 shutting down...
I20240503 23:24:51.175335 400 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240503 23:24:51.175822 400 tablet_replica.cc:330] T f16a349decb2402abf31109ee05bdb50 P fa5c2ac5b68e47649e922a2176ccfd28: stopping tablet replica
I20240503 23:24:51.176043 400 raft_consensus.cc:2229] T f16a349decb2402abf31109ee05bdb50 P fa5c2ac5b68e47649e922a2176ccfd28 [term 1 LEADER]: Raft consensus shutting down.
I20240503 23:24:51.176278 400 raft_consensus.cc:2258] T f16a349decb2402abf31109ee05bdb50 P fa5c2ac5b68e47649e922a2176ccfd28 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 23:24:51.191148 400 tablet_server.cc:194] TabletServer@127.0.100.3:0 shutdown complete.
I20240503 23:24:51.194926 400 master.cc:537] Master@127.0.100.62:46349 shutting down...
I20240503 23:24:51.204910 400 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 7a4cf86e12b74089b8fbea8b8a032a01 [term 1 LEADER]: Raft consensus shutting down.
I20240503 23:24:51.205540 400 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 7a4cf86e12b74089b8fbea8b8a032a01 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 23:24:51.205732 400 tablet_replica.cc:330] T 00000000000000000000000000000000 P 7a4cf86e12b74089b8fbea8b8a032a01: stopping tablet replica
I20240503 23:24:51.231333 400 master.cc:552] Master@127.0.100.62:46349 shutdown complete.
I20240503 23:24:51.237604 400 master.cc:537] Master@127.0.100.61:33979 shutting down...
I20240503 23:24:51.245815 400 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 01d3e7fb2d0344a885db17c0fe4b84a6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 23:24:51.246204 400 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 01d3e7fb2d0344a885db17c0fe4b84a6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 23:24:51.246340 400 tablet_replica.cc:330] T 00000000000000000000000000000000 P 01d3e7fb2d0344a885db17c0fe4b84a6: stopping tablet replica
I20240503 23:24:51.261334 400 master.cc:552] Master@127.0.100.61:33979 shutdown complete.
I20240503 23:24:51.266009 400 master.cc:537] Master@127.0.100.60:33981 shutting down...
I20240503 23:24:51.273954 400 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 2bae372273f24a8e98f912be100fdc47 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 23:24:51.274294 400 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 2bae372273f24a8e98f912be100fdc47 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 23:24:51.274437 400 tablet_replica.cc:330] T 00000000000000000000000000000000 P 2bae372273f24a8e98f912be100fdc47: stopping tablet replica
I20240503 23:24:51.289289 400 master.cc:552] Master@127.0.100.60:33981 shutdown complete.
[ OK ] AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing (2764 ms)
[ RUN ] AutoRebalancerTest.AutoRebalancingTurnOffAndOn
I20240503 23:24:51.304267 400 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.0.100.62:35319
I20240503 23:24:51.304972 400 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 23:24:51.308696 938 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 23:24:51.309451 939 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 23:24:51.310217 941 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 23:24:51.310962 400 server_base.cc:1034] running on GCE node
I20240503 23:24:51.311287 400 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240503 23:24:51.311374 400 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240503 23:24:51.311431 400 hybrid_clock.cc:648] HybridClock initialized: now 1714778691311411 us; error 0 us; skew 500 ppm
I20240503 23:24:51.311653 400 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 23:24:51.313277 400 webserver.cc:460] Webserver started at http://127.0.100.62:42491/ using document root <none> and password file <none>
I20240503 23:24:51.313565 400 fs_manager.cc:359] Metadata directory not provided
I20240503 23:24:51.313657 400 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 23:24:51.313824 400 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 23:24:51.314621 400 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZhIiPz/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714778688451250-400-0/minicluster-data/master-0-root/instance:
uuid: "46860c6a2576449e82ccd4bae9b82d95"
format_stamp: "Formatted at 2024-05-03 23:24:51 on dist-test-slave-jvbp"
I20240503 23:24:51.317826 400 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.003s
I20240503 23:24:51.319876 946 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 23:24:51.320336 400 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.001s
I20240503 23:24:51.320448 400 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZhIiPz/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714778688451250-400-0/minicluster-data/master-0-root
uuid: "46860c6a2576449e82ccd4bae9b82d95"
format_stamp: "Formatted at 2024-05-03 23:24:51 on dist-test-slave-jvbp"
I20240503 23:24:51.320626 400 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZhIiPz/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714778688451250-400-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskZhIiPz/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714778688451250-400-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskZhIiPz/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714778688451250-400-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 23:24:51.335258 400 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 23:24:51.336011 400 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 23:24:51.352509 400 rpc_server.cc:307] RPC server started. Bound to: 127.0.100.62:35319
I20240503 23:24:51.352598 997 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.100.62:35319 every 8 connection(s)
I20240503 23:24:51.354254 998 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 23:24:51.358934 998 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 46860c6a2576449e82ccd4bae9b82d95: Bootstrap starting.
I20240503 23:24:51.360529 998 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 46860c6a2576449e82ccd4bae9b82d95: Neither blocks nor log segments found. Creating new log.
I20240503 23:24:51.362643 998 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 46860c6a2576449e82ccd4bae9b82d95: No bootstrap required, opened a new log
I20240503 23:24:51.363406 998 raft_consensus.cc:348] T 00000000000000000000000000000000 P 46860c6a2576449e82ccd4bae9b82d95 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "46860c6a2576449e82ccd4bae9b82d95" member_type: VOTER }
I20240503 23:24:51.363574 998 raft_consensus.cc:374] T 00000000000000000000000000000000 P 46860c6a2576449e82ccd4bae9b82d95 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 23:24:51.363679 998 raft_consensus.cc:729] T 00000000000000000000000000000000 P 46860c6a2576449e82ccd4bae9b82d95 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 46860c6a2576449e82ccd4bae9b82d95, State: Initialized, Role: FOLLOWER
I20240503 23:24:51.363915 998 consensus_queue.cc:260] T 00000000000000000000000000000000 P 46860c6a2576449e82ccd4bae9b82d95 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "46860c6a2576449e82ccd4bae9b82d95" member_type: VOTER }
I20240503 23:24:51.364074 998 raft_consensus.cc:388] T 00000000000000000000000000000000 P 46860c6a2576449e82ccd4bae9b82d95 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 23:24:51.364157 998 raft_consensus.cc:482] T 00000000000000000000000000000000 P 46860c6a2576449e82ccd4bae9b82d95 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 23:24:51.364262 998 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 46860c6a2576449e82ccd4bae9b82d95 [term 0 FOLLOWER]: Advancing to term 1
I20240503 23:24:51.366405 998 raft_consensus.cc:504] T 00000000000000000000000000000000 P 46860c6a2576449e82ccd4bae9b82d95 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "46860c6a2576449e82ccd4bae9b82d95" member_type: VOTER }
I20240503 23:24:51.366685 998 leader_election.cc:302] T 00000000000000000000000000000000 P 46860c6a2576449e82ccd4bae9b82d95 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 46860c6a2576449e82ccd4bae9b82d95; no voters:
I20240503 23:24:51.367173 998 leader_election.cc:288] T 00000000000000000000000000000000 P 46860c6a2576449e82ccd4bae9b82d95 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240503 23:24:51.367333 1001 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 46860c6a2576449e82ccd4bae9b82d95 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 23:24:51.367867 1001 raft_consensus.cc:686] T 00000000000000000000000000000000 P 46860c6a2576449e82ccd4bae9b82d95 [term 1 LEADER]: Becoming Leader. State: Replica: 46860c6a2576449e82ccd4bae9b82d95, State: Running, Role: LEADER
I20240503 23:24:51.368202 998 sys_catalog.cc:564] T 00000000000000000000000000000000 P 46860c6a2576449e82ccd4bae9b82d95 [sys.catalog]: configured and running, proceeding with master startup.
I20240503 23:24:51.368173 1001 consensus_queue.cc:237] T 00000000000000000000000000000000 P 46860c6a2576449e82ccd4bae9b82d95 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "46860c6a2576449e82ccd4bae9b82d95" member_type: VOTER }
I20240503 23:24:51.369264 1003 sys_catalog.cc:455] T 00000000000000000000000000000000 P 46860c6a2576449e82ccd4bae9b82d95 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 46860c6a2576449e82ccd4bae9b82d95. Latest consensus state: current_term: 1 leader_uuid: "46860c6a2576449e82ccd4bae9b82d95" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "46860c6a2576449e82ccd4bae9b82d95" member_type: VOTER } }
I20240503 23:24:51.369494 1003 sys_catalog.cc:458] T 00000000000000000000000000000000 P 46860c6a2576449e82ccd4bae9b82d95 [sys.catalog]: This master's current role is: LEADER
I20240503 23:24:51.369395 1002 sys_catalog.cc:455] T 00000000000000000000000000000000 P 46860c6a2576449e82ccd4bae9b82d95 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "46860c6a2576449e82ccd4bae9b82d95" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "46860c6a2576449e82ccd4bae9b82d95" member_type: VOTER } }
I20240503 23:24:51.369602 1002 sys_catalog.cc:458] T 00000000000000000000000000000000 P 46860c6a2576449e82ccd4bae9b82d95 [sys.catalog]: This master's current role is: LEADER
I20240503 23:24:51.370952 1006 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240503 23:24:51.372735 1006 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240503 23:24:51.374104 400 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20240503 23:24:51.376129 1006 catalog_manager.cc:1347] Generated new cluster ID: eaa79dcbb2184191b6f7b05037d0e1cc
I20240503 23:24:51.376217 1006 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240503 23:24:51.396574 1006 catalog_manager.cc:1370] Generated new certificate authority record
I20240503 23:24:51.397691 1006 catalog_manager.cc:1504] Loading token signing keys...
I20240503 23:24:51.415724 1006 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 46860c6a2576449e82ccd4bae9b82d95: Generated new TSK 0
I20240503 23:24:51.416042 1006 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240503 23:24:51.439778 400 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 23:24:51.443907 1019 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 23:24:51.444859 1020 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 23:24:51.445415 1022 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 23:24:51.446343 400 server_base.cc:1034] running on GCE node
I20240503 23:24:51.446660 400 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240503 23:24:51.446736 400 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240503 23:24:51.446794 400 hybrid_clock.cc:648] HybridClock initialized: now 1714778691446760 us; error 0 us; skew 500 ppm
I20240503 23:24:51.446949 400 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 23:24:51.448663 400 webserver.cc:460] Webserver started at http://127.0.100.1:38409/ using document root <none> and password file <none>
I20240503 23:24:51.448936 400 fs_manager.cc:359] Metadata directory not provided
I20240503 23:24:51.449033 400 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 23:24:51.449167 400 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 23:24:51.449968 400 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZhIiPz/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714778688451250-400-0/minicluster-data/ts-0-root/instance:
uuid: "5d37fd41fc2c41a99580cc475915a090"
format_stamp: "Formatted at 2024-05-03 23:24:51 on dist-test-slave-jvbp"
I20240503 23:24:51.452996 400 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20240503 23:24:51.454950 1027 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 23:24:51.455358 400 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.001s
I20240503 23:24:51.455482 400 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZhIiPz/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714778688451250-400-0/minicluster-data/ts-0-root
uuid: "5d37fd41fc2c41a99580cc475915a090"
format_stamp: "Formatted at 2024-05-03 23:24:51 on dist-test-slave-jvbp"
I20240503 23:24:51.455648 400 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZhIiPz/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714778688451250-400-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskZhIiPz/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714778688451250-400-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskZhIiPz/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714778688451250-400-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 23:24:51.473276 400 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 23:24:51.474025 400 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 23:24:51.474730 400 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 23:24:51.475821 400 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 23:24:51.475898 400 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 23:24:51.476023 400 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 23:24:51.476102 400 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 23:24:51.493757 400 rpc_server.cc:307] RPC server started. Bound to: 127.0.100.1:38605
I20240503 23:24:51.493791 1089 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.100.1:38605 every 8 connection(s)
I20240503 23:24:51.495711 400 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 23:24:51.500506 1094 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 23:24:51.501317 1095 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 23:24:51.502414 1097 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 23:24:51.503010 400 server_base.cc:1034] running on GCE node
I20240503 23:24:51.503396 400 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240503 23:24:51.503502 400 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240503 23:24:51.503576 400 hybrid_clock.cc:648] HybridClock initialized: now 1714778691503550 us; error 0 us; skew 500 ppm
I20240503 23:24:51.503787 400 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 23:24:51.504125 1090 heartbeater.cc:347] Connected to a master server at 127.0.100.62:35319
I20240503 23:24:51.504299 1090 heartbeater.cc:464] Registering TS with master...
I20240503 23:24:51.504942 1090 heartbeater.cc:511] Master 127.0.100.62:35319 requested a full tablet report, sending...
I20240503 23:24:51.505510 400 webserver.cc:460] Webserver started at http://127.0.100.2:33427/ using document root <none> and password file <none>
I20240503 23:24:51.505846 400 fs_manager.cc:359] Metadata directory not provided
I20240503 23:24:51.505964 400 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 23:24:51.506175 400 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 23:24:51.506189 963 ts_manager.cc:194] Registered new tserver with Master: 5d37fd41fc2c41a99580cc475915a090 (127.0.100.1:38605)
I20240503 23:24:51.507176 400 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZhIiPz/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714778688451250-400-0/minicluster-data/ts-1-root/instance:
uuid: "74775765c2ac4db693fa70319ffd98c9"
format_stamp: "Formatted at 2024-05-03 23:24:51 on dist-test-slave-jvbp"
I20240503 23:24:51.508023 963 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:53530
I20240503 23:24:51.510600 400 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.002s
I20240503 23:24:51.512430 1102 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 23:24:51.512897 400 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.001s
I20240503 23:24:51.513025 400 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZhIiPz/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714778688451250-400-0/minicluster-data/ts-1-root
uuid: "74775765c2ac4db693fa70319ffd98c9"
format_stamp: "Formatted at 2024-05-03 23:24:51 on dist-test-slave-jvbp"
I20240503 23:24:51.513173 400 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZhIiPz/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714778688451250-400-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskZhIiPz/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714778688451250-400-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskZhIiPz/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714778688451250-400-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 23:24:51.544023 400 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 23:24:51.544842 400 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 23:24:51.545606 400 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 23:24:51.546633 400 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 23:24:51.546728 400 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 23:24:51.546834 400 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 23:24:51.546911 400 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 23:24:51.566083 400 rpc_server.cc:307] RPC server started. Bound to: 127.0.100.2:38529
I20240503 23:24:51.566123 1164 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.100.2:38529 every 8 connection(s)
I20240503 23:24:51.568084 400 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 23:24:51.572690 1168 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 23:24:51.573482 1169 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 23:24:51.576587 1165 heartbeater.cc:347] Connected to a master server at 127.0.100.62:35319
I20240503 23:24:51.576781 1165 heartbeater.cc:464] Registering TS with master...
I20240503 23:24:51.577381 1165 heartbeater.cc:511] Master 127.0.100.62:35319 requested a full tablet report, sending...
I20240503 23:24:51.578518 963 ts_manager.cc:194] Registered new tserver with Master: 74775765c2ac4db693fa70319ffd98c9 (127.0.100.2:38529)
I20240503 23:24:51.579352 400 server_base.cc:1034] running on GCE node
I20240503 23:24:51.579430 963 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:53540
W20240503 23:24:51.580117 1172 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 23:24:51.581442 400 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240503 23:24:51.581516 400 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240503 23:24:51.581580 400 hybrid_clock.cc:648] HybridClock initialized: now 1714778691581543 us; error 0 us; skew 500 ppm
I20240503 23:24:51.581738 400 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 23:24:51.583436 400 webserver.cc:460] Webserver started at http://127.0.100.3:42229/ using document root <none> and password file <none>
I20240503 23:24:51.583712 400 fs_manager.cc:359] Metadata directory not provided
I20240503 23:24:51.583819 400 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 23:24:51.583956 400 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 23:24:51.584775 400 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZhIiPz/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714778688451250-400-0/minicluster-data/ts-2-root/instance:
uuid: "bf5fe0b6e639437498453c116a26f95c"
format_stamp: "Formatted at 2024-05-03 23:24:51 on dist-test-slave-jvbp"
I20240503 23:24:51.587697 400 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20240503 23:24:51.589637 1176 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 23:24:51.590009 400 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20240503 23:24:51.590145 400 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZhIiPz/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714778688451250-400-0/minicluster-data/ts-2-root
uuid: "bf5fe0b6e639437498453c116a26f95c"
format_stamp: "Formatted at 2024-05-03 23:24:51 on dist-test-slave-jvbp"
I20240503 23:24:51.590297 400 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZhIiPz/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714778688451250-400-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-taskZhIiPz/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714778688451250-400-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-taskZhIiPz/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714778688451250-400-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 23:24:51.609309 400 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 23:24:51.610108 400 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 23:24:51.611685 400 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 23:24:51.612787 400 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 23:24:51.612869 400 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 23:24:51.612996 400 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 23:24:51.613062 400 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 23:24:51.631248 400 rpc_server.cc:307] RPC server started. Bound to: 127.0.100.3:40377
I20240503 23:24:51.631268 1238 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.100.3:40377 every 8 connection(s)
I20240503 23:24:51.638985 1239 heartbeater.cc:347] Connected to a master server at 127.0.100.62:35319
I20240503 23:24:51.639166 1239 heartbeater.cc:464] Registering TS with master...
I20240503 23:24:51.639660 1239 heartbeater.cc:511] Master 127.0.100.62:35319 requested a full tablet report, sending...
I20240503 23:24:51.640756 963 ts_manager.cc:194] Registered new tserver with Master: bf5fe0b6e639437498453c116a26f95c (127.0.100.3:40377)
I20240503 23:24:51.641510 400 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.009158394s
I20240503 23:24:51.641644 963 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:53546
I20240503 23:24:52.509874 1090 heartbeater.cc:503] Master 127.0.100.62:35319 was elected leader, sending a full tablet report...
I20240503 23:24:52.580950 1165 heartbeater.cc:503] Master 127.0.100.62:35319 was elected leader, sending a full tablet report...
I20240503 23:24:52.643266 1239 heartbeater.cc:503] Master 127.0.100.62:35319 was elected leader, sending a full tablet report...
I20240503 23:24:52.670742 400 test_util.cc:268] Using random seed: 114840805
I20240503 23:24:52.685396 963 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:53558:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "\004\001\000\377\377\377\017\004\001\000\376\377\377\037\004\001\000\375\377\377/\004\001\000\374\377\377?\004\001\000\373\377\377O\004\001\000\372\377\377_\004\001\000\371\377\377o""\004\001\000\377\377\377\017\004\001\000\376\377\377\037\004\001\000\375\377\377/\004\001\000\374\377\377?\004\001\000\373\377\377O\004\001\000\372\377\377_\004\001\000\371\377\377o"
indirect_data: """"
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 23:24:52.686362 963 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-workload in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 23:24:52.718196 1054 tablet_service.cc:1467] Processing CreateTablet for tablet 966a756b35654ba0a0e272a0b6c7d7f6 (DEFAULT_TABLE table=test-workload [id=73a6a60a09904413a4b27971ac6ce555]), partition=RANGE (key) PARTITION 268435455 <= VALUES < 536870910
I20240503 23:24:52.719584 1055 tablet_service.cc:1467] Processing CreateTablet for tablet f3d383cdf48b4e108844ff8a658b9a51 (DEFAULT_TABLE table=test-workload [id=73a6a60a09904413a4b27971ac6ce555]), partition=RANGE (key) PARTITION VALUES < 268435455
I20240503 23:24:52.719895 1054 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 966a756b35654ba0a0e272a0b6c7d7f6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 23:24:52.720335 1053 tablet_service.cc:1467] Processing CreateTablet for tablet 8d775506b875408d80a957dd3926aed8 (DEFAULT_TABLE table=test-workload [id=73a6a60a09904413a4b27971ac6ce555]), partition=RANGE (key) PARTITION 536870910 <= VALUES < 805306365
I20240503 23:24:52.720824 1053 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8d775506b875408d80a957dd3926aed8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 23:24:52.725229 1050 tablet_service.cc:1467] Processing CreateTablet for tablet 26f3c79af13e40878328e02ae549ab0b (DEFAULT_TABLE table=test-workload [id=73a6a60a09904413a4b27971ac6ce555]), partition=RANGE (key) PARTITION 1342177275 <= VALUES < 1610612730
I20240503 23:24:52.725910 1050 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 26f3c79af13e40878328e02ae549ab0b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 23:24:52.726330 1048 tablet_service.cc:1467] Processing CreateTablet for tablet cb8edce628bd466da6c9eaabf7964493 (DEFAULT_TABLE table=test-workload [id=73a6a60a09904413a4b27971ac6ce555]), partition=RANGE (key) PARTITION 1879048185 <= VALUES
I20240503 23:24:52.726827 1048 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cb8edce628bd466da6c9eaabf7964493. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 23:24:52.725925 1049 tablet_service.cc:1467] Processing CreateTablet for tablet 1e6b13b222534155bc03878359c2eb3b (DEFAULT_TABLE table=test-workload [id=73a6a60a09904413a4b27971ac6ce555]), partition=RANGE (key) PARTITION 1610612730 <= VALUES < 1879048185
I20240503 23:24:52.729456 1049 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1e6b13b222534155bc03878359c2eb3b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 23:24:52.725195 1051 tablet_service.cc:1467] Processing CreateTablet for tablet 6743c625d6854920b88764dec9ebd250 (DEFAULT_TABLE table=test-workload [id=73a6a60a09904413a4b27971ac6ce555]), partition=RANGE (key) PARTITION 1073741820 <= VALUES < 1342177275
I20240503 23:24:52.733004 1051 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6743c625d6854920b88764dec9ebd250. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 23:24:52.736796 1055 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f3d383cdf48b4e108844ff8a658b9a51. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 23:24:52.750519 1052 tablet_service.cc:1467] Processing CreateTablet for tablet 98b4ca061f7f4a43a13fbad78dac228a (DEFAULT_TABLE table=test-workload [id=73a6a60a09904413a4b27971ac6ce555]), partition=RANGE (key) PARTITION 805306365 <= VALUES < 1073741820
I20240503 23:24:52.751555 1052 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 98b4ca061f7f4a43a13fbad78dac228a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 23:24:52.755327 1259 tablet_bootstrap.cc:492] T 966a756b35654ba0a0e272a0b6c7d7f6 P 5d37fd41fc2c41a99580cc475915a090: Bootstrap starting.
I20240503 23:24:52.762236 1124 tablet_service.cc:1467] Processing CreateTablet for tablet 1e6b13b222534155bc03878359c2eb3b (DEFAULT_TABLE table=test-workload [id=73a6a60a09904413a4b27971ac6ce555]), partition=RANGE (key) PARTITION 1610612730 <= VALUES < 1879048185
I20240503 23:24:52.760452 1129 tablet_service.cc:1467] Processing CreateTablet for tablet 966a756b35654ba0a0e272a0b6c7d7f6 (DEFAULT_TABLE table=test-workload [id=73a6a60a09904413a4b27971ac6ce555]), partition=RANGE (key) PARTITION 268435455 <= VALUES < 536870910
I20240503 23:24:52.761981 1127 tablet_service.cc:1467] Processing CreateTablet for tablet 98b4ca061f7f4a43a13fbad78dac228a (DEFAULT_TABLE table=test-workload [id=73a6a60a09904413a4b27971ac6ce555]), partition=RANGE (key) PARTITION 805306365 <= VALUES < 1073741820
I20240503 23:24:52.763208 1125 tablet_service.cc:1467] Processing CreateTablet for tablet 26f3c79af13e40878328e02ae549ab0b (DEFAULT_TABLE table=test-workload [id=73a6a60a09904413a4b27971ac6ce555]), partition=RANGE (key) PARTITION 1342177275 <= VALUES < 1610612730
I20240503 23:24:52.763612 1123 tablet_service.cc:1467] Processing CreateTablet for tablet cb8edce628bd466da6c9eaabf7964493 (DEFAULT_TABLE table=test-workload [id=73a6a60a09904413a4b27971ac6ce555]), partition=RANGE (key) PARTITION 1879048185 <= VALUES
I20240503 23:24:52.764257 1124 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1e6b13b222534155bc03878359c2eb3b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 23:24:52.764335 1126 tablet_service.cc:1467] Processing CreateTablet for tablet 6743c625d6854920b88764dec9ebd250 (DEFAULT_TABLE table=test-workload [id=73a6a60a09904413a4b27971ac6ce555]), partition=RANGE (key) PARTITION 1073741820 <= VALUES < 1342177275
I20240503 23:24:52.764462 1125 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 26f3c79af13e40878328e02ae549ab0b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 23:24:52.764151 1128 tablet_service.cc:1467] Processing CreateTablet for tablet 8d775506b875408d80a957dd3926aed8 (DEFAULT_TABLE table=test-workload [id=73a6a60a09904413a4b27971ac6ce555]), partition=RANGE (key) PARTITION 536870910 <= VALUES < 805306365
I20240503 23:24:52.765087 1259 tablet_bootstrap.cc:654] T 966a756b35654ba0a0e272a0b6c7d7f6 P 5d37fd41fc2c41a99580cc475915a090: Neither blocks nor log segments found. Creating new log.
I20240503 23:24:52.765184 1127 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 98b4ca061f7f4a43a13fbad78dac228a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 23:24:52.765586 1130 tablet_service.cc:1467] Processing CreateTablet for tablet f3d383cdf48b4e108844ff8a658b9a51 (DEFAULT_TABLE table=test-workload [id=73a6a60a09904413a4b27971ac6ce555]), partition=RANGE (key) PARTITION VALUES < 268435455
I20240503 23:24:52.765765 1126 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6743c625d6854920b88764dec9ebd250. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 23:24:52.766005 1129 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 966a756b35654ba0a0e272a0b6c7d7f6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 23:24:52.766260 1130 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f3d383cdf48b4e108844ff8a658b9a51. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 23:24:52.766513 1128 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8d775506b875408d80a957dd3926aed8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 23:24:52.767692 1123 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cb8edce628bd466da6c9eaabf7964493. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 23:24:52.777441 1204 tablet_service.cc:1467] Processing CreateTablet for tablet f3d383cdf48b4e108844ff8a658b9a51 (DEFAULT_TABLE table=test-workload [id=73a6a60a09904413a4b27971ac6ce555]), partition=RANGE (key) PARTITION VALUES < 268435455
I20240503 23:24:52.779208 1204 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f3d383cdf48b4e108844ff8a658b9a51. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 23:24:52.782297 1203 tablet_service.cc:1467] Processing CreateTablet for tablet 966a756b35654ba0a0e272a0b6c7d7f6 (DEFAULT_TABLE table=test-workload [id=73a6a60a09904413a4b27971ac6ce555]), partition=RANGE (key) PARTITION 268435455 <= VALUES < 536870910
I20240503 23:24:52.782922 1202 tablet_service.cc:1467] Processing CreateTablet for tablet 8d775506b875408d80a957dd3926aed8 (DEFAULT_TABLE table=test-workload [id=73a6a60a09904413a4b27971ac6ce555]), partition=RANGE (key) PARTITION 536870910 <= VALUES < 805306365
I20240503 23:24:52.784710 1201 tablet_service.cc:1467] Processing CreateTablet for tablet 98b4ca061f7f4a43a13fbad78dac228a (DEFAULT_TABLE table=test-workload [id=73a6a60a09904413a4b27971ac6ce555]), partition=RANGE (key) PARTITION 805306365 <= VALUES < 1073741820
I20240503 23:24:52.788661 1200 tablet_service.cc:1467] Processing CreateTablet for tablet 6743c625d6854920b88764dec9ebd250 (DEFAULT_TABLE table=test-workload [id=73a6a60a09904413a4b27971ac6ce555]), partition=RANGE (key) PARTITION 1073741820 <= VALUES < 1342177275
I20240503 23:24:52.790907 1199 tablet_service.cc:1467] Processing CreateTablet for tablet 26f3c79af13e40878328e02ae549ab0b (DEFAULT_TABLE table=test-workload [id=73a6a60a09904413a4b27971ac6ce555]), partition=RANGE (key) PARTITION 1342177275 <= VALUES < 1610612730
I20240503 23:24:52.791260 1200 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6743c625d6854920b88764dec9ebd250. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 23:24:52.792582 1202 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8d775506b875408d80a957dd3926aed8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 23:24:52.793802 1198 tablet_service.cc:1467] Processing CreateTablet for tablet 1e6b13b222534155bc03878359c2eb3b (DEFAULT_TABLE table=test-workload [id=73a6a60a09904413a4b27971ac6ce555]), partition=RANGE (key) PARTITION 1610612730 <= VALUES < 1879048185
I20240503 23:24:52.795902 1203 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 966a756b35654ba0a0e272a0b6c7d7f6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 23:24:52.794760 1197 tablet_service.cc:1467] Processing CreateTablet for tablet cb8edce628bd466da6c9eaabf7964493 (DEFAULT_TABLE table=test-workload [id=73a6a60a09904413a4b27971ac6ce555]), partition=RANGE (key) PARTITION 1879048185 <= VALUES
I20240503 23:24:52.806789 1201 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 98b4ca061f7f4a43a13fbad78dac228a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 23:24:52.808782 1198 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1e6b13b222534155bc03878359c2eb3b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 23:24:52.811260 1199 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 26f3c79af13e40878328e02ae549ab0b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 23:24:52.812913 1197 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cb8edce628bd466da6c9eaabf7964493. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 23:24:52.822515 1260 tablet_bootstrap.cc:492] T f3d383cdf48b4e108844ff8a658b9a51 P 74775765c2ac4db693fa70319ffd98c9: Bootstrap starting.
I20240503 23:24:52.828667 1259 tablet_bootstrap.cc:492] T 966a756b35654ba0a0e272a0b6c7d7f6 P 5d37fd41fc2c41a99580cc475915a090: No bootstrap required, opened a new log
I20240503 23:24:52.829618 1259 ts_tablet_manager.cc:1367] T 966a756b35654ba0a0e272a0b6c7d7f6 P 5d37fd41fc2c41a99580cc475915a090: Time spent bootstrapping tablet: real 0.074s user 0.016s sys 0.017s
I20240503 23:24:52.838949 1260 tablet_bootstrap.cc:654] T f3d383cdf48b4e108844ff8a658b9a51 P 74775765c2ac4db693fa70319ffd98c9: Neither blocks nor log segments found. Creating new log.
I20240503 23:24:52.841591 1262 tablet_bootstrap.cc:492] T f3d383cdf48b4e108844ff8a658b9a51 P bf5fe0b6e639437498453c116a26f95c: Bootstrap starting.
I20240503 23:24:52.842590 1259 raft_consensus.cc:348] T 966a756b35654ba0a0e272a0b6c7d7f6 P 5d37fd41fc2c41a99580cc475915a090 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d37fd41fc2c41a99580cc475915a090" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 38605 } } peers { permanent_uuid: "74775765c2ac4db693fa70319ffd98c9" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 38529 } } peers { permanent_uuid: "bf5fe0b6e639437498453c116a26f95c" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 40377 } }
I20240503 23:24:52.842939 1259 raft_consensus.cc:374] T 966a756b35654ba0a0e272a0b6c7d7f6 P 5d37fd41fc2c41a99580cc475915a090 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 23:24:52.843079 1259 raft_consensus.cc:729] T 966a756b35654ba0a0e272a0b6c7d7f6 P 5d37fd41fc2c41a99580cc475915a090 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5d37fd41fc2c41a99580cc475915a090, State: Initialized, Role: FOLLOWER
I20240503 23:24:52.843451 1259 consensus_queue.cc:260] T 966a756b35654ba0a0e272a0b6c7d7f6 P 5d37fd41fc2c41a99580cc475915a090 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d37fd41fc2c41a99580cc475915a090" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 38605 } } peers { permanent_uuid: "74775765c2ac4db693fa70319ffd98c9" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 38529 } } peers { permanent_uuid: "bf5fe0b6e639437498453c116a26f95c" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 40377 } }
I20240503 23:24:52.844081 1260 tablet_bootstrap.cc:492] T f3d383cdf48b4e108844ff8a658b9a51 P 74775765c2ac4db693fa70319ffd98c9: No bootstrap required, opened a new log
I20240503 23:24:52.844125 1262 tablet_bootstrap.cc:654] T f3d383cdf48b4e108844ff8a658b9a51 P bf5fe0b6e639437498453c116a26f95c: Neither blocks nor log segments found. Creating new log.
I20240503 23:24:52.844255 1260 ts_tablet_manager.cc:1367] T f3d383cdf48b4e108844ff8a658b9a51 P 74775765c2ac4db693fa70319ffd98c9: Time spent bootstrapping tablet: real 0.028s user 0.008s sys 0.000s
I20240503 23:24:52.845458 1259 ts_tablet_manager.cc:1398] T 966a756b35654ba0a0e272a0b6c7d7f6 P 5d37fd41fc2c41a99580cc475915a090: Time spent starting tablet: real 0.016s user 0.001s sys 0.002s
I20240503 23:24:52.846056 1260 raft_consensus.cc:348] T f3d383cdf48b4e108844ff8a658b9a51 P 74775765c2ac4db693fa70319ffd98c9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d37fd41fc2c41a99580cc475915a090" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 38605 } } peers { permanent_uuid: "bf5fe0b6e639437498453c116a26f95c" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 40377 } } peers { permanent_uuid: "74775765c2ac4db693fa70319ffd98c9" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 38529 } }
I20240503 23:24:52.846243 1260 raft_consensus.cc:374] T f3d383cdf48b4e108844ff8a658b9a51 P 74775765c2ac4db693fa70319ffd98c9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 23:24:52.846346 1260 raft_consensus.cc:729] T f3d383cdf48b4e108844ff8a658b9a51 P 74775765c2ac4db693fa70319ffd98c9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 74775765c2ac4db693fa70319ffd98c9, State: Initialized, Role: FOLLOWER
I20240503 23:24:52.846603 1260 consensus_queue.cc:260] T f3d383cdf48b4e108844ff8a658b9a51 P 74775765c2ac4db693fa70319ffd98c9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d37fd41fc2c41a99580cc475915a090" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 38605 } } peers { permanent_uuid: "bf5fe0b6e639437498453c116a26f95c" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 40377 } } peers { permanent_uuid: "74775765c2ac4db693fa70319ffd98c9" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 38529 } }
I20240503 23:24:52.848281 1259 tablet_bootstrap.cc:492] T 26f3c79af13e40878328e02ae549ab0b P 5d37fd41fc2c41a99580cc475915a090: Bootstrap starting.
I20240503 23:24:52.851042 1259 tablet_bootstrap.cc:654] T 26f3c79af13e40878328e02ae549ab0b P 5d37fd41fc2c41a99580cc475915a090: Neither blocks nor log segments found. Creating new log.
I20240503 23:24:52.854161 1260 ts_tablet_manager.cc:1398] T f3d383cdf48b4e108844ff8a658b9a51 P 74775765c2ac4db693fa70319ffd98c9: Time spent starting tablet: real 0.010s user 0.009s sys 0.000s
I20240503 23:24:52.854516 1260 tablet_bootstrap.cc:492] T 26f3c79af13e40878328e02ae549ab0b P 74775765c2ac4db693fa70319ffd98c9: Bootstrap starting.
I20240503 23:24:52.854542 1262 tablet_bootstrap.cc:492] T f3d383cdf48b4e108844ff8a658b9a51 P bf5fe0b6e639437498453c116a26f95c: No bootstrap required, opened a new log
I20240503 23:24:52.854674 1262 ts_tablet_manager.cc:1367] T f3d383cdf48b4e108844ff8a658b9a51 P bf5fe0b6e639437498453c116a26f95c: Time spent bootstrapping tablet: real 0.013s user 0.010s sys 0.000s
I20240503 23:24:52.855407 1262 raft_consensus.cc:348] T f3d383cdf48b4e108844ff8a658b9a51 P bf5fe0b6e639437498453c116a26f95c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d37fd41fc2c41a99580cc475915a090" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 38605 } } peers { permanent_uuid: "bf5fe0b6e639437498453c116a26f95c" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 40377 } } peers { permanent_uuid: "74775765c2ac4db693fa70319ffd98c9" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 38529 } }
I20240503 23:24:52.855604 1262 raft_consensus.cc:374] T f3d383cdf48b4e108844ff8a658b9a51 P bf5fe0b6e639437498453c116a26f95c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 23:24:52.855675 1262 raft_consensus.cc:729] T f3d383cdf48b4e108844ff8a658b9a51 P bf5fe0b6e639437498453c116a26f95c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bf5fe0b6e639437498453c116a26f95c, State: Initialized, Role: FOLLOWER
I20240503 23:24:52.855931 1262 consensus_queue.cc:260] T f3d383cdf48b4e108844ff8a658b9a51 P bf5fe0b6e639437498453c116a26f95c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d37fd41fc2c41a99580cc475915a090" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 38605 } } peers { permanent_uuid: "bf5fe0b6e639437498453c116a26f95c" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 40377 } } peers { permanent_uuid: "74775765c2ac4db693fa70319ffd98c9" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 38529 } }
I20240503 23:24:52.856732 1260 tablet_bootstrap.cc:654] T 26f3c79af13e40878328e02ae549ab0b P 74775765c2ac4db693fa70319ffd98c9: Neither blocks nor log segments found. Creating new log.
I20240503 23:24:52.857093 1259 tablet_bootstrap.cc:492] T 26f3c79af13e40878328e02ae549ab0b P 5d37fd41fc2c41a99580cc475915a090: No bootstrap required, opened a new log
I20240503 23:24:52.858350 1259 ts_tablet_manager.cc:1367] T 26f3c79af13e40878328e02ae549ab0b P 5d37fd41fc2c41a99580cc475915a090: Time spent bootstrapping tablet: real 0.009s user 0.005s sys 0.000s
I20240503 23:24:52.858434 1262 ts_tablet_manager.cc:1398] T f3d383cdf48b4e108844ff8a658b9a51 P bf5fe0b6e639437498453c116a26f95c: Time spent starting tablet: real 0.004s user 0.003s sys 0.000s
I20240503 23:24:52.858983 1262 tablet_bootstrap.cc:492] T 6743c625d6854920b88764dec9ebd250 P bf5fe0b6e639437498453c116a26f95c: Bootstrap starting.
I20240503 23:24:52.859691 1259 raft_consensus.cc:348] T 26f3c79af13e40878328e02ae549ab0b P 5d37fd41fc2c41a99580cc475915a090 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d37fd41fc2c41a99580cc475915a090" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 38605 } } peers { permanent_uuid: "74775765c2ac4db693fa70319ffd98c9" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 38529 } } peers { permanent_uuid: "bf5fe0b6e639437498453c116a26f95c" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 40377 } }
I20240503 23:24:52.860116 1259 raft_consensus.cc:374] T 26f3c79af13e40878328e02ae549ab0b P 5d37fd41fc2c41a99580cc475915a090 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 23:24:52.860229 1259 raft_consensus.cc:729] T 26f3c79af13e40878328e02ae549ab0b P 5d37fd41fc2c41a99580cc475915a090 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5d37fd41fc2c41a99580cc475915a090, State: Initialized, Role: FOLLOWER
I20240503 23:24:52.860484 1259 consensus_queue.cc:260] T 26f3c79af13e40878328e02ae549ab0b P 5d37fd41fc2c41a99580cc475915a090 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d37fd41fc2c41a99580cc475915a090" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 38605 } } peers { permanent_uuid: "74775765c2ac4db693fa70319ffd98c9" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 38529 } } peers { permanent_uuid: "bf5fe0b6e639437498453c116a26f95c" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 40377 } }
I20240503 23:24:52.862046 1259 ts_tablet_manager.cc:1398] T 26f3c79af13e40878328e02ae549ab0b P 5d37fd41fc2c41a99580cc475915a090: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240503 23:24:52.862375 1259 tablet_bootstrap.cc:492] T 1e6b13b222534155bc03878359c2eb3b P 5d37fd41fc2c41a99580cc475915a090: Bootstrap starting.
I20240503 23:24:52.862780 1262 tablet_bootstrap.cc:654] T 6743c625d6854920b88764dec9ebd250 P bf5fe0b6e639437498453c116a26f95c: Neither blocks nor log segments found. Creating new log.
I20240503 23:24:52.864444 1259 tablet_bootstrap.cc:654] T 1e6b13b222534155bc03878359c2eb3b P 5d37fd41fc2c41a99580cc475915a090: Neither blocks nor log segments found. Creating new log.
I20240503 23:24:52.869338 1260 tablet_bootstrap.cc:492] T 26f3c79af13e40878328e02ae549ab0b P 74775765c2ac4db693fa70319ffd98c9: No bootstrap required, opened a new log
I20240503 23:24:52.869613 1260 ts_tablet_manager.cc:1367] T 26f3c79af13e40878328e02ae549ab0b P 74775765c2ac4db693fa70319ffd98c9: Time spent bootstrapping tablet: real 0.015s user 0.000s sys 0.013s
I20240503 23:24:52.869823 1262 tablet_bootstrap.cc:492] T 6743c625d6854920b88764dec9ebd250 P bf5fe0b6e639437498453c116a26f95c: No bootstrap required, opened a new log
I20240503 23:24:52.870000 1262 ts_tablet_manager.cc:1367] T 6743c625d6854920b88764dec9ebd250 P bf5fe0b6e639437498453c116a26f95c: Time spent bootstrapping tablet: real 0.011s user 0.002s sys 0.006s
I20240503 23:24:52.870291 1259 tablet_bootstrap.cc:492] T 1e6b13b222534155bc03878359c2eb3b P 5d37fd41fc2c41a99580cc475915a090: No bootstrap required, opened a new log
I20240503 23:24:52.870445 1259 ts_tablet_manager.cc:1367] T 1e6b13b222534155bc03878359c2eb3b P 5d37fd41fc2c41a99580cc475915a090: Time spent bootstrapping tablet: real 0.008s user 0.007s sys 0.000s
I20240503 23:24:52.870707 1260 raft_consensus.cc:348] T 26f3c79af13e40878328e02ae549ab0b P 74775765c2ac4db693fa70319ffd98c9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d37fd41fc2c41a99580cc475915a090" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 38605 } } peers { permanent_uuid: "74775765c2ac4db693fa70319ffd98c9" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 38529 } } peers { permanent_uuid: "bf5fe0b6e639437498453c116a26f95c" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 40377 } }
I20240503 23:24:52.870747 1262 raft_consensus.cc:348] T 6743c625d6854920b88764dec9ebd250 P bf5fe0b6e639437498453c116a26f95c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d37fd41fc2c41a99580cc475915a090" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 38605 } } peers { permanent_uuid: "74775765c2ac4db693fa70319ffd98c9" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 38529 } } peers { permanent_uuid: "bf5fe0b6e639437498453c116a26f95c" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 40377 } }
I20240503 23:24:52.871934 1260 raft_consensus.cc:374] T 26f3c79af13e40878328e02ae549ab0b P 74775765c2ac4db693fa70319ffd98c9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 23:24:52.871979 1262 raft_consensus.cc:374] T 6743c625d6854920b88764dec9ebd250 P bf5fe0b6e639437498453c116a26f95c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 23:24:52.872097 1260 raft_consensus.cc:729] T 26f3c79af13e40878328e02ae549ab0b P 74775765c2ac4db693fa70319ffd98c9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 74775765c2ac4db693fa70319ffd98c9, State: Initialized, Role: FOLLOWER
I20240503 23:24:52.872133 1262 raft_consensus.cc:729] T 6743c625d6854920b88764dec9ebd250 P bf5fe0b6e639437498453c116a26f95c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bf5fe0b6e639437498453c116a26f95c, State: Initialized, Role: FOLLOWER
I20240503 23:24:52.872092 1259 raft_consensus.cc:348] T 1e6b13b222534155bc03878359c2eb3b P 5d37fd41fc2c41a99580cc475915a090 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d37fd41fc2c41a99580cc475915a090" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 38605 } } peers { permanent_uuid: "74775765c2ac4db693fa70319ffd98c9" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 38529 } } peers { permanent_uuid: "bf5fe0b6e639437498453c116a26f95c" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 40377 } }
I20240503 23:24:52.872310 1259 raft_consensus.cc:374] T 1e6b13b222534155bc03878359c2eb3b P 5d37fd41fc2c41a99580cc475915a090 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 23:24:52.872416 1259 raft_consensus.cc:729] T 1e6b13b222534155bc03878359c2eb3b P 5d37fd41fc2c41a99580cc475915a090 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5d37fd41fc2c41a99580cc475915a090, State: Initialized, Role: FOLLOWER
I20240503 23:24:52.872499 1260 consensus_queue.cc:260] T 26f3c79af13e40878328e02ae549ab0b P 74775765c2ac4db693fa70319ffd98c9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d37fd41fc2c41a99580cc475915a090" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 38605 } } peers { permanent_uuid: "74775765c2ac4db693fa70319ffd98c9" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 38529 } } peers { permanent_uuid: "bf5fe0b6e639437498453c116a26f95c" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 40377 } }
I20240503 23:24:52.872532 1262 consensus_queue.cc:260] T 6743c625d6854920b88764dec9ebd250 P bf5fe0b6e639437498453c116a26f95c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d37fd41fc2c41a99580cc475915a090" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 38605 } } peers { permanent_uuid: "74775765c2ac4db693fa70319ffd98c9" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 38529 } } peers { permanent_uuid: "bf5fe0b6e639437498453c116a26f95c" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 40377 } }
I20240503 23:24:52.872740 1259 consensus_queue.cc:260] T 1e6b13b222534155bc03878359c2eb3b P 5d37fd41fc2c41a99580cc475915a090 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d37fd41fc2c41a99580cc475915a090" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 38605 } } peers { permanent_uuid: "74775765c2ac4db693fa70319ffd98c9" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 38529 } } peers { permanent_uuid: "bf5fe0b6e639437498453c116a26f95c" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 40377 } }
I20240503 23:24:52.873170 1262 ts_tablet_manager.cc:1398] T 6743c625d6854920b88764dec9ebd250 P bf5fe0b6e639437498453c116a26f95c: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20240503 23:24:52.873481 1259 ts_tablet_manager.cc:1398] T 1e6b13b222534155bc03878359c2eb3b P 5d37fd41fc2c41a99580cc475915a090: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240503 23:24:52.873744 1262 tablet_bootstrap.cc:492] T 98b4ca061f7f4a43a13fbad78dac228a P bf5fe0b6e639437498453c116a26f95c: Bootstrap starting.
I20240503 23:24:52.875181 1259 tablet_bootstrap.cc:492] T cb8edce628bd466da6c9eaabf7964493 P 5d37fd41fc2c41a99580cc475915a090: Bootstrap starting.
I20240503 23:24:52.875521 1260 ts_tablet_manager.cc:1398] T 26f3c79af13e40878328e02ae549ab0b P 74775765c2ac4db693fa70319ffd98c9: Time spent starting tablet: real 0.006s user 0.001s sys 0.002s
I20240503 23:24:52.875859 1260 tablet_bootstrap.cc:492] T 966a756b35654ba0a0e272a0b6c7d7f6 P 74775765c2ac4db693fa70319ffd98c9: Bootstrap starting.
I20240503 23:24:52.876108 1262 tablet_bootstrap.cc:654] T 98b4ca061f7f4a43a13fbad78dac228a P bf5fe0b6e639437498453c116a26f95c: Neither blocks nor log segments found. Creating new log.
I20240503 23:24:52.877269 1259 tablet_bootstrap.cc:654] T cb8edce628bd466da6c9eaabf7964493 P 5d37fd41fc2c41a99580cc475915a090: Neither blocks nor log segments found. Creating new log.
I20240503 23:24:52.877774 1260 tablet_bootstrap.cc:654] T 966a756b35654ba0a0e272a0b6c7d7f6 P 74775765c2ac4db693fa70319ffd98c9: Neither blocks nor log segments found. Creating new log.
I20240503 23:24:52.887694 1262 tablet_bootstrap.cc:492] T 98b4ca061f7f4a43a13fbad78dac228a P bf5fe0b6e639437498453c116a26f95c: No bootstrap required, opened a new log
I20240503 23:24:52.888038 1262 ts_tablet_manager.cc:1367] T 98b4ca061f7f4a43a13fbad78dac228a P bf5fe0b6e639437498453c116a26f95c: Time spent bootstrapping tablet: real 0.014s user 0.005s sys 0.006s
I20240503 23:24:52.888111 1259 tablet_bootstrap.cc:492] T cb8edce628bd466da6c9eaabf7964493 P 5d37fd41fc2c41a99580cc475915a090: No bootstrap required, opened a new log
I20240503 23:24:52.888373 1259 ts_tablet_manager.cc:1367] T cb8edce628bd466da6c9eaabf7964493 P 5d37fd41fc2c41a99580cc475915a090: Time spent bootstrapping tablet: real 0.013s user 0.006s sys 0.006s
I20240503 23:24:52.888942 1260 tablet_bootstrap.cc:492] T 966a756b35654ba0a0e272a0b6c7d7f6 P 74775765c2ac4db693fa70319ffd98c9: No bootstrap required, opened a new log
I20240503 23:24:52.889097 1260 ts_tablet_manager.cc:1367] T 966a756b35654ba0a0e272a0b6c7d7f6 P 74775765c2ac4db693fa70319ffd98c9: Time spent bootstrapping tablet: real 0.013s user 0.000s sys 0.012s
I20240503 23:24:52.889165 1262 raft_consensus.cc:348] T 98b4ca061f7f4a43a13fbad78dac228a P bf5fe0b6e639437498453c116a26f95c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d37fd41fc2c41a99580cc475915a090" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 38605 } } peers { permanent_uuid: "74775765c2ac4db693fa70319ffd98c9" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 38529 } } peers { permanent_uuid: "bf5fe0b6e639437498453c116a26f95c" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 40377 } }
I20240503 23:24:52.889230 1259 raft_consensus.cc:348] T cb8edce628bd466da6c9eaabf7964493 P 5d37fd41fc2c41a99580cc475915a090 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d37fd41fc2c41a99580cc475915a090" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 38605 } } peers { permanent_uuid: "74775765c2ac4db693fa70319ffd98c9" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 38529 } } peers { permanent_uuid: "bf5fe0b6e639437498453c116a26f95c" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 40377 } }
I20240503 23:24:52.889441 1262 raft_consensus.cc:374] T 98b4ca061f7f4a43a13fbad78dac228a P bf5fe0b6e639437498453c116a26f95c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 23:24:52.889475 1259 raft_consensus.cc:374] T cb8edce628bd466da6c9eaabf7964493 P 5d37fd41fc2c41a99580cc475915a090 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 23:24:52.889570 1262 raft_consensus.cc:729] T 98b4ca061f7f4a43a13fbad78dac228a P bf5fe0b6e639437498453c116a26f95c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bf5fe0b6e639437498453c116a26f95c, State: Initialized, Role: FOLLOWER
I20240503 23:24:52.889580 1259 raft_consensus.cc:729] T cb8edce628bd466da6c9eaabf7964493 P 5d37fd41fc2c41a99580cc475915a090 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5d37fd41fc2c41a99580cc475915a090, State: Initialized, Role: FOLLOWER
I20240503 23:24:52.889889 1262 consensus_queue.cc:260] T 98b4ca061f7f4a43a13fbad78dac228a P bf5fe0b6e639437498453c116a26f95c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d37fd41fc2c41a99580cc475915a090" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 38605 } } peers { permanent_uuid: "74775765c2ac4db693fa70319ffd98c9" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 38529 } } peers { permanent_uuid: "bf5fe0b6e639437498453c116a26f95c" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 40377 } }
I20240503 23:24:52.889945 1259 consensus_queue.cc:260] T cb8edce628bd466da6c9eaabf7964493 P 5d37fd41fc2c41a99580cc475915a090 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d37fd41fc2c41a99580cc475915a090" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 38605 } } peers { permanent_uuid: "74775765c2ac4db693fa70319ffd98c9" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 38529 } } peers { permanent_uuid: "bf5fe0b6e639437498453c116a26f95c" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 40377 } }
I20240503 23:24:52.890074 1260 raft_consensus.cc:348] T 966a756b35654ba0a0e272a0b6c7d7f6 P 74775765c2ac4db693fa70319ffd98c9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d37fd41fc2c41a99580cc475915a090" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 38605 } } peers { permanent_uuid: "74775765c2ac4db693fa70319ffd98c9" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 38529 } } peers { permanent_uuid: "bf5fe0b6e639437498453c116a26f95c" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 40377 } }
I20240503 23:24:52.890269 1260 raft_consensus.cc:374] T 966a756b35654ba0a0e272a0b6c7d7f6 P 74775765c2ac4db693fa70319ffd98c9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 23:24:52.890375 1260 raft_consensus.cc:729] T 966a756b35654ba0a0e272a0b6c7d7f6 P 74775765c2ac4db693fa70319ffd98c9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 74775765c2ac4db693fa70319ffd98c9, State: Initialized, Role: FOLLOWER
I20240503 23:24:52.890520 1262 ts_tablet_manager.cc:1398] T 98b4ca061f7f4a43a13fbad78dac228a P bf5fe0b6e639437498453c116a26f95c: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 23:24:52.891418 1262 tablet_bootstrap.cc:492] T 8d775506b875408d80a957dd3926aed8 P bf5fe0b6e639437498453c116a26f95c: Bootstrap starting.
I20240503 23:24:52.890554 1259 ts_tablet_manager.cc:1398] T cb8edce628bd466da6c9eaabf7964493 P 5d37fd41fc2c41a99580cc475915a090: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 23:24:52.890694 1260 consensus_queue.cc:260] T 966a756b35654ba0a0e272a0b6c7d7f6 P 74775765c2ac4db693fa70319ffd98c9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d37fd41fc2c41a99580cc475915a090" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 38605 } } peers { permanent_uuid: "74775765c2ac4db693fa70319ffd98c9" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 38529 } } peers { permanent_uuid: "bf5fe0b6e639437498453c116a26f95c" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 40377 } }
I20240503 23:24:52.891983 1259 tablet_bootstrap.cc:492] T 8d775506b875408d80a957dd3926aed8 P 5d37fd41fc2c41a99580cc475915a090: Bootstrap starting.
I20240503 23:24:52.893761 1262 tablet_bootstrap.cc:654] T 8d775506b875408d80a957dd3926aed8 P bf5fe0b6e639437498453c116a26f95c: Neither blocks nor log segments found. Creating new log.
I20240503 23:24:52.893836 1259 tablet_bootstrap.cc:654] T 8d775506b875408d80a957dd3926aed8 P 5d37fd41fc2c41a99580cc475915a090: Neither blocks nor log segments found. Creating new log.
I20240503 23:24:52.894932 1260 ts_tablet_manager.cc:1398] T 966a756b35654ba0a0e272a0b6c7d7f6 P 74775765c2ac4db693fa70319ffd98c9: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20240503 23:24:52.896440 1260 tablet_bootstrap.cc:492] T 6743c625d6854920b88764dec9ebd250 P 74775765c2ac4db693fa70319ffd98c9: Bootstrap starting.
I20240503 23:24:52.899116 1260 tablet_bootstrap.cc:654] T 6743c625d6854920b88764dec9ebd250 P 74775765c2ac4db693fa70319ffd98c9: Neither blocks nor log segments found. Creating new log.
I20240503 23:24:52.903934 1266 raft_consensus.cc:482] T 966a756b35654ba0a0e272a0b6c7d7f6 P 74775765c2ac4db693fa70319ffd98c9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 23:24:52.904341 1259 tablet_bootstrap.cc:492] T 8d775506b875408d80a957dd3926aed8 P 5d37fd41fc2c41a99580cc475915a090: No bootstrap required, opened a new log
I20240503 23:24:52.904407 1264 raft_consensus.cc:482] T 26f3c79af13e40878328e02ae549ab0b P 5d37fd41fc2c41a99580cc475915a090 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 23:24:52.904588 1259 ts_tablet_manager.cc:1367] T 8d775506b875408d80a957dd3926aed8 P 5d37fd41fc2c41a99580cc475915a090: Time spent bootstrapping tablet: real 0.013s user 0.004s sys 0.007s
I20240503 23:24:52.904399 1266 raft_consensus.cc:504] T 966a756b35654ba0a0e272a0b6c7d7f6 P 74775765c2ac4db693fa70319ffd98c9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d37fd41fc2c41a99580cc475915a090" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 38605 } } peers { permanent_uuid: "74775765c2ac4db693fa70319ffd98c9" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 38529 } } peers { permanent_uuid: "bf5fe0b6e639437498453c116a26f95c" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 40377 } }
I20240503 23:24:52.904666 1264 raft_consensus.cc:504] T 26f3c79af13e40878328e02ae549ab0b P 5d37fd41fc2c41a99580cc475915a090 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d37fd41fc2c41a99580cc475915a090" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 38605 } } peers { permanent_uuid: "74775765c2ac4db693fa70319ffd98c9" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 38529 } } peers { permanent_uuid: "bf5fe0b6e639437498453c116a26f95c" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 40377 } }
I20240503 23:24:52.905586 1259 raft_consensus.cc:348] T 8d775506b875408d80a957dd3926aed8 P 5d37fd41fc2c41a99580cc475915a090 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d37fd41fc2c41a99580cc475915a090" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 38605 } } peers { permanent_uuid: "74775765c2ac4db693fa70319ffd98c9" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 38529 } } peers { permanent_uuid: "bf5fe0b6e639437498453c116a26f95c" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 40377 } }
I20240503 23:24:52.905890 1259 raft_consensus.cc:374] T 8d775506b875408d80a957dd3926aed8 P 5d37fd41fc2c41a99580cc475915a090 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 23:24:52.905997 1259 raft_consensus.cc:729] T 8d775506b875408d80a957dd3926aed8 P 5d37fd41fc2c41a99580cc475915a090 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5d37fd41fc2c41a99580cc475915a090, State: Initialized, Role: FOLLOWER
I20240503 23:24:52.906131 1266 leader_election.cc:288] T 966a756b35654ba0a0e272a0b6c7d7f6 P 74775765c2ac4db693fa70319ffd98c9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5d37fd41fc2c41a99580cc475915a090 (127.0.100.1:38605), bf5fe0b6e639437498453c116a26f95c (127.0.100.3:40377)
I20240503 23:24:52.906404 1259 consensus_queue.cc:260] T 8d775506b875408d80a957dd3926aed8 P 5d37fd41fc2c41a99580cc475915a090 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d37fd41fc2c41a99580cc475915a090" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 38605 } } peers { permanent_uuid: "74775765c2ac4db693fa70319ffd98c9" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 38529 } } peers { permanent_uuid: "bf5fe0b6e639437498453c116a26f95c" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 40377 } }
I20240503 23:24:52.906245 1262 tablet_bootstrap.cc:492] T 8d775506b875408d80a957dd3926aed8 P bf5fe0b6e639437498453c116a26f95c: No bootstrap required, opened a new log
I20240503 23:24:52.907457 1262 ts_tablet_manager.cc:1367] T 8d775506b875408d80a957dd3926aed8 P bf5fe0b6e639437498453c116a26f95c: Time spent bootstrapping tablet: real 0.016s user 0.008s sys 0.003s
I20240503 23:24:52.907866 1264 leader_election.cc:288] T 26f3c79af13e40878328e02ae549ab0b P 5d37fd41fc2c41a99580cc475915a090 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 74775765c2ac4db693fa70319ffd98c9 (127.0.100.2:38529), bf5fe0b6e639437498453c116a26f95c (127.0.100.3:40377)
I20240503 23:24:52.908736 1262 raft_consensus.cc:348] T 8d775506b875408d80a957dd3926aed8 P bf5fe0b6e639437498453c116a26f95c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d37fd41fc2c41a99580cc475915a090" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 38605 } } peers { permanent_uuid: "74775765c2ac4db693fa70319ffd98c9" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 38529 } } peers { permanent_uuid: "bf5fe0b6e639437498453c116a26f95c" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 40377 } }
I20240503 23:24:52.909077 1262 raft_consensus.cc:374] T 8d775506b875408d80a957dd3926aed8 P bf5fe0b6e639437498453c116a26f95c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 23:24:52.909178 1262 raft_consensus.cc:729] T 8d775506b875408d80a957dd3926aed8 P bf5fe0b6e639437498453c116a26f95c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bf5fe0b6e639437498453c116a26f95c, State: Initialized, Role: FOLLOWER
I20240503 23:24:52.909642 1262 consensus_queue.cc:260] T 8d775506b875408d80a957dd3926aed8 P bf5fe0b6e639437498453c116a26f95c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d37fd41fc2c41a99580cc475915a090" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 38605 } } peers { permanent_uuid: "74775765c2ac4db693fa70319ffd98c9" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 38529 } } peers { permanent_uuid: "bf5fe0b6e639437498453c116a26f95c" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 40377 } }
I20240503 23:24:52.911547 1262 ts_tablet_manager.cc:1398] T 8d775506b875408d80a957dd3926aed8 P bf5fe0b6e639437498453c116a26f95c: Time spent starting tablet: real 0.004s user 0.003s sys 0.000s
I20240503 23:24:52.911988 1262 tablet_bootstrap.cc:492] T cb8edce628bd466da6c9eaabf7964493 P bf5fe0b6e639437498453c116a26f95c: Bootstrap starting.
I20240503 23:24:52.914522 1262 tablet_bootstrap.cc:654] T cb8edce628bd466da6c9eaabf7964493 P bf5fe0b6e639437498453c116a26f95c: Neither blocks nor log segments found. Creating new log.
I20240503 23:24:52.930080 1260 tablet_bootstrap.cc:492] T 6743c625d6854920b88764dec9ebd250 P 74775765c2ac4db693fa70319ffd98c9: No bootstrap required, opened a new log
I20240503 23:24:52.930440 1260 ts_tablet_manager.cc:1367] T 6743c625d6854920b88764dec9ebd250 P 74775765c2ac4db693fa70319ffd98c9: Time spent bootstrapping tablet: real 0.034s user 0.004s sys 0.004s
I20240503 23:24:52.931708 1260 raft_consensus.cc:348] T 6743c625d6854920b88764dec9ebd250 P 74775765c2ac4db693fa70319ffd98c9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d37fd41fc2c41a99580cc475915a090" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 38605 } } peers { permanent_uuid: "74775765c2ac4db693fa70319ffd98c9" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 38529 } } peers { permanent_uuid: "bf5fe0b6e639437498453c116a26f95c" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 40377 } }
I20240503 23:24:52.932046 1260 raft_consensus.cc:374] T 6743c625d6854920b88764dec9ebd250 P 74775765c2ac4db693fa70319ffd98c9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 23:24:52.932157 1260 raft_consensus.cc:729] T 6743c625d6854920b88764dec9ebd250 P 74775765c2ac4db693fa70319ffd98c9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 74775765c2ac4db693fa70319ffd98c9, State: Initialized, Role: FOLLOWER
I20240503 23:24:52.932512 1260 consensus_queue.cc:260] T 6743c625d6854920b88764dec9ebd250 P 74775765c2ac4db693fa70319ffd98c9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d37fd41fc2c41a99580cc475915a090" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 38605 } } peers { permanent_uuid: "74775765c2ac4db693fa70319ffd98c9" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 38529 } } peers { permanent_uuid: "bf5fe0b6e639437498453c116a26f95c" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 40377 } }
I20240503 23:24:52.939422 1214 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "26f3c79af13e40878328e02ae549ab0b" candidate_uuid: "5d37fd41fc2c41a99580cc475915a090" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bf5fe0b6e639437498453c116a26f95c" is_pre_election: true
I20240503 23:24:52.940752 1260 ts_tablet_manager.cc:1398] T 6743c625d6854920b88764dec9ebd250 P 74775765c2ac4db693fa70319ffd98c9: Time spent starting tablet: real 0.010s user 0.004s sys 0.000s
I20240503 23:24:52.941193 1260 tablet_bootstrap.cc:492] T 8d775506b875408d80a957dd3926aed8 P 74775765c2ac4db693fa70319ffd98c9: Bootstrap starting.
I20240503 23:24:52.941798 1140 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "26f3c79af13e40878328e02ae549ab0b" candidate_uuid: "5d37fd41fc2c41a99580cc475915a090" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "74775765c2ac4db693fa70319ffd98c9" is_pre_election: true
I20240503 23:24:52.942183 1140 raft_consensus.cc:2454] T 26f3c79af13e40878328e02ae549ab0b P 74775765c2ac4db693fa70319ffd98c9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5d37fd41fc2c41a99580cc475915a090 in term 0.
I20240503 23:24:52.942757 1028 leader_election.cc:302] T 26f3c79af13e40878328e02ae549ab0b P 5d37fd41fc2c41a99580cc475915a090 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5d37fd41fc2c41a99580cc475915a090, 74775765c2ac4db693fa70319ffd98c9; no voters:
I20240503 23:24:52.943115 1264 raft_consensus.cc:2781] T 26f3c79af13e40878328e02ae549ab0b P 5d37fd41fc2c41a99580cc475915a090 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 23:24:52.943239 1264 raft_consensus.cc:482] T 26f3c79af13e40878328e02ae549ab0b P 5d37fd41fc2c41a99580cc475915a090 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 23:24:52.943451 1264 raft_consensus.cc:3037] T 26f3c79af13e40878328e02ae549ab0b P 5d37fd41fc2c41a99580cc475915a090 [term 0 FOLLOWER]: Advancing to term 1
I20240503 23:24:52.944149 1260 tablet_bootstrap.cc:654] T 8d775506b875408d80a957dd3926aed8 P 74775765c2ac4db693fa70319ffd98c9: Neither blocks nor log segments found. Creating new log.
I20240503 23:24:52.946954 1264 raft_consensus.cc:504] T 26f3c79af13e40878328e02ae549ab0b P 5d37fd41fc2c41a99580cc475915a090 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d37fd41fc2c41a99580cc475915a090" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 38605 } } peers { permanent_uuid: "74775765c2ac4db693fa70319ffd98c9" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 38529 } } peers { permanent_uuid: "bf5fe0b6e639437498453c116a26f95c" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 40377 } }
I20240503 23:24:52.947784 1259 ts_tablet_manager.cc:1398] T 8d775506b875408d80a957dd3926aed8 P 5d37fd41fc2c41a99580cc475915a090: Time spent starting tablet: real 0.043s user 0.004s sys 0.017s
I20240503 23:24:52.947870 1264 leader_election.cc:288] T 26f3c79af13e40878328e02ae549ab0b P 5d37fd41fc2c41a99580cc475915a090 [CANDIDATE]: Term 1 election: Requested vote from peers 74775765c2ac4db693fa70319ffd98c9 (127.0.100.2:38529), bf5fe0b6e639437498453c116a26f95c (127.0.100.3:40377)
I20240503 23:24:52.948880 1065 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "966a756b35654ba0a0e272a0b6c7d7f6" candidate_uuid: "74775765c2ac4db693fa70319ffd98c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5d37fd41fc2c41a99580cc475915a090" is_pre_election: true
I20240503 23:24:52.950152 1140 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "26f3c79af13e40878328e02ae549ab0b" candidate_uuid: "5d37fd41fc2c41a99580cc475915a090" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "74775765c2ac4db693fa70319ffd98c9"
I20240503 23:24:52.950696 1140 raft_consensus.cc:3037] T 26f3c79af13e40878328e02ae549ab0b P 74775765c2ac4db693fa70319ffd98c9 [term 0 FOLLOWER]: Advancing to term 1
I20240503 23:24:52.952638 1259 tablet_bootstrap.cc:492] T f3d383cdf48b4e108844ff8a658b9a51 P 5d37fd41fc2c41a99580cc475915a090: Bootstrap starting.
I20240503 23:24:52.953050 1264 raft_consensus.cc:482] T 966a756b35654ba0a0e272a0b6c7d7f6 P 5d37fd41fc2c41a99580cc475915a090 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 23:24:52.952852 1214 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "26f3c79af13e40878328e02ae549ab0b" candidate_uuid: "5d37fd41fc2c41a99580cc475915a090" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bf5fe0b6e639437498453c116a26f95c"
I20240503 23:24:52.954135 1264 raft_consensus.cc:504] T 966a756b35654ba0a0e272a0b6c7d7f6 P 5d37fd41fc2c41a99580cc475915a090 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d37fd41fc2c41a99580cc475915a090" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 38605 } } peers { permanent_uuid: "74775765c2ac4db693fa70319ffd98c9" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 38529 } } peers { permanent_uuid: "bf5fe0b6e639437498453c116a26f95c" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 40377 } }
I20240503 23:24:52.954788 1262 tablet_bootstrap.cc:492] T cb8edce628bd466da6c9eaabf7964493 P bf5fe0b6e639437498453c116a26f95c: No bootstrap required, opened a new log
I20240503 23:24:52.954746 1065 raft_consensus.cc:2454] T 966a756b35654ba0a0e272a0b6c7d7f6 P 5d37fd41fc2c41a99580cc475915a090 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 74775765c2ac4db693fa70319ffd98c9 in term 0.
I20240503 23:24:52.954990 1262 ts_tablet_manager.cc:1367] T cb8edce628bd466da6c9eaabf7964493 P bf5fe0b6e639437498453c116a26f95c: Time spent bootstrapping tablet: real 0.043s user 0.011s sys 0.020s
W20240503 23:24:52.954612 1030 leader_election.cc:341] T 26f3c79af13e40878328e02ae549ab0b P 5d37fd41fc2c41a99580cc475915a090 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer bf5fe0b6e639437498453c116a26f95c (127.0.100.3:40377): Illegal state: must be running to vote when last-logged opid is not known
I20240503 23:24:52.957134 1264 leader_election.cc:288] T 966a756b35654ba0a0e272a0b6c7d7f6 P 5d37fd41fc2c41a99580cc475915a090 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 74775765c2ac4db693fa70319ffd98c9 (127.0.100.2:38529), bf5fe0b6e639437498453c116a26f95c (127.0.100.3:40377)
I20240503 23:24:52.957182 1104 leader_election.cc:302] T 966a756b35654ba0a0e272a0b6c7d7f6 P 74775765c2ac4db693fa70319ffd98c9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5d37fd41fc2c41a99580cc475915a090, 74775765c2ac4db693fa70319ffd98c9; no voters:
W20240503 23:24:52.957952 1030 leader_election.cc:341] T 26f3c79af13e40878328e02ae549ab0b P 5d37fd41fc2c41a99580cc475915a090 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer bf5fe0b6e639437498453c116a26f95c (127.0.100.3:40377): Illegal state: must be running to vote when last-logged opid is not known
I20240503 23:24:52.958078 1214 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "966a756b35654ba0a0e272a0b6c7d7f6" candidate_uuid: "5d37fd41fc2c41a99580cc475915a090" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bf5fe0b6e639437498453c116a26f95c" is_pre_election: true
I20240503 23:24:52.958685 1266 raft_consensus.cc:2781] T 966a756b35654ba0a0e272a0b6c7d7f6 P 74775765c2ac4db693fa70319ffd98c9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 23:24:52.957588 1139 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "966a756b35654ba0a0e272a0b6c7d7f6" candidate_uuid: "5d37fd41fc2c41a99580cc475915a090" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "74775765c2ac4db693fa70319ffd98c9" is_pre_election: true
I20240503 23:24:52.960083 1266 raft_consensus.cc:482] T 966a756b35654ba0a0e272a0b6c7d7f6 P 74775765c2ac4db693fa70319ffd98c9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 23:24:52.960286 1266 raft_consensus.cc:3037] T 966a756b35654ba0a0e272a0b6c7d7f6 P 74775765c2ac4db693fa70319ffd98c9 [term 0 FOLLOWER]: Advancing to term 1
W20240503 23:24:52.960062 1030 leader_election.cc:341] T 966a756b35654ba0a0e272a0b6c7d7f6 P 5d37fd41fc2c41a99580cc475915a090 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer bf5fe0b6e639437498453c116a26f95c (127.0.100.3:40377): Illegal state: must be running to vote when last-logged opid is not known
I20240503 23:24:52.962767 1260 tablet_bootstrap.cc:492] T 8d775506b875408d80a957dd3926aed8 P 74775765c2ac4db693fa70319ffd98c9: No bootstrap required, opened a new log
I20240503 23:24:52.962880 1262 raft_consensus.cc:348] T cb8edce628bd466da6c9eaabf7964493 P bf5fe0b6e639437498453c116a26f95c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d37fd41fc2c41a99580cc475915a090" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 38605 } } peers { permanent_uuid: "74775765c2ac4db693fa70319ffd98c9" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 38529 } } peers { permanent_uuid: "bf5fe0b6e639437498453c116a26f95c" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 40377 } }
I20240503 23:24:52.963024 1260 ts_tablet_manager.cc:1367] T 8d775506b875408d80a957dd3926aed8 P 74775765c2ac4db693fa70319ffd98c9: Time spent bootstrapping tablet: real 0.022s user 0.003s sys 0.006s
I20240503 23:24:52.963160 1262 raft_consensus.cc:374] T cb8edce628bd466da6c9eaabf7964493 P bf5fe0b6e639437498453c116a26f95c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 23:24:52.963275 1262 raft_consensus.cc:729] T cb8edce628bd466da6c9eaabf7964493 P bf5fe0b6e639437498453c116a26f95c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bf5fe0b6e639437498453c116a26f95c, State: Initialized, Role: FOLLOWER
I20240503 23:24:52.963023 1140 raft_consensus.cc:2454] T 26f3c79af13e40878328e02ae549ab0b P 74775765c2ac4db693fa70319ffd98c9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5d37fd41fc2c41a99580cc475915a090 in term 1.
I20240503 23:24:52.963671 1262 consensus_queue.cc:260] T cb8edce628bd466da6c9eaabf7964493 P bf5fe0b6e639437498453c116a26f95c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d37fd41fc2c41a99580cc475915a090" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 38605 } } peers { permanent_uuid: "74775765c2ac4db693fa70319ffd98c9" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 38529 } } peers { permanent_uuid: "bf5fe0b6e639437498453c116a26f95c" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 40377 } }
I20240503 23:24:52.964083 1028 leader_election.cc:302] T 26f3c79af13e40878328e02ae549ab0b P 5d37fd41fc2c41a99580cc475915a090 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5d37fd41fc2c41a99580cc475915a090, 74775765c2ac4db693fa70319ffd98c9; no voters: bf5fe0b6e639437498453c116a26f95c
I20240503 23:24:52.964139 1260 raft_consensus.cc:348] T 8d775506b875408d80a957dd3926aed8 P 74775765c2ac4db693fa70319ffd98c9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d37fd41fc2c41a99580cc475915a090" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 38605 } } peers { permanent_uuid: "74775765c2ac4db693fa70319ffd98c9" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 38529 } } peers { permanent_uuid: "bf5fe0b6e639437498453c116a26f95c" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 40377 } }
I20240503 23:24:52.964329 1260 raft_consensus.cc:374] T 8d775506b875408d80a957dd3926aed8 P 74775765c2ac4db693fa70319ffd98c9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 23:24:52.964428 1260 raft_consensus.cc:729] T 8d775506b875408d80a957dd3926aed8 P 74775765c2ac4db693fa70319ffd98c9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 74775765c2ac4db693fa70319ffd98c9, State: Initialized, Role: FOLLOWER
I20240503 23:24:52.964718 1260 consensus_queue.cc:260] T 8d775506b875408d80a957dd3926aed8 P 74775765c2ac4db693fa70319ffd98c9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d37fd41fc2c41a99580cc475915a090" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 38605 } } peers { permanent_uuid: "74775765c2ac4db693fa70319ffd98c9" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 38529 } } peers { permanent_uuid: "bf5fe0b6e639437498453c116a26f95c" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 40377 } }
I20240503 23:24:52.964875 1264 raft_consensus.cc:2781] T 26f3c79af13e40878328e02ae549ab0b P 5d37fd41fc2c41a99580cc475915a090 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 23:24:52.967386 1264 raft_consensus.cc:686] T 26f3c79af13e40878328e02ae549ab0b P 5d37fd41fc2c41a99580cc475915a090 [term 1 LEADER]: Becoming Leader. State: Replica: 5d37fd41fc2c41a99580cc475915a090, State: Running, Role: LEADER
I20240503 23:24:52.967728 1264 consensus_queue.cc:237] T 26f3c79af13e40878328e02ae549ab0b P 5d37fd41fc2c41a99580cc475915a090 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d37fd41fc2c41a99580cc475915a090" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 38605 } } peers { permanent_uuid: "74775765c2ac4db693fa70319ffd98c9" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 38529 } } peers { permanent_uuid: "bf5fe0b6e639437498453c116a26f95c" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 40377 } }
I20240503 23:24:52.968346 1214 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "966a756b35654ba0a0e272a0b6c7d7f6" candidate_uuid: "74775765c2ac4db693fa70319ffd98c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bf5fe0b6e639437498453c116a26f95c" is_pre_election: true
I20240503 23:24:52.968569 1262 ts_tablet_manager.cc:1398] T cb8edce628bd466da6c9eaabf7964493 P bf5fe0b6e639437498453c116a26f95c: Time spent starting tablet: real 0.013s user 0.002s sys 0.001s
I20240503 23:24:52.969455 1266 raft_consensus.cc:504] T 966a756b35654ba0a0e272a0b6c7d7f6 P 74775765c2ac4db693fa70319ffd98c9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d37fd41fc2c41a99580cc475915a090" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 38605 } } peers { permanent_uuid: "74775765c2ac4db693fa70319ffd98c9" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 38529 } } peers { permanent_uuid: "bf5fe0b6e639437498453c116a26f95c" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 40377 } }
W20240503 23:24:52.969727 1105 leader_election.cc:341] T 966a756b35654ba0a0e272a0b6c7d7f6 P 74775765c2ac4db693fa70319ffd98c9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer bf5fe0b6e639437498453c116a26f95c (127.0.100.3:40377): Illegal state: must be running to vote when last-logged opid is not known
I20240503 23:24:52.970544 1266 leader_election.cc:288] T 966a756b35654ba0a0e272a0b6c7d7f6 P 74775765c2ac4db693fa70319ffd98c9 [CANDIDATE]: Term 1 election: Requested vote from peers 5d37fd41fc2c41a99580cc475915a090 (127.0.100.1:38605), bf5fe0b6e639437498453c116a26f95c (127.0.100.3:40377)
I20240503 23:24:52.971767 1259 tablet_bootstrap.cc:654] T f3d383cdf48b4e108844ff8a658b9a51 P 5d37fd41fc2c41a99580cc475915a090: Neither blocks nor log segments found. Creating new log.
I20240503 23:24:52.973521 1214 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "966a756b35654ba0a0e272a0b6c7d7f6" candidate_uuid: "74775765c2ac4db693fa70319ffd98c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bf5fe0b6e639437498453c116a26f95c"
I20240503 23:24:52.973695 1262 tablet_bootstrap.cc:492] T 26f3c79af13e40878328e02ae549ab0b P bf5fe0b6e639437498453c116a26f95c: Bootstrap starting.
I20240503 23:24:52.975371 1139 raft_consensus.cc:2379] T 966a756b35654ba0a0e272a0b6c7d7f6 P 74775765c2ac4db693fa70319ffd98c9 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5d37fd41fc2c41a99580cc475915a090 in current term 1: Already voted for candidate 74775765c2ac4db693fa70319ffd98c9 in this term.
W20240503 23:24:52.975297 1105 leader_election.cc:341] T 966a756b35654ba0a0e272a0b6c7d7f6 P 74775765c2ac4db693fa70319ffd98c9 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer bf5fe0b6e639437498453c116a26f95c (127.0.100.3:40377): Illegal state: must be running to vote when last-logged opid is not known
I20240503 23:24:52.977057 1065 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "966a756b35654ba0a0e272a0b6c7d7f6" candidate_uuid: "74775765c2ac4db693fa70319ffd98c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5d37fd41fc2c41a99580cc475915a090"
I20240503 23:24:52.978072 1065 raft_consensus.cc:3037] T 966a756b35654ba0a0e272a0b6c7d7f6 P 5d37fd41fc2c41a99580cc475915a090 [term 0 FOLLOWER]: Advancing to term 1
I20240503 23:24:52.978101 1028 leader_election.cc:302] T 966a756b35654ba0a0e272a0b6c7d7f6 P 5d37fd41fc2c41a99580cc475915a090 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5d37fd41fc2c41a99580cc475915a090; no voters: 74775765c2ac4db693fa70319ffd98c9, bf5fe0b6e639437498453c116a26f95c
I20240503 23:24:52.979385 1260 ts_tablet_manager.cc:1398] T 8d775506b875408d80a957dd3926aed8 P 74775765c2ac4db693fa70319ffd98c9: Time spent starting tablet: real 0.015s user 0.002s sys 0.012s
I20240503 23:24:52.980257 1260 tablet_bootstrap.cc:492] T 98b4ca061f7f4a43a13fbad78dac228a P 74775765c2ac4db693fa70319ffd98c9: Bootstrap starting.
I20240503 23:24:52.979887 962 catalog_manager.cc:5484] T 26f3c79af13e40878328e02ae549ab0b P 5d37fd41fc2c41a99580cc475915a090 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5d37fd41fc2c41a99580cc475915a090 (127.0.100.1). New cstate: current_term: 1 leader_uuid: "5d37fd41fc2c41a99580cc475915a090" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d37fd41fc2c41a99580cc475915a090" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 38605 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "74775765c2ac4db693fa70319ffd98c9" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 38529 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bf5fe0b6e639437498453c116a26f95c" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 40377 } health_report { overall_health: UNKNOWN } } }
I20240503 23:24:52.984436 1065 raft_consensus.cc:2454] T 966a756b35654ba0a0e272a0b6c7d7f6 P 5d37fd41fc2c41a99580cc475915a090 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 74775765c2ac4db693fa70319ffd98c9 in term 1.
I20240503 23:24:52.984778 1260 tablet_bootstrap.cc:654] T 98b4ca061f7f4a43a13fbad78dac228a P 74775765c2ac4db693fa70319ffd98c9: Neither blocks nor log segments found. Creating new log.
I20240503 23:24:52.985535 1104 leader_election.cc:302] T 966a756b35654ba0a0e272a0b6c7d7f6 P 74775765c2ac4db693fa70319ffd98c9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5d37fd41fc2c41a99580cc475915a090, 74775765c2ac4db693fa70319ffd98c9; no voters: bf5fe0b6e639437498453c116a26f95c
I20240503 23:24:52.986492 1262 tablet_bootstrap.cc:654] T 26f3c79af13e40878328e02ae549ab0b P bf5fe0b6e639437498453c116a26f95c: Neither blocks nor log segments found. Creating new log.
I20240503 23:24:52.987777 1259 tablet_bootstrap.cc:492] T f3d383cdf48b4e108844ff8a658b9a51 P 5d37fd41fc2c41a99580cc475915a090: No bootstrap required, opened a new log
I20240503 23:24:52.988227 1286 raft_consensus.cc:2781] T 966a756b35654ba0a0e272a0b6c7d7f6 P 74775765c2ac4db693fa70319ffd98c9 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 23:24:52.988399 1286 raft_consensus.cc:686] T 966a756b35654ba0a0e272a0b6c7d7f6 P 74775765c2ac4db693fa70319ffd98c9 [term 1 LEADER]: Becoming Leader. State: Replica: 74775765c2ac4db693fa70319ffd98c9, State: Running, Role: LEADER
I20240503 23:24:52.988839 1286 consensus_queue.cc:237] T 966a756b35654ba0a0e272a0b6c7d7f6 P 74775765c2ac4db693fa70319ffd98c9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d37fd41fc2c41a99580cc475915a090" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 38605 } } peers { permanent_uuid: "74775765c2ac4db693fa70319ffd98c9" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 38529 } } peers { permanent_uuid: "bf5fe0b6e639437498453c116a26f95c" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 40377 } }
I20240503 23:24:52.988106 1264 raft_consensus.cc:2726] T 966a756b35654ba0a0e272a0b6c7d7f6 P 5d37fd41fc2c41a99580cc475915a090 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 23:24:52.989318 1259 ts_tablet_manager.cc:1367] T f3d383cdf48b4e108844ff8a658b9a51 P 5d37fd41fc2c41a99580cc475915a090: Time spent bootstrapping tablet: real 0.039s user 0.003s sys 0.003s
I20240503 23:24:52.990259 1260 tablet_bootstrap.cc:492] T 98b4ca061f7f4a43a13fbad78dac228a P 74775765c2ac4db693fa70319ffd98c9: No bootstrap required, opened a new log
I20240503 23:24:52.990767 1260 ts_tablet_manager.cc:1367] T 98b4ca061f7f4a43a13fbad78dac228a P 74775765c2ac4db693fa70319ffd98c9: Time spent bootstrapping tablet: real 0.011s user 0.003s sys 0.002s
I20240503 23:24:52.990605 1259 raft_consensus.cc:348] T f3d383cdf48b4e108844ff8a658b9a51 P 5d37fd41fc2c41a99580cc475915a090 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d37fd41fc2c41a99580cc475915a090" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 38605 } } peers { permanent_uuid: "bf5fe0b6e639437498453c116a26f95c" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 40377 } } peers { permanent_uuid: "74775765c2ac4db693fa70319ffd98c9" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 38529 } }
I20240503 23:24:52.991041 1259 raft_consensus.cc:374] T f3d383cdf48b4e108844ff8a658b9a51 P 5d37fd41fc2c41a99580cc475915a090 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 23:24:52.991200 1259 raft_consensus.cc:729] T f3d383cdf48b4e108844ff8a658b9a51 P 5d37fd41fc2c41a99580cc475915a090 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5d37fd41fc2c41a99580cc475915a090, State: Initialized, Role: FOLLOWER
I20240503 23:24:52.991530 1259 consensus_queue.cc:260] T f3d383cdf48b4e108844ff8a658b9a51 P 5d37fd41fc2c41a99580cc475915a090 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d37fd41fc2c41a99580cc475915a090" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 38605 } } peers { permanent_uuid: "bf5fe0b6e639437498453c116a26f95c" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 40377 } } peers { permanent_uuid: "74775765c2ac4db693fa70319ffd98c9" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 38529 } }
I20240503 23:24:52.991854 1262 tablet_bootstrap.cc:492] T 26f3c79af13e40878328e02ae549ab0b P bf5fe0b6e639437498453c116a26f95c: No bootstrap required, opened a new log
I20240503 23:24:52.991809 1260 raft_consensus.cc:348] T 98b4ca061f7f4a43a13fbad78dac228a P 74775765c2ac4db693fa70319ffd98c9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d37fd41fc2c41a99580cc475915a090" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 38605 } } peers { permanent_uuid: "74775765c2ac4db693fa70319ffd98c9" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 38529 } } peers { permanent_uuid: "bf5fe0b6e639437498453c116a26f95c" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 40377 } }
I20240503 23:24:52.992056 1260 raft_consensus.cc:374] T 98b4ca061f7f4a43a13fbad78dac228a P 74775765c2ac4db693fa70319ffd98c9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 23:24:52.992089 1262 ts_tablet_manager.cc:1367] T 26f3c79af13e40878328e02ae549ab0b P bf5fe0b6e639437498453c116a26f95c: Time spent bootstrapping tablet: real 0.021s user 0.003s sys 0.004s
I20240503 23:24:52.992170 1260 raft_consensus.cc:729] T 98b4ca061f7f4a43a13fbad78dac228a P 74775765c2ac4db693fa70319ffd98c9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 74775765c2ac4db693fa70319ffd98c9, State: Initialized, Role: FOLLOWER
I20240503 23:24:52.992520 1260 consensus_queue.cc:260] T 98b4ca061f7f4a43a13fbad78dac228a P 74775765c2ac4db693fa70319ffd98c9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d37fd41fc2c41a99580cc475915a090" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 38605 } } peers { permanent_uuid: "74775765c2ac4db693fa70319ffd98c9" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 38529 } } peers { permanent_uuid: "bf5fe0b6e639437498453c116a26f95c" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 40377 } }
I20240503 23:24:52.992651 1259 ts_tablet_manager.cc:1398] T f3d383cdf48b4e108844ff8a658b9a51 P 5d37fd41fc2c41a99580cc475915a090: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20240503 23:24:52.992918 1262 raft_consensus.cc:348] T 26f3c79af13e40878328e02ae549ab0b P bf5fe0b6e639437498453c116a26f95c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d37fd41fc2c41a99580cc475915a090" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 38605 } } peers { permanent_uuid: "74775765c2ac4db693fa70319ffd98c9" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 38529 } } peers { permanent_uuid: "bf5fe0b6e639437498453c116a26f95c" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 40377 } }
I20240503 23:24:52.993062 1260 ts_tablet_manager.cc:1398] T 98b4ca061f7f4a43a13fbad78dac228a P 74775765c2ac4db693fa70319ffd98c9: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240503 23:24:52.993141 1262 raft_consensus.cc:374] T 26f3c79af13e40878328e02ae549ab0b P bf5fe0b6e639437498453c116a26f95c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 23:24:52.993216 1262 raft_consensus.cc:729] T 26f3c79af13e40878328e02ae549ab0b P bf5fe0b6e639437498453c116a26f95c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bf5fe0b6e639437498453c116a26f95c, State: Initialized, Role: FOLLOWER
I20240503 23:24:52.993422 1260 tablet_bootstrap.cc:492] T 1e6b13b222534155bc03878359c2eb3b P 74775765c2ac4db693fa70319ffd98c9: Bootstrap starting.
I20240503 23:24:52.993525 1262 consensus_queue.cc:260] T 26f3c79af13e40878328e02ae549ab0b P bf5fe0b6e639437498453c116a26f95c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d37fd41fc2c41a99580cc475915a090" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 38605 } } peers { permanent_uuid: "74775765c2ac4db693fa70319ffd98c9" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 38529 } } peers { permanent_uuid: "bf5fe0b6e639437498453c116a26f95c" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 40377 } }
I20240503 23:24:52.995518 1260 tablet_bootstrap.cc:654] T 1e6b13b222534155bc03878359c2eb3b P 74775765c2ac4db693fa70319ffd98c9: Neither blocks nor log segments found. Creating new log.
I20240503 23:24:52.997817 1262 ts_tablet_manager.cc:1398] T 26f3c79af13e40878328e02ae549ab0b P bf5fe0b6e639437498453c116a26f95c: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20240503 23:24:52.998307 1262 tablet_bootstrap.cc:492] T 1e6b13b222534155bc03878359c2eb3b P bf5fe0b6e639437498453c116a26f95c: Bootstrap starting.
I20240503 23:24:53.000970 1262 tablet_bootstrap.cc:654] T 1e6b13b222534155bc03878359c2eb3b P bf5fe0b6e639437498453c116a26f95c: Neither blocks nor log segments found. Creating new log.
I20240503 23:24:53.004736 1259 tablet_bootstrap.cc:492] T 6743c625d6854920b88764dec9ebd250 P 5d37fd41fc2c41a99580cc475915a090: Bootstrap starting.
I20240503 23:24:53.004797 961 catalog_manager.cc:5484] T 966a756b35654ba0a0e272a0b6c7d7f6 P 74775765c2ac4db693fa70319ffd98c9 reported cstate change: term changed from 0 to 1, leader changed from <none> to 74775765c2ac4db693fa70319ffd98c9 (127.0.100.2). New cstate: current_term: 1 leader_uuid: "74775765c2ac4db693fa70319ffd98c9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d37fd41fc2c41a99580cc475915a090" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 38605 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "74775765c2ac4db693fa70319ffd98c9" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 38529 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bf5fe0b6e639437498453c116a26f95c" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 40377 } health_report { overall_health: UNKNOWN } } }
I20240503 23:24:53.007437 1262 tablet_bootstrap.cc:492] T 1e6b13b222534155bc03878359c2eb3b P bf5fe0b6e639437498453c116a26f95c: No bootstrap required, opened a new log
I20240503 23:24:53.007472 1260 tablet_bootstrap.cc:492] T 1e6b13b222534155bc03878359c2eb3b P 74775765c2ac4db693fa70319ffd98c9: No bootstrap required, opened a new log
I20240503 23:24:53.007683 1262 ts_tablet_manager.cc:1367] T 1e6b13b222534155bc03878359c2eb3b P bf5fe0b6e639437498453c116a26f95c: Time spent bootstrapping tablet: real 0.009s user 0.000s sys 0.006s
I20240503 23:24:53.007822 1260 ts_tablet_manager.cc:1367] T 1e6b13b222534155bc03878359c2eb3b P 74775765c2ac4db693fa70319ffd98c9: Time spent bootstrapping tablet: real 0.014s user 0.008s sys 0.005s
I20240503 23:24:53.008615 1259 tablet_bootstrap.cc:654] T 6743c625d6854920b88764dec9ebd250 P 5d37fd41fc2c41a99580cc475915a090: Neither blocks nor log segments found. Creating new log.
I20240503 23:24:53.008656 1262 raft_consensus.cc:348] T 1e6b13b222534155bc03878359c2eb3b P bf5fe0b6e639437498453c116a26f95c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d37fd41fc2c41a99580cc475915a090" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 38605 } } peers { permanent_uuid: "74775765c2ac4db693fa70319ffd98c9" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 38529 } } peers { permanent_uuid: "bf5fe0b6e639437498453c116a26f95c" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 40377 } }
I20240503 23:24:53.008919 1262 raft_consensus.cc:374] T 1e6b13b222534155bc03878359c2eb3b P bf5fe0b6e639437498453c116a26f95c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 23:24:53.009064 1262 raft_consensus.cc:729] T 1e6b13b222534155bc03878359c2eb3b P bf5fe0b6e639437498453c116a26f95c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bf5fe0b6e639437498453c116a26f95c, State: Initialized, Role: FOLLOWER
I20240503 23:24:53.008961 1260 raft_consensus.cc:348] T 1e6b13b222534155bc03878359c2eb3b P 74775765c2ac4db693fa70319ffd98c9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d37fd41fc2c41a99580cc475915a090" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 38605 } } peers { permanent_uuid: "74775765c2ac4db693fa70319ffd98c9" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 38529 } } peers { permanent_uuid: "bf5fe0b6e639437498453c116a26f95c" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 40377 } }
I20240503 23:24:53.009372 1260 raft_consensus.cc:374] T 1e6b13b222534155bc03878359c2eb3b P 74775765c2ac4db693fa70319ffd98c9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 23:24:53.009559 1260 raft_consensus.cc:729] T 1e6b13b222534155bc03878359c2eb3b P 74775765c2ac4db693fa70319ffd98c9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 74775765c2ac4db693fa70319ffd98c9, State: Initialized, Role: FOLLOWER
I20240503 23:24:53.009378 1262 consensus_queue.cc:260] T 1e6b13b222534155bc03878359c2eb3b P bf5fe0b6e639437498453c116a26f95c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d37fd41fc2c41a99580cc475915a090" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 38605 } } peers { permanent_uuid: "74775765c2ac4db693fa70319ffd98c9" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 38529 } } peers { permanent_uuid: "bf5fe0b6e639437498453c116a26f95c" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 40377 } }
I20240503 23:24:53.010018 1260 consensus_queue.cc:260] T 1e6b13b222534155bc03878359c2eb3b P 74775765c2ac4db693fa70319ffd98c9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d37fd41fc2c41a99580cc475915a090" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 38605 } } peers { permanent_uuid: "74775765c2ac4db693fa70319ffd98c9" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 38529 } } peers { permanent_uuid: "bf5fe0b6e639437498453c116a26f95c" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 40377 } }
I20240503 23:24:53.010887 1260 ts_tablet_manager.cc:1398] T 1e6b13b222534155bc03878359c2eb3b P 74775765c2ac4db693fa70319ffd98c9: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240503 23:24:53.011386 1260 tablet_bootstrap.cc:492] T cb8edce628bd466da6c9eaabf7964493 P 74775765c2ac4db693fa70319ffd98c9: Bootstrap starting.
I20240503 23:24:53.012532 1262 ts_tablet_manager.cc:1398] T 1e6b13b222534155bc03878359c2eb3b P bf5fe0b6e639437498453c116a26f95c: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240503 23:24:53.012962 1262 tablet_bootstrap.cc:492] T 966a756b35654ba0a0e272a0b6c7d7f6 P bf5fe0b6e639437498453c116a26f95c: Bootstrap starting.
I20240503 23:24:53.014925 1262 tablet_bootstrap.cc:654] T 966a756b35654ba0a0e272a0b6c7d7f6 P bf5fe0b6e639437498453c116a26f95c: Neither blocks nor log segments found. Creating new log.
I20240503 23:24:53.015331 1260 tablet_bootstrap.cc:654] T cb8edce628bd466da6c9eaabf7964493 P 74775765c2ac4db693fa70319ffd98c9: Neither blocks nor log segments found. Creating new log.
I20240503 23:24:53.017490 1259 tablet_bootstrap.cc:492] T 6743c625d6854920b88764dec9ebd250 P 5d37fd41fc2c41a99580cc475915a090: No bootstrap required, opened a new log
I20240503 23:24:53.017674 1259 ts_tablet_manager.cc:1367] T 6743c625d6854920b88764dec9ebd250 P 5d37fd41fc2c41a99580cc475915a090: Time spent bootstrapping tablet: real 0.022s user 0.010s sys 0.000s
I20240503 23:24:53.018769 1262 tablet_bootstrap.cc:492] T 966a756b35654ba0a0e272a0b6c7d7f6 P bf5fe0b6e639437498453c116a26f95c: No bootstrap required, opened a new log
I20240503 23:24:53.018779 1259 raft_consensus.cc:348] T 6743c625d6854920b88764dec9ebd250 P 5d37fd41fc2c41a99580cc475915a090 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d37fd41fc2c41a99580cc475915a090" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 38605 } } peers { permanent_uuid: "74775765c2ac4db693fa70319ffd98c9" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 38529 } } peers { permanent_uuid: "bf5fe0b6e639437498453c116a26f95c" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 40377 } }
I20240503 23:24:53.018996 1262 ts_tablet_manager.cc:1367] T 966a756b35654ba0a0e272a0b6c7d7f6 P bf5fe0b6e639437498453c116a26f95c: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20240503 23:24:53.019049 1259 raft_consensus.cc:374] T 6743c625d6854920b88764dec9ebd250 P 5d37fd41fc2c41a99580cc475915a090 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 23:24:53.019167 1259 raft_consensus.cc:729] T 6743c625d6854920b88764dec9ebd250 P 5d37fd41fc2c41a99580cc475915a090 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5d37fd41fc2c41a99580cc475915a090, State: Initialized, Role: FOLLOWER
I20240503 23:24:53.019423 1259 consensus_queue.cc:260] T 6743c625d6854920b88764dec9ebd250 P 5d37fd41fc2c41a99580cc475915a090 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d37fd41fc2c41a99580cc475915a090" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 38605 } } peers { permanent_uuid: "74775765c2ac4db693fa70319ffd98c9" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 38529 } } peers { permanent_uuid: "bf5fe0b6e639437498453c116a26f95c" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 40377 } }
I20240503 23:24:53.019762 1262 raft_consensus.cc:348] T 966a756b35654ba0a0e272a0b6c7d7f6 P bf5fe0b6e639437498453c116a26f95c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d37fd41fc2c41a99580cc475915a090" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 38605 } } peers { permanent_uuid: "74775765c2ac4db693fa70319ffd98c9" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 38529 } } peers { permanent_uuid: "bf5fe0b6e639437498453c116a26f95c" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 40377 } }
I20240503 23:24:53.019986 1259 ts_tablet_manager.cc:1398] T 6743c625d6854920b88764dec9ebd250 P 5d37fd41fc2c41a99580cc475915a090: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240503 23:24:53.020026 1262 raft_consensus.cc:374] T 966a756b35654ba0a0e272a0b6c7d7f6 P bf5fe0b6e639437498453c116a26f95c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 23:24:53.020102 1262 raft_consensus.cc:729] T 966a756b35654ba0a0e272a0b6c7d7f6 P bf5fe0b6e639437498453c116a26f95c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bf5fe0b6e639437498453c116a26f95c, State: Initialized, Role: FOLLOWER
I20240503 23:24:53.020413 1259 tablet_bootstrap.cc:492] T 98b4ca061f7f4a43a13fbad78dac228a P 5d37fd41fc2c41a99580cc475915a090: Bootstrap starting.
I20240503 23:24:53.021157 1260 tablet_bootstrap.cc:492] T cb8edce628bd466da6c9eaabf7964493 P 74775765c2ac4db693fa70319ffd98c9: No bootstrap required, opened a new log
I20240503 23:24:53.021338 1260 ts_tablet_manager.cc:1367] T cb8edce628bd466da6c9eaabf7964493 P 74775765c2ac4db693fa70319ffd98c9: Time spent bootstrapping tablet: real 0.010s user 0.006s sys 0.000s
I20240503 23:24:53.020411 1262 consensus_queue.cc:260] T 966a756b35654ba0a0e272a0b6c7d7f6 P bf5fe0b6e639437498453c116a26f95c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d37fd41fc2c41a99580cc475915a090" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 38605 } } peers { permanent_uuid: "74775765c2ac4db693fa70319ffd98c9" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 38529 } } peers { permanent_uuid: "bf5fe0b6e639437498453c116a26f95c" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 40377 } }
I20240503 23:24:53.023133 1260 raft_consensus.cc:348] T cb8edce628bd466da6c9eaabf7964493 P 74775765c2ac4db693fa70319ffd98c9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d37fd41fc2c41a99580cc475915a090" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 38605 } } peers { permanent_uuid: "74775765c2ac4db693fa70319ffd98c9" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 38529 } } peers { permanent_uuid: "bf5fe0b6e639437498453c116a26f95c" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 40377 } }
I20240503 23:24:53.023470 1259 tablet_bootstrap.cc:654] T 98b4ca061f7f4a43a13fbad78dac228a P 5d37fd41fc2c41a99580cc475915a090: Neither blocks nor log segments found. Creating new log.
I20240503 23:24:53.023957 1262 ts_tablet_manager.cc:1398] T 966a756b35654ba0a0e272a0b6c7d7f6 P bf5fe0b6e639437498453c116a26f95c: Time spent starting tablet: real 0.005s user 0.001s sys 0.001s
I20240503 23:24:53.023389 1260 raft_consensus.cc:374] T cb8edce628bd466da6c9eaabf7964493 P 74775765c2ac4db693fa70319ffd98c9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 23:24:53.025139 1260 raft_consensus.cc:729] T cb8edce628bd466da6c9eaabf7964493 P 74775765c2ac4db693fa70319ffd98c9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 74775765c2ac4db693fa70319ffd98c9, State: Initialized, Role: FOLLOWER
I20240503 23:24:53.025523 1260 consensus_queue.cc:260] T cb8edce628bd466da6c9eaabf7964493 P 74775765c2ac4db693fa70319ffd98c9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d37fd41fc2c41a99580cc475915a090" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 38605 } } peers { permanent_uuid: "74775765c2ac4db693fa70319ffd98c9" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 38529 } } peers { permanent_uuid: "bf5fe0b6e639437498453c116a26f95c" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 40377 } }
I20240503 23:24:53.026120 1260 ts_tablet_manager.cc:1398] T cb8edce628bd466da6c9eaabf7964493 P 74775765c2ac4db693fa70319ffd98c9: Time spent starting tablet: real 0.005s user 0.003s sys 0.000s
I20240503 23:24:53.027141 1259 tablet_bootstrap.cc:492] T 98b4ca061f7f4a43a13fbad78dac228a P 5d37fd41fc2c41a99580cc475915a090: No bootstrap required, opened a new log
I20240503 23:24:53.027333 1259 ts_tablet_manager.cc:1367] T 98b4ca061f7f4a43a13fbad78dac228a P 5d37fd41fc2c41a99580cc475915a090: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.002s
I20240503 23:24:53.028155 1259 raft_consensus.cc:348] T 98b4ca061f7f4a43a13fbad78dac228a P 5d37fd41fc2c41a99580cc475915a090 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d37fd41fc2c41a99580cc475915a090" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 38605 } } peers { permanent_uuid: "74775765c2ac4db693fa70319ffd98c9" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 38529 } } peers { permanent_uuid: "bf5fe0b6e639437498453c116a26f95c" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 40377 } }
I20240503 23:24:53.028409 1259 raft_consensus.cc:374] T 98b4ca061f7f4a43a13fbad78dac228a P 5d37fd41fc2c41a99580cc475915a090 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 23:24:53.028496 1259 raft_consensus.cc:729] T 98b4ca061f7f4a43a13fbad78dac228a P 5d37fd41fc2c41a99580cc475915a090 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5d37fd41fc2c41a99580cc475915a090, State: Initialized, Role: FOLLOWER
I20240503 23:24:53.028760 1259 consensus_queue.cc:260] T 98b4ca061f7f4a43a13fbad78dac228a P 5d37fd41fc2c41a99580cc475915a090 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d37fd41fc2c41a99580cc475915a090" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 38605 } } peers { permanent_uuid: "74775765c2ac4db693fa70319ffd98c9" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 38529 } } peers { permanent_uuid: "bf5fe0b6e639437498453c116a26f95c" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 40377 } }
I20240503 23:24:53.029302 1259 ts_tablet_manager.cc:1398] T 98b4ca061f7f4a43a13fbad78dac228a P 5d37fd41fc2c41a99580cc475915a090: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 23:24:53.034723 1268 raft_consensus.cc:482] T 26f3c79af13e40878328e02ae549ab0b P bf5fe0b6e639437498453c116a26f95c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 23:24:53.034915 1268 raft_consensus.cc:504] T 26f3c79af13e40878328e02ae549ab0b P bf5fe0b6e639437498453c116a26f95c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d37fd41fc2c41a99580cc475915a090" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 38605 } } peers { permanent_uuid: "74775765c2ac4db693fa70319ffd98c9" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 38529 } } peers { permanent_uuid: "bf5fe0b6e639437498453c116a26f95c" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 40377 } }
I20240503 23:24:53.035916 1268 leader_election.cc:288] T 26f3c79af13e40878328e02ae549ab0b P bf5fe0b6e639437498453c116a26f95c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5d37fd41fc2c41a99580cc475915a090 (127.0.100.1:38605), 74775765c2ac4db693fa70319ffd98c9 (127.0.100.2:38529)
I20240503 23:24:53.046401 1139 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "26f3c79af13e40878328e02ae549ab0b" candidate_uuid: "bf5fe0b6e639437498453c116a26f95c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "74775765c2ac4db693fa70319ffd98c9" is_pre_election: true
I20240503 23:24:53.046823 1139 raft_consensus.cc:2379] T 26f3c79af13e40878328e02ae549ab0b P 74775765c2ac4db693fa70319ffd98c9 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate bf5fe0b6e639437498453c116a26f95c in current term 1: Already voted for candidate 5d37fd41fc2c41a99580cc475915a090 in this term.
I20240503 23:24:53.046823 1065 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "26f3c79af13e40878328e02ae549ab0b" candidate_uuid: "bf5fe0b6e639437498453c116a26f95c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5d37fd41fc2c41a99580cc475915a090" is_pre_election: true
I20240503 23:24:53.047438 1177 leader_election.cc:302] T 26f3c79af13e40878328e02ae549ab0b P bf5fe0b6e639437498453c116a26f95c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bf5fe0b6e639437498453c116a26f95c; no voters: 5d37fd41fc2c41a99580cc475915a090, 74775765c2ac4db693fa70319ffd98c9
I20240503 23:24:53.047725 1268 raft_consensus.cc:3037] T 26f3c79af13e40878328e02ae549ab0b P bf5fe0b6e639437498453c116a26f95c [term 0 FOLLOWER]: Advancing to term 1
I20240503 23:24:53.050068 1268 raft_consensus.cc:2726] T 26f3c79af13e40878328e02ae549ab0b P bf5fe0b6e639437498453c116a26f95c [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 23:24:53.067804 1268 raft_consensus.cc:482] T f3d383cdf48b4e108844ff8a658b9a51 P bf5fe0b6e639437498453c116a26f95c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 23:24:53.068126 1268 raft_consensus.cc:504] T f3d383cdf48b4e108844ff8a658b9a51 P bf5fe0b6e639437498453c116a26f95c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d37fd41fc2c41a99580cc475915a090" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 38605 } } peers { permanent_uuid: "bf5fe0b6e639437498453c116a26f95c" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 40377 } } peers { permanent_uuid: "74775765c2ac4db693fa70319ffd98c9" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 38529 } }
I20240503 23:24:53.069000 1268 leader_election.cc:288] T f3d383cdf48b4e108844ff8a658b9a51 P bf5fe0b6e639437498453c116a26f95c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5d37fd41fc2c41a99580cc475915a090 (127.0.100.1:38605), 74775765c2ac4db693fa70319ffd98c9 (127.0.100.2:38529)
I20240503 23:24:53.069643 1065 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f3d383cdf48b4e108844ff8a658b9a51" candidate_uuid: "bf5fe0b6e639437498453c116a26f95c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5d37fd41fc2c41a99580cc475915a090" is_pre_election: true
I20240503 23:24:53.069692 1139 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f3d383cdf48b4e108844ff8a658b9a51" candidate_uuid: "bf5fe0b6e639437498453c116a26f95c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "74775765c2ac4db693fa70319ffd98c9" is_pre_election: true
I20240503 23:24:53.069898 1065 raft_consensus.cc:2454] T f3d383cdf48b4e108844ff8a658b9a51 P 5d37fd41fc2c41a99580cc475915a090 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bf5fe0b6e639437498453c116a26f95c in term 0.
I20240503 23:24:53.069902 1139 raft_consensus.cc:2454] T f3d383cdf48b4e108844ff8a658b9a51 P 74775765c2ac4db693fa70319ffd98c9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bf5fe0b6e639437498453c116a26f95c in term 0.
I20240503 23:24:53.070300 1177 leader_election.cc:302] T f3d383cdf48b4e108844ff8a658b9a51 P bf5fe0b6e639437498453c116a26f95c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 74775765c2ac4db693fa70319ffd98c9, bf5fe0b6e639437498453c116a26f95c; no voters:
I20240503 23:24:53.070609 1268 raft_consensus.cc:2781] T f3d383cdf48b4e108844ff8a658b9a51 P bf5fe0b6e639437498453c116a26f95c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 23:24:53.070690 1268 raft_consensus.cc:482] T f3d383cdf48b4e108844ff8a658b9a51 P bf5fe0b6e639437498453c116a26f95c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 23:24:53.070828 1268 raft_consensus.cc:3037] T f3d383cdf48b4e108844ff8a658b9a51 P bf5fe0b6e639437498453c116a26f95c [term 0 FOLLOWER]: Advancing to term 1
I20240503 23:24:53.073154 1268 raft_consensus.cc:504] T f3d383cdf48b4e108844ff8a658b9a51 P bf5fe0b6e639437498453c116a26f95c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d37fd41fc2c41a99580cc475915a090" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 38605 } } peers { permanent_uuid: "bf5fe0b6e639437498453c116a26f95c" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 40377 } } peers { permanent_uuid: "74775765c2ac4db693fa70319ffd98c9" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 38529 } }
I20240503 23:24:53.073832 1268 leader_election.cc:288] T f3d383cdf48b4e108844ff8a658b9a51 P bf5fe0b6e639437498453c116a26f95c [CANDIDATE]: Term 1 election: Requested vote from peers 5d37fd41fc2c41a99580cc475915a090 (127.0.100.1:38605), 74775765c2ac4db693fa70319ffd98c9 (127.0.100.2:38529)
I20240503 23:24:53.074415 1139 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f3d383cdf48b4e108844ff8a658b9a51" candidate_uuid: "bf5fe0b6e639437498453c116a26f95c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "74775765c2ac4db693fa70319ffd98c9"
I20240503 23:24:53.074455 1065 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f3d383cdf48b4e108844ff8a658b9a51" candidate_uuid: "bf5fe0b6e639437498453c116a26f95c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5d37fd41fc2c41a99580cc475915a090"
I20240503 23:24:53.074640 1139 raft_consensus.cc:3037] T f3d383cdf48b4e108844ff8a658b9a51 P 74775765c2ac4db693fa70319ffd98c9 [term 0 FOLLOWER]: Advancing to term 1
I20240503 23:24:53.074644 1065 raft_consensus.cc:3037] T f3d383cdf48b4e108844ff8a658b9a51 P 5d37fd41fc2c41a99580cc475915a090 [term 0 FOLLOWER]: Advancing to term 1
I20240503 23:24:53.078015 1139 raft_consensus.cc:2454] T f3d383cdf48b4e108844ff8a658b9a51 P 74775765c2ac4db693fa70319ffd98c9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bf5fe0b6e639437498453c116a26f95c in term 1.
I20240503 23:24:53.078037 1065 raft_consensus.cc:2454] T f3d383cdf48b4e108844ff8a658b9a51 P 5d37fd41fc2c41a99580cc475915a090 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bf5fe0b6e639437498453c116a26f95c in term 1.
I20240503 23:24:53.078758 1178 leader_election.cc:302] T f3d383cdf48b4e108844ff8a658b9a51 P bf5fe0b6e639437498453c116a26f95c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 5d37fd41fc2c41a99580cc475915a090, 74775765c2ac4db693fa70319ffd98c9, bf5fe0b6e639437498453c116a26f95c; no voters:
I20240503 23:24:53.079119 1268 raft_consensus.cc:2781] T f3d383cdf48b4e108844ff8a658b9a51 P bf5fe0b6e639437498453c116a26f95c [term 1 FOLLOWER]: Leader election won for term 1
I20240503 23:24:53.079605 1268 raft_consensus.cc:686] T f3d383cdf48b4e108844ff8a658b9a51 P bf5fe0b6e639437498453c116a26f95c [term 1 LEADER]: Becoming Leader. State: Replica: bf5fe0b6e639437498453c116a26f95c, State: Running, Role: LEADER
I20240503 23:24:53.079829 1300 raft_consensus.cc:482] T 98b4ca061f7f4a43a13fbad78dac228a P bf5fe0b6e639437498453c116a26f95c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 23:24:53.079985 1300 raft_consensus.cc:504] T 98b4ca061f7f4a43a13fbad78dac228a P bf5fe0b6e639437498453c116a26f95c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d37fd41fc2c41a99580cc475915a090" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 38605 } } peers { permanent_uuid: "74775765c2ac4db693fa70319ffd98c9" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 38529 } } peers { permanent_uuid: "bf5fe0b6e639437498453c116a26f95c" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 40377 } }
I20240503 23:24:53.080534 1268 consensus_queue.cc:237] T f3d383cdf48b4e108844ff8a658b9a51 P bf5fe0b6e639437498453c116a26f95c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d37fd41fc2c41a99580cc475915a090" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 38605 } } peers { permanent_uuid: "bf5fe0b6e639437498453c116a26f95c" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 40377 } } peers { permanent_uuid: "74775765c2ac4db693fa70319ffd98c9" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 38529 } }
I20240503 23:24:53.080979 1300 leader_election.cc:288] T 98b4ca061f7f4a43a13fbad78dac228a P bf5fe0b6e639437498453c116a26f95c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5d37fd41fc2c41a99580cc475915a090 (127.0.100.1:38605), 74775765c2ac4db693fa70319ffd98c9 (127.0.100.2:38529)
I20240503 23:24:53.081584 1139 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "98b4ca061f7f4a43a13fbad78dac228a" candidate_uuid: "bf5fe0b6e639437498453c116a26f95c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "74775765c2ac4db693fa70319ffd98c9" is_pre_election: true
I20240503 23:24:53.081660 1065 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "98b4ca061f7f4a43a13fbad78dac228a" candidate_uuid: "bf5fe0b6e639437498453c116a26f95c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5d37fd41fc2c41a99580cc475915a090" is_pre_election: true
I20240503 23:24:53.081809 1139 raft_consensus.cc:2454] T 98b4ca061f7f4a43a13fbad78dac228a P 74775765c2ac4db693fa70319ffd98c9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bf5fe0b6e639437498453c116a26f95c in term 0.
I20240503 23:24:53.081876 1065 raft_consensus.cc:2454] T 98b4ca061f7f4a43a13fbad78dac228a P 5d37fd41fc2c41a99580cc475915a090 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bf5fe0b6e639437498453c116a26f95c in term 0.
I20240503 23:24:53.082258 1177 leader_election.cc:302] T 98b4ca061f7f4a43a13fbad78dac228a P bf5fe0b6e639437498453c116a26f95c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 74775765c2ac4db693fa70319ffd98c9, bf5fe0b6e639437498453c116a26f95c; no voters:
I20240503 23:24:53.082597 1301 raft_consensus.cc:2781] T 98b4ca061f7f4a43a13fbad78dac228a P bf5fe0b6e639437498453c116a26f95c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 23:24:53.082715 1301 raft_consensus.cc:482] T 98b4ca061f7f4a43a13fbad78dac228a P bf5fe0b6e639437498453c116a26f95c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 23:24:53.082847 1301 raft_consensus.cc:3037] T 98b4ca061f7f4a43a13fbad78dac228a P bf5fe0b6e639437498453c116a26f95c [term 0 FOLLOWER]: Advancing to term 1
I20240503 23:24:53.085695 961 catalog_manager.cc:5484] T f3d383cdf48b4e108844ff8a658b9a51 P bf5fe0b6e639437498453c116a26f95c reported cstate change: term changed from 0 to 1, leader changed from <none> to bf5fe0b6e639437498453c116a26f95c (127.0.100.3). New cstate: current_term: 1 leader_uuid: "bf5fe0b6e639437498453c116a26f95c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d37fd41fc2c41a99580cc475915a090" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 38605 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bf5fe0b6e639437498453c116a26f95c" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 40377 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "74775765c2ac4db693fa70319ffd98c9" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 38529 } health_report { overall_health: UNKNOWN } } }
I20240503 23:24:53.085901 1301 raft_consensus.cc:504] T 98b4ca061f7f4a43a13fbad78dac228a P bf5fe0b6e639437498453c116a26f95c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d37fd41fc2c41a99580cc475915a090" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 38605 } } peers { permanent_uuid: "74775765c2ac4db693fa70319ffd98c9" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 38529 } } peers { permanent_uuid: "bf5fe0b6e639437498453c116a26f95c" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 40377 } }
I20240503 23:24:53.086567 1301 leader_election.cc:288] T 98b4ca061f7f4a43a13fbad78dac228a P bf5fe0b6e639437498453c116a26f95c [CANDIDATE]: Term 1 election: Requested vote from peers 5d37fd41fc2c41a99580cc475915a090 (127.0.100.1:38605), 74775765c2ac4db693fa70319ffd98c9 (127.0.100.2:38529)
I20240503 23:24:53.087082 1065 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "98b4ca061f7f4a43a13fbad78dac228a" candidate_uuid: "bf5fe0b6e639437498453c116a26f95c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5d37fd41fc2c41a99580cc475915a090"
I20240503 23:24:53.087150 1139 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "98b4ca061f7f4a43a13fbad78dac228a" candidate_uuid: "bf5fe0b6e639437498453c116a26f95c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "74775765c2ac4db693fa70319ffd98c9"
I20240503 23:24:53.087321 1065 raft_consensus.cc:3037] T 98b4ca061f7f4a43a13fbad78dac228a P 5d37fd41fc2c41a99580cc475915a090 [term 0 FOLLOWER]: Advancing to term 1
I20240503 23:24:53.087358 1139 raft_consensus.cc:3037] T 98b4ca061f7f4a43a13fbad78dac228a P 74775765c2ac4db693fa70319ffd98c9 [term 0 FOLLOWER]: Advancing to term 1
I20240503 23:24:53.091173 1139 raft_consensus.cc:2454] T 98b4ca061f7f4a43a13fbad78dac228a P 74775765c2ac4db693fa70319ffd98c9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bf5fe0b6e639437498453c116a26f95c in term 1.
I20240503 23:24:53.091501 1065 raft_consensus.cc:2454] T 98b4ca061f7f4a43a13fbad78dac228a P 5d37fd41fc2c41a99580cc475915a090 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bf5fe0b6e639437498453c116a26f95c in term 1.
I20240503 23:24:53.091727 1177 leader_election.cc:302] T 98b4ca061f7f4a43a13fbad78dac228a P bf5fe0b6e639437498453c116a26f95c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 74775765c2ac4db693fa70319ffd98c9, bf5fe0b6e639437498453c116a26f95c; no voters:
I20240503 23:24:53.092087 1301 raft_consensus.cc:2781] T 98b4ca061f7f4a43a13fbad78dac228a P bf5fe0b6e639437498453c116a26f95c [term 1 FOLLOWER]: Leader election won for term 1
I20240503 23:24:53.092265 1301 raft_consensus.cc:686] T 98b4ca061f7f4a43a13fbad78dac228a P bf5fe0b6e639437498453c116a26f95c [term 1 LEADER]: Becoming Leader. State: Replica: bf5fe0b6e639437498453c116a26f95c, State: Running, Role: LEADER
I20240503 23:24:53.092639 1301 consensus_queue.cc:237] T 98b4ca061f7f4a43a13fbad78dac228a P bf5fe0b6e639437498453c116a26f95c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d37fd41fc2c41a99580cc475915a090" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 38605 } } peers { permanent_uuid: "74775765c2ac4db693fa70319ffd98c9" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 38529 } } peers { permanent_uuid: "bf5fe0b6e639437498453c116a26f95c" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 40377 } }
I20240503 23:24:53.095579 961 catalog_manager.cc:5484] T 98b4ca061f7f4a43a13fbad78dac228a P bf5fe0b6e639437498453c116a26f95c reported cstate change: term changed from 0 to 1, leader changed from <none> to bf5fe0b6e639437498453c116a26f95c (127.0.100.3). New cstate: current_term: 1 leader_uuid: "bf5fe0b6e639437498453c116a26f95c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d37fd41fc2c41a99580cc475915a090" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 38605 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "74775765c2ac4db693fa70319ffd98c9" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 38529 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bf5fe0b6e639437498453c116a26f95c" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 40377 } health_report { overall_health: HEALTHY } } }
I20240503 23:24:53.110356 1264 raft_consensus.cc:482] T 8d775506b875408d80a957dd3926aed8 P 5d37fd41fc2c41a99580cc475915a090 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 23:24:53.110661 1264 raft_consensus.cc:504] T 8d775506b875408d80a957dd3926aed8 P 5d37fd41fc2c41a99580cc475915a090 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d37fd41fc2c41a99580cc475915a090" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 38605 } } peers { permanent_uuid: "74775765c2ac4db693fa70319ffd98c9" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 38529 } } peers { permanent_uuid: "bf5fe0b6e639437498453c116a26f95c" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 40377 } }
I20240503 23:24:53.111452 1264 leader_election.cc:288] T 8d775506b875408d80a957dd3926aed8 P 5d37fd41fc2c41a99580cc475915a090 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 74775765c2ac4db693fa70319ffd98c9 (127.0.100.2:38529), bf5fe0b6e639437498453c116a26f95c (127.0.100.3:40377)
I20240503 23:24:53.112073 1214 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d775506b875408d80a957dd3926aed8" candidate_uuid: "5d37fd41fc2c41a99580cc475915a090" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bf5fe0b6e639437498453c116a26f95c" is_pre_election: true
I20240503 23:24:53.112082 1139 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d775506b875408d80a957dd3926aed8" candidate_uuid: "5d37fd41fc2c41a99580cc475915a090" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "74775765c2ac4db693fa70319ffd98c9" is_pre_election: true
I20240503 23:24:53.112324 1139 raft_consensus.cc:2454] T 8d775506b875408d80a957dd3926aed8 P 74775765c2ac4db693fa70319ffd98c9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5d37fd41fc2c41a99580cc475915a090 in term 0.
I20240503 23:24:53.112365 1214 raft_consensus.cc:2454] T 8d775506b875408d80a957dd3926aed8 P bf5fe0b6e639437498453c116a26f95c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5d37fd41fc2c41a99580cc475915a090 in term 0.
I20240503 23:24:53.112790 1028 leader_election.cc:302] T 8d775506b875408d80a957dd3926aed8 P 5d37fd41fc2c41a99580cc475915a090 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5d37fd41fc2c41a99580cc475915a090, 74775765c2ac4db693fa70319ffd98c9; no voters:
I20240503 23:24:53.113173 1264 raft_consensus.cc:2781] T 8d775506b875408d80a957dd3926aed8 P 5d37fd41fc2c41a99580cc475915a090 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 23:24:53.113260 1264 raft_consensus.cc:482] T 8d775506b875408d80a957dd3926aed8 P 5d37fd41fc2c41a99580cc475915a090 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 23:24:53.113385 1264 raft_consensus.cc:3037] T 8d775506b875408d80a957dd3926aed8 P 5d37fd41fc2c41a99580cc475915a090 [term 0 FOLLOWER]: Advancing to term 1
I20240503 23:24:53.115767 1264 raft_consensus.cc:504] T 8d775506b875408d80a957dd3926aed8 P 5d37fd41fc2c41a99580cc475915a090 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d37fd41fc2c41a99580cc475915a090" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 38605 } } peers { permanent_uuid: "74775765c2ac4db693fa70319ffd98c9" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 38529 } } peers { permanent_uuid: "bf5fe0b6e639437498453c116a26f95c" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 40377 } }
I20240503 23:24:53.116568 1279 raft_consensus.cc:482] T cb8edce628bd466da6c9eaabf7964493 P 5d37fd41fc2c41a99580cc475915a090 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 23:24:53.116614 1264 leader_election.cc:288] T 8d775506b875408d80a957dd3926aed8 P 5d37fd41fc2c41a99580cc475915a090 [CANDIDATE]: Term 1 election: Requested vote from peers 74775765c2ac4db693fa70319ffd98c9 (127.0.100.2:38529), bf5fe0b6e639437498453c116a26f95c (127.0.100.3:40377)
I20240503 23:24:53.116765 1279 raft_consensus.cc:504] T cb8edce628bd466da6c9eaabf7964493 P 5d37fd41fc2c41a99580cc475915a090 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d37fd41fc2c41a99580cc475915a090" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 38605 } } peers { permanent_uuid: "74775765c2ac4db693fa70319ffd98c9" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 38529 } } peers { permanent_uuid: "bf5fe0b6e639437498453c116a26f95c" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 40377 } }
I20240503 23:24:53.117259 1214 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d775506b875408d80a957dd3926aed8" candidate_uuid: "5d37fd41fc2c41a99580cc475915a090" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bf5fe0b6e639437498453c116a26f95c"
I20240503 23:24:53.117475 1214 raft_consensus.cc:3037] T 8d775506b875408d80a957dd3926aed8 P bf5fe0b6e639437498453c116a26f95c [term 0 FOLLOWER]: Advancing to term 1
I20240503 23:24:53.117637 1279 leader_election.cc:288] T cb8edce628bd466da6c9eaabf7964493 P 5d37fd41fc2c41a99580cc475915a090 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 74775765c2ac4db693fa70319ffd98c9 (127.0.100.2:38529), bf5fe0b6e639437498453c116a26f95c (127.0.100.3:40377)
I20240503 23:24:53.118145 1139 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d775506b875408d80a957dd3926aed8" candidate_uuid: "5d37fd41fc2c41a99580cc475915a090" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "74775765c2ac4db693fa70319ffd98c9"
I20240503 23:24:53.118253 1140 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cb8edce628bd466da6c9eaabf7964493" candidate_uuid: "5d37fd41fc2c41a99580cc475915a090" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "74775765c2ac4db693fa70319ffd98c9" is_pre_election: true
I20240503 23:24:53.118352 1139 raft_consensus.cc:3037] T 8d775506b875408d80a957dd3926aed8 P 74775765c2ac4db693fa70319ffd98c9 [term 0 FOLLOWER]: Advancing to term 1
I20240503 23:24:53.118468 1140 raft_consensus.cc:2454] T cb8edce628bd466da6c9eaabf7964493 P 74775765c2ac4db693fa70319ffd98c9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5d37fd41fc2c41a99580cc475915a090 in term 0.
I20240503 23:24:53.118870 1028 leader_election.cc:302] T cb8edce628bd466da6c9eaabf7964493 P 5d37fd41fc2c41a99580cc475915a090 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5d37fd41fc2c41a99580cc475915a090, 74775765c2ac4db693fa70319ffd98c9; no voters:
I20240503 23:24:53.119143 1279 raft_consensus.cc:2781] T cb8edce628bd466da6c9eaabf7964493 P 5d37fd41fc2c41a99580cc475915a090 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 23:24:53.119263 1279 raft_consensus.cc:482] T cb8edce628bd466da6c9eaabf7964493 P 5d37fd41fc2c41a99580cc475915a090 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 23:24:53.119442 1279 raft_consensus.cc:3037] T cb8edce628bd466da6c9eaabf7964493 P 5d37fd41fc2c41a99580cc475915a090 [term 0 FOLLOWER]: Advancing to term 1
I20240503 23:24:53.121165 1214 raft_consensus.cc:2454] T 8d775506b875408d80a957dd3926aed8 P bf5fe0b6e639437498453c116a26f95c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5d37fd41fc2c41a99580cc475915a090 in term 1.
I20240503 23:24:53.121436 1139 raft_consensus.cc:2454] T 8d775506b875408d80a957dd3926aed8 P 74775765c2ac4db693fa70319ffd98c9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5d37fd41fc2c41a99580cc475915a090 in term 1.
I20240503 23:24:53.121778 1030 leader_election.cc:302] T 8d775506b875408d80a957dd3926aed8 P 5d37fd41fc2c41a99580cc475915a090 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5d37fd41fc2c41a99580cc475915a090, bf5fe0b6e639437498453c116a26f95c; no voters:
I20240503 23:24:53.121986 1214 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cb8edce628bd466da6c9eaabf7964493" candidate_uuid: "5d37fd41fc2c41a99580cc475915a090" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bf5fe0b6e639437498453c116a26f95c" is_pre_election: true
I20240503 23:24:53.122162 1214 raft_consensus.cc:2454] T cb8edce628bd466da6c9eaabf7964493 P bf5fe0b6e639437498453c116a26f95c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5d37fd41fc2c41a99580cc475915a090 in term 0.
I20240503 23:24:53.122725 1264 raft_consensus.cc:2781] T 8d775506b875408d80a957dd3926aed8 P 5d37fd41fc2c41a99580cc475915a090 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 23:24:53.122691 1279 raft_consensus.cc:504] T cb8edce628bd466da6c9eaabf7964493 P 5d37fd41fc2c41a99580cc475915a090 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d37fd41fc2c41a99580cc475915a090" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 38605 } } peers { permanent_uuid: "74775765c2ac4db693fa70319ffd98c9" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 38529 } } peers { permanent_uuid: "bf5fe0b6e639437498453c116a26f95c" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 40377 } }
I20240503 23:24:53.124130 1264 raft_consensus.cc:686] T 8d775506b875408d80a957dd3926aed8 P 5d37fd41fc2c41a99580cc475915a090 [term 1 LEADER]: Becoming Leader. State: Replica: 5d37fd41fc2c41a99580cc475915a090, State: Running, Role: LEADER
I20240503 23:24:53.124482 1279 leader_election.cc:288] T cb8edce628bd466da6c9eaabf7964493 P 5d37fd41fc2c41a99580cc475915a090 [CANDIDATE]: Term 1 election: Requested vote from peers 74775765c2ac4db693fa70319ffd98c9 (127.0.100.2:38529), bf5fe0b6e639437498453c116a26f95c (127.0.100.3:40377)
I20240503 23:24:53.124485 1264 consensus_queue.cc:237] T 8d775506b875408d80a957dd3926aed8 P 5d37fd41fc2c41a99580cc475915a090 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d37fd41fc2c41a99580cc475915a090" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 38605 } } peers { permanent_uuid: "74775765c2ac4db693fa70319ffd98c9" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 38529 } } peers { permanent_uuid: "bf5fe0b6e639437498453c116a26f95c" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 40377 } }
I20240503 23:24:53.124850 1139 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cb8edce628bd466da6c9eaabf7964493" candidate_uuid: "5d37fd41fc2c41a99580cc475915a090" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "74775765c2ac4db693fa70319ffd98c9"
I20240503 23:24:53.124985 1214 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cb8edce628bd466da6c9eaabf7964493" candidate_uuid: "5d37fd41fc2c41a99580cc475915a090" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bf5fe0b6e639437498453c116a26f95c"
I20240503 23:24:53.125104 1139 raft_consensus.cc:3037] T cb8edce628bd466da6c9eaabf7964493 P 74775765c2ac4db693fa70319ffd98c9 [term 0 FOLLOWER]: Advancing to term 1
I20240503 23:24:53.125238 1214 raft_consensus.cc:3037] T cb8edce628bd466da6c9eaabf7964493 P bf5fe0b6e639437498453c116a26f95c [term 0 FOLLOWER]: Advancing to term 1
I20240503 23:24:53.127872 1214 raft_consensus.cc:2454] T cb8edce628bd466da6c9eaabf7964493 P bf5fe0b6e639437498453c116a26f95c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5d37fd41fc2c41a99580cc475915a090 in term 1.
I20240503 23:24:53.128158 1139 raft_consensus.cc:2454] T cb8edce628bd466da6c9eaabf7964493 P 74775765c2ac4db693fa70319ffd98c9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5d37fd41fc2c41a99580cc475915a090 in term 1.
I20240503 23:24:53.128352 1030 leader_election.cc:302] T cb8edce628bd466da6c9eaabf7964493 P 5d37fd41fc2c41a99580cc475915a090 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5d37fd41fc2c41a99580cc475915a090, bf5fe0b6e639437498453c116a26f95c; no voters:
I20240503 23:24:53.128213 961 catalog_manager.cc:5484] T 8d775506b875408d80a957dd3926aed8 P 5d37fd41fc2c41a99580cc475915a090 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5d37fd41fc2c41a99580cc475915a090 (127.0.100.1). New cstate: current_term: 1 leader_uuid: "5d37fd41fc2c41a99580cc475915a090" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d37fd41fc2c41a99580cc475915a090" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 38605 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "74775765c2ac4db693fa70319ffd98c9" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 38529 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bf5fe0b6e639437498453c116a26f95c" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 40377 } health_report { overall_health: UNKNOWN } } }
I20240503 23:24:53.128854 1279 raft_consensus.cc:2781] T cb8edce628bd466da6c9eaabf7964493 P 5d37fd41fc2c41a99580cc475915a090 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 23:24:53.129004 1279 raft_consensus.cc:686] T cb8edce628bd466da6c9eaabf7964493 P 5d37fd41fc2c41a99580cc475915a090 [term 1 LEADER]: Becoming Leader. State: Replica: 5d37fd41fc2c41a99580cc475915a090, State: Running, Role: LEADER
I20240503 23:24:53.129290 1279 consensus_queue.cc:237] T cb8edce628bd466da6c9eaabf7964493 P 5d37fd41fc2c41a99580cc475915a090 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d37fd41fc2c41a99580cc475915a090" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 38605 } } peers { permanent_uuid: "74775765c2ac4db693fa70319ffd98c9" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 38529 } } peers { permanent_uuid: "bf5fe0b6e639437498453c116a26f95c" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 40377 } }
I20240503 23:24:53.131248 1279 raft_consensus.cc:482] T 1e6b13b222534155bc03878359c2eb3b P 5d37fd41fc2c41a99580cc475915a090 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 23:24:53.131426 1279 raft_consensus.cc:504] T 1e6b13b222534155bc03878359c2eb3b P 5d37fd41fc2c41a99580cc475915a090 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d37fd41fc2c41a99580cc475915a090" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 38605 } } peers { permanent_uuid: "74775765c2ac4db693fa70319ffd98c9" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 38529 } } peers { permanent_uuid: "bf5fe0b6e639437498453c116a26f95c" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 40377 } }
I20240503 23:24:53.132046 1279 leader_election.cc:288] T 1e6b13b222534155bc03878359c2eb3b P 5d37fd41fc2c41a99580cc475915a090 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 74775765c2ac4db693fa70319ffd98c9 (127.0.100.2:38529), bf5fe0b6e639437498453c116a26f95c (127.0.100.3:40377)
I20240503 23:24:53.132493 1139 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e6b13b222534155bc03878359c2eb3b" candidate_uuid: "5d37fd41fc2c41a99580cc475915a090" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "74775765c2ac4db693fa70319ffd98c9" is_pre_election: true
I20240503 23:24:53.132506 1214 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e6b13b222534155bc03878359c2eb3b" candidate_uuid: "5d37fd41fc2c41a99580cc475915a090" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bf5fe0b6e639437498453c116a26f95c" is_pre_election: true
I20240503 23:24:53.132766 1139 raft_consensus.cc:2454] T 1e6b13b222534155bc03878359c2eb3b P 74775765c2ac4db693fa70319ffd98c9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5d37fd41fc2c41a99580cc475915a090 in term 0.
I20240503 23:24:53.132833 1214 raft_consensus.cc:2454] T 1e6b13b222534155bc03878359c2eb3b P bf5fe0b6e639437498453c116a26f95c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5d37fd41fc2c41a99580cc475915a090 in term 0.
I20240503 23:24:53.133327 1030 leader_election.cc:302] T 1e6b13b222534155bc03878359c2eb3b P 5d37fd41fc2c41a99580cc475915a090 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5d37fd41fc2c41a99580cc475915a090, bf5fe0b6e639437498453c116a26f95c; no voters:
I20240503 23:24:53.133612 1279 raft_consensus.cc:2781] T 1e6b13b222534155bc03878359c2eb3b P 5d37fd41fc2c41a99580cc475915a090 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 23:24:53.133719 1279 raft_consensus.cc:482] T 1e6b13b222534155bc03878359c2eb3b P 5d37fd41fc2c41a99580cc475915a090 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 23:24:53.133854 1279 raft_consensus.cc:3037] T 1e6b13b222534155bc03878359c2eb3b P 5d37fd41fc2c41a99580cc475915a090 [term 0 FOLLOWER]: Advancing to term 1
I20240503 23:24:53.134081 961 catalog_manager.cc:5484] T cb8edce628bd466da6c9eaabf7964493 P 5d37fd41fc2c41a99580cc475915a090 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5d37fd41fc2c41a99580cc475915a090 (127.0.100.1). New cstate: current_term: 1 leader_uuid: "5d37fd41fc2c41a99580cc475915a090" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d37fd41fc2c41a99580cc475915a090" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 38605 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "74775765c2ac4db693fa70319ffd98c9" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 38529 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bf5fe0b6e639437498453c116a26f95c" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 40377 } health_report { overall_health: UNKNOWN } } }
I20240503 23:24:53.136467 1279 raft_consensus.cc:504] T 1e6b13b222534155bc03878359c2eb3b P 5d37fd41fc2c41a99580cc475915a090 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d37fd41fc2c41a99580cc475915a090" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 38605 } } peers { permanent_uuid: "74775765c2ac4db693fa70319ffd98c9" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 38529 } } peers { permanent_uuid: "bf5fe0b6e639437498453c116a26f95c" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 40377 } }
I20240503 23:24:53.137213 1279 leader_election.cc:288] T 1e6b13b222534155bc03878359c2eb3b P 5d37fd41fc2c41a99580cc475915a090 [CANDIDATE]: Term 1 election: Requested vote from peers 74775765c2ac4db693fa70319ffd98c9 (127.0.100.2:38529), bf5fe0b6e639437498453c116a26f95c (127.0.100.3:40377)
I20240503 23:24:53.137866 1214 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e6b13b222534155bc03878359c2eb3b" candidate_uuid: "5d37fd41fc2c41a99580cc475915a090" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bf5fe0b6e639437498453c116a26f95c"
I20240503 23:24:53.137895 1139 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e6b13b222534155bc03878359c2eb3b" candidate_uuid: "5d37fd41fc2c41a99580cc475915a090" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "74775765c2ac4db693fa70319ffd98c9"
I20240503 23:24:53.138090 1214 raft_consensus.cc:3037] T 1e6b13b222534155bc03878359c2eb3b P bf5fe0b6e639437498453c116a26f95c [term 0 FOLLOWER]: Advancing to term 1
I20240503 23:24:53.138101 1139 raft_consensus.cc:3037] T 1e6b13b222534155bc03878359c2eb3b P 74775765c2ac4db693fa70319ffd98c9 [term 0 FOLLOWER]: Advancing to term 1
I20240503 23:24:53.140614 1214 raft_consensus.cc:2454] T 1e6b13b222534155bc03878359c2eb3b P bf5fe0b6e639437498453c116a26f95c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5d37fd41fc2c41a99580cc475915a090 in term 1.
I20240503 23:24:53.140694 1279 raft_consensus.cc:482] T 6743c625d6854920b88764dec9ebd250 P 5d37fd41fc2c41a99580cc475915a090 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 23:24:53.140609 1139 raft_consensus.cc:2454] T 1e6b13b222534155bc03878359c2eb3b P 74775765c2ac4db693fa70319ffd98c9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5d37fd41fc2c41a99580cc475915a090 in term 1.
I20240503 23:24:53.140916 1279 raft_consensus.cc:504] T 6743c625d6854920b88764dec9ebd250 P 5d37fd41fc2c41a99580cc475915a090 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d37fd41fc2c41a99580cc475915a090" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 38605 } } peers { permanent_uuid: "74775765c2ac4db693fa70319ffd98c9" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 38529 } } peers { permanent_uuid: "bf5fe0b6e639437498453c116a26f95c" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 40377 } }
I20240503 23:24:53.142058 1030 leader_election.cc:302] T 1e6b13b222534155bc03878359c2eb3b P 5d37fd41fc2c41a99580cc475915a090 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5d37fd41fc2c41a99580cc475915a090, bf5fe0b6e639437498453c116a26f95c; no voters:
I20240503 23:24:53.142405 1264 raft_consensus.cc:2781] T 1e6b13b222534155bc03878359c2eb3b P 5d37fd41fc2c41a99580cc475915a090 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 23:24:53.142601 1264 raft_consensus.cc:686] T 1e6b13b222534155bc03878359c2eb3b P 5d37fd41fc2c41a99580cc475915a090 [term 1 LEADER]: Becoming Leader. State: Replica: 5d37fd41fc2c41a99580cc475915a090, State: Running, Role: LEADER
I20240503 23:24:53.142987 1279 leader_election.cc:288] T 6743c625d6854920b88764dec9ebd250 P 5d37fd41fc2c41a99580cc475915a090 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 74775765c2ac4db693fa70319ffd98c9 (127.0.100.2:38529), bf5fe0b6e639437498453c116a26f95c (127.0.100.3:40377)
I20240503 23:24:53.142974 1264 consensus_queue.cc:237] T 1e6b13b222534155bc03878359c2eb3b P 5d37fd41fc2c41a99580cc475915a090 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d37fd41fc2c41a99580cc475915a090" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 38605 } } peers { permanent_uuid: "74775765c2ac4db693fa70319ffd98c9" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 38529 } } peers { permanent_uuid: "bf5fe0b6e639437498453c116a26f95c" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 40377 } }
I20240503 23:24:53.143110 1214 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6743c625d6854920b88764dec9ebd250" candidate_uuid: "5d37fd41fc2c41a99580cc475915a090" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bf5fe0b6e639437498453c116a26f95c" is_pre_election: true
I20240503 23:24:53.143324 1214 raft_consensus.cc:2454] T 6743c625d6854920b88764dec9ebd250 P bf5fe0b6e639437498453c116a26f95c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5d37fd41fc2c41a99580cc475915a090 in term 0.
I20240503 23:24:53.143530 1139 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6743c625d6854920b88764dec9ebd250" candidate_uuid: "5d37fd41fc2c41a99580cc475915a090" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "74775765c2ac4db693fa70319ffd98c9" is_pre_election: true
I20240503 23:24:53.143754 1139 raft_consensus.cc:2454] T 6743c625d6854920b88764dec9ebd250 P 74775765c2ac4db693fa70319ffd98c9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5d37fd41fc2c41a99580cc475915a090 in term 0.
I20240503 23:24:53.143803 1030 leader_election.cc:302] T 6743c625d6854920b88764dec9ebd250 P 5d37fd41fc2c41a99580cc475915a090 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5d37fd41fc2c41a99580cc475915a090, bf5fe0b6e639437498453c116a26f95c; no voters:
I20240503 23:24:53.144083 1279 raft_consensus.cc:2781] T 6743c625d6854920b88764dec9ebd250 P 5d37fd41fc2c41a99580cc475915a090 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 23:24:53.144194 1279 raft_consensus.cc:482] T 6743c625d6854920b88764dec9ebd250 P 5d37fd41fc2c41a99580cc475915a090 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 23:24:53.144315 1279 raft_consensus.cc:3037] T 6743c625d6854920b88764dec9ebd250 P 5d37fd41fc2c41a99580cc475915a090 [term 0 FOLLOWER]: Advancing to term 1
I20240503 23:24:53.146428 961 catalog_manager.cc:5484] T 1e6b13b222534155bc03878359c2eb3b P 5d37fd41fc2c41a99580cc475915a090 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5d37fd41fc2c41a99580cc475915a090 (127.0.100.1). New cstate: current_term: 1 leader_uuid: "5d37fd41fc2c41a99580cc475915a090" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d37fd41fc2c41a99580cc475915a090" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 38605 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "74775765c2ac4db693fa70319ffd98c9" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 38529 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bf5fe0b6e639437498453c116a26f95c" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 40377 } health_report { overall_health: UNKNOWN } } }
I20240503 23:24:53.146781 1279 raft_consensus.cc:504] T 6743c625d6854920b88764dec9ebd250 P 5d37fd41fc2c41a99580cc475915a090 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d37fd41fc2c41a99580cc475915a090" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 38605 } } peers { permanent_uuid: "74775765c2ac4db693fa70319ffd98c9" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 38529 } } peers { permanent_uuid: "bf5fe0b6e639437498453c116a26f95c" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 40377 } }
I20240503 23:24:53.147754 1279 leader_election.cc:288] T 6743c625d6854920b88764dec9ebd250 P 5d37fd41fc2c41a99580cc475915a090 [CANDIDATE]: Term 1 election: Requested vote from peers 74775765c2ac4db693fa70319ffd98c9 (127.0.100.2:38529), bf5fe0b6e639437498453c116a26f95c (127.0.100.3:40377)
I20240503 23:24:53.148288 1139 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6743c625d6854920b88764dec9ebd250" candidate_uuid: "5d37fd41fc2c41a99580cc475915a090" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "74775765c2ac4db693fa70319ffd98c9"
I20240503 23:24:53.148299 1214 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6743c625d6854920b88764dec9ebd250" candidate_uuid: "5d37fd41fc2c41a99580cc475915a090" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bf5fe0b6e639437498453c116a26f95c"
I20240503 23:24:53.148608 1139 raft_consensus.cc:3037] T 6743c625d6854920b88764dec9ebd250 P 74775765c2ac4db693fa70319ffd98c9 [term 0 FOLLOWER]: Advancing to term 1
I20240503 23:24:53.148622 1214 raft_consensus.cc:3037] T 6743c625d6854920b88764dec9ebd250 P bf5fe0b6e639437498453c116a26f95c [term 0 FOLLOWER]: Advancing to term 1
I20240503 23:24:53.151290 1139 raft_consensus.cc:2454] T 6743c625d6854920b88764dec9ebd250 P 74775765c2ac4db693fa70319ffd98c9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5d37fd41fc2c41a99580cc475915a090 in term 1.
I20240503 23:24:53.151660 1214 raft_consensus.cc:2454] T 6743c625d6854920b88764dec9ebd250 P bf5fe0b6e639437498453c116a26f95c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5d37fd41fc2c41a99580cc475915a090 in term 1.
I20240503 23:24:53.151724 1028 leader_election.cc:302] T 6743c625d6854920b88764dec9ebd250 P 5d37fd41fc2c41a99580cc475915a090 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5d37fd41fc2c41a99580cc475915a090, 74775765c2ac4db693fa70319ffd98c9; no voters:
I20240503 23:24:53.152081 1279 raft_consensus.cc:2781] T 6743c625d6854920b88764dec9ebd250 P 5d37fd41fc2c41a99580cc475915a090 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 23:24:53.152252 1279 raft_consensus.cc:686] T 6743c625d6854920b88764dec9ebd250 P 5d37fd41fc2c41a99580cc475915a090 [term 1 LEADER]: Becoming Leader. State: Replica: 5d37fd41fc2c41a99580cc475915a090, State: Running, Role: LEADER
I20240503 23:24:53.152578 1279 consensus_queue.cc:237] T 6743c625d6854920b88764dec9ebd250 P 5d37fd41fc2c41a99580cc475915a090 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d37fd41fc2c41a99580cc475915a090" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 38605 } } peers { permanent_uuid: "74775765c2ac4db693fa70319ffd98c9" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 38529 } } peers { permanent_uuid: "bf5fe0b6e639437498453c116a26f95c" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 40377 } }
I20240503 23:24:53.155514 961 catalog_manager.cc:5484] T 6743c625d6854920b88764dec9ebd250 P 5d37fd41fc2c41a99580cc475915a090 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5d37fd41fc2c41a99580cc475915a090 (127.0.100.1). New cstate: current_term: 1 leader_uuid: "5d37fd41fc2c41a99580cc475915a090" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d37fd41fc2c41a99580cc475915a090" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 38605 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "74775765c2ac4db693fa70319ffd98c9" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 38529 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bf5fe0b6e639437498453c116a26f95c" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 40377 } health_report { overall_health: UNKNOWN } } }
I20240503 23:24:53.207919 400 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 23:24:53.213071 1309 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 23:24:53.213905 1310 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 23:24:53.214639 1312 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 23:24:53.215526 400 server_base.cc:1034] running on GCE node
I20240503 23:24:53.215881 400 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240503 23:24:53.215970 400 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240503 23:24:53.216032 400 hybrid_clock.cc:648] HybridClock initialized: now 1714778693216011 us; error 0 us; skew 500 ppm
I20240503 23:24:53.216279 400 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 23:24:53.218080 400 webserver.cc:460] Webserver started at http://127.0.100.4:33237/ using document root <none> and password file <none>
I20240503 23:24:53.218376 400 fs_manager.cc:359] Metadata directory not provided
I20240503 23:24:53.218466 400 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 23:24:53.218648 400 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 23:24:53.219449 400 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZhIiPz/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714778688451250-400-0/minicluster-data/ts-3-root/instance:
uuid: "3790742adac34de782c07b282754ded4"
format_stamp: "Formatted at 2024-05-03 23:24:53 on dist-test-slave-jvbp"
I20240503 23:24:53.222618 400 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20240503 23:24:53.224644 1317 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 23:24:53.225060 400 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.001s
I20240503 23:24:53.225178 400 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZhIiPz/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714778688451250-400-0/minicluster-data/ts-3-root
uuid: "3790742adac34de782c07b282754ded4"
format_stamp: "Formatted at 2024-05-03 23:24:53 on dist-test-slave-jvbp"
I20240503 23:24:53.225332 400 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZhIiPz/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714778688451250-400-0/minicluster-data/ts-3-root
metadata directory: /tmp/dist-test-taskZhIiPz/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714778688451250-400-0/minicluster-data/ts-3-root
1 data directories: /tmp/dist-test-taskZhIiPz/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714778688451250-400-0/minicluster-data/ts-3-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 23:24:53.234553 400 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 23:24:53.235370 400 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 23:24:53.236109 400 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 23:24:53.237268 400 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 23:24:53.237365 400 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 23:24:53.237480 400 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 23:24:53.237571 400 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 23:24:53.260756 400 rpc_server.cc:307] RPC server started. Bound to: 127.0.100.4:34925
I20240503 23:24:53.260810 1379 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.100.4:34925 every 8 connection(s)
I20240503 23:24:53.270046 1380 heartbeater.cc:347] Connected to a master server at 127.0.100.62:35319
I20240503 23:24:53.270238 1380 heartbeater.cc:464] Registering TS with master...
I20240503 23:24:53.270725 1380 heartbeater.cc:511] Master 127.0.100.62:35319 requested a full tablet report, sending...
I20240503 23:24:53.271723 961 ts_manager.cc:194] Registered new tserver with Master: 3790742adac34de782c07b282754ded4 (127.0.100.4:34925)
I20240503 23:24:53.272800 961 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:53560
I20240503 23:24:53.374086 1016 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240503 23:24:53.385025 1065 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "26f3c79af13e40878328e02ae549ab0b"
dest_uuid: "5d37fd41fc2c41a99580cc475915a090"
mode: GRACEFUL
new_leader_uuid: "bf5fe0b6e639437498453c116a26f95c"
from {username='slave'} at 127.0.0.1:41550
I20240503 23:24:53.385378 1065 raft_consensus.cc:595] T 26f3c79af13e40878328e02ae549ab0b P 5d37fd41fc2c41a99580cc475915a090 [term 1 LEADER]: Received request to transfer leadership to TS bf5fe0b6e639437498453c116a26f95c
I20240503 23:24:53.386982 1065 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "6743c625d6854920b88764dec9ebd250"
dest_uuid: "5d37fd41fc2c41a99580cc475915a090"
mode: GRACEFUL
new_leader_uuid: "74775765c2ac4db693fa70319ffd98c9"
from {username='slave'} at 127.0.0.1:41550
I20240503 23:24:53.387198 1065 raft_consensus.cc:595] T 6743c625d6854920b88764dec9ebd250 P 5d37fd41fc2c41a99580cc475915a090 [term 1 LEADER]: Received request to transfer leadership to TS 74775765c2ac4db693fa70319ffd98c9
I20240503 23:24:53.388283 1065 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "cb8edce628bd466da6c9eaabf7964493"
dest_uuid: "5d37fd41fc2c41a99580cc475915a090"
mode: GRACEFUL
new_leader_uuid: "74775765c2ac4db693fa70319ffd98c9"
from {username='slave'} at 127.0.0.1:41550
I20240503 23:24:53.388470 1065 raft_consensus.cc:595] T cb8edce628bd466da6c9eaabf7964493 P 5d37fd41fc2c41a99580cc475915a090 [term 1 LEADER]: Received request to transfer leadership to TS 74775765c2ac4db693fa70319ffd98c9
I20240503 23:24:53.388959 1016 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 3
I20240503 23:24:53.389333 1016 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240503 23:24:53.396767 1279 consensus_queue.cc:1035] T 26f3c79af13e40878328e02ae549ab0b P 5d37fd41fc2c41a99580cc475915a090 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bf5fe0b6e639437498453c116a26f95c" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 40377 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 23:24:53.400143 1264 raft_consensus.cc:979] T 26f3c79af13e40878328e02ae549ab0b P 5d37fd41fc2c41a99580cc475915a090: : Instructing follower bf5fe0b6e639437498453c116a26f95c to start an election
I20240503 23:24:53.400415 1264 raft_consensus.cc:1067] T 26f3c79af13e40878328e02ae549ab0b P 5d37fd41fc2c41a99580cc475915a090 [term 1 LEADER]: Signalling peer bf5fe0b6e639437498453c116a26f95c to start an election
I20240503 23:24:53.402426 1213 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "26f3c79af13e40878328e02ae549ab0b"
dest_uuid: "bf5fe0b6e639437498453c116a26f95c"
from {username='slave'} at 127.0.0.1:60194
I20240503 23:24:53.404368 1213 raft_consensus.cc:482] T 26f3c79af13e40878328e02ae549ab0b P bf5fe0b6e639437498453c116a26f95c [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 23:24:53.404901 1213 raft_consensus.cc:3037] T 26f3c79af13e40878328e02ae549ab0b P bf5fe0b6e639437498453c116a26f95c [term 1 FOLLOWER]: Advancing to term 2
I20240503 23:24:53.407672 1213 raft_consensus.cc:504] T 26f3c79af13e40878328e02ae549ab0b P bf5fe0b6e639437498453c116a26f95c [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d37fd41fc2c41a99580cc475915a090" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 38605 } } peers { permanent_uuid: "74775765c2ac4db693fa70319ffd98c9" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 38529 } } peers { permanent_uuid: "bf5fe0b6e639437498453c116a26f95c" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 40377 } }
I20240503 23:24:53.409018 1139 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "26f3c79af13e40878328e02ae549ab0b" candidate_uuid: "bf5fe0b6e639437498453c116a26f95c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "74775765c2ac4db693fa70319ffd98c9"
I20240503 23:24:53.409049 1065 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "26f3c79af13e40878328e02ae549ab0b" candidate_uuid: "bf5fe0b6e639437498453c116a26f95c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "5d37fd41fc2c41a99580cc475915a090"
I20240503 23:24:53.409307 1139 raft_consensus.cc:3037] T 26f3c79af13e40878328e02ae549ab0b P 74775765c2ac4db693fa70319ffd98c9 [term 1 FOLLOWER]: Advancing to term 2
I20240503 23:24:53.410441 1213 leader_election.cc:288] T 26f3c79af13e40878328e02ae549ab0b P bf5fe0b6e639437498453c116a26f95c [CANDIDATE]: Term 2 election: Requested vote from peers 5d37fd41fc2c41a99580cc475915a090 (127.0.100.1:38605), 74775765c2ac4db693fa70319ffd98c9 (127.0.100.2:38529)
I20240503 23:24:53.411708 1065 raft_consensus.cc:3032] T 26f3c79af13e40878328e02ae549ab0b P 5d37fd41fc2c41a99580cc475915a090 [term 1 LEADER]: Stepping down as leader of term 1
I20240503 23:24:53.411862 1065 raft_consensus.cc:729] T 26f3c79af13e40878328e02ae549ab0b P 5d37fd41fc2c41a99580cc475915a090 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 5d37fd41fc2c41a99580cc475915a090, State: Running, Role: LEADER
I20240503 23:24:53.412074 1139 raft_consensus.cc:2454] T 26f3c79af13e40878328e02ae549ab0b P 74775765c2ac4db693fa70319ffd98c9 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bf5fe0b6e639437498453c116a26f95c in term 2.
I20240503 23:24:53.412207 1065 consensus_queue.cc:260] T 26f3c79af13e40878328e02ae549ab0b P 5d37fd41fc2c41a99580cc475915a090 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d37fd41fc2c41a99580cc475915a090" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 38605 } } peers { permanent_uuid: "74775765c2ac4db693fa70319ffd98c9" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 38529 } } peers { permanent_uuid: "bf5fe0b6e639437498453c116a26f95c" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 40377 } }
I20240503 23:24:53.412534 1177 leader_election.cc:302] T 26f3c79af13e40878328e02ae549ab0b P bf5fe0b6e639437498453c116a26f95c [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 74775765c2ac4db693fa70319ffd98c9, bf5fe0b6e639437498453c116a26f95c; no voters:
I20240503 23:24:53.412707 1065 raft_consensus.cc:3037] T 26f3c79af13e40878328e02ae549ab0b P 5d37fd41fc2c41a99580cc475915a090 [term 1 FOLLOWER]: Advancing to term 2
I20240503 23:24:53.412932 1301 raft_consensus.cc:2781] T 26f3c79af13e40878328e02ae549ab0b P bf5fe0b6e639437498453c116a26f95c [term 2 FOLLOWER]: Leader election won for term 2
I20240503 23:24:53.413089 1301 raft_consensus.cc:686] T 26f3c79af13e40878328e02ae549ab0b P bf5fe0b6e639437498453c116a26f95c [term 2 LEADER]: Becoming Leader. State: Replica: bf5fe0b6e639437498453c116a26f95c, State: Running, Role: LEADER
I20240503 23:24:53.413386 1301 consensus_queue.cc:237] T 26f3c79af13e40878328e02ae549ab0b P bf5fe0b6e639437498453c116a26f95c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d37fd41fc2c41a99580cc475915a090" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 38605 } } peers { permanent_uuid: "74775765c2ac4db693fa70319ffd98c9" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 38529 } } peers { permanent_uuid: "bf5fe0b6e639437498453c116a26f95c" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 40377 } }
I20240503 23:24:53.415557 1065 raft_consensus.cc:2454] T 26f3c79af13e40878328e02ae549ab0b P 5d37fd41fc2c41a99580cc475915a090 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bf5fe0b6e639437498453c116a26f95c in term 2.
I20240503 23:24:53.416185 961 catalog_manager.cc:5484] T 26f3c79af13e40878328e02ae549ab0b P bf5fe0b6e639437498453c116a26f95c reported cstate change: term changed from 1 to 2, leader changed from 5d37fd41fc2c41a99580cc475915a090 (127.0.100.1) to bf5fe0b6e639437498453c116a26f95c (127.0.100.3). New cstate: current_term: 2 leader_uuid: "bf5fe0b6e639437498453c116a26f95c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d37fd41fc2c41a99580cc475915a090" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 38605 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "74775765c2ac4db693fa70319ffd98c9" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 38529 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bf5fe0b6e639437498453c116a26f95c" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 40377 } health_report { overall_health: HEALTHY } } }
I20240503 23:24:53.416664 1139 raft_consensus.cc:1226] T 26f3c79af13e40878328e02ae549ab0b P 74775765c2ac4db693fa70319ffd98c9 [term 2 FOLLOWER]: Rejecting Update request from peer 5d37fd41fc2c41a99580cc475915a090 for earlier term 1. Current term is 2. Ops: []
I20240503 23:24:53.463212 1301 consensus_queue.cc:1035] T f3d383cdf48b4e108844ff8a658b9a51 P bf5fe0b6e639437498453c116a26f95c [LEADER]: Connected to new peer: Peer: permanent_uuid: "5d37fd41fc2c41a99580cc475915a090" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 38605 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 23:24:53.464468 1301 raft_consensus.cc:482] T 966a756b35654ba0a0e272a0b6c7d7f6 P bf5fe0b6e639437498453c116a26f95c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 23:24:53.464726 1301 raft_consensus.cc:504] T 966a756b35654ba0a0e272a0b6c7d7f6 P bf5fe0b6e639437498453c116a26f95c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d37fd41fc2c41a99580cc475915a090" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 38605 } } peers { permanent_uuid: "74775765c2ac4db693fa70319ffd98c9" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 38529 } } peers { permanent_uuid: "bf5fe0b6e639437498453c116a26f95c" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 40377 } }
I20240503 23:24:53.465998 1064 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "966a756b35654ba0a0e272a0b6c7d7f6" candidate_uuid: "bf5fe0b6e639437498453c116a26f95c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5d37fd41fc2c41a99580cc475915a090" is_pre_election: true
I20240503 23:24:53.465493 1301 leader_election.cc:288] T 966a756b35654ba0a0e272a0b6c7d7f6 P bf5fe0b6e639437498453c116a26f95c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5d37fd41fc2c41a99580cc475915a090 (127.0.100.1:38605), 74775765c2ac4db693fa70319ffd98c9 (127.0.100.2:38529)
I20240503 23:24:53.466336 1064 raft_consensus.cc:2379] T 966a756b35654ba0a0e272a0b6c7d7f6 P 5d37fd41fc2c41a99580cc475915a090 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate bf5fe0b6e639437498453c116a26f95c in current term 1: Already voted for candidate 74775765c2ac4db693fa70319ffd98c9 in this term.
I20240503 23:24:53.466476 1139 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "966a756b35654ba0a0e272a0b6c7d7f6" candidate_uuid: "bf5fe0b6e639437498453c116a26f95c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "74775765c2ac4db693fa70319ffd98c9" is_pre_election: true
I20240503 23:24:53.466995 1177 leader_election.cc:302] T 966a756b35654ba0a0e272a0b6c7d7f6 P bf5fe0b6e639437498453c116a26f95c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bf5fe0b6e639437498453c116a26f95c; no voters: 5d37fd41fc2c41a99580cc475915a090, 74775765c2ac4db693fa70319ffd98c9
I20240503 23:24:53.467542 1300 raft_consensus.cc:3037] T 966a756b35654ba0a0e272a0b6c7d7f6 P bf5fe0b6e639437498453c116a26f95c [term 0 FOLLOWER]: Advancing to term 1
I20240503 23:24:53.471587 1300 raft_consensus.cc:2726] T 966a756b35654ba0a0e272a0b6c7d7f6 P bf5fe0b6e639437498453c116a26f95c [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 23:24:53.471690 1391 consensus_queue.cc:1035] T f3d383cdf48b4e108844ff8a658b9a51 P bf5fe0b6e639437498453c116a26f95c [LEADER]: Connected to new peer: Peer: permanent_uuid: "74775765c2ac4db693fa70319ffd98c9" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 38529 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 23:24:53.515076 1304 consensus_queue.cc:1035] T cb8edce628bd466da6c9eaabf7964493 P 5d37fd41fc2c41a99580cc475915a090 [LEADER]: Connected to new peer: Peer: permanent_uuid: "74775765c2ac4db693fa70319ffd98c9" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 38529 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 23:24:53.518307 1279 raft_consensus.cc:979] T cb8edce628bd466da6c9eaabf7964493 P 5d37fd41fc2c41a99580cc475915a090: : Instructing follower 74775765c2ac4db693fa70319ffd98c9 to start an election
I20240503 23:24:53.518560 1279 raft_consensus.cc:1067] T cb8edce628bd466da6c9eaabf7964493 P 5d37fd41fc2c41a99580cc475915a090 [term 1 LEADER]: Signalling peer 74775765c2ac4db693fa70319ffd98c9 to start an election
I20240503 23:24:53.519662 1140 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "cb8edce628bd466da6c9eaabf7964493"
dest_uuid: "74775765c2ac4db693fa70319ffd98c9"
from {username='slave'} at 127.0.0.1:55494
I20240503 23:24:53.519942 1140 raft_consensus.cc:482] T cb8edce628bd466da6c9eaabf7964493 P 74775765c2ac4db693fa70319ffd98c9 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 23:24:53.520083 1140 raft_consensus.cc:3037] T cb8edce628bd466da6c9eaabf7964493 P 74775765c2ac4db693fa70319ffd98c9 [term 1 FOLLOWER]: Advancing to term 2
I20240503 23:24:53.520571 1279 consensus_queue.cc:1035] T cb8edce628bd466da6c9eaabf7964493 P 5d37fd41fc2c41a99580cc475915a090 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bf5fe0b6e639437498453c116a26f95c" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 40377 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 23:24:53.522312 1391 consensus_queue.cc:1035] T 98b4ca061f7f4a43a13fbad78dac228a P bf5fe0b6e639437498453c116a26f95c [LEADER]: Connected to new peer: Peer: permanent_uuid: "5d37fd41fc2c41a99580cc475915a090" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 38605 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 23:24:53.523103 1140 raft_consensus.cc:504] T cb8edce628bd466da6c9eaabf7964493 P 74775765c2ac4db693fa70319ffd98c9 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d37fd41fc2c41a99580cc475915a090" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 38605 } } peers { permanent_uuid: "74775765c2ac4db693fa70319ffd98c9" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 38529 } } peers { permanent_uuid: "bf5fe0b6e639437498453c116a26f95c" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 40377 } }
I20240503 23:24:53.523500 1139 raft_consensus.cc:1226] T cb8edce628bd466da6c9eaabf7964493 P 74775765c2ac4db693fa70319ffd98c9 [term 2 FOLLOWER]: Rejecting Update request from peer 5d37fd41fc2c41a99580cc475915a090 for earlier term 1. Current term is 2. Ops: []
I20240503 23:24:53.524130 1140 leader_election.cc:288] T cb8edce628bd466da6c9eaabf7964493 P 74775765c2ac4db693fa70319ffd98c9 [CANDIDATE]: Term 2 election: Requested vote from peers 5d37fd41fc2c41a99580cc475915a090 (127.0.100.1:38605), bf5fe0b6e639437498453c116a26f95c (127.0.100.3:40377)
I20240503 23:24:53.525703 1279 consensus_queue.cc:1046] T cb8edce628bd466da6c9eaabf7964493 P 5d37fd41fc2c41a99580cc475915a090 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "74775765c2ac4db693fa70319ffd98c9" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 38529 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 0, Time since last communication: 0.001s
I20240503 23:24:53.526122 1279 raft_consensus.cc:3032] T cb8edce628bd466da6c9eaabf7964493 P 5d37fd41fc2c41a99580cc475915a090 [term 1 LEADER]: Stepping down as leader of term 1
I20240503 23:24:53.526307 1279 raft_consensus.cc:729] T cb8edce628bd466da6c9eaabf7964493 P 5d37fd41fc2c41a99580cc475915a090 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 5d37fd41fc2c41a99580cc475915a090, State: Running, Role: LEADER
I20240503 23:24:53.529929 1279 consensus_queue.cc:260] T cb8edce628bd466da6c9eaabf7964493 P 5d37fd41fc2c41a99580cc475915a090 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d37fd41fc2c41a99580cc475915a090" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 38605 } } peers { permanent_uuid: "74775765c2ac4db693fa70319ffd98c9" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 38529 } } peers { permanent_uuid: "bf5fe0b6e639437498453c116a26f95c" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 40377 } }
I20240503 23:24:53.530611 1213 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cb8edce628bd466da6c9eaabf7964493" candidate_uuid: "74775765c2ac4db693fa70319ffd98c9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "bf5fe0b6e639437498453c116a26f95c"
I20240503 23:24:53.531076 1213 raft_consensus.cc:3037] T cb8edce628bd466da6c9eaabf7964493 P bf5fe0b6e639437498453c116a26f95c [term 1 FOLLOWER]: Advancing to term 2
I20240503 23:24:53.530637 1065 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cb8edce628bd466da6c9eaabf7964493" candidate_uuid: "74775765c2ac4db693fa70319ffd98c9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "5d37fd41fc2c41a99580cc475915a090"
I20240503 23:24:53.532349 1286 consensus_queue.cc:1035] T 966a756b35654ba0a0e272a0b6c7d7f6 P 74775765c2ac4db693fa70319ffd98c9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bf5fe0b6e639437498453c116a26f95c" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 40377 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 23:24:53.533365 1279 raft_consensus.cc:3037] T cb8edce628bd466da6c9eaabf7964493 P 5d37fd41fc2c41a99580cc475915a090 [term 1 FOLLOWER]: Advancing to term 2
I20240503 23:24:53.534621 1213 raft_consensus.cc:2454] T cb8edce628bd466da6c9eaabf7964493 P bf5fe0b6e639437498453c116a26f95c [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 74775765c2ac4db693fa70319ffd98c9 in term 2.
I20240503 23:24:53.535421 1105 leader_election.cc:302] T cb8edce628bd466da6c9eaabf7964493 P 74775765c2ac4db693fa70319ffd98c9 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 74775765c2ac4db693fa70319ffd98c9, bf5fe0b6e639437498453c116a26f95c; no voters:
I20240503 23:24:53.535909 1286 raft_consensus.cc:2781] T cb8edce628bd466da6c9eaabf7964493 P 74775765c2ac4db693fa70319ffd98c9 [term 2 FOLLOWER]: Leader election won for term 2
I20240503 23:24:53.540653 1065 raft_consensus.cc:2454] T cb8edce628bd466da6c9eaabf7964493 P 5d37fd41fc2c41a99580cc475915a090 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 74775765c2ac4db693fa70319ffd98c9 in term 2.
I20240503 23:24:53.540813 1301 consensus_queue.cc:1035] T 98b4ca061f7f4a43a13fbad78dac228a P bf5fe0b6e639437498453c116a26f95c [LEADER]: Connected to new peer: Peer: permanent_uuid: "74775765c2ac4db693fa70319ffd98c9" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 38529 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 23:24:53.541633 1286 raft_consensus.cc:686] T cb8edce628bd466da6c9eaabf7964493 P 74775765c2ac4db693fa70319ffd98c9 [term 2 LEADER]: Becoming Leader. State: Replica: 74775765c2ac4db693fa70319ffd98c9, State: Running, Role: LEADER
I20240503 23:24:53.542021 1286 consensus_queue.cc:237] T cb8edce628bd466da6c9eaabf7964493 P 74775765c2ac4db693fa70319ffd98c9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d37fd41fc2c41a99580cc475915a090" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 38605 } } peers { permanent_uuid: "74775765c2ac4db693fa70319ffd98c9" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 38529 } } peers { permanent_uuid: "bf5fe0b6e639437498453c116a26f95c" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 40377 } }
I20240503 23:24:53.542447 1279 consensus_queue.cc:1035] T 1e6b13b222534155bc03878359c2eb3b P 5d37fd41fc2c41a99580cc475915a090 [LEADER]: Connected to new peer: Peer: permanent_uuid: "74775765c2ac4db693fa70319ffd98c9" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 38529 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 23:24:53.545995 963 catalog_manager.cc:5484] T cb8edce628bd466da6c9eaabf7964493 P 74775765c2ac4db693fa70319ffd98c9 reported cstate change: term changed from 1 to 2, leader changed from 5d37fd41fc2c41a99580cc475915a090 (127.0.100.1) to 74775765c2ac4db693fa70319ffd98c9 (127.0.100.2). New cstate: current_term: 2 leader_uuid: "74775765c2ac4db693fa70319ffd98c9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d37fd41fc2c41a99580cc475915a090" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 38605 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "74775765c2ac4db693fa70319ffd98c9" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 38529 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bf5fe0b6e639437498453c116a26f95c" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 40377 } health_report { overall_health: UNKNOWN } } }
I20240503 23:24:53.554970 1398 consensus_queue.cc:1035] T 966a756b35654ba0a0e272a0b6c7d7f6 P 74775765c2ac4db693fa70319ffd98c9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5d37fd41fc2c41a99580cc475915a090" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 38605 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 23:24:53.556823 1264 consensus_queue.cc:1035] T 1e6b13b222534155bc03878359c2eb3b P 5d37fd41fc2c41a99580cc475915a090 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bf5fe0b6e639437498453c116a26f95c" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 40377 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 23:24:53.615186 1279 consensus_queue.cc:1035] T 6743c625d6854920b88764dec9ebd250 P 5d37fd41fc2c41a99580cc475915a090 [LEADER]: Connected to new peer: Peer: permanent_uuid: "74775765c2ac4db693fa70319ffd98c9" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 38529 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 23:24:53.618494 1264 raft_consensus.cc:979] T 6743c625d6854920b88764dec9ebd250 P 5d37fd41fc2c41a99580cc475915a090: : Instructing follower 74775765c2ac4db693fa70319ffd98c9 to start an election
I20240503 23:24:53.618701 1264 raft_consensus.cc:1067] T 6743c625d6854920b88764dec9ebd250 P 5d37fd41fc2c41a99580cc475915a090 [term 1 LEADER]: Signalling peer 74775765c2ac4db693fa70319ffd98c9 to start an election
I20240503 23:24:53.619431 1138 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "6743c625d6854920b88764dec9ebd250"
dest_uuid: "74775765c2ac4db693fa70319ffd98c9"
from {username='slave'} at 127.0.0.1:55494
I20240503 23:24:53.620085 1138 raft_consensus.cc:482] T 6743c625d6854920b88764dec9ebd250 P 74775765c2ac4db693fa70319ffd98c9 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 23:24:53.620249 1138 raft_consensus.cc:3037] T 6743c625d6854920b88764dec9ebd250 P 74775765c2ac4db693fa70319ffd98c9 [term 1 FOLLOWER]: Advancing to term 2
I20240503 23:24:53.620935 1264 consensus_queue.cc:1035] T 6743c625d6854920b88764dec9ebd250 P 5d37fd41fc2c41a99580cc475915a090 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bf5fe0b6e639437498453c116a26f95c" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 40377 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 23:24:53.623867 1138 raft_consensus.cc:504] T 6743c625d6854920b88764dec9ebd250 P 74775765c2ac4db693fa70319ffd98c9 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d37fd41fc2c41a99580cc475915a090" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 38605 } } peers { permanent_uuid: "74775765c2ac4db693fa70319ffd98c9" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 38529 } } peers { permanent_uuid: "bf5fe0b6e639437498453c116a26f95c" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 40377 } }
I20240503 23:24:53.625233 1065 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6743c625d6854920b88764dec9ebd250" candidate_uuid: "74775765c2ac4db693fa70319ffd98c9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "5d37fd41fc2c41a99580cc475915a090"
I20240503 23:24:53.625494 1065 raft_consensus.cc:3032] T 6743c625d6854920b88764dec9ebd250 P 5d37fd41fc2c41a99580cc475915a090 [term 1 LEADER]: Stepping down as leader of term 1
I20240503 23:24:53.625617 1065 raft_consensus.cc:729] T 6743c625d6854920b88764dec9ebd250 P 5d37fd41fc2c41a99580cc475915a090 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 5d37fd41fc2c41a99580cc475915a090, State: Running, Role: LEADER
I20240503 23:24:53.625895 1065 consensus_queue.cc:260] T 6743c625d6854920b88764dec9ebd250 P 5d37fd41fc2c41a99580cc475915a090 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d37fd41fc2c41a99580cc475915a090" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 38605 } } peers { permanent_uuid: "74775765c2ac4db693fa70319ffd98c9" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 38529 } } peers { permanent_uuid: "bf5fe0b6e639437498453c116a26f95c" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 40377 } }
I20240503 23:24:53.626255 1065 raft_consensus.cc:3037] T 6743c625d6854920b88764dec9ebd250 P 5d37fd41fc2c41a99580cc475915a090 [term 1 FOLLOWER]: Advancing to term 2
I20240503 23:24:53.628648 1065 raft_consensus.cc:2454] T 6743c625d6854920b88764dec9ebd250 P 5d37fd41fc2c41a99580cc475915a090 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 74775765c2ac4db693fa70319ffd98c9 in term 2.
I20240503 23:24:53.625232 1138 leader_election.cc:288] T 6743c625d6854920b88764dec9ebd250 P 74775765c2ac4db693fa70319ffd98c9 [CANDIDATE]: Term 2 election: Requested vote from peers 5d37fd41fc2c41a99580cc475915a090 (127.0.100.1:38605), bf5fe0b6e639437498453c116a26f95c (127.0.100.3:40377)
I20240503 23:24:53.629114 1104 leader_election.cc:302] T 6743c625d6854920b88764dec9ebd250 P 74775765c2ac4db693fa70319ffd98c9 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5d37fd41fc2c41a99580cc475915a090, 74775765c2ac4db693fa70319ffd98c9; no voters:
I20240503 23:24:53.629216 1214 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6743c625d6854920b88764dec9ebd250" candidate_uuid: "74775765c2ac4db693fa70319ffd98c9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "bf5fe0b6e639437498453c116a26f95c"
I20240503 23:24:53.629451 1398 raft_consensus.cc:2781] T 6743c625d6854920b88764dec9ebd250 P 74775765c2ac4db693fa70319ffd98c9 [term 2 FOLLOWER]: Leader election won for term 2
I20240503 23:24:53.629653 1398 raft_consensus.cc:686] T 6743c625d6854920b88764dec9ebd250 P 74775765c2ac4db693fa70319ffd98c9 [term 2 LEADER]: Becoming Leader. State: Replica: 74775765c2ac4db693fa70319ffd98c9, State: Running, Role: LEADER
I20240503 23:24:53.629979 1398 consensus_queue.cc:237] T 6743c625d6854920b88764dec9ebd250 P 74775765c2ac4db693fa70319ffd98c9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d37fd41fc2c41a99580cc475915a090" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 38605 } } peers { permanent_uuid: "74775765c2ac4db693fa70319ffd98c9" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 38529 } } peers { permanent_uuid: "bf5fe0b6e639437498453c116a26f95c" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 40377 } }
I20240503 23:24:53.629451 1214 raft_consensus.cc:3037] T 6743c625d6854920b88764dec9ebd250 P bf5fe0b6e639437498453c116a26f95c [term 1 FOLLOWER]: Advancing to term 2
I20240503 23:24:53.632477 963 catalog_manager.cc:5484] T 6743c625d6854920b88764dec9ebd250 P 74775765c2ac4db693fa70319ffd98c9 reported cstate change: term changed from 1 to 2, leader changed from 5d37fd41fc2c41a99580cc475915a090 (127.0.100.1) to 74775765c2ac4db693fa70319ffd98c9 (127.0.100.2). New cstate: current_term: 2 leader_uuid: "74775765c2ac4db693fa70319ffd98c9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d37fd41fc2c41a99580cc475915a090" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 38605 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "74775765c2ac4db693fa70319ffd98c9" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 38529 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bf5fe0b6e639437498453c116a26f95c" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 40377 } health_report { overall_health: UNKNOWN } } }
I20240503 23:24:53.633307 1214 raft_consensus.cc:2454] T 6743c625d6854920b88764dec9ebd250 P bf5fe0b6e639437498453c116a26f95c [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 74775765c2ac4db693fa70319ffd98c9 in term 2.
I20240503 23:24:53.659018 1264 consensus_queue.cc:1035] T 8d775506b875408d80a957dd3926aed8 P 5d37fd41fc2c41a99580cc475915a090 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bf5fe0b6e639437498453c116a26f95c" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 40377 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 23:24:53.663667 1279 consensus_queue.cc:1035] T 8d775506b875408d80a957dd3926aed8 P 5d37fd41fc2c41a99580cc475915a090 [LEADER]: Connected to new peer: Peer: permanent_uuid: "74775765c2ac4db693fa70319ffd98c9" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 38529 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 23:24:53.930744 1138 raft_consensus.cc:1261] T 26f3c79af13e40878328e02ae549ab0b P 74775765c2ac4db693fa70319ffd98c9 [term 2 FOLLOWER]: Refusing update from remote peer bf5fe0b6e639437498453c116a26f95c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240503 23:24:53.931659 1389 consensus_queue.cc:1035] T 26f3c79af13e40878328e02ae549ab0b P bf5fe0b6e639437498453c116a26f95c [LEADER]: Connected to new peer: Peer: permanent_uuid: "74775765c2ac4db693fa70319ffd98c9" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 38529 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 23:24:53.935585 1065 raft_consensus.cc:1261] T 26f3c79af13e40878328e02ae549ab0b P 5d37fd41fc2c41a99580cc475915a090 [term 2 FOLLOWER]: Refusing update from remote peer bf5fe0b6e639437498453c116a26f95c: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240503 23:24:53.936398 1391 consensus_queue.cc:1035] T 26f3c79af13e40878328e02ae549ab0b P bf5fe0b6e639437498453c116a26f95c [LEADER]: Connected to new peer: Peer: permanent_uuid: "5d37fd41fc2c41a99580cc475915a090" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 38605 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 23:24:53.950623 1065 raft_consensus.cc:1261] T cb8edce628bd466da6c9eaabf7964493 P 5d37fd41fc2c41a99580cc475915a090 [term 2 FOLLOWER]: Refusing update from remote peer 74775765c2ac4db693fa70319ffd98c9: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240503 23:24:53.951319 1398 consensus_queue.cc:1035] T cb8edce628bd466da6c9eaabf7964493 P 74775765c2ac4db693fa70319ffd98c9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5d37fd41fc2c41a99580cc475915a090" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 38605 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 23:24:53.955020 1214 raft_consensus.cc:1261] T cb8edce628bd466da6c9eaabf7964493 P bf5fe0b6e639437498453c116a26f95c [term 2 FOLLOWER]: Refusing update from remote peer 74775765c2ac4db693fa70319ffd98c9: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240503 23:24:53.955641 1398 consensus_queue.cc:1035] T cb8edce628bd466da6c9eaabf7964493 P 74775765c2ac4db693fa70319ffd98c9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bf5fe0b6e639437498453c116a26f95c" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 40377 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 23:24:54.131565 1214 raft_consensus.cc:1261] T 6743c625d6854920b88764dec9ebd250 P bf5fe0b6e639437498453c116a26f95c [term 2 FOLLOWER]: Refusing update from remote peer 74775765c2ac4db693fa70319ffd98c9: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240503 23:24:54.132169 1398 consensus_queue.cc:1035] T 6743c625d6854920b88764dec9ebd250 P 74775765c2ac4db693fa70319ffd98c9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bf5fe0b6e639437498453c116a26f95c" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 40377 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 23:24:54.135697 1065 raft_consensus.cc:1261] T 6743c625d6854920b88764dec9ebd250 P 5d37fd41fc2c41a99580cc475915a090 [term 2 FOLLOWER]: Refusing update from remote peer 74775765c2ac4db693fa70319ffd98c9: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240503 23:24:54.136302 1398 consensus_queue.cc:1035] T 6743c625d6854920b88764dec9ebd250 P 74775765c2ac4db693fa70319ffd98c9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5d37fd41fc2c41a99580cc475915a090" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 38605 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 23:24:54.274219 1380 heartbeater.cc:503] Master 127.0.100.62:35319 was elected leader, sending a full tablet report...
I20240503 23:24:55.389884 1016 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240503 23:24:55.391522 1016 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240503 23:24:55.391731 1016 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240503 23:24:57.392280 1016 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240503 23:24:57.395246 1016 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240503 23:24:57.395697 1016 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240503 23:24:58.406459 1065 consensus_queue.cc:237] T 1e6b13b222534155bc03878359c2eb3b P 5d37fd41fc2c41a99580cc475915a090 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5d37fd41fc2c41a99580cc475915a090" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 38605 } } peers { permanent_uuid: "74775765c2ac4db693fa70319ffd98c9" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 38529 } attrs { replace: true } } peers { permanent_uuid: "bf5fe0b6e639437498453c116a26f95c" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 40377 } } peers { permanent_uuid: "3790742adac34de782c07b282754ded4" member_type: NON_VOTER last_known_addr { host: "127.0.100.4" port: 34925 } attrs { promote: true } }
I20240503 23:24:58.411756 1138 raft_consensus.cc:1261] T 1e6b13b222534155bc03878359c2eb3b P 74775765c2ac4db693fa70319ffd98c9 [term 1 FOLLOWER]: Refusing update from remote peer 5d37fd41fc2c41a99580cc475915a090: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 23:24:58.412717 1214 raft_consensus.cc:1261] T 1e6b13b222534155bc03878359c2eb3b P bf5fe0b6e639437498453c116a26f95c [term 1 FOLLOWER]: Refusing update from remote peer 5d37fd41fc2c41a99580cc475915a090: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 23:24:58.412830 1279 consensus_queue.cc:1035] T 1e6b13b222534155bc03878359c2eb3b P 5d37fd41fc2c41a99580cc475915a090 [LEADER]: Connected to new peer: Peer: permanent_uuid: "74775765c2ac4db693fa70319ffd98c9" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 38529 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 23:24:58.413674 1279 consensus_queue.cc:1035] T 1e6b13b222534155bc03878359c2eb3b P 5d37fd41fc2c41a99580cc475915a090 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bf5fe0b6e639437498453c116a26f95c" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 40377 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 23:24:58.428359 1279 raft_consensus.cc:2932] T 1e6b13b222534155bc03878359c2eb3b P 5d37fd41fc2c41a99580cc475915a090 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 3790742adac34de782c07b282754ded4 (127.0.100.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5d37fd41fc2c41a99580cc475915a090" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 38605 } } peers { permanent_uuid: "74775765c2ac4db693fa70319ffd98c9" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 38529 } attrs { replace: true } } peers { permanent_uuid: "bf5fe0b6e639437498453c116a26f95c" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 40377 } } peers { permanent_uuid: "3790742adac34de782c07b282754ded4" member_type: NON_VOTER last_known_addr { host: "127.0.100.4" port: 34925 } attrs { promote: true } } }
I20240503 23:24:58.428903 1138 raft_consensus.cc:2932] T 1e6b13b222534155bc03878359c2eb3b P 74775765c2ac4db693fa70319ffd98c9 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 3790742adac34de782c07b282754ded4 (127.0.100.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5d37fd41fc2c41a99580cc475915a090" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 38605 } } peers { permanent_uuid: "74775765c2ac4db693fa70319ffd98c9" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 38529 } attrs { replace: true } } peers { permanent_uuid: "bf5fe0b6e639437498453c116a26f95c" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 40377 } } peers { permanent_uuid: "3790742adac34de782c07b282754ded4" member_type: NON_VOTER last_known_addr { host: "127.0.100.4" port: 34925 } attrs { promote: true } } }
I20240503 23:24:58.438318 962 catalog_manager.cc:5484] T 1e6b13b222534155bc03878359c2eb3b P 5d37fd41fc2c41a99580cc475915a090 reported cstate change: config changed from index -1 to 2, NON_VOTER 3790742adac34de782c07b282754ded4 (127.0.100.4) added. New cstate: current_term: 1 leader_uuid: "5d37fd41fc2c41a99580cc475915a090" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5d37fd41fc2c41a99580cc475915a090" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 38605 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "74775765c2ac4db693fa70319ffd98c9" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 38529 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bf5fe0b6e639437498453c116a26f95c" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 40377 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3790742adac34de782c07b282754ded4" member_type: NON_VOTER last_known_addr { host: "127.0.100.4" port: 34925 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20240503 23:24:58.439488 1030 consensus_peers.cc:485] T 1e6b13b222534155bc03878359c2eb3b P 5d37fd41fc2c41a99580cc475915a090 -> Peer 3790742adac34de782c07b282754ded4 (127.0.100.4:34925): Couldn't send request to peer 3790742adac34de782c07b282754ded4. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 1e6b13b222534155bc03878359c2eb3b. This is attempt 1: this message will repeat every 5th retry.
I20240503 23:24:58.456267 1214 raft_consensus.cc:2932] T 1e6b13b222534155bc03878359c2eb3b P bf5fe0b6e639437498453c116a26f95c [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 3790742adac34de782c07b282754ded4 (127.0.100.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5d37fd41fc2c41a99580cc475915a090" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 38605 } } peers { permanent_uuid: "74775765c2ac4db693fa70319ffd98c9" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 38529 } attrs { replace: true } } peers { permanent_uuid: "bf5fe0b6e639437498453c116a26f95c" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 40377 } } peers { permanent_uuid: "3790742adac34de782c07b282754ded4" member_type: NON_VOTER last_known_addr { host: "127.0.100.4" port: 34925 } attrs { promote: true } } }
I20240503 23:24:58.461596 1213 consensus_queue.cc:237] T 98b4ca061f7f4a43a13fbad78dac228a P bf5fe0b6e639437498453c116a26f95c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5d37fd41fc2c41a99580cc475915a090" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 38605 } attrs { replace: true } } peers { permanent_uuid: "74775765c2ac4db693fa70319ffd98c9" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 38529 } } peers { permanent_uuid: "bf5fe0b6e639437498453c116a26f95c" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 40377 } } peers { permanent_uuid: "3790742adac34de782c07b282754ded4" member_type: NON_VOTER last_known_addr { host: "127.0.100.4" port: 34925 } attrs { promote: true } }
I20240503 23:24:58.467644 1065 raft_consensus.cc:1261] T 98b4ca061f7f4a43a13fbad78dac228a P 5d37fd41fc2c41a99580cc475915a090 [term 1 FOLLOWER]: Refusing update from remote peer bf5fe0b6e639437498453c116a26f95c: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 23:24:58.468808 1301 consensus_queue.cc:1035] T 98b4ca061f7f4a43a13fbad78dac228a P bf5fe0b6e639437498453c116a26f95c [LEADER]: Connected to new peer: Peer: permanent_uuid: "5d37fd41fc2c41a99580cc475915a090" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 38605 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 23:24:58.475983 1301 raft_consensus.cc:2932] T 98b4ca061f7f4a43a13fbad78dac228a P bf5fe0b6e639437498453c116a26f95c [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 3790742adac34de782c07b282754ded4 (127.0.100.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5d37fd41fc2c41a99580cc475915a090" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 38605 } attrs { replace: true } } peers { permanent_uuid: "74775765c2ac4db693fa70319ffd98c9" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 38529 } } peers { permanent_uuid: "bf5fe0b6e639437498453c116a26f95c" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 40377 } } peers { permanent_uuid: "3790742adac34de782c07b282754ded4" member_type: NON_VOTER last_known_addr { host: "127.0.100.4" port: 34925 } attrs { promote: true } } }
I20240503 23:24:58.477953 1063 raft_consensus.cc:2932] T 98b4ca061f7f4a43a13fbad78dac228a P 5d37fd41fc2c41a99580cc475915a090 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 3790742adac34de782c07b282754ded4 (127.0.100.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5d37fd41fc2c41a99580cc475915a090" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 38605 } attrs { replace: true } } peers { permanent_uuid: "74775765c2ac4db693fa70319ffd98c9" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 38529 } } peers { permanent_uuid: "bf5fe0b6e639437498453c116a26f95c" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 40377 } } peers { permanent_uuid: "3790742adac34de782c07b282754ded4" member_type: NON_VOTER last_known_addr { host: "127.0.100.4" port: 34925 } attrs { promote: true } } }
I20240503 23:24:58.485265 1138 raft_consensus.cc:1261] T 98b4ca061f7f4a43a13fbad78dac228a P 74775765c2ac4db693fa70319ffd98c9 [term 1 FOLLOWER]: Refusing update from remote peer bf5fe0b6e639437498453c116a26f95c: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240503 23:24:58.487628 1179 consensus_peers.cc:485] T 98b4ca061f7f4a43a13fbad78dac228a P bf5fe0b6e639437498453c116a26f95c -> Peer 3790742adac34de782c07b282754ded4 (127.0.100.4:34925): Couldn't send request to peer 3790742adac34de782c07b282754ded4. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 98b4ca061f7f4a43a13fbad78dac228a. This is attempt 1: this message will repeat every 5th retry.
I20240503 23:24:58.490433 1428 consensus_queue.cc:1035] T 98b4ca061f7f4a43a13fbad78dac228a P bf5fe0b6e639437498453c116a26f95c [LEADER]: Connected to new peer: Peer: permanent_uuid: "74775765c2ac4db693fa70319ffd98c9" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 38529 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 23:24:58.489898 963 catalog_manager.cc:5484] T 98b4ca061f7f4a43a13fbad78dac228a P 5d37fd41fc2c41a99580cc475915a090 reported cstate change: config changed from index -1 to 2, NON_VOTER 3790742adac34de782c07b282754ded4 (127.0.100.4) added. New cstate: current_term: 1 leader_uuid: "bf5fe0b6e639437498453c116a26f95c" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5d37fd41fc2c41a99580cc475915a090" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 38605 } attrs { replace: true } } peers { permanent_uuid: "74775765c2ac4db693fa70319ffd98c9" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 38529 } } peers { permanent_uuid: "bf5fe0b6e639437498453c116a26f95c" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 40377 } } peers { permanent_uuid: "3790742adac34de782c07b282754ded4" member_type: NON_VOTER last_known_addr { host: "127.0.100.4" port: 34925 } attrs { promote: true } } }
I20240503 23:24:58.495769 1063 consensus_queue.cc:237] T 8d775506b875408d80a957dd3926aed8 P 5d37fd41fc2c41a99580cc475915a090 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5d37fd41fc2c41a99580cc475915a090" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 38605 } } peers { permanent_uuid: "74775765c2ac4db693fa70319ffd98c9" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 38529 } } peers { permanent_uuid: "bf5fe0b6e639437498453c116a26f95c" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 40377 } attrs { replace: true } } peers { permanent_uuid: "3790742adac34de782c07b282754ded4" member_type: NON_VOTER last_known_addr { host: "127.0.100.4" port: 34925 } attrs { promote: true } }
I20240503 23:24:58.500355 1139 raft_consensus.cc:1261] T 8d775506b875408d80a957dd3926aed8 P 74775765c2ac4db693fa70319ffd98c9 [term 1 FOLLOWER]: Refusing update from remote peer 5d37fd41fc2c41a99580cc475915a090: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240503 23:24:58.501117 1030 consensus_peers.cc:485] T 8d775506b875408d80a957dd3926aed8 P 5d37fd41fc2c41a99580cc475915a090 -> Peer 3790742adac34de782c07b282754ded4 (127.0.100.4:34925): Couldn't send request to peer 3790742adac34de782c07b282754ded4. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 8d775506b875408d80a957dd3926aed8. This is attempt 1: this message will repeat every 5th retry.
I20240503 23:24:58.501544 1420 consensus_queue.cc:1035] T 8d775506b875408d80a957dd3926aed8 P 5d37fd41fc2c41a99580cc475915a090 [LEADER]: Connected to new peer: Peer: permanent_uuid: "74775765c2ac4db693fa70319ffd98c9" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 38529 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 23:24:58.502606 1213 raft_consensus.cc:1261] T 8d775506b875408d80a957dd3926aed8 P bf5fe0b6e639437498453c116a26f95c [term 1 FOLLOWER]: Refusing update from remote peer 5d37fd41fc2c41a99580cc475915a090: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 23:24:58.503527 1420 consensus_queue.cc:1035] T 8d775506b875408d80a957dd3926aed8 P 5d37fd41fc2c41a99580cc475915a090 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bf5fe0b6e639437498453c116a26f95c" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 40377 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 23:24:58.512650 1138 raft_consensus.cc:2932] T 98b4ca061f7f4a43a13fbad78dac228a P 74775765c2ac4db693fa70319ffd98c9 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 3790742adac34de782c07b282754ded4 (127.0.100.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5d37fd41fc2c41a99580cc475915a090" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 38605 } attrs { replace: true } } peers { permanent_uuid: "74775765c2ac4db693fa70319ffd98c9" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 38529 } } peers { permanent_uuid: "bf5fe0b6e639437498453c116a26f95c" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 40377 } } peers { permanent_uuid: "3790742adac34de782c07b282754ded4" member_type: NON_VOTER last_known_addr { host: "127.0.100.4" port: 34925 } attrs { promote: true } } }
I20240503 23:24:58.522096 1414 raft_consensus.cc:2932] T 8d775506b875408d80a957dd3926aed8 P 5d37fd41fc2c41a99580cc475915a090 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 3790742adac34de782c07b282754ded4 (127.0.100.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5d37fd41fc2c41a99580cc475915a090" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 38605 } } peers { permanent_uuid: "74775765c2ac4db693fa70319ffd98c9" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 38529 } } peers { permanent_uuid: "bf5fe0b6e639437498453c116a26f95c" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 40377 } attrs { replace: true } } peers { permanent_uuid: "3790742adac34de782c07b282754ded4" member_type: NON_VOTER last_known_addr { host: "127.0.100.4" port: 34925 } attrs { promote: true } } }
I20240503 23:24:58.523051 1213 raft_consensus.cc:2932] T 8d775506b875408d80a957dd3926aed8 P bf5fe0b6e639437498453c116a26f95c [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 3790742adac34de782c07b282754ded4 (127.0.100.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5d37fd41fc2c41a99580cc475915a090" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 38605 } } peers { permanent_uuid: "74775765c2ac4db693fa70319ffd98c9" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 38529 } } peers { permanent_uuid: "bf5fe0b6e639437498453c116a26f95c" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 40377 } attrs { replace: true } } peers { permanent_uuid: "3790742adac34de782c07b282754ded4" member_type: NON_VOTER last_known_addr { host: "127.0.100.4" port: 34925 } attrs { promote: true } } }
I20240503 23:24:58.523821 1139 raft_consensus.cc:2932] T 8d775506b875408d80a957dd3926aed8 P 74775765c2ac4db693fa70319ffd98c9 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 3790742adac34de782c07b282754ded4 (127.0.100.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5d37fd41fc2c41a99580cc475915a090" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 38605 } } peers { permanent_uuid: "74775765c2ac4db693fa70319ffd98c9" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 38529 } } peers { permanent_uuid: "bf5fe0b6e639437498453c116a26f95c" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 40377 } attrs { replace: true } } peers { permanent_uuid: "3790742adac34de782c07b282754ded4" member_type: NON_VOTER last_known_addr { host: "127.0.100.4" port: 34925 } attrs { promote: true } } }
I20240503 23:24:58.530071 1214 consensus_queue.cc:237] T 26f3c79af13e40878328e02ae549ab0b P bf5fe0b6e639437498453c116a26f95c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "5d37fd41fc2c41a99580cc475915a090" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 38605 } } peers { permanent_uuid: "74775765c2ac4db693fa70319ffd98c9" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 38529 } attrs { replace: true } } peers { permanent_uuid: "bf5fe0b6e639437498453c116a26f95c" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 40377 } } peers { permanent_uuid: "3790742adac34de782c07b282754ded4" member_type: NON_VOTER last_known_addr { host: "127.0.100.4" port: 34925 } attrs { promote: true } }
I20240503 23:24:58.530061 963 catalog_manager.cc:5484] T 8d775506b875408d80a957dd3926aed8 P 5d37fd41fc2c41a99580cc475915a090 reported cstate change: config changed from index -1 to 2, NON_VOTER 3790742adac34de782c07b282754ded4 (127.0.100.4) added. New cstate: current_term: 1 leader_uuid: "5d37fd41fc2c41a99580cc475915a090" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5d37fd41fc2c41a99580cc475915a090" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 38605 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "74775765c2ac4db693fa70319ffd98c9" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 38529 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bf5fe0b6e639437498453c116a26f95c" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 40377 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3790742adac34de782c07b282754ded4" member_type: NON_VOTER last_known_addr { host: "127.0.100.4" port: 34925 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 23:24:58.545074 1063 raft_consensus.cc:1261] T 26f3c79af13e40878328e02ae549ab0b P 5d37fd41fc2c41a99580cc475915a090 [term 2 FOLLOWER]: Refusing update from remote peer bf5fe0b6e639437498453c116a26f95c: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20240503 23:24:58.545123 1139 raft_consensus.cc:1261] T 26f3c79af13e40878328e02ae549ab0b P 74775765c2ac4db693fa70319ffd98c9 [term 2 FOLLOWER]: Refusing update from remote peer bf5fe0b6e639437498453c116a26f95c: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
W20240503 23:24:58.545662 1179 consensus_peers.cc:485] T 26f3c79af13e40878328e02ae549ab0b P bf5fe0b6e639437498453c116a26f95c -> Peer 3790742adac34de782c07b282754ded4 (127.0.100.4:34925): Couldn't send request to peer 3790742adac34de782c07b282754ded4. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 26f3c79af13e40878328e02ae549ab0b. This is attempt 1: this message will repeat every 5th retry.
I20240503 23:24:58.545838 1427 consensus_queue.cc:1035] T 26f3c79af13e40878328e02ae549ab0b P bf5fe0b6e639437498453c116a26f95c [LEADER]: Connected to new peer: Peer: permanent_uuid: "5d37fd41fc2c41a99580cc475915a090" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 38605 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 23:24:58.546346 1301 consensus_queue.cc:1035] T 26f3c79af13e40878328e02ae549ab0b P bf5fe0b6e639437498453c116a26f95c [LEADER]: Connected to new peer: Peer: permanent_uuid: "74775765c2ac4db693fa70319ffd98c9" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 38529 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 23:24:58.550037 1301 raft_consensus.cc:2932] T 26f3c79af13e40878328e02ae549ab0b P bf5fe0b6e639437498453c116a26f95c [term 2 LEADER]: Committing config change with OpId 2.3: config changed from index -1 to 3, NON_VOTER 3790742adac34de782c07b282754ded4 (127.0.100.4) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "5d37fd41fc2c41a99580cc475915a090" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 38605 } } peers { permanent_uuid: "74775765c2ac4db693fa70319ffd98c9" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 38529 } attrs { replace: true } } peers { permanent_uuid: "bf5fe0b6e639437498453c116a26f95c" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 40377 } } peers { permanent_uuid: "3790742adac34de782c07b282754ded4" member_type: NON_VOTER last_known_addr { host: "127.0.100.4" port: 34925 } attrs { promote: true } } }
I20240503 23:24:58.551028 1139 raft_consensus.cc:2932] T 26f3c79af13e40878328e02ae549ab0b P 74775765c2ac4db693fa70319ffd98c9 [term 2 FOLLOWER]: Committing config change with OpId 2.3: config changed from index -1 to 3, NON_VOTER 3790742adac34de782c07b282754ded4 (127.0.100.4) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "5d37fd41fc2c41a99580cc475915a090" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 38605 } } peers { permanent_uuid: "74775765c2ac4db693fa70319ffd98c9" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 38529 } attrs { replace: true } } peers { permanent_uuid: "bf5fe0b6e639437498453c116a26f95c" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 40377 } } peers { permanent_uuid: "3790742adac34de782c07b282754ded4" member_type: NON_VOTER last_known_addr { host: "127.0.100.4" port: 34925 } attrs { promote: true } } }
I20240503 23:24:58.555770 1063 raft_consensus.cc:2932] T 26f3c79af13e40878328e02ae549ab0b P 5d37fd41fc2c41a99580cc475915a090 [term 2 FOLLOWER]: Committing config change with OpId 2.3: config changed from index -1 to 3, NON_VOTER 3790742adac34de782c07b282754ded4 (127.0.100.4) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "5d37fd41fc2c41a99580cc475915a090" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 38605 } } peers { permanent_uuid: "74775765c2ac4db693fa70319ffd98c9" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 38529 } attrs { replace: true } } peers { permanent_uuid: "bf5fe0b6e639437498453c116a26f95c" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 40377 } } peers { permanent_uuid: "3790742adac34de782c07b282754ded4" member_type: NON_VOTER last_known_addr { host: "127.0.100.4" port: 34925 } attrs { promote: true } } }
I20240503 23:24:58.563736 961 catalog_manager.cc:5484] T 26f3c79af13e40878328e02ae549ab0b P bf5fe0b6e639437498453c116a26f95c reported cstate change: config changed from index -1 to 3, NON_VOTER 3790742adac34de782c07b282754ded4 (127.0.100.4) added. New cstate: current_term: 2 leader_uuid: "bf5fe0b6e639437498453c116a26f95c" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "5d37fd41fc2c41a99580cc475915a090" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 38605 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "74775765c2ac4db693fa70319ffd98c9" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 38529 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bf5fe0b6e639437498453c116a26f95c" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 40377 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3790742adac34de782c07b282754ded4" member_type: NON_VOTER last_known_addr { host: "127.0.100.4" port: 34925 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 23:24:58.797861 400 tablet_server.cc:178] TabletServer@127.0.100.1:0 shutting down...
W20240503 23:24:58.805580 948 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer [suppressed 2 similar messages]
W20240503 23:24:58.808507 1011 auto_rebalancer.cc:663] Could not move replica: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20240503 23:24:58.808709 1011 auto_rebalancer.cc:264] scheduled replica move failed to complete: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20240503 23:24:58.811367 1437 ts_tablet_manager.cc:902] T 1e6b13b222534155bc03878359c2eb3b P 3790742adac34de782c07b282754ded4: Initiating tablet copy from peer 5d37fd41fc2c41a99580cc475915a090 (127.0.100.1:38605)
W20240503 23:24:58.811470 1011 auto_rebalancer.cc:663] Could not move replica: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20240503 23:24:58.811733 1011 auto_rebalancer.cc:264] scheduled replica move failed to complete: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20240503 23:24:58.815711 400 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240503 23:24:58.816720 400 tablet_replica.cc:330] T 6743c625d6854920b88764dec9ebd250 P 5d37fd41fc2c41a99580cc475915a090: stopping tablet replica
I20240503 23:24:58.817171 400 raft_consensus.cc:2229] T 6743c625d6854920b88764dec9ebd250 P 5d37fd41fc2c41a99580cc475915a090 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 23:24:58.817647 400 raft_consensus.cc:2258] T 6743c625d6854920b88764dec9ebd250 P 5d37fd41fc2c41a99580cc475915a090 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 23:24:58.818871 400 tablet_replica.cc:330] T 26f3c79af13e40878328e02ae549ab0b P 5d37fd41fc2c41a99580cc475915a090: stopping tablet replica
I20240503 23:24:58.819300 400 raft_consensus.cc:2229] T 26f3c79af13e40878328e02ae549ab0b P 5d37fd41fc2c41a99580cc475915a090 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 23:24:58.819778 400 raft_consensus.cc:2258] T 26f3c79af13e40878328e02ae549ab0b P 5d37fd41fc2c41a99580cc475915a090 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 23:24:58.821503 400 tablet_replica.cc:330] T 1e6b13b222534155bc03878359c2eb3b P 5d37fd41fc2c41a99580cc475915a090: stopping tablet replica
I20240503 23:24:58.821892 400 raft_consensus.cc:2229] T 1e6b13b222534155bc03878359c2eb3b P 5d37fd41fc2c41a99580cc475915a090 [term 1 LEADER]: Raft consensus shutting down.
I20240503 23:24:58.822609 400 raft_consensus.cc:2258] T 1e6b13b222534155bc03878359c2eb3b P 5d37fd41fc2c41a99580cc475915a090 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 23:24:58.823904 400 tablet_replica.cc:330] T f3d383cdf48b4e108844ff8a658b9a51 P 5d37fd41fc2c41a99580cc475915a090: stopping tablet replica
I20240503 23:24:58.824291 400 raft_consensus.cc:2229] T f3d383cdf48b4e108844ff8a658b9a51 P 5d37fd41fc2c41a99580cc475915a090 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 23:24:58.824638 400 raft_consensus.cc:2258] T f3d383cdf48b4e108844ff8a658b9a51 P 5d37fd41fc2c41a99580cc475915a090 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 23:24:58.827459 400 tablet_replica.cc:330] T 98b4ca061f7f4a43a13fbad78dac228a P 5d37fd41fc2c41a99580cc475915a090: stopping tablet replica
I20240503 23:24:58.827853 400 raft_consensus.cc:2229] T 98b4ca061f7f4a43a13fbad78dac228a P 5d37fd41fc2c41a99580cc475915a090 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 23:24:58.828264 400 raft_consensus.cc:2258] T 98b4ca061f7f4a43a13fbad78dac228a P 5d37fd41fc2c41a99580cc475915a090 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 23:24:58.829366 400 tablet_replica.cc:330] T 966a756b35654ba0a0e272a0b6c7d7f6 P 5d37fd41fc2c41a99580cc475915a090: stopping tablet replica
I20240503 23:24:58.829656 400 raft_consensus.cc:2229] T 966a756b35654ba0a0e272a0b6c7d7f6 P 5d37fd41fc2c41a99580cc475915a090 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 23:24:58.829869 400 raft_consensus.cc:2258] T 966a756b35654ba0a0e272a0b6c7d7f6 P 5d37fd41fc2c41a99580cc475915a090 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 23:24:58.830668 400 tablet_replica.cc:330] T cb8edce628bd466da6c9eaabf7964493 P 5d37fd41fc2c41a99580cc475915a090: stopping tablet replica
I20240503 23:24:58.830886 400 raft_consensus.cc:2229] T cb8edce628bd466da6c9eaabf7964493 P 5d37fd41fc2c41a99580cc475915a090 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 23:24:58.831126 400 raft_consensus.cc:2258] T cb8edce628bd466da6c9eaabf7964493 P 5d37fd41fc2c41a99580cc475915a090 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 23:24:58.831817 400 tablet_replica.cc:330] T 8d775506b875408d80a957dd3926aed8 P 5d37fd41fc2c41a99580cc475915a090: stopping tablet replica
I20240503 23:24:58.832060 400 raft_consensus.cc:2229] T 8d775506b875408d80a957dd3926aed8 P 5d37fd41fc2c41a99580cc475915a090 [term 1 LEADER]: Raft consensus shutting down.
I20240503 23:24:58.832911 400 raft_consensus.cc:2258] T 8d775506b875408d80a957dd3926aed8 P 5d37fd41fc2c41a99580cc475915a090 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 23:24:58.834605 1437 tablet_copy_client.cc:316] T 1e6b13b222534155bc03878359c2eb3b P 3790742adac34de782c07b282754ded4: tablet copy: Beginning tablet copy session from remote peer at address 127.0.100.1:38605
W20240503 23:24:58.853744 1318 connection.cc:441] server connection from 127.0.0.1:35718 torn down before Call kudu.consensus.ConsensusService.StartTabletCopy from 127.0.0.1:35718 (request call id 2) could send its response
I20240503 23:24:58.859423 400 tablet_server.cc:194] TabletServer@127.0.100.1:0 shutdown complete.
I20240503 23:24:58.865468 400 tablet_server.cc:178] TabletServer@127.0.100.2:0 shutting down...
W20240503 23:24:58.877691 1177 consensus_peers.cc:485] T 98b4ca061f7f4a43a13fbad78dac228a P bf5fe0b6e639437498453c116a26f95c -> Peer 74775765c2ac4db693fa70319ffd98c9 (127.0.100.2:38529): Couldn't send request to peer 74775765c2ac4db693fa70319ffd98c9. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20240503 23:24:58.883430 400 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240503 23:24:58.884017 400 tablet_replica.cc:330] T 1e6b13b222534155bc03878359c2eb3b P 74775765c2ac4db693fa70319ffd98c9: stopping tablet replica
I20240503 23:24:58.884385 400 raft_consensus.cc:2229] T 1e6b13b222534155bc03878359c2eb3b P 74775765c2ac4db693fa70319ffd98c9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 23:24:58.884768 400 raft_consensus.cc:2258] T 1e6b13b222534155bc03878359c2eb3b P 74775765c2ac4db693fa70319ffd98c9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 23:24:58.885951 400 tablet_replica.cc:330] T 26f3c79af13e40878328e02ae549ab0b P 74775765c2ac4db693fa70319ffd98c9: stopping tablet replica
I20240503 23:24:58.886277 400 raft_consensus.cc:2229] T 26f3c79af13e40878328e02ae549ab0b P 74775765c2ac4db693fa70319ffd98c9 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 23:24:58.886613 400 raft_consensus.cc:2258] T 26f3c79af13e40878328e02ae549ab0b P 74775765c2ac4db693fa70319ffd98c9 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 23:24:58.887672 400 tablet_replica.cc:330] T f3d383cdf48b4e108844ff8a658b9a51 P 74775765c2ac4db693fa70319ffd98c9: stopping tablet replica
I20240503 23:24:58.888010 400 raft_consensus.cc:2229] T f3d383cdf48b4e108844ff8a658b9a51 P 74775765c2ac4db693fa70319ffd98c9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 23:24:58.888214 400 raft_consensus.cc:2258] T f3d383cdf48b4e108844ff8a658b9a51 P 74775765c2ac4db693fa70319ffd98c9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 23:24:58.888980 400 tablet_replica.cc:330] T 6743c625d6854920b88764dec9ebd250 P 74775765c2ac4db693fa70319ffd98c9: stopping tablet replica
I20240503 23:24:58.889272 400 raft_consensus.cc:2229] T 6743c625d6854920b88764dec9ebd250 P 74775765c2ac4db693fa70319ffd98c9 [term 2 LEADER]: Raft consensus shutting down.
I20240503 23:24:58.889691 400 raft_consensus.cc:2258] T 6743c625d6854920b88764dec9ebd250 P 74775765c2ac4db693fa70319ffd98c9 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 23:24:58.890458 400 tablet_replica.cc:330] T cb8edce628bd466da6c9eaabf7964493 P 74775765c2ac4db693fa70319ffd98c9: stopping tablet replica
I20240503 23:24:58.890750 400 raft_consensus.cc:2229] T cb8edce628bd466da6c9eaabf7964493 P 74775765c2ac4db693fa70319ffd98c9 [term 2 LEADER]: Raft consensus shutting down.
I20240503 23:24:58.891178 400 raft_consensus.cc:2258] T cb8edce628bd466da6c9eaabf7964493 P 74775765c2ac4db693fa70319ffd98c9 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 23:24:58.891876 400 tablet_replica.cc:330] T 966a756b35654ba0a0e272a0b6c7d7f6 P 74775765c2ac4db693fa70319ffd98c9: stopping tablet replica
I20240503 23:24:58.892155 400 raft_consensus.cc:2229] T 966a756b35654ba0a0e272a0b6c7d7f6 P 74775765c2ac4db693fa70319ffd98c9 [term 1 LEADER]: Raft consensus shutting down.
I20240503 23:24:58.892526 400 raft_consensus.cc:2258] T 966a756b35654ba0a0e272a0b6c7d7f6 P 74775765c2ac4db693fa70319ffd98c9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 23:24:58.893289 400 tablet_replica.cc:330] T 8d775506b875408d80a957dd3926aed8 P 74775765c2ac4db693fa70319ffd98c9: stopping tablet replica
I20240503 23:24:58.893519 400 raft_consensus.cc:2229] T 8d775506b875408d80a957dd3926aed8 P 74775765c2ac4db693fa70319ffd98c9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 23:24:58.893755 400 raft_consensus.cc:2258] T 8d775506b875408d80a957dd3926aed8 P 74775765c2ac4db693fa70319ffd98c9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 23:24:58.894588 400 tablet_replica.cc:330] T 98b4ca061f7f4a43a13fbad78dac228a P 74775765c2ac4db693fa70319ffd98c9: stopping tablet replica
I20240503 23:24:58.894855 400 raft_consensus.cc:2229] T 98b4ca061f7f4a43a13fbad78dac228a P 74775765c2ac4db693fa70319ffd98c9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 23:24:58.895123 400 raft_consensus.cc:2258] T 98b4ca061f7f4a43a13fbad78dac228a P 74775765c2ac4db693fa70319ffd98c9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 23:24:58.903981 400 tablet_server.cc:194] TabletServer@127.0.100.2:0 shutdown complete.
I20240503 23:24:58.904317 1437 ts_tablet_manager.cc:902] T 98b4ca061f7f4a43a13fbad78dac228a P 3790742adac34de782c07b282754ded4: Initiating tablet copy from peer bf5fe0b6e639437498453c116a26f95c (127.0.100.3:40377)
I20240503 23:24:58.905067 1437 tablet_copy_client.cc:316] T 98b4ca061f7f4a43a13fbad78dac228a P 3790742adac34de782c07b282754ded4: tablet copy: Beginning tablet copy session from remote peer at address 127.0.100.3:40377
I20240503 23:24:58.912190 1224 tablet_copy_service.cc:140] P bf5fe0b6e639437498453c116a26f95c: Received BeginTabletCopySession request for tablet 98b4ca061f7f4a43a13fbad78dac228a from peer 3790742adac34de782c07b282754ded4 ({username='slave'} at 127.0.0.1:60226)
I20240503 23:24:58.912448 1224 tablet_copy_service.cc:161] P bf5fe0b6e639437498453c116a26f95c: Beginning new tablet copy session on tablet 98b4ca061f7f4a43a13fbad78dac228a from peer 3790742adac34de782c07b282754ded4 at {username='slave'} at 127.0.0.1:60226: session id = 3790742adac34de782c07b282754ded4-98b4ca061f7f4a43a13fbad78dac228a
I20240503 23:24:58.916134 1224 tablet_copy_source_session.cc:205] T 98b4ca061f7f4a43a13fbad78dac228a P bf5fe0b6e639437498453c116a26f95c: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 23:24:58.918531 1437 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 98b4ca061f7f4a43a13fbad78dac228a. 1 dirs total, 0 dirs full, 0 dirs failed
W20240503 23:24:58.922366 1177 consensus_peers.cc:485] T f3d383cdf48b4e108844ff8a658b9a51 P bf5fe0b6e639437498453c116a26f95c -> Peer 74775765c2ac4db693fa70319ffd98c9 (127.0.100.2:38529): Couldn't send request to peer 74775765c2ac4db693fa70319ffd98c9. Status: Network error: Client connection negotiation failed: client connection to 127.0.100.2:38529: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 23:24:58.923372 400 tablet_server.cc:178] TabletServer@127.0.100.3:0 shutting down...
W20240503 23:24:58.926939 1178 consensus_peers.cc:485] T 98b4ca061f7f4a43a13fbad78dac228a P bf5fe0b6e639437498453c116a26f95c -> Peer 5d37fd41fc2c41a99580cc475915a090 (127.0.100.1:38605): Couldn't send request to peer 5d37fd41fc2c41a99580cc475915a090. Status: Network error: Client connection negotiation failed: client connection to 127.0.100.1:38605: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 23:24:58.927906 1011 auto_rebalancer.cc:663] Could not move replica: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20240503 23:24:58.928005 1011 auto_rebalancer.cc:264] scheduled replica move failed to complete: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20240503 23:24:58.928323 1437 tablet_copy_client.cc:791] T 98b4ca061f7f4a43a13fbad78dac228a P 3790742adac34de782c07b282754ded4: tablet copy: Starting download of 0 data blocks...
I20240503 23:24:58.929410 1437 tablet_copy_client.cc:655] T 98b4ca061f7f4a43a13fbad78dac228a P 3790742adac34de782c07b282754ded4: tablet copy: Starting download of 1 WAL segments...
I20240503 23:24:58.929986 400 tablet_copy_service.cc:362] P bf5fe0b6e639437498453c116a26f95c: Destroying tablet copy session 3790742adac34de782c07b282754ded4-98b4ca061f7f4a43a13fbad78dac228a due to service shutdown
I20240503 23:24:58.930215 400 tablet_copy_service.cc:433] P bf5fe0b6e639437498453c116a26f95c: ending tablet copy session 3790742adac34de782c07b282754ded4-98b4ca061f7f4a43a13fbad78dac228a on tablet 98b4ca061f7f4a43a13fbad78dac228a with peer 3790742adac34de782c07b282754ded4
W20240503 23:24:58.931152 1011 auto_rebalancer.cc:663] Could not move replica: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20240503 23:24:58.931290 1011 auto_rebalancer.cc:264] scheduled replica move failed to complete: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20240503 23:24:58.939247 400 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240503 23:24:58.939594 400 tablet_replica.cc:330] T 1e6b13b222534155bc03878359c2eb3b P bf5fe0b6e639437498453c116a26f95c: stopping tablet replica
I20240503 23:24:58.939813 400 raft_consensus.cc:2229] T 1e6b13b222534155bc03878359c2eb3b P bf5fe0b6e639437498453c116a26f95c [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 23:24:58.940093 400 raft_consensus.cc:2258] T 1e6b13b222534155bc03878359c2eb3b P bf5fe0b6e639437498453c116a26f95c [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 23:24:58.941057 400 tablet_replica.cc:330] T 6743c625d6854920b88764dec9ebd250 P bf5fe0b6e639437498453c116a26f95c: stopping tablet replica
I20240503 23:24:58.941247 400 raft_consensus.cc:2229] T 6743c625d6854920b88764dec9ebd250 P bf5fe0b6e639437498453c116a26f95c [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 23:24:58.941419 400 raft_consensus.cc:2258] T 6743c625d6854920b88764dec9ebd250 P bf5fe0b6e639437498453c116a26f95c [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 23:24:58.941959 400 tablet_replica.cc:330] T f3d383cdf48b4e108844ff8a658b9a51 P bf5fe0b6e639437498453c116a26f95c: stopping tablet replica
I20240503 23:24:58.942150 400 raft_consensus.cc:2229] T f3d383cdf48b4e108844ff8a658b9a51 P bf5fe0b6e639437498453c116a26f95c [term 1 LEADER]: Raft consensus shutting down.
I20240503 23:24:58.942487 400 raft_consensus.cc:2258] T f3d383cdf48b4e108844ff8a658b9a51 P bf5fe0b6e639437498453c116a26f95c [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 23:24:58.942945 400 tablet_replica.cc:330] T 966a756b35654ba0a0e272a0b6c7d7f6 P bf5fe0b6e639437498453c116a26f95c: stopping tablet replica
I20240503 23:24:58.943118 400 raft_consensus.cc:2229] T 966a756b35654ba0a0e272a0b6c7d7f6 P bf5fe0b6e639437498453c116a26f95c [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 23:24:58.943266 400 raft_consensus.cc:2258] T 966a756b35654ba0a0e272a0b6c7d7f6 P bf5fe0b6e639437498453c116a26f95c [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 23:24:58.943711 400 tablet_replica.cc:330] T 98b4ca061f7f4a43a13fbad78dac228a P bf5fe0b6e639437498453c116a26f95c: stopping tablet replica
I20240503 23:24:58.943867 400 raft_consensus.cc:2229] T 98b4ca061f7f4a43a13fbad78dac228a P bf5fe0b6e639437498453c116a26f95c [term 1 LEADER]: Raft consensus shutting down.
I20240503 23:24:58.944273 400 raft_consensus.cc:2258] T 98b4ca061f7f4a43a13fbad78dac228a P bf5fe0b6e639437498453c116a26f95c [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 23:24:58.945000 400 tablet_replica.cc:330] T 8d775506b875408d80a957dd3926aed8 P bf5fe0b6e639437498453c116a26f95c: stopping tablet replica
I20240503 23:24:58.945168 400 raft_consensus.cc:2229] T 8d775506b875408d80a957dd3926aed8 P bf5fe0b6e639437498453c116a26f95c [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 23:24:58.945344 400 raft_consensus.cc:2258] T 8d775506b875408d80a957dd3926aed8 P bf5fe0b6e639437498453c116a26f95c [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 23:24:58.946050 400 tablet_replica.cc:330] T cb8edce628bd466da6c9eaabf7964493 P bf5fe0b6e639437498453c116a26f95c: stopping tablet replica
I20240503 23:24:58.946200 400 raft_consensus.cc:2229] T cb8edce628bd466da6c9eaabf7964493 P bf5fe0b6e639437498453c116a26f95c [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 23:24:58.946364 400 raft_consensus.cc:2258] T cb8edce628bd466da6c9eaabf7964493 P bf5fe0b6e639437498453c116a26f95c [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 23:24:58.946794 400 tablet_replica.cc:330] T 26f3c79af13e40878328e02ae549ab0b P bf5fe0b6e639437498453c116a26f95c: stopping tablet replica
I20240503 23:24:58.946954 400 raft_consensus.cc:2229] T 26f3c79af13e40878328e02ae549ab0b P bf5fe0b6e639437498453c116a26f95c [term 2 LEADER]: Raft consensus shutting down.
I20240503 23:24:58.947307 400 raft_consensus.cc:2258] T 26f3c79af13e40878328e02ae549ab0b P bf5fe0b6e639437498453c116a26f95c [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 23:24:58.954581 400 tablet_server.cc:194] TabletServer@127.0.100.3:0 shutdown complete.
I20240503 23:24:58.959729 400 tablet_server.cc:178] TabletServer@127.0.100.4:0 shutting down...
W20240503 23:24:58.966128 1437 ts_tablet_manager.cc:980] T 98b4ca061f7f4a43a13fbad78dac228a P 3790742adac34de782c07b282754ded4: Tablet Copy: Unable to fetch data from remote peer bf5fe0b6e639437498453c116a26f95c (127.0.100.3:40377): Network error: Unable to download WAL segment with seq. number 1: unable to fetch data from remote: Client connection negotiation failed: client connection to 127.0.100.3:40377: connect: Connection refused (error 111)
I20240503 23:24:58.966455 1437 tablet_replica.cc:330] T 98b4ca061f7f4a43a13fbad78dac228a P 3790742adac34de782c07b282754ded4: stopping tablet replica
I20240503 23:24:58.966588 1437 raft_consensus.cc:2229] T 98b4ca061f7f4a43a13fbad78dac228a P 3790742adac34de782c07b282754ded4 [term 1 LEARNER]: Raft consensus shutting down.
I20240503 23:24:58.966754 1437 raft_consensus.cc:2258] T 98b4ca061f7f4a43a13fbad78dac228a P 3790742adac34de782c07b282754ded4 [term 1 LEARNER]: Raft consensus is shut down!
W20240503 23:24:58.968705 1437 tablet_copy_client.cc:1116] T 98b4ca061f7f4a43a13fbad78dac228a P 3790742adac34de782c07b282754ded4: tablet copy: Unable to close tablet copy session: Network error: failure ending tablet copy session: Client connection negotiation failed: client connection to 127.0.100.3:40377: connect: Connection refused (error 111)
W20240503 23:24:58.971107 1437 tablet_metadata.cc:476] failed to load DataDirGroup from superblock: Already present: tried to load directory group for tablet 98b4ca061f7f4a43a13fbad78dac228a but one is already registered
I20240503 23:24:58.971649 1437 ts_tablet_manager.cc:1875] T 98b4ca061f7f4a43a13fbad78dac228a P 3790742adac34de782c07b282754ded4: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 23:24:58.972399 400 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240503 23:24:58.981772 1437 ts_tablet_manager.cc:1888] T 98b4ca061f7f4a43a13fbad78dac228a P 3790742adac34de782c07b282754ded4: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.0
I20240503 23:24:58.982069 1437 log.cc:1196] T 98b4ca061f7f4a43a13fbad78dac228a P 3790742adac34de782c07b282754ded4: Deleting WAL directory at /tmp/dist-test-taskZhIiPz/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714778688451250-400-0/minicluster-data/ts-3-root/wals/98b4ca061f7f4a43a13fbad78dac228a
I20240503 23:24:58.997295 400 tablet_server.cc:194] TabletServer@127.0.100.4:0 shutdown complete.
I20240503 23:24:59.000694 400 master.cc:537] Master@127.0.100.62:35319 shutting down...
I20240503 23:24:59.009850 400 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 46860c6a2576449e82ccd4bae9b82d95 [term 1 LEADER]: Raft consensus shutting down.
I20240503 23:24:59.010174 400 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 46860c6a2576449e82ccd4bae9b82d95 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 23:24:59.010283 400 tablet_replica.cc:330] T 00000000000000000000000000000000 P 46860c6a2576449e82ccd4bae9b82d95: stopping tablet replica
I20240503 23:24:59.026094 400 master.cc:552] Master@127.0.100.62:35319 shutdown complete.
[ OK ] AutoRebalancerTest.AutoRebalancingTurnOffAndOn (7740 ms)
[ RUN ] AutoRebalancerTest.NextLeaderResumesAutoRebalancing
I20240503 23:24:59.045105 400 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.0.100.62:46733,127.0.100.61:45861,127.0.100.60:38057
I20240503 23:24:59.045861 400 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 23:24:59.049624 1444 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 23:24:59.050395 1445 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 23:24:59.051152 1447 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 23:24:59.051913 400 server_base.cc:1034] running on GCE node
I20240503 23:24:59.052279 400 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240503 23:24:59.052345 400 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240503 23:24:59.052417 400 hybrid_clock.cc:648] HybridClock initialized: now 1714778699052382 us; error 0 us; skew 500 ppm
I20240503 23:24:59.052734 400 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 23:24:59.060712 400 webserver.cc:460] Webserver started at http://127.0.100.62:41617/ using document root <none> and password file <none>
I20240503 23:24:59.061108 400 fs_manager.cc:359] Metadata directory not provided
I20240503 23:24:59.061231 400 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 23:24:59.061403 400 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 23:24:59.062289 400 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZhIiPz/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714778688451250-400-0/minicluster-data/master-0-root/instance:
uuid: "e29ee7fe7b864b3d84f43581d1debeb3"
format_stamp: "Formatted at 2024-05-03 23:24:59 on dist-test-slave-jvbp"
I20240503 23:24:59.065461 400 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20240503 23:24:59.067605 1452 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 23:24:59.068053 400 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.002s sys 0.000s
I20240503 23:24:59.068195 400 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZhIiPz/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714778688451250-400-0/minicluster-data/master-0-root
uuid: "e29ee7fe7b864b3d84f43581d1debeb3"
format_stamp: "Formatted at 2024-05-03 23:24:59 on dist-test-slave-jvbp"
I20240503 23:24:59.068361 400 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZhIiPz/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714778688451250-400-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskZhIiPz/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714778688451250-400-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskZhIiPz/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714778688451250-400-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 23:24:59.089478 400 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 23:24:59.090255 400 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 23:24:59.105737 400 rpc_server.cc:307] RPC server started. Bound to: 127.0.100.62:46733
I20240503 23:24:59.105751 1503 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.100.62:46733 every 8 connection(s)
I20240503 23:24:59.107895 1504 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 23:24:59.107972 400 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 23:24:59.111964 1506 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 23:24:59.112118 1504 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.100.62" port: 46733 } has no permanent_uuid. Determining permanent_uuid...
W20240503 23:24:59.112993 1507 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 23:24:59.119793 400 server_base.cc:1034] running on GCE node
W20240503 23:24:59.121549 1509 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 23:24:59.122241 400 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240503 23:24:59.122362 400 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240503 23:24:59.122495 400 hybrid_clock.cc:648] HybridClock initialized: now 1714778699122453 us; error 0 us; skew 500 ppm
I20240503 23:24:59.122702 400 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 23:24:59.124694 400 webserver.cc:460] Webserver started at http://127.0.100.61:37027/ using document root <none> and password file <none>
I20240503 23:24:59.124971 400 fs_manager.cc:359] Metadata directory not provided
I20240503 23:24:59.125088 400 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 23:24:59.125357 400 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 23:24:59.126220 1504 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.100.61" port: 45861 } has no permanent_uuid. Determining permanent_uuid...
I20240503 23:24:59.126318 400 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZhIiPz/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714778688451250-400-0/minicluster-data/master-1-root/instance:
uuid: "a3b81f85f3594536ade9e155ca2d1cb4"
format_stamp: "Formatted at 2024-05-03 23:24:59 on dist-test-slave-jvbp"
W20240503 23:24:59.128474 1504 consensus_peers.cc:644] Error getting permanent uuid from config peer 127.0.100.61:45861: Network error: Client connection negotiation failed: client connection to 127.0.100.61:45861: connect: Connection refused (error 111)
I20240503 23:24:59.129268 400 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.002s
I20240503 23:24:59.131119 1517 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 23:24:59.131493 400 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.001s
I20240503 23:24:59.131611 400 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZhIiPz/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714778688451250-400-0/minicluster-data/master-1-root
uuid: "a3b81f85f3594536ade9e155ca2d1cb4"
format_stamp: "Formatted at 2024-05-03 23:24:59 on dist-test-slave-jvbp"
I20240503 23:24:59.131757 400 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZhIiPz/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714778688451250-400-0/minicluster-data/master-1-root
metadata directory: /tmp/dist-test-taskZhIiPz/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714778688451250-400-0/minicluster-data/master-1-root
1 data directories: /tmp/dist-test-taskZhIiPz/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714778688451250-400-0/minicluster-data/master-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 23:24:59.163774 400 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 23:24:59.164528 400 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 23:24:59.179738 1504 consensus_peers.cc:654] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.100.61" port: 45861 } attempt: 1
I20240503 23:24:59.180423 400 rpc_server.cc:307] RPC server started. Bound to: 127.0.100.61:45861
I20240503 23:24:59.180454 1568 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.100.61:45861 every 8 connection(s)
I20240503 23:24:59.182665 1570 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 23:24:59.182919 400 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 23:24:59.187381 1572 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 23:24:59.187820 1573 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 23:24:59.187779 1570 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.100.62" port: 46733 } has no permanent_uuid. Determining permanent_uuid...
W20240503 23:24:59.190450 1575 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 23:24:59.191138 400 server_base.cc:1034] running on GCE node
I20240503 23:24:59.191637 1504 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.100.60" port: 38057 } has no permanent_uuid. Determining permanent_uuid...
I20240503 23:24:59.191740 400 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240503 23:24:59.191838 400 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240503 23:24:59.191933 400 hybrid_clock.cc:648] HybridClock initialized: now 1714778699191891 us; error 0 us; skew 500 ppm
I20240503 23:24:59.192152 400 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
W20240503 23:24:59.193914 1504 consensus_peers.cc:644] Error getting permanent uuid from config peer 127.0.100.60:38057: Network error: Client connection negotiation failed: client connection to 127.0.100.60:38057: connect: Connection refused (error 111)
I20240503 23:24:59.194234 400 webserver.cc:460] Webserver started at http://127.0.100.60:42931/ using document root <none> and password file <none>
I20240503 23:24:59.194480 400 fs_manager.cc:359] Metadata directory not provided
I20240503 23:24:59.194578 400 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 23:24:59.194700 400 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 23:24:59.195619 400 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZhIiPz/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714778688451250-400-0/minicluster-data/master-2-root/instance:
uuid: "1a6c6637a2d64ca098e1d774053fd1ce"
format_stamp: "Formatted at 2024-05-03 23:24:59 on dist-test-slave-jvbp"
I20240503 23:24:59.197227 1570 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.100.61" port: 45861 } has no permanent_uuid. Determining permanent_uuid...
I20240503 23:24:59.199244 400 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20240503 23:24:59.201884 1581 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 23:24:59.202401 400 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20240503 23:24:59.202569 400 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZhIiPz/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714778688451250-400-0/minicluster-data/master-2-root
uuid: "1a6c6637a2d64ca098e1d774053fd1ce"
format_stamp: "Formatted at 2024-05-03 23:24:59 on dist-test-slave-jvbp"
I20240503 23:24:59.202782 400 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZhIiPz/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714778688451250-400-0/minicluster-data/master-2-root
metadata directory: /tmp/dist-test-taskZhIiPz/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714778688451250-400-0/minicluster-data/master-2-root
1 data directories: /tmp/dist-test-taskZhIiPz/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714778688451250-400-0/minicluster-data/master-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 23:24:59.204757 1570 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.100.60" port: 38057 } has no permanent_uuid. Determining permanent_uuid...
W20240503 23:24:59.207283 1570 consensus_peers.cc:644] Error getting permanent uuid from config peer 127.0.100.60:38057: Network error: Client connection negotiation failed: client connection to 127.0.100.60:38057: connect: Connection refused (error 111)
I20240503 23:24:59.218241 1504 consensus_peers.cc:654] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.100.60" port: 38057 } attempt: 1
W20240503 23:24:59.220666 1504 consensus_peers.cc:644] Error getting permanent uuid from config peer 127.0.100.60:38057: Network error: Client connection negotiation failed: client connection to 127.0.100.60:38057: connect: Connection refused (error 111)
I20240503 23:24:59.228580 1570 consensus_peers.cc:654] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.100.60" port: 38057 } attempt: 1
W20240503 23:24:59.230975 1570 consensus_peers.cc:644] Error getting permanent uuid from config peer 127.0.100.60:38057: Network error: Client connection negotiation failed: client connection to 127.0.100.60:38057: connect: Connection refused (error 111)
I20240503 23:24:59.242760 400 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 23:24:59.243552 400 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 23:24:59.260427 400 rpc_server.cc:307] RPC server started. Bound to: 127.0.100.60:38057
I20240503 23:24:59.260453 1633 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.100.60:38057 every 8 connection(s)
I20240503 23:24:59.261606 400 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20240503 23:24:59.262199 1634 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 23:24:59.263942 1504 consensus_peers.cc:654] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.100.60" port: 38057 } attempt: 2
I20240503 23:24:59.265359 1634 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.100.62" port: 46733 } has no permanent_uuid. Determining permanent_uuid...
I20240503 23:24:59.273787 1634 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.100.61" port: 45861 } has no permanent_uuid. Determining permanent_uuid...
I20240503 23:24:59.276530 1504 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e29ee7fe7b864b3d84f43581d1debeb3: Bootstrap starting.
I20240503 23:24:59.278249 1504 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e29ee7fe7b864b3d84f43581d1debeb3: Neither blocks nor log segments found. Creating new log.
I20240503 23:24:59.280512 1634 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.100.60" port: 38057 } has no permanent_uuid. Determining permanent_uuid...
I20240503 23:24:59.280779 1504 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e29ee7fe7b864b3d84f43581d1debeb3: No bootstrap required, opened a new log
I20240503 23:24:59.282296 1504 raft_consensus.cc:348] T 00000000000000000000000000000000 P e29ee7fe7b864b3d84f43581d1debeb3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e29ee7fe7b864b3d84f43581d1debeb3" member_type: VOTER last_known_addr { host: "127.0.100.62" port: 46733 } } peers { permanent_uuid: "a3b81f85f3594536ade9e155ca2d1cb4" member_type: VOTER last_known_addr { host: "127.0.100.61" port: 45861 } } peers { permanent_uuid: "1a6c6637a2d64ca098e1d774053fd1ce" member_type: VOTER last_known_addr { host: "127.0.100.60" port: 38057 } }
I20240503 23:24:59.282559 1504 raft_consensus.cc:374] T 00000000000000000000000000000000 P e29ee7fe7b864b3d84f43581d1debeb3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 23:24:59.282644 1504 raft_consensus.cc:729] T 00000000000000000000000000000000 P e29ee7fe7b864b3d84f43581d1debeb3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e29ee7fe7b864b3d84f43581d1debeb3, State: Initialized, Role: FOLLOWER
I20240503 23:24:59.282963 1504 consensus_queue.cc:260] T 00000000000000000000000000000000 P e29ee7fe7b864b3d84f43581d1debeb3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e29ee7fe7b864b3d84f43581d1debeb3" member_type: VOTER last_known_addr { host: "127.0.100.62" port: 46733 } } peers { permanent_uuid: "a3b81f85f3594536ade9e155ca2d1cb4" member_type: VOTER last_known_addr { host: "127.0.100.61" port: 45861 } } peers { permanent_uuid: "1a6c6637a2d64ca098e1d774053fd1ce" member_type: VOTER last_known_addr { host: "127.0.100.60" port: 38057 } }
I20240503 23:24:59.283859 1639 sys_catalog.cc:455] T 00000000000000000000000000000000 P e29ee7fe7b864b3d84f43581d1debeb3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e29ee7fe7b864b3d84f43581d1debeb3" member_type: VOTER last_known_addr { host: "127.0.100.62" port: 46733 } } peers { permanent_uuid: "a3b81f85f3594536ade9e155ca2d1cb4" member_type: VOTER last_known_addr { host: "127.0.100.61" port: 45861 } } peers { permanent_uuid: "1a6c6637a2d64ca098e1d774053fd1ce" member_type: VOTER last_known_addr { host: "127.0.100.60" port: 38057 } } }
I20240503 23:24:59.284042 1504 sys_catalog.cc:564] T 00000000000000000000000000000000 P e29ee7fe7b864b3d84f43581d1debeb3 [sys.catalog]: configured and running, proceeding with master startup.
I20240503 23:24:59.284173 1639 sys_catalog.cc:458] T 00000000000000000000000000000000 P e29ee7fe7b864b3d84f43581d1debeb3 [sys.catalog]: This master's current role is: FOLLOWER
I20240503 23:24:59.291544 400 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 1
W20240503 23:24:59.292866 1650 catalog_manager.cc:1558] T 00000000000000000000000000000000 P e29ee7fe7b864b3d84f43581d1debeb3: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20240503 23:24:59.292987 1650 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20240503 23:24:59.293124 1634 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1a6c6637a2d64ca098e1d774053fd1ce: Bootstrap starting.
I20240503 23:24:59.294622 1634 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1a6c6637a2d64ca098e1d774053fd1ce: Neither blocks nor log segments found. Creating new log.
I20240503 23:24:59.296859 1634 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1a6c6637a2d64ca098e1d774053fd1ce: No bootstrap required, opened a new log
I20240503 23:24:59.297698 1634 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1a6c6637a2d64ca098e1d774053fd1ce [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e29ee7fe7b864b3d84f43581d1debeb3" member_type: VOTER last_known_addr { host: "127.0.100.62" port: 46733 } } peers { permanent_uuid: "a3b81f85f3594536ade9e155ca2d1cb4" member_type: VOTER last_known_addr { host: "127.0.100.61" port: 45861 } } peers { permanent_uuid: "1a6c6637a2d64ca098e1d774053fd1ce" member_type: VOTER last_known_addr { host: "127.0.100.60" port: 38057 } }
I20240503 23:24:59.297933 1634 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1a6c6637a2d64ca098e1d774053fd1ce [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 23:24:59.298024 1634 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1a6c6637a2d64ca098e1d774053fd1ce [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1a6c6637a2d64ca098e1d774053fd1ce, State: Initialized, Role: FOLLOWER
I20240503 23:24:59.298318 1634 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1a6c6637a2d64ca098e1d774053fd1ce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e29ee7fe7b864b3d84f43581d1debeb3" member_type: VOTER last_known_addr { host: "127.0.100.62" port: 46733 } } peers { permanent_uuid: "a3b81f85f3594536ade9e155ca2d1cb4" member_type: VOTER last_known_addr { host: "127.0.100.61" port: 45861 } } peers { permanent_uuid: "1a6c6637a2d64ca098e1d774053fd1ce" member_type: VOTER last_known_addr { host: "127.0.100.60" port: 38057 } }
I20240503 23:24:59.299116 1652 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1a6c6637a2d64ca098e1d774053fd1ce [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e29ee7fe7b864b3d84f43581d1debeb3" member_type: VOTER last_known_addr { host: "127.0.100.62" port: 46733 } } peers { permanent_uuid: "a3b81f85f3594536ade9e155ca2d1cb4" member_type: VOTER last_known_addr { host: "127.0.100.61" port: 45861 } } peers { permanent_uuid: "1a6c6637a2d64ca098e1d774053fd1ce" member_type: VOTER last_known_addr { host: "127.0.100.60" port: 38057 } } }
I20240503 23:24:59.299333 1652 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1a6c6637a2d64ca098e1d774053fd1ce [sys.catalog]: This master's current role is: FOLLOWER
I20240503 23:24:59.299335 1634 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1a6c6637a2d64ca098e1d774053fd1ce [sys.catalog]: configured and running, proceeding with master startup.
I20240503 23:24:59.302282 1570 consensus_peers.cc:654] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.100.60" port: 38057 } attempt: 2
W20240503 23:24:59.307627 1663 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 1a6c6637a2d64ca098e1d774053fd1ce: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20240503 23:24:59.307775 1663 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20240503 23:24:59.311899 1570 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a3b81f85f3594536ade9e155ca2d1cb4: Bootstrap starting.
I20240503 23:24:59.313575 1570 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a3b81f85f3594536ade9e155ca2d1cb4: Neither blocks nor log segments found. Creating new log.
I20240503 23:24:59.315683 1570 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a3b81f85f3594536ade9e155ca2d1cb4: No bootstrap required, opened a new log
I20240503 23:24:59.316465 1570 raft_consensus.cc:348] T 00000000000000000000000000000000 P a3b81f85f3594536ade9e155ca2d1cb4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e29ee7fe7b864b3d84f43581d1debeb3" member_type: VOTER last_known_addr { host: "127.0.100.62" port: 46733 } } peers { permanent_uuid: "a3b81f85f3594536ade9e155ca2d1cb4" member_type: VOTER last_known_addr { host: "127.0.100.61" port: 45861 } } peers { permanent_uuid: "1a6c6637a2d64ca098e1d774053fd1ce" member_type: VOTER last_known_addr { host: "127.0.100.60" port: 38057 } }
I20240503 23:24:59.316748 1570 raft_consensus.cc:374] T 00000000000000000000000000000000 P a3b81f85f3594536ade9e155ca2d1cb4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 23:24:59.316835 1570 raft_consensus.cc:729] T 00000000000000000000000000000000 P a3b81f85f3594536ade9e155ca2d1cb4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a3b81f85f3594536ade9e155ca2d1cb4, State: Initialized, Role: FOLLOWER
I20240503 23:24:59.317138 1570 consensus_queue.cc:260] T 00000000000000000000000000000000 P a3b81f85f3594536ade9e155ca2d1cb4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e29ee7fe7b864b3d84f43581d1debeb3" member_type: VOTER last_known_addr { host: "127.0.100.62" port: 46733 } } peers { permanent_uuid: "a3b81f85f3594536ade9e155ca2d1cb4" member_type: VOTER last_known_addr { host: "127.0.100.61" port: 45861 } } peers { permanent_uuid: "1a6c6637a2d64ca098e1d774053fd1ce" member_type: VOTER last_known_addr { host: "127.0.100.60" port: 38057 } }
I20240503 23:24:59.318675 1665 sys_catalog.cc:455] T 00000000000000000000000000000000 P a3b81f85f3594536ade9e155ca2d1cb4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e29ee7fe7b864b3d84f43581d1debeb3" member_type: VOTER last_known_addr { host: "127.0.100.62" port: 46733 } } peers { permanent_uuid: "a3b81f85f3594536ade9e155ca2d1cb4" member_type: VOTER last_known_addr { host: "127.0.100.61" port: 45861 } } peers { permanent_uuid: "1a6c6637a2d64ca098e1d774053fd1ce" member_type: VOTER last_known_addr { host: "127.0.100.60" port: 38057 } } }
I20240503 23:24:59.318876 1665 sys_catalog.cc:458] T 00000000000000000000000000000000 P a3b81f85f3594536ade9e155ca2d1cb4 [sys.catalog]: This master's current role is: FOLLOWER
I20240503 23:24:59.318953 1570 sys_catalog.cc:564] T 00000000000000000000000000000000 P a3b81f85f3594536ade9e155ca2d1cb4 [sys.catalog]: configured and running, proceeding with master startup.
I20240503 23:24:59.323220 400 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 2
I20240503 23:24:59.324213 400 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 23:24:59.324635 1676 catalog_manager.cc:1558] T 00000000000000000000000000000000 P a3b81f85f3594536ade9e155ca2d1cb4: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20240503 23:24:59.324751 1676 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
W20240503 23:24:59.328588 1677 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 23:24:59.329182 1678 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 23:24:59.330811 400 server_base.cc:1034] running on GCE node
W20240503 23:24:59.331380 1680 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 23:24:59.331872 400 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240503 23:24:59.331981 400 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240503 23:24:59.332058 400 hybrid_clock.cc:648] HybridClock initialized: now 1714778699332030 us; error 0 us; skew 500 ppm
I20240503 23:24:59.332273 400 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 23:24:59.334071 400 webserver.cc:460] Webserver started at http://127.0.100.1:33449/ using document root <none> and password file <none>
I20240503 23:24:59.334352 400 fs_manager.cc:359] Metadata directory not provided
I20240503 23:24:59.334444 400 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 23:24:59.334623 400 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 23:24:59.335414 400 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZhIiPz/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714778688451250-400-0/minicluster-data/ts-0-root/instance:
uuid: "0fc5931f4b97441b92926f9ca3f0ea1b"
format_stamp: "Formatted at 2024-05-03 23:24:59 on dist-test-slave-jvbp"
I20240503 23:24:59.338521 400 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20240503 23:24:59.340539 1685 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 23:24:59.341053 400 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.001s
I20240503 23:24:59.341187 400 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZhIiPz/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714778688451250-400-0/minicluster-data/ts-0-root
uuid: "0fc5931f4b97441b92926f9ca3f0ea1b"
format_stamp: "Formatted at 2024-05-03 23:24:59 on dist-test-slave-jvbp"
I20240503 23:24:59.341317 400 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZhIiPz/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714778688451250-400-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskZhIiPz/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714778688451250-400-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskZhIiPz/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714778688451250-400-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 23:24:59.360283 400 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 23:24:59.361099 400 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 23:24:59.361788 400 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 23:24:59.362861 400 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 23:24:59.362949 400 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 23:24:59.363037 400 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 23:24:59.363096 400 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 23:24:59.383905 400 rpc_server.cc:307] RPC server started. Bound to: 127.0.100.1:45121
I20240503 23:24:59.383946 1747 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.100.1:45121 every 8 connection(s)
I20240503 23:24:59.390805 1665 raft_consensus.cc:482] T 00000000000000000000000000000000 P a3b81f85f3594536ade9e155ca2d1cb4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 23:24:59.391150 1665 raft_consensus.cc:504] T 00000000000000000000000000000000 P a3b81f85f3594536ade9e155ca2d1cb4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e29ee7fe7b864b3d84f43581d1debeb3" member_type: VOTER last_known_addr { host: "127.0.100.62" port: 46733 } } peers { permanent_uuid: "a3b81f85f3594536ade9e155ca2d1cb4" member_type: VOTER last_known_addr { host: "127.0.100.61" port: 45861 } } peers { permanent_uuid: "1a6c6637a2d64ca098e1d774053fd1ce" member_type: VOTER last_known_addr { host: "127.0.100.60" port: 38057 } }
I20240503 23:24:59.392565 1665 leader_election.cc:288] T 00000000000000000000000000000000 P a3b81f85f3594536ade9e155ca2d1cb4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e29ee7fe7b864b3d84f43581d1debeb3 (127.0.100.62:46733), 1a6c6637a2d64ca098e1d774053fd1ce (127.0.100.60:38057)
I20240503 23:24:59.393812 1479 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "a3b81f85f3594536ade9e155ca2d1cb4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e29ee7fe7b864b3d84f43581d1debeb3" is_pre_election: true
I20240503 23:24:59.394263 1479 raft_consensus.cc:2454] T 00000000000000000000000000000000 P e29ee7fe7b864b3d84f43581d1debeb3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a3b81f85f3594536ade9e155ca2d1cb4 in term 0.
I20240503 23:24:59.394863 1609 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "a3b81f85f3594536ade9e155ca2d1cb4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1a6c6637a2d64ca098e1d774053fd1ce" is_pre_election: true
I20240503 23:24:59.395223 1609 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 1a6c6637a2d64ca098e1d774053fd1ce [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a3b81f85f3594536ade9e155ca2d1cb4 in term 0.
I20240503 23:24:59.396667 1519 leader_election.cc:302] T 00000000000000000000000000000000 P a3b81f85f3594536ade9e155ca2d1cb4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1a6c6637a2d64ca098e1d774053fd1ce, a3b81f85f3594536ade9e155ca2d1cb4; no voters:
I20240503 23:24:59.398020 1665 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a3b81f85f3594536ade9e155ca2d1cb4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 23:24:59.398211 1665 raft_consensus.cc:482] T 00000000000000000000000000000000 P a3b81f85f3594536ade9e155ca2d1cb4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 23:24:59.398329 1665 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a3b81f85f3594536ade9e155ca2d1cb4 [term 0 FOLLOWER]: Advancing to term 1
I20240503 23:24:59.402824 1665 raft_consensus.cc:504] T 00000000000000000000000000000000 P a3b81f85f3594536ade9e155ca2d1cb4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e29ee7fe7b864b3d84f43581d1debeb3" member_type: VOTER last_known_addr { host: "127.0.100.62" port: 46733 } } peers { permanent_uuid: "a3b81f85f3594536ade9e155ca2d1cb4" member_type: VOTER last_known_addr { host: "127.0.100.61" port: 45861 } } peers { permanent_uuid: "1a6c6637a2d64ca098e1d774053fd1ce" member_type: VOTER last_known_addr { host: "127.0.100.60" port: 38057 } }
I20240503 23:24:59.404076 1665 leader_election.cc:288] T 00000000000000000000000000000000 P a3b81f85f3594536ade9e155ca2d1cb4 [CANDIDATE]: Term 1 election: Requested vote from peers e29ee7fe7b864b3d84f43581d1debeb3 (127.0.100.62:46733), 1a6c6637a2d64ca098e1d774053fd1ce (127.0.100.60:38057)
I20240503 23:24:59.404253 1609 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "a3b81f85f3594536ade9e155ca2d1cb4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1a6c6637a2d64ca098e1d774053fd1ce"
I20240503 23:24:59.404300 1479 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "a3b81f85f3594536ade9e155ca2d1cb4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e29ee7fe7b864b3d84f43581d1debeb3"
I20240503 23:24:59.404507 1609 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1a6c6637a2d64ca098e1d774053fd1ce [term 0 FOLLOWER]: Advancing to term 1
I20240503 23:24:59.404642 1479 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e29ee7fe7b864b3d84f43581d1debeb3 [term 0 FOLLOWER]: Advancing to term 1
I20240503 23:24:59.409036 1609 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 1a6c6637a2d64ca098e1d774053fd1ce [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a3b81f85f3594536ade9e155ca2d1cb4 in term 1.
I20240503 23:24:59.410259 1479 raft_consensus.cc:2454] T 00000000000000000000000000000000 P e29ee7fe7b864b3d84f43581d1debeb3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a3b81f85f3594536ade9e155ca2d1cb4 in term 1.
I20240503 23:24:59.410408 1519 leader_election.cc:302] T 00000000000000000000000000000000 P a3b81f85f3594536ade9e155ca2d1cb4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1a6c6637a2d64ca098e1d774053fd1ce, a3b81f85f3594536ade9e155ca2d1cb4; no voters:
I20240503 23:24:59.410888 400 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20240503 23:24:59.411060 1665 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a3b81f85f3594536ade9e155ca2d1cb4 [term 1 FOLLOWER]: Leader election won for term 1
W20240503 23:24:59.419703 1756 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 23:24:59.421169 1665 raft_consensus.cc:686] T 00000000000000000000000000000000 P a3b81f85f3594536ade9e155ca2d1cb4 [term 1 LEADER]: Becoming Leader. State: Replica: a3b81f85f3594536ade9e155ca2d1cb4, State: Running, Role: LEADER
I20240503 23:24:59.421869 1665 consensus_queue.cc:237] T 00000000000000000000000000000000 P a3b81f85f3594536ade9e155ca2d1cb4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e29ee7fe7b864b3d84f43581d1debeb3" member_type: VOTER last_known_addr { host: "127.0.100.62" port: 46733 } } peers { permanent_uuid: "a3b81f85f3594536ade9e155ca2d1cb4" member_type: VOTER last_known_addr { host: "127.0.100.61" port: 45861 } } peers { permanent_uuid: "1a6c6637a2d64ca098e1d774053fd1ce" member_type: VOTER last_known_addr { host: "127.0.100.60" port: 38057 } }
I20240503 23:24:59.423609 1750 heartbeater.cc:347] Connected to a master server at 127.0.100.61:45861
I20240503 23:24:59.423832 1750 heartbeater.cc:464] Registering TS with master...
I20240503 23:24:59.424640 1750 heartbeater.cc:511] Master 127.0.100.61:45861 requested a full tablet report, sending...
I20240503 23:24:59.424851 1749 heartbeater.cc:347] Connected to a master server at 127.0.100.62:46733
I20240503 23:24:59.425061 1749 heartbeater.cc:464] Registering TS with master...
I20240503 23:24:59.425696 1749 heartbeater.cc:511] Master 127.0.100.62:46733 requested a full tablet report, sending...
I20240503 23:24:59.427230 1469 ts_manager.cc:194] Registered new tserver with Master: 0fc5931f4b97441b92926f9ca3f0ea1b (127.0.100.1:45121)
I20240503 23:24:59.429170 1755 sys_catalog.cc:455] T 00000000000000000000000000000000 P a3b81f85f3594536ade9e155ca2d1cb4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a3b81f85f3594536ade9e155ca2d1cb4. Latest consensus state: current_term: 1 leader_uuid: "a3b81f85f3594536ade9e155ca2d1cb4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e29ee7fe7b864b3d84f43581d1debeb3" member_type: VOTER last_known_addr { host: "127.0.100.62" port: 46733 } } peers { permanent_uuid: "a3b81f85f3594536ade9e155ca2d1cb4" member_type: VOTER last_known_addr { host: "127.0.100.61" port: 45861 } } peers { permanent_uuid: "1a6c6637a2d64ca098e1d774053fd1ce" member_type: VOTER last_known_addr { host: "127.0.100.60" port: 38057 } } }
I20240503 23:24:59.429912 1755 sys_catalog.cc:458] T 00000000000000000000000000000000 P a3b81f85f3594536ade9e155ca2d1cb4 [sys.catalog]: This master's current role is: LEADER
I20240503 23:24:59.430266 1534 ts_manager.cc:194] Registered new tserver with Master: 0fc5931f4b97441b92926f9ca3f0ea1b (127.0.100.1:45121)
W20240503 23:24:59.432276 1757 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 23:24:59.433218 1748 heartbeater.cc:347] Connected to a master server at 127.0.100.60:38057
I20240503 23:24:59.433444 1748 heartbeater.cc:464] Registering TS with master...
I20240503 23:24:59.434219 1748 heartbeater.cc:511] Master 127.0.100.60:38057 requested a full tablet report, sending...
W20240503 23:24:59.434793 1760 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 23:24:59.435200 1761 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240503 23:24:59.435518 1599 ts_manager.cc:194] Registered new tserver with Master: 0fc5931f4b97441b92926f9ca3f0ea1b (127.0.100.1:45121)
I20240503 23:24:59.436203 400 server_base.cc:1034] running on GCE node
I20240503 23:24:59.436601 400 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240503 23:24:59.436693 400 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240503 23:24:59.436774 400 hybrid_clock.cc:648] HybridClock initialized: now 1714778699436731 us; error 0 us; skew 500 ppm
I20240503 23:24:59.436954 400 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 23:24:59.437290 1761 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240503 23:24:59.438724 400 webserver.cc:460] Webserver started at http://127.0.100.2:42917/ using document root <none> and password file <none>
I20240503 23:24:59.439020 400 fs_manager.cc:359] Metadata directory not provided
I20240503 23:24:59.439147 400 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 23:24:59.439296 400 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 23:24:59.440110 400 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZhIiPz/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714778688451250-400-0/minicluster-data/ts-1-root/instance:
uuid: "7d4c7dd4fc344655bf4fd1a67948bb3a"
format_stamp: "Formatted at 2024-05-03 23:24:59 on dist-test-slave-jvbp"
I20240503 23:24:59.441114 1479 raft_consensus.cc:1261] T 00000000000000000000000000000000 P e29ee7fe7b864b3d84f43581d1debeb3 [term 1 FOLLOWER]: Refusing update from remote peer a3b81f85f3594536ade9e155ca2d1cb4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 23:24:59.441145 1609 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 1a6c6637a2d64ca098e1d774053fd1ce [term 1 FOLLOWER]: Refusing update from remote peer a3b81f85f3594536ade9e155ca2d1cb4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 23:24:59.441783 1665 consensus_queue.cc:1035] T 00000000000000000000000000000000 P a3b81f85f3594536ade9e155ca2d1cb4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e29ee7fe7b864b3d84f43581d1debeb3" member_type: VOTER last_known_addr { host: "127.0.100.62" port: 46733 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 23:24:59.442108 1755 consensus_queue.cc:1035] T 00000000000000000000000000000000 P a3b81f85f3594536ade9e155ca2d1cb4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1a6c6637a2d64ca098e1d774053fd1ce" member_type: VOTER last_known_addr { host: "127.0.100.60" port: 38057 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 23:24:59.444059 400 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.002s
I20240503 23:24:59.450003 1652 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1a6c6637a2d64ca098e1d774053fd1ce [sys.catalog]: SysCatalogTable state changed. Reason: New leader a3b81f85f3594536ade9e155ca2d1cb4. Latest consensus state: current_term: 1 leader_uuid: "a3b81f85f3594536ade9e155ca2d1cb4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e29ee7fe7b864b3d84f43581d1debeb3" member_type: VOTER last_known_addr { host: "127.0.100.62" port: 46733 } } peers { permanent_uuid: "a3b81f85f3594536ade9e155ca2d1cb4" member_type: VOTER last_known_addr { host: "127.0.100.61" port: 45861 } } peers { permanent_uuid: "1a6c6637a2d64ca098e1d774053fd1ce" member_type: VOTER last_known_addr { host: "127.0.100.60" port: 38057 } } }
I20240503 23:24:59.450495 1652 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1a6c6637a2d64ca098e1d774053fd1ce [sys.catalog]: This master's current role is: FOLLOWER
I20240503 23:24:59.453591 1639 sys_catalog.cc:455] T 00000000000000000000000000000000 P e29ee7fe7b864b3d84f43581d1debeb3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a3b81f85f3594536ade9e155ca2d1cb4. Latest consensus state: current_term: 1 leader_uuid: "a3b81f85f3594536ade9e155ca2d1cb4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e29ee7fe7b864b3d84f43581d1debeb3" member_type: VOTER last_known_addr { host: "127.0.100.62" port: 46733 } } peers { permanent_uuid: "a3b81f85f3594536ade9e155ca2d1cb4" member_type: VOTER last_known_addr { host: "127.0.100.61" port: 45861 } } peers { permanent_uuid: "1a6c6637a2d64ca098e1d774053fd1ce" member_type: VOTER last_known_addr { host: "127.0.100.60" port: 38057 } } }
I20240503 23:24:59.453938 1639 sys_catalog.cc:458] T 00000000000000000000000000000000 P e29ee7fe7b864b3d84f43581d1debeb3 [sys.catalog]: This master's current role is: FOLLOWER
I20240503 23:24:59.454458 1770 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 23:24:59.455811 1755 sys_catalog.cc:455] T 00000000000000000000000000000000 P a3b81f85f3594536ade9e155ca2d1cb4 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "a3b81f85f3594536ade9e155ca2d1cb4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e29ee7fe7b864b3d84f43581d1debeb3" member_type: VOTER last_known_addr { host: "127.0.100.62" port: 46733 } } peers { permanent_uuid: "a3b81f85f3594536ade9e155ca2d1cb4" member_type: VOTER last_known_addr { host: "127.0.100.61" port: 45861 } } peers { permanent_uuid: "1a6c6637a2d64ca098e1d774053fd1ce" member_type: VOTER last_known_addr { host: "127.0.100.60" port: 38057 } } }
I20240503 23:24:59.456122 1755 sys_catalog.cc:458] T 00000000000000000000000000000000 P a3b81f85f3594536ade9e155ca2d1cb4 [sys.catalog]: This master's current role is: LEADER
I20240503 23:24:59.457248 1761 catalog_manager.cc:1347] Generated new cluster ID: 0a1435ca4f9c436d8c5957e687a4b867
I20240503 23:24:59.457388 1761 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240503 23:24:59.460801 1639 sys_catalog.cc:455] T 00000000000000000000000000000000 P e29ee7fe7b864b3d84f43581d1debeb3 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "a3b81f85f3594536ade9e155ca2d1cb4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e29ee7fe7b864b3d84f43581d1debeb3" member_type: VOTER last_known_addr { host: "127.0.100.62" port: 46733 } } peers { permanent_uuid: "a3b81f85f3594536ade9e155ca2d1cb4" member_type: VOTER last_known_addr { host: "127.0.100.61" port: 45861 } } peers { permanent_uuid: "1a6c6637a2d64ca098e1d774053fd1ce" member_type: VOTER last_known_addr { host: "127.0.100.60" port: 38057 } } }
I20240503 23:24:59.460778 1665 sys_catalog.cc:455] T 00000000000000000000000000000000 P a3b81f85f3594536ade9e155ca2d1cb4 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "a3b81f85f3594536ade9e155ca2d1cb4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e29ee7fe7b864b3d84f43581d1debeb3" member_type: VOTER last_known_addr { host: "127.0.100.62" port: 46733 } } peers { permanent_uuid: "a3b81f85f3594536ade9e155ca2d1cb4" member_type: VOTER last_known_addr { host: "127.0.100.61" port: 45861 } } peers { permanent_uuid: "1a6c6637a2d64ca098e1d774053fd1ce" member_type: VOTER last_known_addr { host: "127.0.100.60" port: 38057 } } }
I20240503 23:24:59.461125 1639 sys_catalog.cc:458] T 00000000000000000000000000000000 P e29ee7fe7b864b3d84f43581d1debeb3 [sys.catalog]: This master's current role is: FOLLOWER
I20240503 23:24:59.461150 1665 sys_catalog.cc:458] T 00000000000000000000000000000000 P a3b81f85f3594536ade9e155ca2d1cb4 [sys.catalog]: This master's current role is: LEADER
I20240503 23:24:59.461316 400 fs_manager.cc:725] Time spent opening block manager: real 0.011s user 0.002s sys 0.003s
I20240503 23:24:59.461477 400 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZhIiPz/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714778688451250-400-0/minicluster-data/ts-1-root
uuid: "7d4c7dd4fc344655bf4fd1a67948bb3a"
format_stamp: "Formatted at 2024-05-03 23:24:59 on dist-test-slave-jvbp"
I20240503 23:24:59.461629 400 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZhIiPz/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714778688451250-400-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskZhIiPz/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714778688451250-400-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskZhIiPz/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714778688451250-400-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 23:24:59.464078 1652 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1a6c6637a2d64ca098e1d774053fd1ce [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "a3b81f85f3594536ade9e155ca2d1cb4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e29ee7fe7b864b3d84f43581d1debeb3" member_type: VOTER last_known_addr { host: "127.0.100.62" port: 46733 } } peers { permanent_uuid: "a3b81f85f3594536ade9e155ca2d1cb4" member_type: VOTER last_known_addr { host: "127.0.100.61" port: 45861 } } peers { permanent_uuid: "1a6c6637a2d64ca098e1d774053fd1ce" member_type: VOTER last_known_addr { host: "127.0.100.60" port: 38057 } } }
I20240503 23:24:59.464341 1652 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1a6c6637a2d64ca098e1d774053fd1ce [sys.catalog]: This master's current role is: FOLLOWER
I20240503 23:24:59.481624 400 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 23:24:59.482398 400 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 23:24:59.483100 400 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 23:24:59.484310 400 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 23:24:59.484395 400 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 23:24:59.484532 400 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 23:24:59.484628 400 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.001s
I20240503 23:24:59.513608 400 rpc_server.cc:307] RPC server started. Bound to: 127.0.100.2:38443
I20240503 23:24:59.513634 1836 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.100.2:38443 every 8 connection(s)
I20240503 23:24:59.520072 1761 catalog_manager.cc:1370] Generated new certificate authority record
I20240503 23:24:59.521518 1761 catalog_manager.cc:1504] Loading token signing keys...
I20240503 23:24:59.536271 1761 catalog_manager.cc:5857] T 00000000000000000000000000000000 P a3b81f85f3594536ade9e155ca2d1cb4: Generated new TSK 0
I20240503 23:24:59.536715 1761 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240503 23:24:59.542089 1838 heartbeater.cc:347] Connected to a master server at 127.0.100.62:46733
I20240503 23:24:59.542357 1838 heartbeater.cc:464] Registering TS with master...
I20240503 23:24:59.542982 1838 heartbeater.cc:511] Master 127.0.100.62:46733 requested a full tablet report, sending...
I20240503 23:24:59.545790 1469 ts_manager.cc:194] Registered new tserver with Master: 7d4c7dd4fc344655bf4fd1a67948bb3a (127.0.100.2:38443)
I20240503 23:24:59.547912 400 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20240503 23:24:59.548450 1837 heartbeater.cc:347] Connected to a master server at 127.0.100.60:38057
I20240503 23:24:59.548835 1837 heartbeater.cc:464] Registering TS with master...
I20240503 23:24:59.549016 1840 heartbeater.cc:347] Connected to a master server at 127.0.100.61:45861
I20240503 23:24:59.549202 1840 heartbeater.cc:464] Registering TS with master...
I20240503 23:24:59.549530 1837 heartbeater.cc:511] Master 127.0.100.60:38057 requested a full tablet report, sending...
I20240503 23:24:59.549778 1840 heartbeater.cc:511] Master 127.0.100.61:45861 requested a full tablet report, sending...
I20240503 23:24:59.550652 1534 ts_manager.cc:194] Registered new tserver with Master: 7d4c7dd4fc344655bf4fd1a67948bb3a (127.0.100.2:38443)
I20240503 23:24:59.550899 1599 ts_manager.cc:194] Registered new tserver with Master: 7d4c7dd4fc344655bf4fd1a67948bb3a (127.0.100.2:38443)
I20240503 23:24:59.552249 1534 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:41314
W20240503 23:24:59.553715 1845 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 23:24:59.555648 1846 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 23:24:59.557214 1848 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 23:24:59.557960 400 server_base.cc:1034] running on GCE node
I20240503 23:24:59.558254 400 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240503 23:24:59.559157 400 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240503 23:24:59.559217 400 hybrid_clock.cc:648] HybridClock initialized: now 1714778699559201 us; error 0 us; skew 500 ppm
I20240503 23:24:59.559381 400 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 23:24:59.561264 400 webserver.cc:460] Webserver started at http://127.0.100.3:33715/ using document root <none> and password file <none>
I20240503 23:24:59.561523 400 fs_manager.cc:359] Metadata directory not provided
I20240503 23:24:59.561605 400 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 23:24:59.561736 400 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 23:24:59.562528 400 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZhIiPz/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714778688451250-400-0/minicluster-data/ts-2-root/instance:
uuid: "2a4b81befdbb4672a8fb3a6b854aad63"
format_stamp: "Formatted at 2024-05-03 23:24:59 on dist-test-slave-jvbp"
I20240503 23:24:59.565306 400 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.004s sys 0.000s
I20240503 23:24:59.567198 1853 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 23:24:59.567656 400 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.002s sys 0.000s
I20240503 23:24:59.567775 400 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZhIiPz/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714778688451250-400-0/minicluster-data/ts-2-root
uuid: "2a4b81befdbb4672a8fb3a6b854aad63"
format_stamp: "Formatted at 2024-05-03 23:24:59 on dist-test-slave-jvbp"
I20240503 23:24:59.567909 400 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZhIiPz/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714778688451250-400-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-taskZhIiPz/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714778688451250-400-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-taskZhIiPz/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714778688451250-400-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 23:24:59.580533 400 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 23:24:59.581396 400 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 23:24:59.582193 400 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 23:24:59.583209 400 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 23:24:59.583310 400 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 23:24:59.583422 400 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 23:24:59.583499 400 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 23:24:59.605674 400 rpc_server.cc:307] RPC server started. Bound to: 127.0.100.3:39279
I20240503 23:24:59.605713 1915 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.100.3:39279 every 8 connection(s)
I20240503 23:24:59.623029 1917 heartbeater.cc:347] Connected to a master server at 127.0.100.62:46733
I20240503 23:24:59.623327 1918 heartbeater.cc:347] Connected to a master server at 127.0.100.61:45861
I20240503 23:24:59.623330 1917 heartbeater.cc:464] Registering TS with master...
I20240503 23:24:59.623361 1916 heartbeater.cc:347] Connected to a master server at 127.0.100.60:38057
I20240503 23:24:59.623514 1918 heartbeater.cc:464] Registering TS with master...
I20240503 23:24:59.623569 1916 heartbeater.cc:464] Registering TS with master...
I20240503 23:24:59.624169 1918 heartbeater.cc:511] Master 127.0.100.61:45861 requested a full tablet report, sending...
I20240503 23:24:59.624243 1917 heartbeater.cc:511] Master 127.0.100.62:46733 requested a full tablet report, sending...
I20240503 23:24:59.624270 1916 heartbeater.cc:511] Master 127.0.100.60:38057 requested a full tablet report, sending...
I20240503 23:24:59.625249 1534 ts_manager.cc:194] Registered new tserver with Master: 2a4b81befdbb4672a8fb3a6b854aad63 (127.0.100.3:39279)
I20240503 23:24:59.625377 1599 ts_manager.cc:194] Registered new tserver with Master: 2a4b81befdbb4672a8fb3a6b854aad63 (127.0.100.3:39279)
I20240503 23:24:59.625413 1469 ts_manager.cc:194] Registered new tserver with Master: 2a4b81befdbb4672a8fb3a6b854aad63 (127.0.100.3:39279)
I20240503 23:24:59.626310 1534 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:41330
I20240503 23:24:59.626407 400 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.016601218s
I20240503 23:25:00.294776 1650 catalog_manager.cc:1259] Loaded cluster ID: 0a1435ca4f9c436d8c5957e687a4b867
I20240503 23:25:00.294943 1650 catalog_manager.cc:1552] T 00000000000000000000000000000000 P e29ee7fe7b864b3d84f43581d1debeb3: loading cluster ID for follower catalog manager: success
I20240503 23:25:00.297250 1650 catalog_manager.cc:1574] T 00000000000000000000000000000000 P e29ee7fe7b864b3d84f43581d1debeb3: acquiring CA information for follower catalog manager: success
I20240503 23:25:00.298535 1650 catalog_manager.cc:1602] T 00000000000000000000000000000000 P e29ee7fe7b864b3d84f43581d1debeb3: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20240503 23:25:00.309561 1663 catalog_manager.cc:1259] Loaded cluster ID: 0a1435ca4f9c436d8c5957e687a4b867
I20240503 23:25:00.309708 1663 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 1a6c6637a2d64ca098e1d774053fd1ce: loading cluster ID for follower catalog manager: success
I20240503 23:25:00.311874 1663 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 1a6c6637a2d64ca098e1d774053fd1ce: acquiring CA information for follower catalog manager: success
I20240503 23:25:00.313319 1663 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 1a6c6637a2d64ca098e1d774053fd1ce: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20240503 23:25:00.433113 1534 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:45254
I20240503 23:25:00.554608 1840 heartbeater.cc:503] Master 127.0.100.61:45861 was elected leader, sending a full tablet report...
I20240503 23:25:00.627627 1918 heartbeater.cc:503] Master 127.0.100.61:45861 was elected leader, sending a full tablet report...
I20240503 23:25:01.323202 1675 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240503 23:25:01.434736 1750 heartbeater.cc:503] Master 127.0.100.61:45861 was elected leader, sending a full tablet report...
I20240503 23:25:01.689296 400 test_util.cc:268] Using random seed: 123859381
I20240503 23:25:01.718669 1534 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41346:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 1
split_rows_range_bounds {
rows: "\004\001\000\252\252\252*\004\001\000TUUU""\004\001\000\252\252\252*\004\001\000TUUU"
indirect_data: """"
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240503 23:25:01.767095 1802 tablet_service.cc:1467] Processing CreateTablet for tablet b8a96c4240ba4dd1b68d61cdf7dc8239 (DEFAULT_TABLE table=test-workload [id=fcb70a042c934bcf8c6de2d9332e1477]), partition=RANGE (key) PARTITION VALUES < 715827882
I20240503 23:25:01.767846 1802 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b8a96c4240ba4dd1b68d61cdf7dc8239. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 23:25:01.771716 1881 tablet_service.cc:1467] Processing CreateTablet for tablet d90b129a95954e83a2d1d4c5def6a82f (DEFAULT_TABLE table=test-workload [id=fcb70a042c934bcf8c6de2d9332e1477]), partition=RANGE (key) PARTITION 715827882 <= VALUES < 1431655764
I20240503 23:25:01.772387 1881 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d90b129a95954e83a2d1d4c5def6a82f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 23:25:01.772276 1713 tablet_service.cc:1467] Processing CreateTablet for tablet 27acd48c50e34ea58f455356b3e47eca (DEFAULT_TABLE table=test-workload [id=fcb70a042c934bcf8c6de2d9332e1477]), partition=RANGE (key) PARTITION 1431655764 <= VALUES
I20240503 23:25:01.772845 1713 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 27acd48c50e34ea58f455356b3e47eca. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 23:25:01.778898 1952 tablet_bootstrap.cc:492] T b8a96c4240ba4dd1b68d61cdf7dc8239 P 7d4c7dd4fc344655bf4fd1a67948bb3a: Bootstrap starting.
I20240503 23:25:01.780208 1953 tablet_bootstrap.cc:492] T 27acd48c50e34ea58f455356b3e47eca P 0fc5931f4b97441b92926f9ca3f0ea1b: Bootstrap starting.
I20240503 23:25:01.781294 1952 tablet_bootstrap.cc:654] T b8a96c4240ba4dd1b68d61cdf7dc8239 P 7d4c7dd4fc344655bf4fd1a67948bb3a: Neither blocks nor log segments found. Creating new log.
I20240503 23:25:01.784219 1953 tablet_bootstrap.cc:654] T 27acd48c50e34ea58f455356b3e47eca P 0fc5931f4b97441b92926f9ca3f0ea1b: Neither blocks nor log segments found. Creating new log.
I20240503 23:25:01.788591 1954 tablet_bootstrap.cc:492] T d90b129a95954e83a2d1d4c5def6a82f P 2a4b81befdbb4672a8fb3a6b854aad63: Bootstrap starting.
I20240503 23:25:01.790237 1953 tablet_bootstrap.cc:492] T 27acd48c50e34ea58f455356b3e47eca P 0fc5931f4b97441b92926f9ca3f0ea1b: No bootstrap required, opened a new log
I20240503 23:25:01.790530 1953 ts_tablet_manager.cc:1367] T 27acd48c50e34ea58f455356b3e47eca P 0fc5931f4b97441b92926f9ca3f0ea1b: Time spent bootstrapping tablet: real 0.010s user 0.005s sys 0.003s
I20240503 23:25:01.791857 1954 tablet_bootstrap.cc:654] T d90b129a95954e83a2d1d4c5def6a82f P 2a4b81befdbb4672a8fb3a6b854aad63: Neither blocks nor log segments found. Creating new log.
I20240503 23:25:01.791762 1953 raft_consensus.cc:348] T 27acd48c50e34ea58f455356b3e47eca P 0fc5931f4b97441b92926f9ca3f0ea1b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0fc5931f4b97441b92926f9ca3f0ea1b" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 45121 } }
I20240503 23:25:01.792675 1953 raft_consensus.cc:374] T 27acd48c50e34ea58f455356b3e47eca P 0fc5931f4b97441b92926f9ca3f0ea1b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 23:25:01.792840 1953 raft_consensus.cc:729] T 27acd48c50e34ea58f455356b3e47eca P 0fc5931f4b97441b92926f9ca3f0ea1b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0fc5931f4b97441b92926f9ca3f0ea1b, State: Initialized, Role: FOLLOWER
I20240503 23:25:01.793120 1953 consensus_queue.cc:260] T 27acd48c50e34ea58f455356b3e47eca P 0fc5931f4b97441b92926f9ca3f0ea1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0fc5931f4b97441b92926f9ca3f0ea1b" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 45121 } }
I20240503 23:25:01.793324 1953 raft_consensus.cc:388] T 27acd48c50e34ea58f455356b3e47eca P 0fc5931f4b97441b92926f9ca3f0ea1b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 23:25:01.793429 1953 raft_consensus.cc:482] T 27acd48c50e34ea58f455356b3e47eca P 0fc5931f4b97441b92926f9ca3f0ea1b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 23:25:01.793550 1952 tablet_bootstrap.cc:492] T b8a96c4240ba4dd1b68d61cdf7dc8239 P 7d4c7dd4fc344655bf4fd1a67948bb3a: No bootstrap required, opened a new log
I20240503 23:25:01.793710 1953 raft_consensus.cc:3037] T 27acd48c50e34ea58f455356b3e47eca P 0fc5931f4b97441b92926f9ca3f0ea1b [term 0 FOLLOWER]: Advancing to term 1
I20240503 23:25:01.793809 1952 ts_tablet_manager.cc:1367] T b8a96c4240ba4dd1b68d61cdf7dc8239 P 7d4c7dd4fc344655bf4fd1a67948bb3a: Time spent bootstrapping tablet: real 0.015s user 0.009s sys 0.005s
I20240503 23:25:01.794786 1954 tablet_bootstrap.cc:492] T d90b129a95954e83a2d1d4c5def6a82f P 2a4b81befdbb4672a8fb3a6b854aad63: No bootstrap required, opened a new log
I20240503 23:25:01.794862 1952 raft_consensus.cc:348] T b8a96c4240ba4dd1b68d61cdf7dc8239 P 7d4c7dd4fc344655bf4fd1a67948bb3a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7d4c7dd4fc344655bf4fd1a67948bb3a" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 38443 } }
I20240503 23:25:01.795080 1954 ts_tablet_manager.cc:1367] T d90b129a95954e83a2d1d4c5def6a82f P 2a4b81befdbb4672a8fb3a6b854aad63: Time spent bootstrapping tablet: real 0.007s user 0.000s sys 0.005s
I20240503 23:25:01.795100 1952 raft_consensus.cc:374] T b8a96c4240ba4dd1b68d61cdf7dc8239 P 7d4c7dd4fc344655bf4fd1a67948bb3a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 23:25:01.795318 1952 raft_consensus.cc:729] T b8a96c4240ba4dd1b68d61cdf7dc8239 P 7d4c7dd4fc344655bf4fd1a67948bb3a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7d4c7dd4fc344655bf4fd1a67948bb3a, State: Initialized, Role: FOLLOWER
I20240503 23:25:01.795679 1952 consensus_queue.cc:260] T b8a96c4240ba4dd1b68d61cdf7dc8239 P 7d4c7dd4fc344655bf4fd1a67948bb3a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7d4c7dd4fc344655bf4fd1a67948bb3a" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 38443 } }
I20240503 23:25:01.795913 1952 raft_consensus.cc:388] T b8a96c4240ba4dd1b68d61cdf7dc8239 P 7d4c7dd4fc344655bf4fd1a67948bb3a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 23:25:01.796013 1952 raft_consensus.cc:482] T b8a96c4240ba4dd1b68d61cdf7dc8239 P 7d4c7dd4fc344655bf4fd1a67948bb3a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 23:25:01.796166 1952 raft_consensus.cc:3037] T b8a96c4240ba4dd1b68d61cdf7dc8239 P 7d4c7dd4fc344655bf4fd1a67948bb3a [term 0 FOLLOWER]: Advancing to term 1
I20240503 23:25:01.796156 1954 raft_consensus.cc:348] T d90b129a95954e83a2d1d4c5def6a82f P 2a4b81befdbb4672a8fb3a6b854aad63 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2a4b81befdbb4672a8fb3a6b854aad63" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 39279 } }
I20240503 23:25:01.796329 1954 raft_consensus.cc:374] T d90b129a95954e83a2d1d4c5def6a82f P 2a4b81befdbb4672a8fb3a6b854aad63 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 23:25:01.796428 1954 raft_consensus.cc:729] T d90b129a95954e83a2d1d4c5def6a82f P 2a4b81befdbb4672a8fb3a6b854aad63 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2a4b81befdbb4672a8fb3a6b854aad63, State: Initialized, Role: FOLLOWER
I20240503 23:25:01.796722 1954 consensus_queue.cc:260] T d90b129a95954e83a2d1d4c5def6a82f P 2a4b81befdbb4672a8fb3a6b854aad63 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2a4b81befdbb4672a8fb3a6b854aad63" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 39279 } }
I20240503 23:25:01.796964 1953 raft_consensus.cc:504] T 27acd48c50e34ea58f455356b3e47eca P 0fc5931f4b97441b92926f9ca3f0ea1b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0fc5931f4b97441b92926f9ca3f0ea1b" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 45121 } }
I20240503 23:25:01.797020 1954 raft_consensus.cc:388] T d90b129a95954e83a2d1d4c5def6a82f P 2a4b81befdbb4672a8fb3a6b854aad63 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 23:25:01.798142 1954 raft_consensus.cc:482] T d90b129a95954e83a2d1d4c5def6a82f P 2a4b81befdbb4672a8fb3a6b854aad63 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 23:25:01.798146 1953 leader_election.cc:302] T 27acd48c50e34ea58f455356b3e47eca P 0fc5931f4b97441b92926f9ca3f0ea1b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0fc5931f4b97441b92926f9ca3f0ea1b; no voters:
I20240503 23:25:01.798326 1954 raft_consensus.cc:3037] T d90b129a95954e83a2d1d4c5def6a82f P 2a4b81befdbb4672a8fb3a6b854aad63 [term 0 FOLLOWER]: Advancing to term 1
I20240503 23:25:01.800654 1952 raft_consensus.cc:504] T b8a96c4240ba4dd1b68d61cdf7dc8239 P 7d4c7dd4fc344655bf4fd1a67948bb3a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7d4c7dd4fc344655bf4fd1a67948bb3a" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 38443 } }
I20240503 23:25:01.801008 1952 leader_election.cc:302] T b8a96c4240ba4dd1b68d61cdf7dc8239 P 7d4c7dd4fc344655bf4fd1a67948bb3a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7d4c7dd4fc344655bf4fd1a67948bb3a; no voters:
I20240503 23:25:01.802484 1954 raft_consensus.cc:504] T d90b129a95954e83a2d1d4c5def6a82f P 2a4b81befdbb4672a8fb3a6b854aad63 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2a4b81befdbb4672a8fb3a6b854aad63" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 39279 } }
I20240503 23:25:01.802866 1954 leader_election.cc:302] T d90b129a95954e83a2d1d4c5def6a82f P 2a4b81befdbb4672a8fb3a6b854aad63 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2a4b81befdbb4672a8fb3a6b854aad63; no voters:
I20240503 23:25:01.803628 1954 leader_election.cc:288] T d90b129a95954e83a2d1d4c5def6a82f P 2a4b81befdbb4672a8fb3a6b854aad63 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240503 23:25:01.804100 1960 raft_consensus.cc:2781] T d90b129a95954e83a2d1d4c5def6a82f P 2a4b81befdbb4672a8fb3a6b854aad63 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 23:25:01.805843 1953 leader_election.cc:288] T 27acd48c50e34ea58f455356b3e47eca P 0fc5931f4b97441b92926f9ca3f0ea1b [CANDIDATE]: Term 1 election: Requested vote from peers
I20240503 23:25:01.806092 1958 raft_consensus.cc:2781] T 27acd48c50e34ea58f455356b3e47eca P 0fc5931f4b97441b92926f9ca3f0ea1b [term 1 FOLLOWER]: Leader election won for term 1
I20240503 23:25:01.809837 1952 leader_election.cc:288] T b8a96c4240ba4dd1b68d61cdf7dc8239 P 7d4c7dd4fc344655bf4fd1a67948bb3a [CANDIDATE]: Term 1 election: Requested vote from peers
I20240503 23:25:01.810014 1959 raft_consensus.cc:2781] T b8a96c4240ba4dd1b68d61cdf7dc8239 P 7d4c7dd4fc344655bf4fd1a67948bb3a [term 1 FOLLOWER]: Leader election won for term 1
I20240503 23:25:01.812328 1958 raft_consensus.cc:686] T 27acd48c50e34ea58f455356b3e47eca P 0fc5931f4b97441b92926f9ca3f0ea1b [term 1 LEADER]: Becoming Leader. State: Replica: 0fc5931f4b97441b92926f9ca3f0ea1b, State: Running, Role: LEADER
I20240503 23:25:01.814283 1958 consensus_queue.cc:237] T 27acd48c50e34ea58f455356b3e47eca P 0fc5931f4b97441b92926f9ca3f0ea1b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0fc5931f4b97441b92926f9ca3f0ea1b" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 45121 } }
I20240503 23:25:01.816097 1959 raft_consensus.cc:686] T b8a96c4240ba4dd1b68d61cdf7dc8239 P 7d4c7dd4fc344655bf4fd1a67948bb3a [term 1 LEADER]: Becoming Leader. State: Replica: 7d4c7dd4fc344655bf4fd1a67948bb3a, State: Running, Role: LEADER
I20240503 23:25:01.816485 1959 consensus_queue.cc:237] T b8a96c4240ba4dd1b68d61cdf7dc8239 P 7d4c7dd4fc344655bf4fd1a67948bb3a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7d4c7dd4fc344655bf4fd1a67948bb3a" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 38443 } }
I20240503 23:25:01.825052 1534 catalog_manager.cc:5484] T 27acd48c50e34ea58f455356b3e47eca P 0fc5931f4b97441b92926f9ca3f0ea1b reported cstate change: term changed from 0 to 1, leader changed from <none> to 0fc5931f4b97441b92926f9ca3f0ea1b (127.0.100.1). New cstate: current_term: 1 leader_uuid: "0fc5931f4b97441b92926f9ca3f0ea1b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0fc5931f4b97441b92926f9ca3f0ea1b" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 45121 } health_report { overall_health: HEALTHY } } }
I20240503 23:25:01.835758 1532 catalog_manager.cc:5484] T b8a96c4240ba4dd1b68d61cdf7dc8239 P 7d4c7dd4fc344655bf4fd1a67948bb3a reported cstate change: term changed from 0 to 1, leader changed from <none> to 7d4c7dd4fc344655bf4fd1a67948bb3a (127.0.100.2). New cstate: current_term: 1 leader_uuid: "7d4c7dd4fc344655bf4fd1a67948bb3a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7d4c7dd4fc344655bf4fd1a67948bb3a" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 38443 } health_report { overall_health: HEALTHY } } }
I20240503 23:25:01.836575 1960 raft_consensus.cc:686] T d90b129a95954e83a2d1d4c5def6a82f P 2a4b81befdbb4672a8fb3a6b854aad63 [term 1 LEADER]: Becoming Leader. State: Replica: 2a4b81befdbb4672a8fb3a6b854aad63, State: Running, Role: LEADER
I20240503 23:25:01.837108 1954 ts_tablet_manager.cc:1398] T d90b129a95954e83a2d1d4c5def6a82f P 2a4b81befdbb4672a8fb3a6b854aad63: Time spent starting tablet: real 0.028s user 0.006s sys 0.007s
I20240503 23:25:01.837021 1960 consensus_queue.cc:237] T d90b129a95954e83a2d1d4c5def6a82f P 2a4b81befdbb4672a8fb3a6b854aad63 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2a4b81befdbb4672a8fb3a6b854aad63" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 39279 } }
I20240503 23:25:01.837801 1953 ts_tablet_manager.cc:1398] T 27acd48c50e34ea58f455356b3e47eca P 0fc5931f4b97441b92926f9ca3f0ea1b: Time spent starting tablet: real 0.047s user 0.017s sys 0.012s
I20240503 23:25:01.835812 1952 ts_tablet_manager.cc:1398] T b8a96c4240ba4dd1b68d61cdf7dc8239 P 7d4c7dd4fc344655bf4fd1a67948bb3a: Time spent starting tablet: real 0.042s user 0.015s sys 0.011s
I20240503 23:25:01.843374 1533 catalog_manager.cc:5484] T d90b129a95954e83a2d1d4c5def6a82f P 2a4b81befdbb4672a8fb3a6b854aad63 reported cstate change: term changed from 0 to 1, leader changed from <none> to 2a4b81befdbb4672a8fb3a6b854aad63 (127.0.100.3). New cstate: current_term: 1 leader_uuid: "2a4b81befdbb4672a8fb3a6b854aad63" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2a4b81befdbb4672a8fb3a6b854aad63" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 39279 } health_report { overall_health: HEALTHY } } }
I20240503 23:25:01.870128 400 master.cc:537] Master@127.0.100.61:45861 shutting down...
I20240503 23:25:01.877135 400 raft_consensus.cc:2229] T 00000000000000000000000000000000 P a3b81f85f3594536ade9e155ca2d1cb4 [term 1 LEADER]: Raft consensus shutting down.
I20240503 23:25:01.877597 400 raft_consensus.cc:2258] T 00000000000000000000000000000000 P a3b81f85f3594536ade9e155ca2d1cb4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 23:25:01.877717 400 tablet_replica.cc:330] T 00000000000000000000000000000000 P a3b81f85f3594536ade9e155ca2d1cb4: stopping tablet replica
I20240503 23:25:01.894197 400 master.cc:552] Master@127.0.100.61:45861 shutdown complete.
W20240503 23:25:02.862377 1918 heartbeater.cc:644] Failed to heartbeat to 127.0.100.61:45861 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.100.61:45861: connect: Connection refused (error 111)
I20240503 23:25:03.368245 1979 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1a6c6637a2d64ca098e1d774053fd1ce [term 1 FOLLOWER]: Starting pre-election (detected failure of leader a3b81f85f3594536ade9e155ca2d1cb4)
I20240503 23:25:03.368522 1979 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1a6c6637a2d64ca098e1d774053fd1ce [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e29ee7fe7b864b3d84f43581d1debeb3" member_type: VOTER last_known_addr { host: "127.0.100.62" port: 46733 } } peers { permanent_uuid: "a3b81f85f3594536ade9e155ca2d1cb4" member_type: VOTER last_known_addr { host: "127.0.100.61" port: 45861 } } peers { permanent_uuid: "1a6c6637a2d64ca098e1d774053fd1ce" member_type: VOTER last_known_addr { host: "127.0.100.60" port: 38057 } }
I20240503 23:25:03.369475 1979 leader_election.cc:288] T 00000000000000000000000000000000 P 1a6c6637a2d64ca098e1d774053fd1ce [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers e29ee7fe7b864b3d84f43581d1debeb3 (127.0.100.62:46733), a3b81f85f3594536ade9e155ca2d1cb4 (127.0.100.61:45861)
I20240503 23:25:03.370280 1980 raft_consensus.cc:482] T 00000000000000000000000000000000 P e29ee7fe7b864b3d84f43581d1debeb3 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader a3b81f85f3594536ade9e155ca2d1cb4)
I20240503 23:25:03.370549 1479 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "1a6c6637a2d64ca098e1d774053fd1ce" candidate_term: 2 candidate_status { last_received { term: 1 index: 11 } } ignore_live_leader: false dest_uuid: "e29ee7fe7b864b3d84f43581d1debeb3" is_pre_election: true
I20240503 23:25:03.370635 1980 raft_consensus.cc:504] T 00000000000000000000000000000000 P e29ee7fe7b864b3d84f43581d1debeb3 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e29ee7fe7b864b3d84f43581d1debeb3" member_type: VOTER last_known_addr { host: "127.0.100.62" port: 46733 } } peers { permanent_uuid: "a3b81f85f3594536ade9e155ca2d1cb4" member_type: VOTER last_known_addr { host: "127.0.100.61" port: 45861 } } peers { permanent_uuid: "1a6c6637a2d64ca098e1d774053fd1ce" member_type: VOTER last_known_addr { host: "127.0.100.60" port: 38057 } }
I20240503 23:25:03.371151 1479 raft_consensus.cc:2454] T 00000000000000000000000000000000 P e29ee7fe7b864b3d84f43581d1debeb3 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1a6c6637a2d64ca098e1d774053fd1ce in term 1.
I20240503 23:25:03.372280 1585 leader_election.cc:302] T 00000000000000000000000000000000 P 1a6c6637a2d64ca098e1d774053fd1ce [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1a6c6637a2d64ca098e1d774053fd1ce, e29ee7fe7b864b3d84f43581d1debeb3; no voters:
W20240503 23:25:03.372705 1582 leader_election.cc:334] T 00000000000000000000000000000000 P 1a6c6637a2d64ca098e1d774053fd1ce [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer a3b81f85f3594536ade9e155ca2d1cb4 (127.0.100.61:45861): Network error: Client connection negotiation failed: client connection to 127.0.100.61:45861: connect: Connection refused (error 111)
I20240503 23:25:03.372844 1980 leader_election.cc:288] T 00000000000000000000000000000000 P e29ee7fe7b864b3d84f43581d1debeb3 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a3b81f85f3594536ade9e155ca2d1cb4 (127.0.100.61:45861), 1a6c6637a2d64ca098e1d774053fd1ce (127.0.100.60:38057)
I20240503 23:25:03.372706 1979 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1a6c6637a2d64ca098e1d774053fd1ce [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240503 23:25:03.373428 1979 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1a6c6637a2d64ca098e1d774053fd1ce [term 1 FOLLOWER]: Starting leader election (detected failure of leader a3b81f85f3594536ade9e155ca2d1cb4)
I20240503 23:25:03.373454 1608 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "e29ee7fe7b864b3d84f43581d1debeb3" candidate_term: 2 candidate_status { last_received { term: 1 index: 11 } } ignore_live_leader: false dest_uuid: "1a6c6637a2d64ca098e1d774053fd1ce" is_pre_election: true
I20240503 23:25:03.373598 1979 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1a6c6637a2d64ca098e1d774053fd1ce [term 1 FOLLOWER]: Advancing to term 2
W20240503 23:25:03.373606 1453 leader_election.cc:334] T 00000000000000000000000000000000 P e29ee7fe7b864b3d84f43581d1debeb3 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer a3b81f85f3594536ade9e155ca2d1cb4 (127.0.100.61:45861): Network error: Client connection negotiation failed: client connection to 127.0.100.61:45861: connect: Connection refused (error 111)
I20240503 23:25:03.375725 1979 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1a6c6637a2d64ca098e1d774053fd1ce [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e29ee7fe7b864b3d84f43581d1debeb3" member_type: VOTER last_known_addr { host: "127.0.100.62" port: 46733 } } peers { permanent_uuid: "a3b81f85f3594536ade9e155ca2d1cb4" member_type: VOTER last_known_addr { host: "127.0.100.61" port: 45861 } } peers { permanent_uuid: "1a6c6637a2d64ca098e1d774053fd1ce" member_type: VOTER last_known_addr { host: "127.0.100.60" port: 38057 } }
I20240503 23:25:03.376158 1608 raft_consensus.cc:2379] T 00000000000000000000000000000000 P 1a6c6637a2d64ca098e1d774053fd1ce [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e29ee7fe7b864b3d84f43581d1debeb3 in current term 2: Already voted for candidate 1a6c6637a2d64ca098e1d774053fd1ce in this term.
I20240503 23:25:03.376605 1979 leader_election.cc:288] T 00000000000000000000000000000000 P 1a6c6637a2d64ca098e1d774053fd1ce [CANDIDATE]: Term 2 election: Requested vote from peers e29ee7fe7b864b3d84f43581d1debeb3 (127.0.100.62:46733), a3b81f85f3594536ade9e155ca2d1cb4 (127.0.100.61:45861)
I20240503 23:25:03.377172 1454 leader_election.cc:302] T 00000000000000000000000000000000 P e29ee7fe7b864b3d84f43581d1debeb3 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e29ee7fe7b864b3d84f43581d1debeb3; no voters: 1a6c6637a2d64ca098e1d774053fd1ce, a3b81f85f3594536ade9e155ca2d1cb4
I20240503 23:25:03.377148 1479 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "1a6c6637a2d64ca098e1d774053fd1ce" candidate_term: 2 candidate_status { last_received { term: 1 index: 11 } } ignore_live_leader: false dest_uuid: "e29ee7fe7b864b3d84f43581d1debeb3"
I20240503 23:25:03.377388 1479 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e29ee7fe7b864b3d84f43581d1debeb3 [term 1 FOLLOWER]: Advancing to term 2
W20240503 23:25:03.378343 1582 leader_election.cc:334] T 00000000000000000000000000000000 P 1a6c6637a2d64ca098e1d774053fd1ce [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer a3b81f85f3594536ade9e155ca2d1cb4 (127.0.100.61:45861): Network error: Client connection negotiation failed: client connection to 127.0.100.61:45861: connect: Connection refused (error 111)
I20240503 23:25:03.379844 1479 raft_consensus.cc:2454] T 00000000000000000000000000000000 P e29ee7fe7b864b3d84f43581d1debeb3 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1a6c6637a2d64ca098e1d774053fd1ce in term 2.
I20240503 23:25:03.380074 1980 raft_consensus.cc:2726] T 00000000000000000000000000000000 P e29ee7fe7b864b3d84f43581d1debeb3 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240503 23:25:03.380354 1585 leader_election.cc:302] T 00000000000000000000000000000000 P 1a6c6637a2d64ca098e1d774053fd1ce [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1a6c6637a2d64ca098e1d774053fd1ce, e29ee7fe7b864b3d84f43581d1debeb3; no voters: a3b81f85f3594536ade9e155ca2d1cb4
I20240503 23:25:03.380692 1979 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1a6c6637a2d64ca098e1d774053fd1ce [term 2 FOLLOWER]: Leader election won for term 2
I20240503 23:25:03.381165 1979 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1a6c6637a2d64ca098e1d774053fd1ce [term 2 LEADER]: Becoming Leader. State: Replica: 1a6c6637a2d64ca098e1d774053fd1ce, State: Running, Role: LEADER
I20240503 23:25:03.381454 1979 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1a6c6637a2d64ca098e1d774053fd1ce [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 1.11, Last appended by leader: 11, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e29ee7fe7b864b3d84f43581d1debeb3" member_type: VOTER last_known_addr { host: "127.0.100.62" port: 46733 } } peers { permanent_uuid: "a3b81f85f3594536ade9e155ca2d1cb4" member_type: VOTER last_known_addr { host: "127.0.100.61" port: 45861 } } peers { permanent_uuid: "1a6c6637a2d64ca098e1d774053fd1ce" member_type: VOTER last_known_addr { host: "127.0.100.60" port: 38057 } }
I20240503 23:25:03.382733 1985 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1a6c6637a2d64ca098e1d774053fd1ce [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1a6c6637a2d64ca098e1d774053fd1ce. Latest consensus state: current_term: 2 leader_uuid: "1a6c6637a2d64ca098e1d774053fd1ce" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e29ee7fe7b864b3d84f43581d1debeb3" member_type: VOTER last_known_addr { host: "127.0.100.62" port: 46733 } } peers { permanent_uuid: "a3b81f85f3594536ade9e155ca2d1cb4" member_type: VOTER last_known_addr { host: "127.0.100.61" port: 45861 } } peers { permanent_uuid: "1a6c6637a2d64ca098e1d774053fd1ce" member_type: VOTER last_known_addr { host: "127.0.100.60" port: 38057 } } }
I20240503 23:25:03.382978 1985 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1a6c6637a2d64ca098e1d774053fd1ce [sys.catalog]: This master's current role is: LEADER
I20240503 23:25:03.383471 1987 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240503 23:25:03.385668 1987 catalog_manager.cc:669] Loaded metadata for table test-workload [id=fcb70a042c934bcf8c6de2d9332e1477]
I20240503 23:25:03.387374 1987 tablet_loader.cc:96] loaded metadata for tablet 27acd48c50e34ea58f455356b3e47eca (table test-workload [id=fcb70a042c934bcf8c6de2d9332e1477])
I20240503 23:25:03.387814 1987 tablet_loader.cc:96] loaded metadata for tablet b8a96c4240ba4dd1b68d61cdf7dc8239 (table test-workload [id=fcb70a042c934bcf8c6de2d9332e1477])
I20240503 23:25:03.388144 1987 tablet_loader.cc:96] loaded metadata for tablet d90b129a95954e83a2d1d4c5def6a82f (table test-workload [id=fcb70a042c934bcf8c6de2d9332e1477])
I20240503 23:25:03.388410 1987 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240503 23:25:03.389535 1987 catalog_manager.cc:1259] Loaded cluster ID: 0a1435ca4f9c436d8c5957e687a4b867
I20240503 23:25:03.389640 1987 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240503 23:25:03.391156 1987 catalog_manager.cc:1504] Loading token signing keys...
I20240503 23:25:03.392155 1987 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1a6c6637a2d64ca098e1d774053fd1ce: Loaded TSK: 0
I20240503 23:25:03.393024 1987 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20240503 23:25:03.812216 1582 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.100.61:45861: connect: Connection refused (error 111) [suppressed 15 similar messages]
W20240503 23:25:03.813586 1582 consensus_peers.cc:485] T 00000000000000000000000000000000 P 1a6c6637a2d64ca098e1d774053fd1ce -> Peer a3b81f85f3594536ade9e155ca2d1cb4 (127.0.100.61:45861): Couldn't send request to peer a3b81f85f3594536ade9e155ca2d1cb4. Status: Network error: Client connection negotiation failed: client connection to 127.0.100.61:45861: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 23:25:03.896466 1479 raft_consensus.cc:1261] T 00000000000000000000000000000000 P e29ee7fe7b864b3d84f43581d1debeb3 [term 2 FOLLOWER]: Refusing update from remote peer 1a6c6637a2d64ca098e1d774053fd1ce: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 2 index: 12. (index mismatch)
I20240503 23:25:03.897447 1985 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 1a6c6637a2d64ca098e1d774053fd1ce [LEADER]: Connected to new peer: Peer: permanent_uuid: "e29ee7fe7b864b3d84f43581d1debeb3" member_type: VOTER last_known_addr { host: "127.0.100.62" port: 46733 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240503 23:25:03.900079 1991 sys_catalog.cc:455] T 00000000000000000000000000000000 P e29ee7fe7b864b3d84f43581d1debeb3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1a6c6637a2d64ca098e1d774053fd1ce. Latest consensus state: current_term: 2 leader_uuid: "1a6c6637a2d64ca098e1d774053fd1ce" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e29ee7fe7b864b3d84f43581d1debeb3" member_type: VOTER last_known_addr { host: "127.0.100.62" port: 46733 } } peers { permanent_uuid: "a3b81f85f3594536ade9e155ca2d1cb4" member_type: VOTER last_known_addr { host: "127.0.100.61" port: 45861 } } peers { permanent_uuid: "1a6c6637a2d64ca098e1d774053fd1ce" member_type: VOTER last_known_addr { host: "127.0.100.60" port: 38057 } } }
I20240503 23:25:03.900759 1991 sys_catalog.cc:458] T 00000000000000000000000000000000 P e29ee7fe7b864b3d84f43581d1debeb3 [sys.catalog]: This master's current role is: FOLLOWER
I20240503 23:25:03.902338 1993 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1a6c6637a2d64ca098e1d774053fd1ce [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 2 leader_uuid: "1a6c6637a2d64ca098e1d774053fd1ce" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e29ee7fe7b864b3d84f43581d1debeb3" member_type: VOTER last_known_addr { host: "127.0.100.62" port: 46733 } } peers { permanent_uuid: "a3b81f85f3594536ade9e155ca2d1cb4" member_type: VOTER last_known_addr { host: "127.0.100.61" port: 45861 } } peers { permanent_uuid: "1a6c6637a2d64ca098e1d774053fd1ce" member_type: VOTER last_known_addr { host: "127.0.100.60" port: 38057 } } }
I20240503 23:25:03.903077 1991 sys_catalog.cc:455] T 00000000000000000000000000000000 P e29ee7fe7b864b3d84f43581d1debeb3 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 2 leader_uuid: "1a6c6637a2d64ca098e1d774053fd1ce" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e29ee7fe7b864b3d84f43581d1debeb3" member_type: VOTER last_known_addr { host: "127.0.100.62" port: 46733 } } peers { permanent_uuid: "a3b81f85f3594536ade9e155ca2d1cb4" member_type: VOTER last_known_addr { host: "127.0.100.61" port: 45861 } } peers { permanent_uuid: "1a6c6637a2d64ca098e1d774053fd1ce" member_type: VOTER last_known_addr { host: "127.0.100.60" port: 38057 } } }
I20240503 23:25:03.903347 1991 sys_catalog.cc:458] T 00000000000000000000000000000000 P e29ee7fe7b864b3d84f43581d1debeb3 [sys.catalog]: This master's current role is: FOLLOWER
I20240503 23:25:03.902869 1993 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1a6c6637a2d64ca098e1d774053fd1ce [sys.catalog]: This master's current role is: LEADER
I20240503 23:25:04.849445 1916 heartbeater.cc:503] Master 127.0.100.60:38057 was elected leader, sending a full tablet report...
I20240503 23:25:04.858633 1748 heartbeater.cc:503] Master 127.0.100.60:38057 was elected leader, sending a full tablet report...
I20240503 23:25:04.863380 1837 heartbeater.cc:503] Master 127.0.100.60:38057 was elected leader, sending a full tablet report...
I20240503 23:25:05.307157 1662 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240503 23:25:05.307338 1662 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240503 23:25:05.444666 400 tablet_server.cc:178] TabletServer@127.0.100.1:0 shutting down...
I20240503 23:25:05.454742 400 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240503 23:25:05.455116 400 tablet_replica.cc:330] T 27acd48c50e34ea58f455356b3e47eca P 0fc5931f4b97441b92926f9ca3f0ea1b: stopping tablet replica
I20240503 23:25:05.455338 400 raft_consensus.cc:2229] T 27acd48c50e34ea58f455356b3e47eca P 0fc5931f4b97441b92926f9ca3f0ea1b [term 1 LEADER]: Raft consensus shutting down.
I20240503 23:25:05.455572 400 raft_consensus.cc:2258] T 27acd48c50e34ea58f455356b3e47eca P 0fc5931f4b97441b92926f9ca3f0ea1b [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 23:25:05.470628 400 tablet_server.cc:194] TabletServer@127.0.100.1:0 shutdown complete.
I20240503 23:25:05.474206 400 tablet_server.cc:178] TabletServer@127.0.100.2:0 shutting down...
I20240503 23:25:05.486387 400 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240503 23:25:05.486760 400 tablet_replica.cc:330] T b8a96c4240ba4dd1b68d61cdf7dc8239 P 7d4c7dd4fc344655bf4fd1a67948bb3a: stopping tablet replica
I20240503 23:25:05.486987 400 raft_consensus.cc:2229] T b8a96c4240ba4dd1b68d61cdf7dc8239 P 7d4c7dd4fc344655bf4fd1a67948bb3a [term 1 LEADER]: Raft consensus shutting down.
I20240503 23:25:05.487221 400 raft_consensus.cc:2258] T b8a96c4240ba4dd1b68d61cdf7dc8239 P 7d4c7dd4fc344655bf4fd1a67948bb3a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 23:25:05.501878 400 tablet_server.cc:194] TabletServer@127.0.100.2:0 shutdown complete.
I20240503 23:25:05.505896 400 tablet_server.cc:178] TabletServer@127.0.100.3:0 shutting down...
I20240503 23:25:05.515816 400 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240503 23:25:05.516151 400 tablet_replica.cc:330] T d90b129a95954e83a2d1d4c5def6a82f P 2a4b81befdbb4672a8fb3a6b854aad63: stopping tablet replica
I20240503 23:25:05.516367 400 raft_consensus.cc:2229] T d90b129a95954e83a2d1d4c5def6a82f P 2a4b81befdbb4672a8fb3a6b854aad63 [term 1 LEADER]: Raft consensus shutting down.
I20240503 23:25:05.516630 400 raft_consensus.cc:2258] T d90b129a95954e83a2d1d4c5def6a82f P 2a4b81befdbb4672a8fb3a6b854aad63 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 23:25:05.531080 400 tablet_server.cc:194] TabletServer@127.0.100.3:0 shutdown complete.
I20240503 23:25:05.534794 400 master.cc:537] Master@127.0.100.62:46733 shutting down...
I20240503 23:25:05.544344 400 raft_consensus.cc:2229] T 00000000000000000000000000000000 P e29ee7fe7b864b3d84f43581d1debeb3 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 23:25:05.544744 400 raft_consensus.cc:2258] T 00000000000000000000000000000000 P e29ee7fe7b864b3d84f43581d1debeb3 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 23:25:05.544932 400 tablet_replica.cc:330] T 00000000000000000000000000000000 P e29ee7fe7b864b3d84f43581d1debeb3: stopping tablet replica
I20240503 23:25:05.559715 400 master.cc:552] Master@127.0.100.62:46733 shutdown complete.
I20240503 23:25:05.565414 400 master.cc:537] Master@127.0.100.60:38057 shutting down...
I20240503 23:25:05.573561 400 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 1a6c6637a2d64ca098e1d774053fd1ce [term 2 LEADER]: Raft consensus shutting down.
I20240503 23:25:05.574154 400 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 1a6c6637a2d64ca098e1d774053fd1ce [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 23:25:05.574302 400 tablet_replica.cc:330] T 00000000000000000000000000000000 P 1a6c6637a2d64ca098e1d774053fd1ce: stopping tablet replica
I20240503 23:25:05.588961 400 master.cc:552] Master@127.0.100.60:38057 shutdown complete.
[ OK ] AutoRebalancerTest.NextLeaderResumesAutoRebalancing (6559 ms)
[ RUN ] AutoRebalancerTest.MovesScheduledIfAddTserver
I20240503 23:25:05.604161 400 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.0.100.62:37107
I20240503 23:25:05.604887 400 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 23:25:05.608753 2002 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 23:25:05.609558 2003 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 23:25:05.610396 2005 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 23:25:05.611248 400 server_base.cc:1034] running on GCE node
I20240503 23:25:05.611613 400 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240503 23:25:05.611689 400 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240503 23:25:05.611750 400 hybrid_clock.cc:648] HybridClock initialized: now 1714778705611720 us; error 0 us; skew 500 ppm
I20240503 23:25:05.611910 400 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 23:25:05.618883 400 webserver.cc:460] Webserver started at http://127.0.100.62:41597/ using document root <none> and password file <none>
I20240503 23:25:05.619204 400 fs_manager.cc:359] Metadata directory not provided
I20240503 23:25:05.619303 400 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 23:25:05.619424 400 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 23:25:05.620227 400 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZhIiPz/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714778688451250-400-0/minicluster-data/master-0-root/instance:
uuid: "54e22664d9d64eb8ad0fd21ef1aa7b60"
format_stamp: "Formatted at 2024-05-03 23:25:05 on dist-test-slave-jvbp"
I20240503 23:25:05.623322 400 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20240503 23:25:05.625237 2010 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 23:25:05.625663 400 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.001s
I20240503 23:25:05.625792 400 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZhIiPz/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714778688451250-400-0/minicluster-data/master-0-root
uuid: "54e22664d9d64eb8ad0fd21ef1aa7b60"
format_stamp: "Formatted at 2024-05-03 23:25:05 on dist-test-slave-jvbp"
I20240503 23:25:05.625986 400 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZhIiPz/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714778688451250-400-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskZhIiPz/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714778688451250-400-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskZhIiPz/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714778688451250-400-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 23:25:05.668684 400 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 23:25:05.669401 400 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 23:25:05.686647 400 rpc_server.cc:307] RPC server started. Bound to: 127.0.100.62:37107
I20240503 23:25:05.686686 2061 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.100.62:37107 every 8 connection(s)
I20240503 23:25:05.688457 2062 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 23:25:05.693253 2062 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 54e22664d9d64eb8ad0fd21ef1aa7b60: Bootstrap starting.
I20240503 23:25:05.694782 2062 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 54e22664d9d64eb8ad0fd21ef1aa7b60: Neither blocks nor log segments found. Creating new log.
I20240503 23:25:05.696803 2062 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 54e22664d9d64eb8ad0fd21ef1aa7b60: No bootstrap required, opened a new log
I20240503 23:25:05.697543 2062 raft_consensus.cc:348] T 00000000000000000000000000000000 P 54e22664d9d64eb8ad0fd21ef1aa7b60 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "54e22664d9d64eb8ad0fd21ef1aa7b60" member_type: VOTER }
I20240503 23:25:05.697700 2062 raft_consensus.cc:374] T 00000000000000000000000000000000 P 54e22664d9d64eb8ad0fd21ef1aa7b60 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 23:25:05.697754 2062 raft_consensus.cc:729] T 00000000000000000000000000000000 P 54e22664d9d64eb8ad0fd21ef1aa7b60 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 54e22664d9d64eb8ad0fd21ef1aa7b60, State: Initialized, Role: FOLLOWER
I20240503 23:25:05.697986 2062 consensus_queue.cc:260] T 00000000000000000000000000000000 P 54e22664d9d64eb8ad0fd21ef1aa7b60 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "54e22664d9d64eb8ad0fd21ef1aa7b60" member_type: VOTER }
I20240503 23:25:05.698104 2062 raft_consensus.cc:388] T 00000000000000000000000000000000 P 54e22664d9d64eb8ad0fd21ef1aa7b60 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 23:25:05.698141 2062 raft_consensus.cc:482] T 00000000000000000000000000000000 P 54e22664d9d64eb8ad0fd21ef1aa7b60 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 23:25:05.698217 2062 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 54e22664d9d64eb8ad0fd21ef1aa7b60 [term 0 FOLLOWER]: Advancing to term 1
I20240503 23:25:05.700198 2062 raft_consensus.cc:504] T 00000000000000000000000000000000 P 54e22664d9d64eb8ad0fd21ef1aa7b60 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "54e22664d9d64eb8ad0fd21ef1aa7b60" member_type: VOTER }
I20240503 23:25:05.700415 2062 leader_election.cc:302] T 00000000000000000000000000000000 P 54e22664d9d64eb8ad0fd21ef1aa7b60 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 54e22664d9d64eb8ad0fd21ef1aa7b60; no voters:
I20240503 23:25:05.700901 2062 leader_election.cc:288] T 00000000000000000000000000000000 P 54e22664d9d64eb8ad0fd21ef1aa7b60 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240503 23:25:05.701086 2065 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 54e22664d9d64eb8ad0fd21ef1aa7b60 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 23:25:05.701589 2065 raft_consensus.cc:686] T 00000000000000000000000000000000 P 54e22664d9d64eb8ad0fd21ef1aa7b60 [term 1 LEADER]: Becoming Leader. State: Replica: 54e22664d9d64eb8ad0fd21ef1aa7b60, State: Running, Role: LEADER
I20240503 23:25:05.701851 2065 consensus_queue.cc:237] T 00000000000000000000000000000000 P 54e22664d9d64eb8ad0fd21ef1aa7b60 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "54e22664d9d64eb8ad0fd21ef1aa7b60" member_type: VOTER }
I20240503 23:25:05.701915 2062 sys_catalog.cc:564] T 00000000000000000000000000000000 P 54e22664d9d64eb8ad0fd21ef1aa7b60 [sys.catalog]: configured and running, proceeding with master startup.
I20240503 23:25:05.703074 2067 sys_catalog.cc:455] T 00000000000000000000000000000000 P 54e22664d9d64eb8ad0fd21ef1aa7b60 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 54e22664d9d64eb8ad0fd21ef1aa7b60. Latest consensus state: current_term: 1 leader_uuid: "54e22664d9d64eb8ad0fd21ef1aa7b60" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "54e22664d9d64eb8ad0fd21ef1aa7b60" member_type: VOTER } }
I20240503 23:25:05.703287 2067 sys_catalog.cc:458] T 00000000000000000000000000000000 P 54e22664d9d64eb8ad0fd21ef1aa7b60 [sys.catalog]: This master's current role is: LEADER
I20240503 23:25:05.703658 2066 sys_catalog.cc:455] T 00000000000000000000000000000000 P 54e22664d9d64eb8ad0fd21ef1aa7b60 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "54e22664d9d64eb8ad0fd21ef1aa7b60" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "54e22664d9d64eb8ad0fd21ef1aa7b60" member_type: VOTER } }
I20240503 23:25:05.703969 2066 sys_catalog.cc:458] T 00000000000000000000000000000000 P 54e22664d9d64eb8ad0fd21ef1aa7b60 [sys.catalog]: This master's current role is: LEADER
I20240503 23:25:05.705240 2072 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240503 23:25:05.706701 2072 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240503 23:25:05.707811 400 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20240503 23:25:05.710211 2072 catalog_manager.cc:1347] Generated new cluster ID: b3f5f551031744089da6423c318b7f50
I20240503 23:25:05.710314 2072 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240503 23:25:05.726287 2072 catalog_manager.cc:1370] Generated new certificate authority record
I20240503 23:25:05.727262 2072 catalog_manager.cc:1504] Loading token signing keys...
I20240503 23:25:05.740042 2072 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 54e22664d9d64eb8ad0fd21ef1aa7b60: Generated new TSK 0
I20240503 23:25:05.740403 2072 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240503 23:25:05.773865 400 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 23:25:05.777952 2083 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 23:25:05.778813 2084 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 23:25:05.779695 2086 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 23:25:05.780483 400 server_base.cc:1034] running on GCE node
I20240503 23:25:05.780856 400 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240503 23:25:05.780928 400 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240503 23:25:05.780957 400 hybrid_clock.cc:648] HybridClock initialized: now 1714778705780952 us; error 0 us; skew 500 ppm
I20240503 23:25:05.781102 400 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 23:25:05.782888 400 webserver.cc:460] Webserver started at http://127.0.100.1:46367/ using document root <none> and password file <none>
I20240503 23:25:05.783133 400 fs_manager.cc:359] Metadata directory not provided
I20240503 23:25:05.783195 400 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 23:25:05.783306 400 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 23:25:05.784300 400 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZhIiPz/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714778688451250-400-0/minicluster-data/ts-0-root/instance:
uuid: "d81282cf09ac4506a76e8fb57e71ec21"
format_stamp: "Formatted at 2024-05-03 23:25:05 on dist-test-slave-jvbp"
I20240503 23:25:05.787356 400 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20240503 23:25:05.789394 2091 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 23:25:05.790042 400 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20240503 23:25:05.790231 400 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZhIiPz/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714778688451250-400-0/minicluster-data/ts-0-root
uuid: "d81282cf09ac4506a76e8fb57e71ec21"
format_stamp: "Formatted at 2024-05-03 23:25:05 on dist-test-slave-jvbp"
I20240503 23:25:05.790378 400 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZhIiPz/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714778688451250-400-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskZhIiPz/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714778688451250-400-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskZhIiPz/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714778688451250-400-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 23:25:05.805455 400 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 23:25:05.806143 400 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 23:25:05.806797 400 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 23:25:05.807823 400 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 23:25:05.807886 400 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 23:25:05.807950 400 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 23:25:05.807993 400 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 23:25:05.824740 2153 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.100.1:34233 every 8 connection(s)
I20240503 23:25:05.824712 400 rpc_server.cc:307] RPC server started. Bound to: 127.0.100.1:34233
I20240503 23:25:05.826741 400 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 23:25:05.830940 2158 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 23:25:05.832067 2159 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 23:25:05.834156 2154 heartbeater.cc:347] Connected to a master server at 127.0.100.62:37107
I20240503 23:25:05.834370 2154 heartbeater.cc:464] Registering TS with master...
W20240503 23:25:05.834342 2161 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 23:25:05.834818 2154 heartbeater.cc:511] Master 127.0.100.62:37107 requested a full tablet report, sending...
I20240503 23:25:05.835049 400 server_base.cc:1034] running on GCE node
I20240503 23:25:05.835381 400 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240503 23:25:05.835443 400 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240503 23:25:05.835469 400 hybrid_clock.cc:648] HybridClock initialized: now 1714778705835464 us; error 0 us; skew 500 ppm
I20240503 23:25:05.835626 400 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 23:25:05.835877 2027 ts_manager.cc:194] Registered new tserver with Master: d81282cf09ac4506a76e8fb57e71ec21 (127.0.100.1:34233)
I20240503 23:25:05.837118 400 webserver.cc:460] Webserver started at http://127.0.100.2:35577/ using document root <none> and password file <none>
I20240503 23:25:05.837272 2027 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:39876
I20240503 23:25:05.837354 400 fs_manager.cc:359] Metadata directory not provided
I20240503 23:25:05.837442 400 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 23:25:05.837567 400 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 23:25:05.838480 400 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZhIiPz/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714778688451250-400-0/minicluster-data/ts-1-root/instance:
uuid: "c9bf353c47b34903be4da8e7878d6633"
format_stamp: "Formatted at 2024-05-03 23:25:05 on dist-test-slave-jvbp"
I20240503 23:25:05.841208 400 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.004s sys 0.000s
I20240503 23:25:05.842897 2166 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 23:25:05.843276 400 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.002s sys 0.000s
I20240503 23:25:05.843366 400 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZhIiPz/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714778688451250-400-0/minicluster-data/ts-1-root
uuid: "c9bf353c47b34903be4da8e7878d6633"
format_stamp: "Formatted at 2024-05-03 23:25:05 on dist-test-slave-jvbp"
I20240503 23:25:05.843483 400 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZhIiPz/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714778688451250-400-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskZhIiPz/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714778688451250-400-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskZhIiPz/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714778688451250-400-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 23:25:05.859423 400 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 23:25:05.860121 400 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 23:25:05.860821 400 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 23:25:05.861737 400 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 23:25:05.861799 400 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 23:25:05.861883 400 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 23:25:05.861923 400 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 23:25:05.877769 400 rpc_server.cc:307] RPC server started. Bound to: 127.0.100.2:44703
I20240503 23:25:05.877789 2228 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.100.2:44703 every 8 connection(s)
I20240503 23:25:05.879596 400 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 23:25:05.883479 2233 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 23:25:05.884321 2232 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 23:25:05.885033 2235 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 23:25:05.885794 400 server_base.cc:1034] running on GCE node
I20240503 23:25:05.886130 400 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240503 23:25:05.886193 400 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240503 23:25:05.886238 400 hybrid_clock.cc:648] HybridClock initialized: now 1714778705886232 us; error 0 us; skew 500 ppm
I20240503 23:25:05.886432 400 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 23:25:05.887100 2229 heartbeater.cc:347] Connected to a master server at 127.0.100.62:37107
I20240503 23:25:05.887255 2229 heartbeater.cc:464] Registering TS with master...
I20240503 23:25:05.887761 2229 heartbeater.cc:511] Master 127.0.100.62:37107 requested a full tablet report, sending...
I20240503 23:25:05.888054 400 webserver.cc:460] Webserver started at http://127.0.100.3:35411/ using document root <none> and password file <none>
I20240503 23:25:05.888327 400 fs_manager.cc:359] Metadata directory not provided
I20240503 23:25:05.888648 400 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 23:25:05.888901 400 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 23:25:05.889181 2027 ts_manager.cc:194] Registered new tserver with Master: c9bf353c47b34903be4da8e7878d6633 (127.0.100.2:44703)
I20240503 23:25:05.890120 400 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZhIiPz/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714778688451250-400-0/minicluster-data/ts-2-root/instance:
uuid: "011b13632d074888950aae2ce091fc86"
format_stamp: "Formatted at 2024-05-03 23:25:05 on dist-test-slave-jvbp"
I20240503 23:25:05.890538 2027 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:39878
I20240503 23:25:05.893260 400 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.002s
I20240503 23:25:05.894992 2240 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 23:25:05.895463 400 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.001s
I20240503 23:25:05.895573 400 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZhIiPz/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714778688451250-400-0/minicluster-data/ts-2-root
uuid: "011b13632d074888950aae2ce091fc86"
format_stamp: "Formatted at 2024-05-03 23:25:05 on dist-test-slave-jvbp"
I20240503 23:25:05.895706 400 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZhIiPz/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714778688451250-400-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-taskZhIiPz/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714778688451250-400-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-taskZhIiPz/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714778688451250-400-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 23:25:05.904894 400 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 23:25:05.905534 400 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 23:25:05.906180 400 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 23:25:05.907076 400 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 23:25:05.907133 400 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 23:25:05.907197 400 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 23:25:05.907243 400 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 23:25:05.924506 400 rpc_server.cc:307] RPC server started. Bound to: 127.0.100.3:41395
I20240503 23:25:05.924567 2302 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.100.3:41395 every 8 connection(s)
I20240503 23:25:05.932178 2303 heartbeater.cc:347] Connected to a master server at 127.0.100.62:37107
I20240503 23:25:05.932350 2303 heartbeater.cc:464] Registering TS with master...
I20240503 23:25:05.932806 2303 heartbeater.cc:511] Master 127.0.100.62:37107 requested a full tablet report, sending...
I20240503 23:25:05.933812 2027 ts_manager.cc:194] Registered new tserver with Master: 011b13632d074888950aae2ce091fc86 (127.0.100.3:41395)
I20240503 23:25:05.934738 400 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.009155483s
I20240503 23:25:05.934762 2027 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:39886
I20240503 23:25:06.839067 2154 heartbeater.cc:503] Master 127.0.100.62:37107 was elected leader, sending a full tablet report...
I20240503 23:25:06.892200 2229 heartbeater.cc:503] Master 127.0.100.62:37107 was elected leader, sending a full tablet report...
I20240503 23:25:06.937256 2303 heartbeater.cc:503] Master 127.0.100.62:37107 was elected leader, sending a full tablet report...
I20240503 23:25:06.972812 400 test_util.cc:268] Using random seed: 129142915
I20240503 23:25:06.990015 2027 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39888:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "\004\001\000\377\377\377?""\004\001\000\377\377\377?"
indirect_data: """"
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 23:25:06.990913 2027 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-workload in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 23:25:07.013200 2268 tablet_service.cc:1467] Processing CreateTablet for tablet 44a56d068f5d4f38bfff50e277b12541 (DEFAULT_TABLE table=test-workload [id=141458155540495281d7c93613f1e996]), partition=RANGE (key) PARTITION VALUES < 1073741823
I20240503 23:25:07.013489 2118 tablet_service.cc:1467] Processing CreateTablet for tablet 9d1b3bdd6dee4f37aecfcc05a395bed0 (DEFAULT_TABLE table=test-workload [id=141458155540495281d7c93613f1e996]), partition=RANGE (key) PARTITION 1073741823 <= VALUES
I20240503 23:25:07.013890 2268 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 44a56d068f5d4f38bfff50e277b12541. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 23:25:07.014012 2118 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9d1b3bdd6dee4f37aecfcc05a395bed0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 23:25:07.017258 2267 tablet_service.cc:1467] Processing CreateTablet for tablet 9d1b3bdd6dee4f37aecfcc05a395bed0 (DEFAULT_TABLE table=test-workload [id=141458155540495281d7c93613f1e996]), partition=RANGE (key) PARTITION 1073741823 <= VALUES
I20240503 23:25:07.017802 2267 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9d1b3bdd6dee4f37aecfcc05a395bed0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 23:25:07.018992 2194 tablet_service.cc:1467] Processing CreateTablet for tablet 44a56d068f5d4f38bfff50e277b12541 (DEFAULT_TABLE table=test-workload [id=141458155540495281d7c93613f1e996]), partition=RANGE (key) PARTITION VALUES < 1073741823
I20240503 23:25:07.019577 2194 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 44a56d068f5d4f38bfff50e277b12541. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 23:25:07.020965 2323 tablet_bootstrap.cc:492] T 9d1b3bdd6dee4f37aecfcc05a395bed0 P d81282cf09ac4506a76e8fb57e71ec21: Bootstrap starting.
I20240503 23:25:07.023218 2323 tablet_bootstrap.cc:654] T 9d1b3bdd6dee4f37aecfcc05a395bed0 P d81282cf09ac4506a76e8fb57e71ec21: Neither blocks nor log segments found. Creating new log.
I20240503 23:25:07.024296 2193 tablet_service.cc:1467] Processing CreateTablet for tablet 9d1b3bdd6dee4f37aecfcc05a395bed0 (DEFAULT_TABLE table=test-workload [id=141458155540495281d7c93613f1e996]), partition=RANGE (key) PARTITION 1073741823 <= VALUES
I20240503 23:25:07.013219 2119 tablet_service.cc:1467] Processing CreateTablet for tablet 44a56d068f5d4f38bfff50e277b12541 (DEFAULT_TABLE table=test-workload [id=141458155540495281d7c93613f1e996]), partition=RANGE (key) PARTITION VALUES < 1073741823
I20240503 23:25:07.025494 2193 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9d1b3bdd6dee4f37aecfcc05a395bed0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 23:25:07.026190 2119 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 44a56d068f5d4f38bfff50e277b12541. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 23:25:07.028872 2324 tablet_bootstrap.cc:492] T 9d1b3bdd6dee4f37aecfcc05a395bed0 P 011b13632d074888950aae2ce091fc86: Bootstrap starting.
I20240503 23:25:07.031765 2324 tablet_bootstrap.cc:654] T 9d1b3bdd6dee4f37aecfcc05a395bed0 P 011b13632d074888950aae2ce091fc86: Neither blocks nor log segments found. Creating new log.
I20240503 23:25:07.037796 2326 tablet_bootstrap.cc:492] T 44a56d068f5d4f38bfff50e277b12541 P c9bf353c47b34903be4da8e7878d6633: Bootstrap starting.
I20240503 23:25:07.038121 2324 tablet_bootstrap.cc:492] T 9d1b3bdd6dee4f37aecfcc05a395bed0 P 011b13632d074888950aae2ce091fc86: No bootstrap required, opened a new log
I20240503 23:25:07.038280 2324 ts_tablet_manager.cc:1367] T 9d1b3bdd6dee4f37aecfcc05a395bed0 P 011b13632d074888950aae2ce091fc86: Time spent bootstrapping tablet: real 0.010s user 0.007s sys 0.000s
I20240503 23:25:07.039064 2324 raft_consensus.cc:348] T 9d1b3bdd6dee4f37aecfcc05a395bed0 P 011b13632d074888950aae2ce091fc86 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d81282cf09ac4506a76e8fb57e71ec21" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 34233 } } peers { permanent_uuid: "c9bf353c47b34903be4da8e7878d6633" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 44703 } } peers { permanent_uuid: "011b13632d074888950aae2ce091fc86" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 41395 } }
I20240503 23:25:07.039474 2324 raft_consensus.cc:374] T 9d1b3bdd6dee4f37aecfcc05a395bed0 P 011b13632d074888950aae2ce091fc86 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 23:25:07.039590 2324 raft_consensus.cc:729] T 9d1b3bdd6dee4f37aecfcc05a395bed0 P 011b13632d074888950aae2ce091fc86 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 011b13632d074888950aae2ce091fc86, State: Initialized, Role: FOLLOWER
I20240503 23:25:07.039955 2324 consensus_queue.cc:260] T 9d1b3bdd6dee4f37aecfcc05a395bed0 P 011b13632d074888950aae2ce091fc86 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d81282cf09ac4506a76e8fb57e71ec21" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 34233 } } peers { permanent_uuid: "c9bf353c47b34903be4da8e7878d6633" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 44703 } } peers { permanent_uuid: "011b13632d074888950aae2ce091fc86" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 41395 } }
I20240503 23:25:07.040943 2324 ts_tablet_manager.cc:1398] T 9d1b3bdd6dee4f37aecfcc05a395bed0 P 011b13632d074888950aae2ce091fc86: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240503 23:25:07.041033 2323 tablet_bootstrap.cc:492] T 9d1b3bdd6dee4f37aecfcc05a395bed0 P d81282cf09ac4506a76e8fb57e71ec21: No bootstrap required, opened a new log
I20240503 23:25:07.041208 2323 ts_tablet_manager.cc:1367] T 9d1b3bdd6dee4f37aecfcc05a395bed0 P d81282cf09ac4506a76e8fb57e71ec21: Time spent bootstrapping tablet: real 0.020s user 0.007s sys 0.005s
I20240503 23:25:07.041275 2324 tablet_bootstrap.cc:492] T 44a56d068f5d4f38bfff50e277b12541 P 011b13632d074888950aae2ce091fc86: Bootstrap starting.
I20240503 23:25:07.042070 2323 raft_consensus.cc:348] T 9d1b3bdd6dee4f37aecfcc05a395bed0 P d81282cf09ac4506a76e8fb57e71ec21 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d81282cf09ac4506a76e8fb57e71ec21" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 34233 } } peers { permanent_uuid: "c9bf353c47b34903be4da8e7878d6633" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 44703 } } peers { permanent_uuid: "011b13632d074888950aae2ce091fc86" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 41395 } }
I20240503 23:25:07.043167 2323 raft_consensus.cc:374] T 9d1b3bdd6dee4f37aecfcc05a395bed0 P d81282cf09ac4506a76e8fb57e71ec21 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 23:25:07.043356 2323 raft_consensus.cc:729] T 9d1b3bdd6dee4f37aecfcc05a395bed0 P d81282cf09ac4506a76e8fb57e71ec21 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d81282cf09ac4506a76e8fb57e71ec21, State: Initialized, Role: FOLLOWER
I20240503 23:25:07.043710 2323 consensus_queue.cc:260] T 9d1b3bdd6dee4f37aecfcc05a395bed0 P d81282cf09ac4506a76e8fb57e71ec21 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d81282cf09ac4506a76e8fb57e71ec21" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 34233 } } peers { permanent_uuid: "c9bf353c47b34903be4da8e7878d6633" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 44703 } } peers { permanent_uuid: "011b13632d074888950aae2ce091fc86" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 41395 } }
I20240503 23:25:07.044227 2324 tablet_bootstrap.cc:654] T 44a56d068f5d4f38bfff50e277b12541 P 011b13632d074888950aae2ce091fc86: Neither blocks nor log segments found. Creating new log.
I20240503 23:25:07.044721 2326 tablet_bootstrap.cc:654] T 44a56d068f5d4f38bfff50e277b12541 P c9bf353c47b34903be4da8e7878d6633: Neither blocks nor log segments found. Creating new log.
I20240503 23:25:07.045357 2323 ts_tablet_manager.cc:1398] T 9d1b3bdd6dee4f37aecfcc05a395bed0 P d81282cf09ac4506a76e8fb57e71ec21: Time spent starting tablet: real 0.004s user 0.003s sys 0.001s
I20240503 23:25:07.045886 2323 tablet_bootstrap.cc:492] T 44a56d068f5d4f38bfff50e277b12541 P d81282cf09ac4506a76e8fb57e71ec21: Bootstrap starting.
I20240503 23:25:07.047895 2323 tablet_bootstrap.cc:654] T 44a56d068f5d4f38bfff50e277b12541 P d81282cf09ac4506a76e8fb57e71ec21: Neither blocks nor log segments found. Creating new log.
I20240503 23:25:07.048534 2326 tablet_bootstrap.cc:492] T 44a56d068f5d4f38bfff50e277b12541 P c9bf353c47b34903be4da8e7878d6633: No bootstrap required, opened a new log
I20240503 23:25:07.048767 2326 ts_tablet_manager.cc:1367] T 44a56d068f5d4f38bfff50e277b12541 P c9bf353c47b34903be4da8e7878d6633: Time spent bootstrapping tablet: real 0.011s user 0.002s sys 0.002s
I20240503 23:25:07.050246 2326 raft_consensus.cc:348] T 44a56d068f5d4f38bfff50e277b12541 P c9bf353c47b34903be4da8e7878d6633 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d81282cf09ac4506a76e8fb57e71ec21" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 34233 } } peers { permanent_uuid: "011b13632d074888950aae2ce091fc86" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 41395 } } peers { permanent_uuid: "c9bf353c47b34903be4da8e7878d6633" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 44703 } }
I20240503 23:25:07.050568 2326 raft_consensus.cc:374] T 44a56d068f5d4f38bfff50e277b12541 P c9bf353c47b34903be4da8e7878d6633 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 23:25:07.050897 2326 raft_consensus.cc:729] T 44a56d068f5d4f38bfff50e277b12541 P c9bf353c47b34903be4da8e7878d6633 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c9bf353c47b34903be4da8e7878d6633, State: Initialized, Role: FOLLOWER
I20240503 23:25:07.051296 2326 consensus_queue.cc:260] T 44a56d068f5d4f38bfff50e277b12541 P c9bf353c47b34903be4da8e7878d6633 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d81282cf09ac4506a76e8fb57e71ec21" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 34233 } } peers { permanent_uuid: "011b13632d074888950aae2ce091fc86" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 41395 } } peers { permanent_uuid: "c9bf353c47b34903be4da8e7878d6633" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 44703 } }
I20240503 23:25:07.052464 2324 tablet_bootstrap.cc:492] T 44a56d068f5d4f38bfff50e277b12541 P 011b13632d074888950aae2ce091fc86: No bootstrap required, opened a new log
I20240503 23:25:07.052774 2324 ts_tablet_manager.cc:1367] T 44a56d068f5d4f38bfff50e277b12541 P 011b13632d074888950aae2ce091fc86: Time spent bootstrapping tablet: real 0.012s user 0.001s sys 0.008s
I20240503 23:25:07.053107 2326 ts_tablet_manager.cc:1398] T 44a56d068f5d4f38bfff50e277b12541 P c9bf353c47b34903be4da8e7878d6633: Time spent starting tablet: real 0.004s user 0.002s sys 0.001s
I20240503 23:25:07.053117 2323 tablet_bootstrap.cc:492] T 44a56d068f5d4f38bfff50e277b12541 P d81282cf09ac4506a76e8fb57e71ec21: No bootstrap required, opened a new log
I20240503 23:25:07.053339 2323 ts_tablet_manager.cc:1367] T 44a56d068f5d4f38bfff50e277b12541 P d81282cf09ac4506a76e8fb57e71ec21: Time spent bootstrapping tablet: real 0.008s user 0.003s sys 0.002s
I20240503 23:25:07.053393 2326 tablet_bootstrap.cc:492] T 9d1b3bdd6dee4f37aecfcc05a395bed0 P c9bf353c47b34903be4da8e7878d6633: Bootstrap starting.
I20240503 23:25:07.053761 2324 raft_consensus.cc:348] T 44a56d068f5d4f38bfff50e277b12541 P 011b13632d074888950aae2ce091fc86 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d81282cf09ac4506a76e8fb57e71ec21" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 34233 } } peers { permanent_uuid: "011b13632d074888950aae2ce091fc86" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 41395 } } peers { permanent_uuid: "c9bf353c47b34903be4da8e7878d6633" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 44703 } }
I20240503 23:25:07.054018 2324 raft_consensus.cc:374] T 44a56d068f5d4f38bfff50e277b12541 P 011b13632d074888950aae2ce091fc86 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 23:25:07.054118 2323 raft_consensus.cc:348] T 44a56d068f5d4f38bfff50e277b12541 P d81282cf09ac4506a76e8fb57e71ec21 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d81282cf09ac4506a76e8fb57e71ec21" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 34233 } } peers { permanent_uuid: "011b13632d074888950aae2ce091fc86" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 41395 } } peers { permanent_uuid: "c9bf353c47b34903be4da8e7878d6633" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 44703 } }
I20240503 23:25:07.054319 2323 raft_consensus.cc:374] T 44a56d068f5d4f38bfff50e277b12541 P d81282cf09ac4506a76e8fb57e71ec21 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 23:25:07.054404 2323 raft_consensus.cc:729] T 44a56d068f5d4f38bfff50e277b12541 P d81282cf09ac4506a76e8fb57e71ec21 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d81282cf09ac4506a76e8fb57e71ec21, State: Initialized, Role: FOLLOWER
I20240503 23:25:07.054689 2323 consensus_queue.cc:260] T 44a56d068f5d4f38bfff50e277b12541 P d81282cf09ac4506a76e8fb57e71ec21 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d81282cf09ac4506a76e8fb57e71ec21" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 34233 } } peers { permanent_uuid: "011b13632d074888950aae2ce091fc86" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 41395 } } peers { permanent_uuid: "c9bf353c47b34903be4da8e7878d6633" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 44703 } }
I20240503 23:25:07.054129 2324 raft_consensus.cc:729] T 44a56d068f5d4f38bfff50e277b12541 P 011b13632d074888950aae2ce091fc86 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 011b13632d074888950aae2ce091fc86, State: Initialized, Role: FOLLOWER
I20240503 23:25:07.055250 2323 ts_tablet_manager.cc:1398] T 44a56d068f5d4f38bfff50e277b12541 P d81282cf09ac4506a76e8fb57e71ec21: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240503 23:25:07.055433 2324 consensus_queue.cc:260] T 44a56d068f5d4f38bfff50e277b12541 P 011b13632d074888950aae2ce091fc86 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d81282cf09ac4506a76e8fb57e71ec21" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 34233 } } peers { permanent_uuid: "011b13632d074888950aae2ce091fc86" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 41395 } } peers { permanent_uuid: "c9bf353c47b34903be4da8e7878d6633" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 44703 } }
I20240503 23:25:07.055907 2324 ts_tablet_manager.cc:1398] T 44a56d068f5d4f38bfff50e277b12541 P 011b13632d074888950aae2ce091fc86: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240503 23:25:07.056382 2326 tablet_bootstrap.cc:654] T 9d1b3bdd6dee4f37aecfcc05a395bed0 P c9bf353c47b34903be4da8e7878d6633: Neither blocks nor log segments found. Creating new log.
I20240503 23:25:07.061753 2326 tablet_bootstrap.cc:492] T 9d1b3bdd6dee4f37aecfcc05a395bed0 P c9bf353c47b34903be4da8e7878d6633: No bootstrap required, opened a new log
I20240503 23:25:07.061995 2326 ts_tablet_manager.cc:1367] T 9d1b3bdd6dee4f37aecfcc05a395bed0 P c9bf353c47b34903be4da8e7878d6633: Time spent bootstrapping tablet: real 0.009s user 0.003s sys 0.003s
I20240503 23:25:07.062731 2326 raft_consensus.cc:348] T 9d1b3bdd6dee4f37aecfcc05a395bed0 P c9bf353c47b34903be4da8e7878d6633 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d81282cf09ac4506a76e8fb57e71ec21" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 34233 } } peers { permanent_uuid: "c9bf353c47b34903be4da8e7878d6633" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 44703 } } peers { permanent_uuid: "011b13632d074888950aae2ce091fc86" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 41395 } }
I20240503 23:25:07.062988 2326 raft_consensus.cc:374] T 9d1b3bdd6dee4f37aecfcc05a395bed0 P c9bf353c47b34903be4da8e7878d6633 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 23:25:07.063076 2326 raft_consensus.cc:729] T 9d1b3bdd6dee4f37aecfcc05a395bed0 P c9bf353c47b34903be4da8e7878d6633 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c9bf353c47b34903be4da8e7878d6633, State: Initialized, Role: FOLLOWER
I20240503 23:25:07.063336 2326 consensus_queue.cc:260] T 9d1b3bdd6dee4f37aecfcc05a395bed0 P c9bf353c47b34903be4da8e7878d6633 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d81282cf09ac4506a76e8fb57e71ec21" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 34233 } } peers { permanent_uuid: "c9bf353c47b34903be4da8e7878d6633" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 44703 } } peers { permanent_uuid: "011b13632d074888950aae2ce091fc86" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 41395 } }
I20240503 23:25:07.063802 2326 ts_tablet_manager.cc:1398] T 9d1b3bdd6dee4f37aecfcc05a395bed0 P c9bf353c47b34903be4da8e7878d6633: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240503 23:25:07.098649 2328 raft_consensus.cc:482] T 44a56d068f5d4f38bfff50e277b12541 P 011b13632d074888950aae2ce091fc86 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 23:25:07.098942 2328 raft_consensus.cc:504] T 44a56d068f5d4f38bfff50e277b12541 P 011b13632d074888950aae2ce091fc86 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d81282cf09ac4506a76e8fb57e71ec21" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 34233 } } peers { permanent_uuid: "011b13632d074888950aae2ce091fc86" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 41395 } } peers { permanent_uuid: "c9bf353c47b34903be4da8e7878d6633" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 44703 } }
I20240503 23:25:07.100109 2328 leader_election.cc:288] T 44a56d068f5d4f38bfff50e277b12541 P 011b13632d074888950aae2ce091fc86 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d81282cf09ac4506a76e8fb57e71ec21 (127.0.100.1:34233), c9bf353c47b34903be4da8e7878d6633 (127.0.100.2:44703)
I20240503 23:25:07.109529 2129 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "44a56d068f5d4f38bfff50e277b12541" candidate_uuid: "011b13632d074888950aae2ce091fc86" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d81282cf09ac4506a76e8fb57e71ec21" is_pre_election: true
I20240503 23:25:07.109750 2204 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "44a56d068f5d4f38bfff50e277b12541" candidate_uuid: "011b13632d074888950aae2ce091fc86" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c9bf353c47b34903be4da8e7878d6633" is_pre_election: true
I20240503 23:25:07.109930 2204 raft_consensus.cc:2454] T 44a56d068f5d4f38bfff50e277b12541 P c9bf353c47b34903be4da8e7878d6633 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 011b13632d074888950aae2ce091fc86 in term 0.
I20240503 23:25:07.109930 2129 raft_consensus.cc:2454] T 44a56d068f5d4f38bfff50e277b12541 P d81282cf09ac4506a76e8fb57e71ec21 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 011b13632d074888950aae2ce091fc86 in term 0.
I20240503 23:25:07.110563 2243 leader_election.cc:302] T 44a56d068f5d4f38bfff50e277b12541 P 011b13632d074888950aae2ce091fc86 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 011b13632d074888950aae2ce091fc86, d81282cf09ac4506a76e8fb57e71ec21; no voters:
I20240503 23:25:07.110879 2328 raft_consensus.cc:2781] T 44a56d068f5d4f38bfff50e277b12541 P 011b13632d074888950aae2ce091fc86 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 23:25:07.111004 2328 raft_consensus.cc:482] T 44a56d068f5d4f38bfff50e277b12541 P 011b13632d074888950aae2ce091fc86 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 23:25:07.111094 2328 raft_consensus.cc:3037] T 44a56d068f5d4f38bfff50e277b12541 P 011b13632d074888950aae2ce091fc86 [term 0 FOLLOWER]: Advancing to term 1
I20240503 23:25:07.114259 2328 raft_consensus.cc:504] T 44a56d068f5d4f38bfff50e277b12541 P 011b13632d074888950aae2ce091fc86 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d81282cf09ac4506a76e8fb57e71ec21" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 34233 } } peers { permanent_uuid: "011b13632d074888950aae2ce091fc86" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 41395 } } peers { permanent_uuid: "c9bf353c47b34903be4da8e7878d6633" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 44703 } }
I20240503 23:25:07.115223 2328 leader_election.cc:288] T 44a56d068f5d4f38bfff50e277b12541 P 011b13632d074888950aae2ce091fc86 [CANDIDATE]: Term 1 election: Requested vote from peers d81282cf09ac4506a76e8fb57e71ec21 (127.0.100.1:34233), c9bf353c47b34903be4da8e7878d6633 (127.0.100.2:44703)
I20240503 23:25:07.115888 2129 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "44a56d068f5d4f38bfff50e277b12541" candidate_uuid: "011b13632d074888950aae2ce091fc86" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d81282cf09ac4506a76e8fb57e71ec21"
I20240503 23:25:07.115940 2204 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "44a56d068f5d4f38bfff50e277b12541" candidate_uuid: "011b13632d074888950aae2ce091fc86" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c9bf353c47b34903be4da8e7878d6633"
I20240503 23:25:07.117002 2204 raft_consensus.cc:3037] T 44a56d068f5d4f38bfff50e277b12541 P c9bf353c47b34903be4da8e7878d6633 [term 0 FOLLOWER]: Advancing to term 1
I20240503 23:25:07.116997 2129 raft_consensus.cc:3037] T 44a56d068f5d4f38bfff50e277b12541 P d81282cf09ac4506a76e8fb57e71ec21 [term 0 FOLLOWER]: Advancing to term 1
I20240503 23:25:07.119486 2129 raft_consensus.cc:2454] T 44a56d068f5d4f38bfff50e277b12541 P d81282cf09ac4506a76e8fb57e71ec21 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 011b13632d074888950aae2ce091fc86 in term 1.
I20240503 23:25:07.119545 2204 raft_consensus.cc:2454] T 44a56d068f5d4f38bfff50e277b12541 P c9bf353c47b34903be4da8e7878d6633 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 011b13632d074888950aae2ce091fc86 in term 1.
I20240503 23:25:07.120071 2243 leader_election.cc:302] T 44a56d068f5d4f38bfff50e277b12541 P 011b13632d074888950aae2ce091fc86 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 011b13632d074888950aae2ce091fc86, d81282cf09ac4506a76e8fb57e71ec21; no voters:
I20240503 23:25:07.120524 2328 raft_consensus.cc:2781] T 44a56d068f5d4f38bfff50e277b12541 P 011b13632d074888950aae2ce091fc86 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 23:25:07.121065 2328 raft_consensus.cc:686] T 44a56d068f5d4f38bfff50e277b12541 P 011b13632d074888950aae2ce091fc86 [term 1 LEADER]: Becoming Leader. State: Replica: 011b13632d074888950aae2ce091fc86, State: Running, Role: LEADER
I20240503 23:25:07.121416 2328 consensus_queue.cc:237] T 44a56d068f5d4f38bfff50e277b12541 P 011b13632d074888950aae2ce091fc86 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d81282cf09ac4506a76e8fb57e71ec21" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 34233 } } peers { permanent_uuid: "011b13632d074888950aae2ce091fc86" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 41395 } } peers { permanent_uuid: "c9bf353c47b34903be4da8e7878d6633" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 44703 } }
I20240503 23:25:07.124753 2026 catalog_manager.cc:5484] T 44a56d068f5d4f38bfff50e277b12541 P 011b13632d074888950aae2ce091fc86 reported cstate change: term changed from 0 to 1, leader changed from <none> to 011b13632d074888950aae2ce091fc86 (127.0.100.3). New cstate: current_term: 1 leader_uuid: "011b13632d074888950aae2ce091fc86" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d81282cf09ac4506a76e8fb57e71ec21" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 34233 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "011b13632d074888950aae2ce091fc86" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 41395 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c9bf353c47b34903be4da8e7878d6633" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 44703 } health_report { overall_health: UNKNOWN } } }
I20240503 23:25:07.193603 2328 raft_consensus.cc:482] T 9d1b3bdd6dee4f37aecfcc05a395bed0 P 011b13632d074888950aae2ce091fc86 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 23:25:07.193893 2328 raft_consensus.cc:504] T 9d1b3bdd6dee4f37aecfcc05a395bed0 P 011b13632d074888950aae2ce091fc86 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d81282cf09ac4506a76e8fb57e71ec21" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 34233 } } peers { permanent_uuid: "c9bf353c47b34903be4da8e7878d6633" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 44703 } } peers { permanent_uuid: "011b13632d074888950aae2ce091fc86" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 41395 } }
I20240503 23:25:07.194703 2328 leader_election.cc:288] T 9d1b3bdd6dee4f37aecfcc05a395bed0 P 011b13632d074888950aae2ce091fc86 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d81282cf09ac4506a76e8fb57e71ec21 (127.0.100.1:34233), c9bf353c47b34903be4da8e7878d6633 (127.0.100.2:44703)
I20240503 23:25:07.195447 2129 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9d1b3bdd6dee4f37aecfcc05a395bed0" candidate_uuid: "011b13632d074888950aae2ce091fc86" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d81282cf09ac4506a76e8fb57e71ec21" is_pre_election: true
I20240503 23:25:07.195452 2204 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9d1b3bdd6dee4f37aecfcc05a395bed0" candidate_uuid: "011b13632d074888950aae2ce091fc86" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c9bf353c47b34903be4da8e7878d6633" is_pre_election: true
I20240503 23:25:07.195792 2129 raft_consensus.cc:2454] T 9d1b3bdd6dee4f37aecfcc05a395bed0 P d81282cf09ac4506a76e8fb57e71ec21 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 011b13632d074888950aae2ce091fc86 in term 0.
I20240503 23:25:07.195824 2204 raft_consensus.cc:2454] T 9d1b3bdd6dee4f37aecfcc05a395bed0 P c9bf353c47b34903be4da8e7878d6633 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 011b13632d074888950aae2ce091fc86 in term 0.
I20240503 23:25:07.196244 2243 leader_election.cc:302] T 9d1b3bdd6dee4f37aecfcc05a395bed0 P 011b13632d074888950aae2ce091fc86 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 011b13632d074888950aae2ce091fc86, c9bf353c47b34903be4da8e7878d6633; no voters:
I20240503 23:25:07.196605 2328 raft_consensus.cc:2781] T 9d1b3bdd6dee4f37aecfcc05a395bed0 P 011b13632d074888950aae2ce091fc86 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 23:25:07.196722 2328 raft_consensus.cc:482] T 9d1b3bdd6dee4f37aecfcc05a395bed0 P 011b13632d074888950aae2ce091fc86 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 23:25:07.196838 2328 raft_consensus.cc:3037] T 9d1b3bdd6dee4f37aecfcc05a395bed0 P 011b13632d074888950aae2ce091fc86 [term 0 FOLLOWER]: Advancing to term 1
I20240503 23:25:07.199023 2328 raft_consensus.cc:504] T 9d1b3bdd6dee4f37aecfcc05a395bed0 P 011b13632d074888950aae2ce091fc86 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d81282cf09ac4506a76e8fb57e71ec21" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 34233 } } peers { permanent_uuid: "c9bf353c47b34903be4da8e7878d6633" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 44703 } } peers { permanent_uuid: "011b13632d074888950aae2ce091fc86" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 41395 } }
I20240503 23:25:07.199790 2328 leader_election.cc:288] T 9d1b3bdd6dee4f37aecfcc05a395bed0 P 011b13632d074888950aae2ce091fc86 [CANDIDATE]: Term 1 election: Requested vote from peers d81282cf09ac4506a76e8fb57e71ec21 (127.0.100.1:34233), c9bf353c47b34903be4da8e7878d6633 (127.0.100.2:44703)
I20240503 23:25:07.200423 2129 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9d1b3bdd6dee4f37aecfcc05a395bed0" candidate_uuid: "011b13632d074888950aae2ce091fc86" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d81282cf09ac4506a76e8fb57e71ec21"
I20240503 23:25:07.200536 2204 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9d1b3bdd6dee4f37aecfcc05a395bed0" candidate_uuid: "011b13632d074888950aae2ce091fc86" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c9bf353c47b34903be4da8e7878d6633"
I20240503 23:25:07.200716 2129 raft_consensus.cc:3037] T 9d1b3bdd6dee4f37aecfcc05a395bed0 P d81282cf09ac4506a76e8fb57e71ec21 [term 0 FOLLOWER]: Advancing to term 1
I20240503 23:25:07.200805 2204 raft_consensus.cc:3037] T 9d1b3bdd6dee4f37aecfcc05a395bed0 P c9bf353c47b34903be4da8e7878d6633 [term 0 FOLLOWER]: Advancing to term 1
I20240503 23:25:07.203055 2204 raft_consensus.cc:2454] T 9d1b3bdd6dee4f37aecfcc05a395bed0 P c9bf353c47b34903be4da8e7878d6633 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 011b13632d074888950aae2ce091fc86 in term 1.
I20240503 23:25:07.203050 2129 raft_consensus.cc:2454] T 9d1b3bdd6dee4f37aecfcc05a395bed0 P d81282cf09ac4506a76e8fb57e71ec21 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 011b13632d074888950aae2ce091fc86 in term 1.
I20240503 23:25:07.203631 2243 leader_election.cc:302] T 9d1b3bdd6dee4f37aecfcc05a395bed0 P 011b13632d074888950aae2ce091fc86 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 011b13632d074888950aae2ce091fc86, d81282cf09ac4506a76e8fb57e71ec21; no voters:
I20240503 23:25:07.204012 2328 raft_consensus.cc:2781] T 9d1b3bdd6dee4f37aecfcc05a395bed0 P 011b13632d074888950aae2ce091fc86 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 23:25:07.204196 2328 raft_consensus.cc:686] T 9d1b3bdd6dee4f37aecfcc05a395bed0 P 011b13632d074888950aae2ce091fc86 [term 1 LEADER]: Becoming Leader. State: Replica: 011b13632d074888950aae2ce091fc86, State: Running, Role: LEADER
I20240503 23:25:07.204474 2328 consensus_queue.cc:237] T 9d1b3bdd6dee4f37aecfcc05a395bed0 P 011b13632d074888950aae2ce091fc86 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d81282cf09ac4506a76e8fb57e71ec21" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 34233 } } peers { permanent_uuid: "c9bf353c47b34903be4da8e7878d6633" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 44703 } } peers { permanent_uuid: "011b13632d074888950aae2ce091fc86" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 41395 } }
I20240503 23:25:07.207885 2026 catalog_manager.cc:5484] T 9d1b3bdd6dee4f37aecfcc05a395bed0 P 011b13632d074888950aae2ce091fc86 reported cstate change: term changed from 0 to 1, leader changed from <none> to 011b13632d074888950aae2ce091fc86 (127.0.100.3). New cstate: current_term: 1 leader_uuid: "011b13632d074888950aae2ce091fc86" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d81282cf09ac4506a76e8fb57e71ec21" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 34233 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c9bf353c47b34903be4da8e7878d6633" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 44703 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "011b13632d074888950aae2ce091fc86" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 41395 } health_report { overall_health: HEALTHY } } }
I20240503 23:25:07.544339 2328 consensus_queue.cc:1035] T 44a56d068f5d4f38bfff50e277b12541 P 011b13632d074888950aae2ce091fc86 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d81282cf09ac4506a76e8fb57e71ec21" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 34233 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 23:25:07.551162 2342 consensus_queue.cc:1035] T 44a56d068f5d4f38bfff50e277b12541 P 011b13632d074888950aae2ce091fc86 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c9bf353c47b34903be4da8e7878d6633" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 44703 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 23:25:07.688788 2342 consensus_queue.cc:1035] T 9d1b3bdd6dee4f37aecfcc05a395bed0 P 011b13632d074888950aae2ce091fc86 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c9bf353c47b34903be4da8e7878d6633" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 44703 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 23:25:07.694232 2328 consensus_queue.cc:1035] T 9d1b3bdd6dee4f37aecfcc05a395bed0 P 011b13632d074888950aae2ce091fc86 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d81282cf09ac4506a76e8fb57e71ec21" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 34233 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 23:25:07.707741 2080 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240503 23:25:07.716485 2278 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "44a56d068f5d4f38bfff50e277b12541"
dest_uuid: "011b13632d074888950aae2ce091fc86"
mode: GRACEFUL
new_leader_uuid: "d81282cf09ac4506a76e8fb57e71ec21"
from {username='slave'} at 127.0.0.1:33470
I20240503 23:25:07.716854 2278 raft_consensus.cc:595] T 44a56d068f5d4f38bfff50e277b12541 P 011b13632d074888950aae2ce091fc86 [term 1 LEADER]: Received request to transfer leadership to TS d81282cf09ac4506a76e8fb57e71ec21
I20240503 23:25:07.717456 2080 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 1
I20240503 23:25:07.717657 2080 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240503 23:25:08.068743 2337 raft_consensus.cc:979] T 44a56d068f5d4f38bfff50e277b12541 P 011b13632d074888950aae2ce091fc86: : Instructing follower d81282cf09ac4506a76e8fb57e71ec21 to start an election
I20240503 23:25:08.068958 2337 raft_consensus.cc:1067] T 44a56d068f5d4f38bfff50e277b12541 P 011b13632d074888950aae2ce091fc86 [term 1 LEADER]: Signalling peer d81282cf09ac4506a76e8fb57e71ec21 to start an election
I20240503 23:25:08.069713 2129 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "44a56d068f5d4f38bfff50e277b12541"
dest_uuid: "d81282cf09ac4506a76e8fb57e71ec21"
from {username='slave'} at 127.0.0.1:47562
I20240503 23:25:08.069936 2129 raft_consensus.cc:482] T 44a56d068f5d4f38bfff50e277b12541 P d81282cf09ac4506a76e8fb57e71ec21 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 23:25:08.070055 2129 raft_consensus.cc:3037] T 44a56d068f5d4f38bfff50e277b12541 P d81282cf09ac4506a76e8fb57e71ec21 [term 1 FOLLOWER]: Advancing to term 2
I20240503 23:25:08.072188 2129 raft_consensus.cc:504] T 44a56d068f5d4f38bfff50e277b12541 P d81282cf09ac4506a76e8fb57e71ec21 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d81282cf09ac4506a76e8fb57e71ec21" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 34233 } } peers { permanent_uuid: "011b13632d074888950aae2ce091fc86" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 41395 } } peers { permanent_uuid: "c9bf353c47b34903be4da8e7878d6633" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 44703 } }
I20240503 23:25:08.073154 2129 leader_election.cc:288] T 44a56d068f5d4f38bfff50e277b12541 P d81282cf09ac4506a76e8fb57e71ec21 [CANDIDATE]: Term 2 election: Requested vote from peers 011b13632d074888950aae2ce091fc86 (127.0.100.3:41395), c9bf353c47b34903be4da8e7878d6633 (127.0.100.2:44703)
I20240503 23:25:08.080442 2278 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "44a56d068f5d4f38bfff50e277b12541" candidate_uuid: "d81282cf09ac4506a76e8fb57e71ec21" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "011b13632d074888950aae2ce091fc86"
I20240503 23:25:08.080726 2204 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "44a56d068f5d4f38bfff50e277b12541" candidate_uuid: "d81282cf09ac4506a76e8fb57e71ec21" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "c9bf353c47b34903be4da8e7878d6633"
I20240503 23:25:08.080933 2204 raft_consensus.cc:3037] T 44a56d068f5d4f38bfff50e277b12541 P c9bf353c47b34903be4da8e7878d6633 [term 1 FOLLOWER]: Advancing to term 2
I20240503 23:25:08.080929 2278 raft_consensus.cc:3032] T 44a56d068f5d4f38bfff50e277b12541 P 011b13632d074888950aae2ce091fc86 [term 1 LEADER]: Stepping down as leader of term 1
I20240503 23:25:08.081137 2278 raft_consensus.cc:729] T 44a56d068f5d4f38bfff50e277b12541 P 011b13632d074888950aae2ce091fc86 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 011b13632d074888950aae2ce091fc86, State: Running, Role: LEADER
I20240503 23:25:08.081511 2278 consensus_queue.cc:260] T 44a56d068f5d4f38bfff50e277b12541 P 011b13632d074888950aae2ce091fc86 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d81282cf09ac4506a76e8fb57e71ec21" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 34233 } } peers { permanent_uuid: "011b13632d074888950aae2ce091fc86" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 41395 } } peers { permanent_uuid: "c9bf353c47b34903be4da8e7878d6633" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 44703 } }
I20240503 23:25:08.081980 2278 raft_consensus.cc:3037] T 44a56d068f5d4f38bfff50e277b12541 P 011b13632d074888950aae2ce091fc86 [term 1 FOLLOWER]: Advancing to term 2
I20240503 23:25:08.084264 2204 raft_consensus.cc:2454] T 44a56d068f5d4f38bfff50e277b12541 P c9bf353c47b34903be4da8e7878d6633 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d81282cf09ac4506a76e8fb57e71ec21 in term 2.
I20240503 23:25:08.084709 2278 raft_consensus.cc:2454] T 44a56d068f5d4f38bfff50e277b12541 P 011b13632d074888950aae2ce091fc86 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d81282cf09ac4506a76e8fb57e71ec21 in term 2.
I20240503 23:25:08.084951 2094 leader_election.cc:302] T 44a56d068f5d4f38bfff50e277b12541 P d81282cf09ac4506a76e8fb57e71ec21 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c9bf353c47b34903be4da8e7878d6633, d81282cf09ac4506a76e8fb57e71ec21; no voters:
I20240503 23:25:08.085367 2329 raft_consensus.cc:2781] T 44a56d068f5d4f38bfff50e277b12541 P d81282cf09ac4506a76e8fb57e71ec21 [term 2 FOLLOWER]: Leader election won for term 2
I20240503 23:25:08.085855 2329 raft_consensus.cc:686] T 44a56d068f5d4f38bfff50e277b12541 P d81282cf09ac4506a76e8fb57e71ec21 [term 2 LEADER]: Becoming Leader. State: Replica: d81282cf09ac4506a76e8fb57e71ec21, State: Running, Role: LEADER
I20240503 23:25:08.086207 2329 consensus_queue.cc:237] T 44a56d068f5d4f38bfff50e277b12541 P d81282cf09ac4506a76e8fb57e71ec21 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d81282cf09ac4506a76e8fb57e71ec21" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 34233 } } peers { permanent_uuid: "011b13632d074888950aae2ce091fc86" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 41395 } } peers { permanent_uuid: "c9bf353c47b34903be4da8e7878d6633" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 44703 } }
I20240503 23:25:08.089818 2026 catalog_manager.cc:5484] T 44a56d068f5d4f38bfff50e277b12541 P d81282cf09ac4506a76e8fb57e71ec21 reported cstate change: term changed from 1 to 2, leader changed from 011b13632d074888950aae2ce091fc86 (127.0.100.3) to d81282cf09ac4506a76e8fb57e71ec21 (127.0.100.1). New cstate: current_term: 2 leader_uuid: "d81282cf09ac4506a76e8fb57e71ec21" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d81282cf09ac4506a76e8fb57e71ec21" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 34233 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "011b13632d074888950aae2ce091fc86" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 41395 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c9bf353c47b34903be4da8e7878d6633" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 44703 } health_report { overall_health: UNKNOWN } } }
I20240503 23:25:08.589344 2278 raft_consensus.cc:1261] T 44a56d068f5d4f38bfff50e277b12541 P 011b13632d074888950aae2ce091fc86 [term 2 FOLLOWER]: Refusing update from remote peer d81282cf09ac4506a76e8fb57e71ec21: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240503 23:25:08.590273 2357 consensus_queue.cc:1035] T 44a56d068f5d4f38bfff50e277b12541 P d81282cf09ac4506a76e8fb57e71ec21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "011b13632d074888950aae2ce091fc86" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 41395 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 23:25:08.595639 2204 raft_consensus.cc:1261] T 44a56d068f5d4f38bfff50e277b12541 P c9bf353c47b34903be4da8e7878d6633 [term 2 FOLLOWER]: Refusing update from remote peer d81282cf09ac4506a76e8fb57e71ec21: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240503 23:25:08.597468 2359 consensus_queue.cc:1035] T 44a56d068f5d4f38bfff50e277b12541 P d81282cf09ac4506a76e8fb57e71ec21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c9bf353c47b34903be4da8e7878d6633" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 44703 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 23:25:09.718257 2080 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240503 23:25:09.718988 2080 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240503 23:25:09.719117 2080 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240503 23:25:11.719703 2080 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240503 23:25:11.720436 2080 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240503 23:25:11.720825 2080 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240503 23:25:13.721402 2080 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240503 23:25:13.722132 2080 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240503 23:25:13.722260 2080 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240503 23:25:15.722755 2080 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240503 23:25:15.723460 2080 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240503 23:25:15.723619 2080 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240503 23:25:17.724066 2080 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240503 23:25:17.724889 2080 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240503 23:25:17.725068 2080 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240503 23:25:18.278528 400 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 23:25:18.283075 2368 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 23:25:18.283790 2369 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 23:25:18.284495 2371 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 23:25:18.285297 400 server_base.cc:1034] running on GCE node
I20240503 23:25:18.286195 400 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240503 23:25:18.286288 400 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240503 23:25:18.286343 400 hybrid_clock.cc:648] HybridClock initialized: now 1714778718286314 us; error 0 us; skew 500 ppm
I20240503 23:25:18.286517 400 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 23:25:18.293325 400 webserver.cc:460] Webserver started at http://127.0.100.4:32799/ using document root <none> and password file <none>
I20240503 23:25:18.293641 400 fs_manager.cc:359] Metadata directory not provided
I20240503 23:25:18.293742 400 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 23:25:18.293879 400 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 23:25:18.294662 400 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZhIiPz/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714778688451250-400-0/minicluster-data/ts-3-root/instance:
uuid: "4e023fc160224de49ee6346a6827ac29"
format_stamp: "Formatted at 2024-05-03 23:25:18 on dist-test-slave-jvbp"
I20240503 23:25:18.297910 400 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.003s
I20240503 23:25:18.300017 2376 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 23:25:18.300602 400 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.001s
I20240503 23:25:18.300765 400 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZhIiPz/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714778688451250-400-0/minicluster-data/ts-3-root
uuid: "4e023fc160224de49ee6346a6827ac29"
format_stamp: "Formatted at 2024-05-03 23:25:18 on dist-test-slave-jvbp"
I20240503 23:25:18.300956 400 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZhIiPz/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714778688451250-400-0/minicluster-data/ts-3-root
metadata directory: /tmp/dist-test-taskZhIiPz/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714778688451250-400-0/minicluster-data/ts-3-root
1 data directories: /tmp/dist-test-taskZhIiPz/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714778688451250-400-0/minicluster-data/ts-3-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 23:25:18.311712 400 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 23:25:18.312708 400 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 23:25:18.313536 400 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 23:25:18.315667 400 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 23:25:18.315762 400 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 23:25:18.315862 400 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 23:25:18.315943 400 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 23:25:18.338099 400 rpc_server.cc:307] RPC server started. Bound to: 127.0.100.4:39433
I20240503 23:25:18.338114 2438 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.100.4:39433 every 8 connection(s)
I20240503 23:25:18.348048 2439 heartbeater.cc:347] Connected to a master server at 127.0.100.62:37107
I20240503 23:25:18.348256 2439 heartbeater.cc:464] Registering TS with master...
I20240503 23:25:18.348809 2439 heartbeater.cc:511] Master 127.0.100.62:37107 requested a full tablet report, sending...
I20240503 23:25:18.349872 2026 ts_manager.cc:194] Registered new tserver with Master: 4e023fc160224de49ee6346a6827ac29 (127.0.100.4:39433)
I20240503 23:25:18.350838 2026 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:43236
I20240503 23:25:18.735991 2129 consensus_queue.cc:237] T 44a56d068f5d4f38bfff50e277b12541 P d81282cf09ac4506a76e8fb57e71ec21 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d81282cf09ac4506a76e8fb57e71ec21" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 34233 } } peers { permanent_uuid: "011b13632d074888950aae2ce091fc86" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 41395 } } peers { permanent_uuid: "c9bf353c47b34903be4da8e7878d6633" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 44703 } attrs { replace: true } } peers { permanent_uuid: "4e023fc160224de49ee6346a6827ac29" member_type: NON_VOTER last_known_addr { host: "127.0.100.4" port: 39433 } attrs { promote: true } }
I20240503 23:25:18.739400 2278 raft_consensus.cc:1261] T 44a56d068f5d4f38bfff50e277b12541 P 011b13632d074888950aae2ce091fc86 [term 2 FOLLOWER]: Refusing update from remote peer d81282cf09ac4506a76e8fb57e71ec21: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20240503 23:25:18.740375 2361 consensus_queue.cc:1035] T 44a56d068f5d4f38bfff50e277b12541 P d81282cf09ac4506a76e8fb57e71ec21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "011b13632d074888950aae2ce091fc86" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 41395 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 23:25:18.742012 2204 raft_consensus.cc:1261] T 44a56d068f5d4f38bfff50e277b12541 P c9bf353c47b34903be4da8e7878d6633 [term 2 FOLLOWER]: Refusing update from remote peer d81282cf09ac4506a76e8fb57e71ec21: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20240503 23:25:18.742650 2446 consensus_queue.cc:1035] T 44a56d068f5d4f38bfff50e277b12541 P d81282cf09ac4506a76e8fb57e71ec21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c9bf353c47b34903be4da8e7878d6633" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 44703 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 23:25:18.745808 2361 raft_consensus.cc:2932] T 44a56d068f5d4f38bfff50e277b12541 P d81282cf09ac4506a76e8fb57e71ec21 [term 2 LEADER]: Committing config change with OpId 2.3: config changed from index -1 to 3, NON_VOTER 4e023fc160224de49ee6346a6827ac29 (127.0.100.4) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d81282cf09ac4506a76e8fb57e71ec21" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 34233 } } peers { permanent_uuid: "011b13632d074888950aae2ce091fc86" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 41395 } } peers { permanent_uuid: "c9bf353c47b34903be4da8e7878d6633" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 44703 } attrs { replace: true } } peers { permanent_uuid: "4e023fc160224de49ee6346a6827ac29" member_type: NON_VOTER last_known_addr { host: "127.0.100.4" port: 39433 } attrs { promote: true } } }
W20240503 23:25:18.751410 2092 consensus_peers.cc:485] T 44a56d068f5d4f38bfff50e277b12541 P d81282cf09ac4506a76e8fb57e71ec21 -> Peer 4e023fc160224de49ee6346a6827ac29 (127.0.100.4:39433): Couldn't send request to peer 4e023fc160224de49ee6346a6827ac29. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 44a56d068f5d4f38bfff50e277b12541. This is attempt 1: this message will repeat every 5th retry.
I20240503 23:25:18.752103 2278 raft_consensus.cc:2932] T 44a56d068f5d4f38bfff50e277b12541 P 011b13632d074888950aae2ce091fc86 [term 2 FOLLOWER]: Committing config change with OpId 2.3: config changed from index -1 to 3, NON_VOTER 4e023fc160224de49ee6346a6827ac29 (127.0.100.4) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d81282cf09ac4506a76e8fb57e71ec21" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 34233 } } peers { permanent_uuid: "011b13632d074888950aae2ce091fc86" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 41395 } } peers { permanent_uuid: "c9bf353c47b34903be4da8e7878d6633" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 44703 } attrs { replace: true } } peers { permanent_uuid: "4e023fc160224de49ee6346a6827ac29" member_type: NON_VOTER last_known_addr { host: "127.0.100.4" port: 39433 } attrs { promote: true } } }
I20240503 23:25:18.754186 2203 raft_consensus.cc:2932] T 44a56d068f5d4f38bfff50e277b12541 P c9bf353c47b34903be4da8e7878d6633 [term 2 FOLLOWER]: Committing config change with OpId 2.3: config changed from index -1 to 3, NON_VOTER 4e023fc160224de49ee6346a6827ac29 (127.0.100.4) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d81282cf09ac4506a76e8fb57e71ec21" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 34233 } } peers { permanent_uuid: "011b13632d074888950aae2ce091fc86" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 41395 } } peers { permanent_uuid: "c9bf353c47b34903be4da8e7878d6633" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 44703 } attrs { replace: true } } peers { permanent_uuid: "4e023fc160224de49ee6346a6827ac29" member_type: NON_VOTER last_known_addr { host: "127.0.100.4" port: 39433 } attrs { promote: true } } }
I20240503 23:25:18.758091 2026 catalog_manager.cc:5484] T 44a56d068f5d4f38bfff50e277b12541 P 011b13632d074888950aae2ce091fc86 reported cstate change: config changed from index -1 to 3, NON_VOTER 4e023fc160224de49ee6346a6827ac29 (127.0.100.4) added. New cstate: current_term: 2 leader_uuid: "d81282cf09ac4506a76e8fb57e71ec21" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d81282cf09ac4506a76e8fb57e71ec21" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 34233 } } peers { permanent_uuid: "011b13632d074888950aae2ce091fc86" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 41395 } } peers { permanent_uuid: "c9bf353c47b34903be4da8e7878d6633" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 44703 } attrs { replace: true } } peers { permanent_uuid: "4e023fc160224de49ee6346a6827ac29" member_type: NON_VOTER last_known_addr { host: "127.0.100.4" port: 39433 } attrs { promote: true } } }
I20240503 23:25:19.311986 2457 ts_tablet_manager.cc:902] T 44a56d068f5d4f38bfff50e277b12541 P 4e023fc160224de49ee6346a6827ac29: Initiating tablet copy from peer d81282cf09ac4506a76e8fb57e71ec21 (127.0.100.1:34233)
I20240503 23:25:19.312851 2457 tablet_copy_client.cc:316] T 44a56d068f5d4f38bfff50e277b12541 P 4e023fc160224de49ee6346a6827ac29: tablet copy: Beginning tablet copy session from remote peer at address 127.0.100.1:34233
I20240503 23:25:19.320995 2139 tablet_copy_service.cc:140] P d81282cf09ac4506a76e8fb57e71ec21: Received BeginTabletCopySession request for tablet 44a56d068f5d4f38bfff50e277b12541 from peer 4e023fc160224de49ee6346a6827ac29 ({username='slave'} at 127.0.0.1:60668)
I20240503 23:25:19.321271 2139 tablet_copy_service.cc:161] P d81282cf09ac4506a76e8fb57e71ec21: Beginning new tablet copy session on tablet 44a56d068f5d4f38bfff50e277b12541 from peer 4e023fc160224de49ee6346a6827ac29 at {username='slave'} at 127.0.0.1:60668: session id = 4e023fc160224de49ee6346a6827ac29-44a56d068f5d4f38bfff50e277b12541
I20240503 23:25:19.324282 2139 tablet_copy_source_session.cc:205] T 44a56d068f5d4f38bfff50e277b12541 P d81282cf09ac4506a76e8fb57e71ec21: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 23:25:19.325820 2457 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 44a56d068f5d4f38bfff50e277b12541. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 23:25:19.331512 2457 tablet_copy_client.cc:791] T 44a56d068f5d4f38bfff50e277b12541 P 4e023fc160224de49ee6346a6827ac29: tablet copy: Starting download of 0 data blocks...
I20240503 23:25:19.331877 2457 tablet_copy_client.cc:655] T 44a56d068f5d4f38bfff50e277b12541 P 4e023fc160224de49ee6346a6827ac29: tablet copy: Starting download of 1 WAL segments...
I20240503 23:25:19.334522 2457 tablet_copy_client.cc:531] T 44a56d068f5d4f38bfff50e277b12541 P 4e023fc160224de49ee6346a6827ac29: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 23:25:19.338449 2457 tablet_bootstrap.cc:492] T 44a56d068f5d4f38bfff50e277b12541 P 4e023fc160224de49ee6346a6827ac29: Bootstrap starting.
I20240503 23:25:19.348161 2457 tablet_bootstrap.cc:492] T 44a56d068f5d4f38bfff50e277b12541 P 4e023fc160224de49ee6346a6827ac29: Bootstrap replayed 1/1 log segments. Stats: ops{read=3 overwritten=0 applied=3 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 23:25:19.348680 2457 tablet_bootstrap.cc:492] T 44a56d068f5d4f38bfff50e277b12541 P 4e023fc160224de49ee6346a6827ac29: Bootstrap complete.
I20240503 23:25:19.348995 2457 ts_tablet_manager.cc:1367] T 44a56d068f5d4f38bfff50e277b12541 P 4e023fc160224de49ee6346a6827ac29: Time spent bootstrapping tablet: real 0.011s user 0.009s sys 0.002s
I20240503 23:25:19.349797 2457 raft_consensus.cc:348] T 44a56d068f5d4f38bfff50e277b12541 P 4e023fc160224de49ee6346a6827ac29 [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d81282cf09ac4506a76e8fb57e71ec21" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 34233 } } peers { permanent_uuid: "011b13632d074888950aae2ce091fc86" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 41395 } } peers { permanent_uuid: "c9bf353c47b34903be4da8e7878d6633" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 44703 } attrs { replace: true } } peers { permanent_uuid: "4e023fc160224de49ee6346a6827ac29" member_type: NON_VOTER last_known_addr { host: "127.0.100.4" port: 39433 } attrs { promote: true } }
I20240503 23:25:19.350062 2457 raft_consensus.cc:729] T 44a56d068f5d4f38bfff50e277b12541 P 4e023fc160224de49ee6346a6827ac29 [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: 4e023fc160224de49ee6346a6827ac29, State: Initialized, Role: LEARNER
I20240503 23:25:19.350407 2457 consensus_queue.cc:260] T 44a56d068f5d4f38bfff50e277b12541 P 4e023fc160224de49ee6346a6827ac29 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 2.3, Last appended by leader: 3, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d81282cf09ac4506a76e8fb57e71ec21" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 34233 } } peers { permanent_uuid: "011b13632d074888950aae2ce091fc86" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 41395 } } peers { permanent_uuid: "c9bf353c47b34903be4da8e7878d6633" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 44703 } attrs { replace: true } } peers { permanent_uuid: "4e023fc160224de49ee6346a6827ac29" member_type: NON_VOTER last_known_addr { host: "127.0.100.4" port: 39433 } attrs { promote: true } }
I20240503 23:25:19.351662 2439 heartbeater.cc:503] Master 127.0.100.62:37107 was elected leader, sending a full tablet report...
I20240503 23:25:19.352330 2457 ts_tablet_manager.cc:1398] T 44a56d068f5d4f38bfff50e277b12541 P 4e023fc160224de49ee6346a6827ac29: Time spent starting tablet: real 0.003s user 0.002s sys 0.001s
I20240503 23:25:19.353143 2139 tablet_copy_service.cc:342] P d81282cf09ac4506a76e8fb57e71ec21: Request end of tablet copy session 4e023fc160224de49ee6346a6827ac29-44a56d068f5d4f38bfff50e277b12541 received from {username='slave'} at 127.0.0.1:60668
I20240503 23:25:19.353257 2139 tablet_copy_service.cc:433] P d81282cf09ac4506a76e8fb57e71ec21: ending tablet copy session 4e023fc160224de49ee6346a6827ac29-44a56d068f5d4f38bfff50e277b12541 on tablet 44a56d068f5d4f38bfff50e277b12541 with peer 4e023fc160224de49ee6346a6827ac29
I20240503 23:25:19.725469 2080 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240503 23:25:19.726220 2080 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240503 23:25:19.726351 2080 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240503 23:25:19.821990 2414 raft_consensus.cc:1203] T 44a56d068f5d4f38bfff50e277b12541 P 4e023fc160224de49ee6346a6827ac29 [term 2 LEARNER]: Deduplicated request from leader. Original: 2.2->[2.3-2.3] Dedup: 2.3->[]
I20240503 23:25:20.421389 2464 raft_consensus.cc:1050] T 44a56d068f5d4f38bfff50e277b12541 P d81282cf09ac4506a76e8fb57e71ec21: attempting to promote NON_VOTER 4e023fc160224de49ee6346a6827ac29 to VOTER
I20240503 23:25:20.422398 2464 consensus_queue.cc:237] T 44a56d068f5d4f38bfff50e277b12541 P d81282cf09ac4506a76e8fb57e71ec21 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.3, Last appended by leader: 1, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d81282cf09ac4506a76e8fb57e71ec21" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 34233 } } peers { permanent_uuid: "011b13632d074888950aae2ce091fc86" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 41395 } } peers { permanent_uuid: "c9bf353c47b34903be4da8e7878d6633" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 44703 } attrs { replace: true } } peers { permanent_uuid: "4e023fc160224de49ee6346a6827ac29" member_type: VOTER last_known_addr { host: "127.0.100.4" port: 39433 } attrs { promote: false } }
I20240503 23:25:20.425084 2414 raft_consensus.cc:1261] T 44a56d068f5d4f38bfff50e277b12541 P 4e023fc160224de49ee6346a6827ac29 [term 2 LEARNER]: Refusing update from remote peer d81282cf09ac4506a76e8fb57e71ec21: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240503 23:25:20.425983 2467 consensus_queue.cc:1035] T 44a56d068f5d4f38bfff50e277b12541 P d81282cf09ac4506a76e8fb57e71ec21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4e023fc160224de49ee6346a6827ac29" member_type: VOTER last_known_addr { host: "127.0.100.4" port: 39433 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 23:25:20.426437 2203 raft_consensus.cc:1261] T 44a56d068f5d4f38bfff50e277b12541 P c9bf353c47b34903be4da8e7878d6633 [term 2 FOLLOWER]: Refusing update from remote peer d81282cf09ac4506a76e8fb57e71ec21: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240503 23:25:20.427019 2278 raft_consensus.cc:1261] T 44a56d068f5d4f38bfff50e277b12541 P 011b13632d074888950aae2ce091fc86 [term 2 FOLLOWER]: Refusing update from remote peer d81282cf09ac4506a76e8fb57e71ec21: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240503 23:25:20.427466 2468 consensus_queue.cc:1035] T 44a56d068f5d4f38bfff50e277b12541 P d81282cf09ac4506a76e8fb57e71ec21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c9bf353c47b34903be4da8e7878d6633" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 44703 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 23:25:20.427930 2465 consensus_queue.cc:1035] T 44a56d068f5d4f38bfff50e277b12541 P d81282cf09ac4506a76e8fb57e71ec21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "011b13632d074888950aae2ce091fc86" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 41395 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 23:25:20.433528 2468 raft_consensus.cc:2932] T 44a56d068f5d4f38bfff50e277b12541 P d81282cf09ac4506a76e8fb57e71ec21 [term 2 LEADER]: Committing config change with OpId 2.4: config changed from index 3 to 4, 4e023fc160224de49ee6346a6827ac29 (127.0.100.4) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d81282cf09ac4506a76e8fb57e71ec21" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 34233 } } peers { permanent_uuid: "011b13632d074888950aae2ce091fc86" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 41395 } } peers { permanent_uuid: "c9bf353c47b34903be4da8e7878d6633" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 44703 } attrs { replace: true } } peers { permanent_uuid: "4e023fc160224de49ee6346a6827ac29" member_type: VOTER last_known_addr { host: "127.0.100.4" port: 39433 } attrs { promote: false } } }
I20240503 23:25:20.434274 2203 raft_consensus.cc:2932] T 44a56d068f5d4f38bfff50e277b12541 P c9bf353c47b34903be4da8e7878d6633 [term 2 FOLLOWER]: Committing config change with OpId 2.4: config changed from index 3 to 4, 4e023fc160224de49ee6346a6827ac29 (127.0.100.4) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d81282cf09ac4506a76e8fb57e71ec21" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 34233 } } peers { permanent_uuid: "011b13632d074888950aae2ce091fc86" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 41395 } } peers { permanent_uuid: "c9bf353c47b34903be4da8e7878d6633" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 44703 } attrs { replace: true } } peers { permanent_uuid: "4e023fc160224de49ee6346a6827ac29" member_type: VOTER last_known_addr { host: "127.0.100.4" port: 39433 } attrs { promote: false } } }
I20240503 23:25:20.438519 2278 raft_consensus.cc:2932] T 44a56d068f5d4f38bfff50e277b12541 P 011b13632d074888950aae2ce091fc86 [term 2 FOLLOWER]: Committing config change with OpId 2.4: config changed from index 3 to 4, 4e023fc160224de49ee6346a6827ac29 (127.0.100.4) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d81282cf09ac4506a76e8fb57e71ec21" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 34233 } } peers { permanent_uuid: "011b13632d074888950aae2ce091fc86" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 41395 } } peers { permanent_uuid: "c9bf353c47b34903be4da8e7878d6633" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 44703 } attrs { replace: true } } peers { permanent_uuid: "4e023fc160224de49ee6346a6827ac29" member_type: VOTER last_known_addr { host: "127.0.100.4" port: 39433 } attrs { promote: false } } }
I20240503 23:25:20.439541 2027 catalog_manager.cc:5484] T 44a56d068f5d4f38bfff50e277b12541 P d81282cf09ac4506a76e8fb57e71ec21 reported cstate change: config changed from index 3 to 4, 4e023fc160224de49ee6346a6827ac29 (127.0.100.4) changed from NON_VOTER to VOTER. New cstate: current_term: 2 leader_uuid: "d81282cf09ac4506a76e8fb57e71ec21" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d81282cf09ac4506a76e8fb57e71ec21" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 34233 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "011b13632d074888950aae2ce091fc86" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 41395 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c9bf353c47b34903be4da8e7878d6633" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 44703 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4e023fc160224de49ee6346a6827ac29" member_type: VOTER last_known_addr { host: "127.0.100.4" port: 39433 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 23:25:20.443322 2414 raft_consensus.cc:2932] T 44a56d068f5d4f38bfff50e277b12541 P 4e023fc160224de49ee6346a6827ac29 [term 2 FOLLOWER]: Committing config change with OpId 2.4: config changed from index 3 to 4, 4e023fc160224de49ee6346a6827ac29 (127.0.100.4) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d81282cf09ac4506a76e8fb57e71ec21" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 34233 } } peers { permanent_uuid: "011b13632d074888950aae2ce091fc86" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 41395 } } peers { permanent_uuid: "c9bf353c47b34903be4da8e7878d6633" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 44703 } attrs { replace: true } } peers { permanent_uuid: "4e023fc160224de49ee6346a6827ac29" member_type: VOTER last_known_addr { host: "127.0.100.4" port: 39433 } attrs { promote: false } } }
I20240503 23:25:20.449467 2129 consensus_queue.cc:237] T 44a56d068f5d4f38bfff50e277b12541 P d81282cf09ac4506a76e8fb57e71ec21 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.4, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d81282cf09ac4506a76e8fb57e71ec21" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 34233 } } peers { permanent_uuid: "011b13632d074888950aae2ce091fc86" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 41395 } } peers { permanent_uuid: "4e023fc160224de49ee6346a6827ac29" member_type: VOTER last_known_addr { host: "127.0.100.4" port: 39433 } attrs { promote: false } }
I20240503 23:25:20.454061 2278 raft_consensus.cc:1261] T 44a56d068f5d4f38bfff50e277b12541 P 011b13632d074888950aae2ce091fc86 [term 2 FOLLOWER]: Refusing update from remote peer d81282cf09ac4506a76e8fb57e71ec21: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240503 23:25:20.454833 2464 consensus_queue.cc:1035] T 44a56d068f5d4f38bfff50e277b12541 P d81282cf09ac4506a76e8fb57e71ec21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "011b13632d074888950aae2ce091fc86" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 41395 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 23:25:20.458377 2413 raft_consensus.cc:1261] T 44a56d068f5d4f38bfff50e277b12541 P 4e023fc160224de49ee6346a6827ac29 [term 2 FOLLOWER]: Refusing update from remote peer d81282cf09ac4506a76e8fb57e71ec21: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240503 23:25:20.459391 2464 consensus_queue.cc:1035] T 44a56d068f5d4f38bfff50e277b12541 P d81282cf09ac4506a76e8fb57e71ec21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4e023fc160224de49ee6346a6827ac29" member_type: VOTER last_known_addr { host: "127.0.100.4" port: 39433 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 23:25:20.460633 2465 raft_consensus.cc:2932] T 44a56d068f5d4f38bfff50e277b12541 P d81282cf09ac4506a76e8fb57e71ec21 [term 2 LEADER]: Committing config change with OpId 2.5: config changed from index 4 to 5, VOTER c9bf353c47b34903be4da8e7878d6633 (127.0.100.2) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d81282cf09ac4506a76e8fb57e71ec21" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 34233 } } peers { permanent_uuid: "011b13632d074888950aae2ce091fc86" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 41395 } } peers { permanent_uuid: "4e023fc160224de49ee6346a6827ac29" member_type: VOTER last_known_addr { host: "127.0.100.4" port: 39433 } attrs { promote: false } } }
I20240503 23:25:20.463393 2413 raft_consensus.cc:2932] T 44a56d068f5d4f38bfff50e277b12541 P 4e023fc160224de49ee6346a6827ac29 [term 2 FOLLOWER]: Committing config change with OpId 2.5: config changed from index 4 to 5, VOTER c9bf353c47b34903be4da8e7878d6633 (127.0.100.2) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d81282cf09ac4506a76e8fb57e71ec21" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 34233 } } peers { permanent_uuid: "011b13632d074888950aae2ce091fc86" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 41395 } } peers { permanent_uuid: "4e023fc160224de49ee6346a6827ac29" member_type: VOTER last_known_addr { host: "127.0.100.4" port: 39433 } attrs { promote: false } } }
I20240503 23:25:20.464715 2013 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 44a56d068f5d4f38bfff50e277b12541 with cas_config_opid_index 4: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 23:25:20.465596 2278 raft_consensus.cc:2932] T 44a56d068f5d4f38bfff50e277b12541 P 011b13632d074888950aae2ce091fc86 [term 2 FOLLOWER]: Committing config change with OpId 2.5: config changed from index 4 to 5, VOTER c9bf353c47b34903be4da8e7878d6633 (127.0.100.2) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d81282cf09ac4506a76e8fb57e71ec21" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 34233 } } peers { permanent_uuid: "011b13632d074888950aae2ce091fc86" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 41395 } } peers { permanent_uuid: "4e023fc160224de49ee6346a6827ac29" member_type: VOTER last_known_addr { host: "127.0.100.4" port: 39433 } attrs { promote: false } } }
I20240503 23:25:20.466405 2025 catalog_manager.cc:5484] T 44a56d068f5d4f38bfff50e277b12541 P d81282cf09ac4506a76e8fb57e71ec21 reported cstate change: config changed from index 4 to 5, VOTER c9bf353c47b34903be4da8e7878d6633 (127.0.100.2) evicted. New cstate: current_term: 2 leader_uuid: "d81282cf09ac4506a76e8fb57e71ec21" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d81282cf09ac4506a76e8fb57e71ec21" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 34233 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "011b13632d074888950aae2ce091fc86" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 41395 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4e023fc160224de49ee6346a6827ac29" member_type: VOTER last_known_addr { host: "127.0.100.4" port: 39433 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 23:25:20.472970 2193 tablet_service.cc:1514] Processing DeleteTablet for tablet 44a56d068f5d4f38bfff50e277b12541 with delete_type TABLET_DATA_TOMBSTONED (TS c9bf353c47b34903be4da8e7878d6633 not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:45126
I20240503 23:25:20.474231 2479 tablet_replica.cc:330] T 44a56d068f5d4f38bfff50e277b12541 P c9bf353c47b34903be4da8e7878d6633: stopping tablet replica
I20240503 23:25:20.474561 2479 raft_consensus.cc:2229] T 44a56d068f5d4f38bfff50e277b12541 P c9bf353c47b34903be4da8e7878d6633 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 23:25:20.475106 2479 raft_consensus.cc:2258] T 44a56d068f5d4f38bfff50e277b12541 P c9bf353c47b34903be4da8e7878d6633 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 23:25:20.476081 2479 ts_tablet_manager.cc:1875] T 44a56d068f5d4f38bfff50e277b12541 P c9bf353c47b34903be4da8e7878d6633: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 23:25:20.482301 2479 ts_tablet_manager.cc:1888] T 44a56d068f5d4f38bfff50e277b12541 P c9bf353c47b34903be4da8e7878d6633: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.4
I20240503 23:25:20.482457 2479 log.cc:1196] T 44a56d068f5d4f38bfff50e277b12541 P c9bf353c47b34903be4da8e7878d6633: Deleting WAL directory at /tmp/dist-test-taskZhIiPz/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714778688451250-400-0/minicluster-data/ts-1-root/wals/44a56d068f5d4f38bfff50e277b12541
I20240503 23:25:20.483428 2013 catalog_manager.cc:4830] TS c9bf353c47b34903be4da8e7878d6633 (127.0.100.2:44703): tablet 44a56d068f5d4f38bfff50e277b12541 (table test-workload [id=141458155540495281d7c93613f1e996]) successfully deleted
I20240503 23:25:21.726981 2080 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240503 23:25:21.727716 2080 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240503 23:25:21.727861 2080 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240503 23:25:23.728305 2080 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240503 23:25:23.729027 2080 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240503 23:25:23.729142 2080 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240503 23:25:25.729657 2080 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240503 23:25:25.730479 2080 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240503 23:25:25.730599 2080 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240503 23:25:27.731165 2080 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240503 23:25:27.732038 2080 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240503 23:25:27.732184 2080 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240503 23:25:29.732842 2080 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240503 23:25:29.733543 2080 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240503 23:25:29.733652 2080 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240503 23:25:31.734131 2080 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240503 23:25:31.734897 2080 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240503 23:25:31.735016 2080 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240503 23:25:32.425467 400 tablet_server.cc:178] TabletServer@127.0.100.1:0 shutting down...
I20240503 23:25:32.443596 400 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240503 23:25:32.444118 400 tablet_replica.cc:330] T 44a56d068f5d4f38bfff50e277b12541 P d81282cf09ac4506a76e8fb57e71ec21: stopping tablet replica
I20240503 23:25:32.444388 400 raft_consensus.cc:2229] T 44a56d068f5d4f38bfff50e277b12541 P d81282cf09ac4506a76e8fb57e71ec21 [term 2 LEADER]: Raft consensus shutting down.
I20240503 23:25:32.444857 400 raft_consensus.cc:2258] T 44a56d068f5d4f38bfff50e277b12541 P d81282cf09ac4506a76e8fb57e71ec21 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 23:25:32.445663 400 tablet_replica.cc:330] T 9d1b3bdd6dee4f37aecfcc05a395bed0 P d81282cf09ac4506a76e8fb57e71ec21: stopping tablet replica
I20240503 23:25:32.445856 400 raft_consensus.cc:2229] T 9d1b3bdd6dee4f37aecfcc05a395bed0 P d81282cf09ac4506a76e8fb57e71ec21 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 23:25:32.446051 400 raft_consensus.cc:2258] T 9d1b3bdd6dee4f37aecfcc05a395bed0 P d81282cf09ac4506a76e8fb57e71ec21 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 23:25:32.465389 400 tablet_server.cc:194] TabletServer@127.0.100.1:0 shutdown complete.
I20240503 23:25:32.469771 400 tablet_server.cc:178] TabletServer@127.0.100.2:0 shutting down...
I20240503 23:25:32.482182 400 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240503 23:25:32.482534 400 tablet_replica.cc:330] T 9d1b3bdd6dee4f37aecfcc05a395bed0 P c9bf353c47b34903be4da8e7878d6633: stopping tablet replica
I20240503 23:25:32.482761 400 raft_consensus.cc:2229] T 9d1b3bdd6dee4f37aecfcc05a395bed0 P c9bf353c47b34903be4da8e7878d6633 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 23:25:32.483023 400 raft_consensus.cc:2258] T 9d1b3bdd6dee4f37aecfcc05a395bed0 P c9bf353c47b34903be4da8e7878d6633 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 23:25:32.497927 400 tablet_server.cc:194] TabletServer@127.0.100.2:0 shutdown complete.
I20240503 23:25:32.502574 400 tablet_server.cc:178] TabletServer@127.0.100.3:0 shutting down...
I20240503 23:25:32.513309 400 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240503 23:25:32.513703 400 tablet_replica.cc:330] T 44a56d068f5d4f38bfff50e277b12541 P 011b13632d074888950aae2ce091fc86: stopping tablet replica
I20240503 23:25:32.513928 400 raft_consensus.cc:2229] T 44a56d068f5d4f38bfff50e277b12541 P 011b13632d074888950aae2ce091fc86 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 23:25:32.514266 400 raft_consensus.cc:2258] T 44a56d068f5d4f38bfff50e277b12541 P 011b13632d074888950aae2ce091fc86 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 23:25:32.515020 400 tablet_replica.cc:330] T 9d1b3bdd6dee4f37aecfcc05a395bed0 P 011b13632d074888950aae2ce091fc86: stopping tablet replica
I20240503 23:25:32.515215 400 raft_consensus.cc:2229] T 9d1b3bdd6dee4f37aecfcc05a395bed0 P 011b13632d074888950aae2ce091fc86 [term 1 LEADER]: Raft consensus shutting down.
I20240503 23:25:32.515589 400 raft_consensus.cc:2258] T 9d1b3bdd6dee4f37aecfcc05a395bed0 P 011b13632d074888950aae2ce091fc86 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 23:25:32.530679 400 tablet_server.cc:194] TabletServer@127.0.100.3:0 shutdown complete.
I20240503 23:25:32.535420 400 tablet_server.cc:178] TabletServer@127.0.100.4:0 shutting down...
I20240503 23:25:32.545984 400 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240503 23:25:32.546339 400 tablet_replica.cc:330] T 44a56d068f5d4f38bfff50e277b12541 P 4e023fc160224de49ee6346a6827ac29: stopping tablet replica
I20240503 23:25:32.546535 400 raft_consensus.cc:2229] T 44a56d068f5d4f38bfff50e277b12541 P 4e023fc160224de49ee6346a6827ac29 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 23:25:32.546833 400 raft_consensus.cc:2258] T 44a56d068f5d4f38bfff50e277b12541 P 4e023fc160224de49ee6346a6827ac29 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 23:25:32.561283 400 tablet_server.cc:194] TabletServer@127.0.100.4:0 shutdown complete.
I20240503 23:25:32.571832 400 master.cc:537] Master@127.0.100.62:37107 shutting down...
I20240503 23:25:32.581171 400 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 54e22664d9d64eb8ad0fd21ef1aa7b60 [term 1 LEADER]: Raft consensus shutting down.
I20240503 23:25:32.581503 400 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 54e22664d9d64eb8ad0fd21ef1aa7b60 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 23:25:32.581640 400 tablet_replica.cc:330] T 00000000000000000000000000000000 P 54e22664d9d64eb8ad0fd21ef1aa7b60: stopping tablet replica
I20240503 23:25:32.596438 400 master.cc:552] Master@127.0.100.62:37107 shutdown complete.
[ OK ] AutoRebalancerTest.MovesScheduledIfAddTserver (27006 ms)
[ RUN ] AutoRebalancerTest.NoReplicaMovesIfNoTservers
I20240503 23:25:32.611285 400 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.0.100.62:39877
I20240503 23:25:32.611950 400 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 23:25:32.616184 2482 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 23:25:32.616916 2483 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 23:25:32.617657 2485 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 23:25:32.618466 400 server_base.cc:1034] running on GCE node
I20240503 23:25:32.618726 400 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240503 23:25:32.618777 400 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240503 23:25:32.618803 400 hybrid_clock.cc:648] HybridClock initialized: now 1714778732618798 us; error 0 us; skew 500 ppm
I20240503 23:25:32.618944 400 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 23:25:32.626374 400 webserver.cc:460] Webserver started at http://127.0.100.62:34353/ using document root <none> and password file <none>
I20240503 23:25:32.626669 400 fs_manager.cc:359] Metadata directory not provided
I20240503 23:25:32.626736 400 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 23:25:32.626844 400 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 23:25:32.627621 400 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZhIiPz/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTservers.1714778688451250-400-0/minicluster-data/master-0-root/instance:
uuid: "588245fdd94b46a4b3186b42e0472a5e"
format_stamp: "Formatted at 2024-05-03 23:25:32 on dist-test-slave-jvbp"
I20240503 23:25:32.631206 400 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.004s
I20240503 23:25:32.633198 2490 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 23:25:32.633687 400 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.001s
I20240503 23:25:32.633793 400 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZhIiPz/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTservers.1714778688451250-400-0/minicluster-data/master-0-root
uuid: "588245fdd94b46a4b3186b42e0472a5e"
format_stamp: "Formatted at 2024-05-03 23:25:32 on dist-test-slave-jvbp"
I20240503 23:25:32.633908 400 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZhIiPz/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTservers.1714778688451250-400-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskZhIiPz/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTservers.1714778688451250-400-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskZhIiPz/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTservers.1714778688451250-400-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 23:25:32.643124 400 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 23:25:32.643808 400 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 23:25:32.656853 400 rpc_server.cc:307] RPC server started. Bound to: 127.0.100.62:39877
I20240503 23:25:32.656862 2541 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.100.62:39877 every 8 connection(s)
I20240503 23:25:32.658730 2542 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 23:25:32.664367 2542 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 588245fdd94b46a4b3186b42e0472a5e: Bootstrap starting.
I20240503 23:25:32.666169 2542 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 588245fdd94b46a4b3186b42e0472a5e: Neither blocks nor log segments found. Creating new log.
I20240503 23:25:32.668612 2542 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 588245fdd94b46a4b3186b42e0472a5e: No bootstrap required, opened a new log
I20240503 23:25:32.669338 2542 raft_consensus.cc:348] T 00000000000000000000000000000000 P 588245fdd94b46a4b3186b42e0472a5e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "588245fdd94b46a4b3186b42e0472a5e" member_type: VOTER }
I20240503 23:25:32.669476 2542 raft_consensus.cc:374] T 00000000000000000000000000000000 P 588245fdd94b46a4b3186b42e0472a5e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 23:25:32.669528 2542 raft_consensus.cc:729] T 00000000000000000000000000000000 P 588245fdd94b46a4b3186b42e0472a5e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 588245fdd94b46a4b3186b42e0472a5e, State: Initialized, Role: FOLLOWER
I20240503 23:25:32.669736 2542 consensus_queue.cc:260] T 00000000000000000000000000000000 P 588245fdd94b46a4b3186b42e0472a5e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "588245fdd94b46a4b3186b42e0472a5e" member_type: VOTER }
I20240503 23:25:32.670094 2542 raft_consensus.cc:388] T 00000000000000000000000000000000 P 588245fdd94b46a4b3186b42e0472a5e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 23:25:32.670184 2542 raft_consensus.cc:482] T 00000000000000000000000000000000 P 588245fdd94b46a4b3186b42e0472a5e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 23:25:32.670279 2542 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 588245fdd94b46a4b3186b42e0472a5e [term 0 FOLLOWER]: Advancing to term 1
I20240503 23:25:32.672431 2542 raft_consensus.cc:504] T 00000000000000000000000000000000 P 588245fdd94b46a4b3186b42e0472a5e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "588245fdd94b46a4b3186b42e0472a5e" member_type: VOTER }
I20240503 23:25:32.672811 2542 leader_election.cc:302] T 00000000000000000000000000000000 P 588245fdd94b46a4b3186b42e0472a5e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 588245fdd94b46a4b3186b42e0472a5e; no voters:
I20240503 23:25:32.673473 2542 leader_election.cc:288] T 00000000000000000000000000000000 P 588245fdd94b46a4b3186b42e0472a5e [CANDIDATE]: Term 1 election: Requested vote from peers
I20240503 23:25:32.673592 2545 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 588245fdd94b46a4b3186b42e0472a5e [term 1 FOLLOWER]: Leader election won for term 1
I20240503 23:25:32.674167 2545 raft_consensus.cc:686] T 00000000000000000000000000000000 P 588245fdd94b46a4b3186b42e0472a5e [term 1 LEADER]: Becoming Leader. State: Replica: 588245fdd94b46a4b3186b42e0472a5e, State: Running, Role: LEADER
I20240503 23:25:32.674486 2545 consensus_queue.cc:237] T 00000000000000000000000000000000 P 588245fdd94b46a4b3186b42e0472a5e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "588245fdd94b46a4b3186b42e0472a5e" member_type: VOTER }
I20240503 23:25:32.674580 2542 sys_catalog.cc:564] T 00000000000000000000000000000000 P 588245fdd94b46a4b3186b42e0472a5e [sys.catalog]: configured and running, proceeding with master startup.
I20240503 23:25:32.675678 2546 sys_catalog.cc:455] T 00000000000000000000000000000000 P 588245fdd94b46a4b3186b42e0472a5e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "588245fdd94b46a4b3186b42e0472a5e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "588245fdd94b46a4b3186b42e0472a5e" member_type: VOTER } }
I20240503 23:25:32.675655 2547 sys_catalog.cc:455] T 00000000000000000000000000000000 P 588245fdd94b46a4b3186b42e0472a5e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 588245fdd94b46a4b3186b42e0472a5e. Latest consensus state: current_term: 1 leader_uuid: "588245fdd94b46a4b3186b42e0472a5e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "588245fdd94b46a4b3186b42e0472a5e" member_type: VOTER } }
I20240503 23:25:32.675889 2546 sys_catalog.cc:458] T 00000000000000000000000000000000 P 588245fdd94b46a4b3186b42e0472a5e [sys.catalog]: This master's current role is: LEADER
I20240503 23:25:32.675889 2547 sys_catalog.cc:458] T 00000000000000000000000000000000 P 588245fdd94b46a4b3186b42e0472a5e [sys.catalog]: This master's current role is: LEADER
I20240503 23:25:32.677517 2550 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240503 23:25:32.679010 2550 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240503 23:25:32.680814 400 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20240503 23:25:32.682751 2550 catalog_manager.cc:1347] Generated new cluster ID: 2a2941f4dddf43c3a79789aa5d26ed5f
I20240503 23:25:32.682854 2550 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240503 23:25:32.698742 2550 catalog_manager.cc:1370] Generated new certificate authority record
I20240503 23:25:32.699702 2550 catalog_manager.cc:1504] Loading token signing keys...
I20240503 23:25:32.708297 2550 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 588245fdd94b46a4b3186b42e0472a5e: Generated new TSK 0
I20240503 23:25:32.708673 2550 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240503 23:25:32.713347 400 internal_mini_cluster.cc:371] 0 TS(s) registered with all masters after 0.0001032s
I20240503 23:25:44.800634 400 master.cc:537] Master@127.0.100.62:39877 shutting down...
I20240503 23:25:44.816260 400 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 588245fdd94b46a4b3186b42e0472a5e [term 1 LEADER]: Raft consensus shutting down.
I20240503 23:25:44.816627 400 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 588245fdd94b46a4b3186b42e0472a5e [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 23:25:44.816771 400 tablet_replica.cc:330] T 00000000000000000000000000000000 P 588245fdd94b46a4b3186b42e0472a5e: stopping tablet replica
I20240503 23:25:44.831327 400 master.cc:552] Master@127.0.100.62:39877 shutdown complete.
[ OK ] AutoRebalancerTest.NoReplicaMovesIfNoTservers (12230 ms)
[ RUN ] AutoRebalancerTest.NoReplicaMovesIfNoTablets
I20240503 23:25:44.841584 400 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.0.100.62:38247
I20240503 23:25:44.842252 400 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 23:25:44.846307 2564 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 23:25:44.847085 2565 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 23:25:44.847858 2567 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 23:25:44.848724 400 server_base.cc:1034] running on GCE node
I20240503 23:25:44.850729 400 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240503 23:25:44.850839 400 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240503 23:25:44.850900 400 hybrid_clock.cc:648] HybridClock initialized: now 1714778744850874 us; error 0 us; skew 500 ppm
I20240503 23:25:44.851111 400 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 23:25:44.852818 400 webserver.cc:460] Webserver started at http://127.0.100.62:42023/ using document root <none> and password file <none>
I20240503 23:25:44.853132 400 fs_manager.cc:359] Metadata directory not provided
I20240503 23:25:44.853237 400 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 23:25:44.853402 400 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 23:25:44.854209 400 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZhIiPz/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1714778688451250-400-0/minicluster-data/master-0-root/instance:
uuid: "55ccc811bdc54f7c9d30ad7dd73a4bc7"
format_stamp: "Formatted at 2024-05-03 23:25:44 on dist-test-slave-jvbp"
I20240503 23:25:44.857249 400 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.004s
I20240503 23:25:44.859222 2572 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 23:25:44.859670 400 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.002s
I20240503 23:25:44.859802 400 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZhIiPz/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1714778688451250-400-0/minicluster-data/master-0-root
uuid: "55ccc811bdc54f7c9d30ad7dd73a4bc7"
format_stamp: "Formatted at 2024-05-03 23:25:44 on dist-test-slave-jvbp"
I20240503 23:25:44.859966 400 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZhIiPz/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1714778688451250-400-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskZhIiPz/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1714778688451250-400-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskZhIiPz/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1714778688451250-400-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 23:25:44.885298 400 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 23:25:44.886009 400 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 23:25:44.900182 400 rpc_server.cc:307] RPC server started. Bound to: 127.0.100.62:38247
I20240503 23:25:44.900185 2623 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.100.62:38247 every 8 connection(s)
I20240503 23:25:44.902138 2624 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 23:25:44.907960 2624 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 55ccc811bdc54f7c9d30ad7dd73a4bc7: Bootstrap starting.
I20240503 23:25:44.909782 2624 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 55ccc811bdc54f7c9d30ad7dd73a4bc7: Neither blocks nor log segments found. Creating new log.
I20240503 23:25:44.911924 2624 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 55ccc811bdc54f7c9d30ad7dd73a4bc7: No bootstrap required, opened a new log
I20240503 23:25:44.912846 2624 raft_consensus.cc:348] T 00000000000000000000000000000000 P 55ccc811bdc54f7c9d30ad7dd73a4bc7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "55ccc811bdc54f7c9d30ad7dd73a4bc7" member_type: VOTER }
I20240503 23:25:44.913034 2624 raft_consensus.cc:374] T 00000000000000000000000000000000 P 55ccc811bdc54f7c9d30ad7dd73a4bc7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 23:25:44.913089 2624 raft_consensus.cc:729] T 00000000000000000000000000000000 P 55ccc811bdc54f7c9d30ad7dd73a4bc7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 55ccc811bdc54f7c9d30ad7dd73a4bc7, State: Initialized, Role: FOLLOWER
I20240503 23:25:44.913383 2624 consensus_queue.cc:260] T 00000000000000000000000000000000 P 55ccc811bdc54f7c9d30ad7dd73a4bc7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "55ccc811bdc54f7c9d30ad7dd73a4bc7" member_type: VOTER }
I20240503 23:25:44.913545 2624 raft_consensus.cc:388] T 00000000000000000000000000000000 P 55ccc811bdc54f7c9d30ad7dd73a4bc7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 23:25:44.913642 2624 raft_consensus.cc:482] T 00000000000000000000000000000000 P 55ccc811bdc54f7c9d30ad7dd73a4bc7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 23:25:44.913736 2624 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 55ccc811bdc54f7c9d30ad7dd73a4bc7 [term 0 FOLLOWER]: Advancing to term 1
I20240503 23:25:44.916000 2624 raft_consensus.cc:504] T 00000000000000000000000000000000 P 55ccc811bdc54f7c9d30ad7dd73a4bc7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "55ccc811bdc54f7c9d30ad7dd73a4bc7" member_type: VOTER }
I20240503 23:25:44.916276 2624 leader_election.cc:302] T 00000000000000000000000000000000 P 55ccc811bdc54f7c9d30ad7dd73a4bc7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 55ccc811bdc54f7c9d30ad7dd73a4bc7; no voters:
I20240503 23:25:44.916929 2624 leader_election.cc:288] T 00000000000000000000000000000000 P 55ccc811bdc54f7c9d30ad7dd73a4bc7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240503 23:25:44.917126 2627 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 55ccc811bdc54f7c9d30ad7dd73a4bc7 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 23:25:44.917730 2627 raft_consensus.cc:686] T 00000000000000000000000000000000 P 55ccc811bdc54f7c9d30ad7dd73a4bc7 [term 1 LEADER]: Becoming Leader. State: Replica: 55ccc811bdc54f7c9d30ad7dd73a4bc7, State: Running, Role: LEADER
I20240503 23:25:44.918104 2624 sys_catalog.cc:564] T 00000000000000000000000000000000 P 55ccc811bdc54f7c9d30ad7dd73a4bc7 [sys.catalog]: configured and running, proceeding with master startup.
I20240503 23:25:44.918064 2627 consensus_queue.cc:237] T 00000000000000000000000000000000 P 55ccc811bdc54f7c9d30ad7dd73a4bc7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "55ccc811bdc54f7c9d30ad7dd73a4bc7" member_type: VOTER }
I20240503 23:25:44.919592 2629 sys_catalog.cc:455] T 00000000000000000000000000000000 P 55ccc811bdc54f7c9d30ad7dd73a4bc7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 55ccc811bdc54f7c9d30ad7dd73a4bc7. Latest consensus state: current_term: 1 leader_uuid: "55ccc811bdc54f7c9d30ad7dd73a4bc7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "55ccc811bdc54f7c9d30ad7dd73a4bc7" member_type: VOTER } }
I20240503 23:25:44.919903 2629 sys_catalog.cc:458] T 00000000000000000000000000000000 P 55ccc811bdc54f7c9d30ad7dd73a4bc7 [sys.catalog]: This master's current role is: LEADER
I20240503 23:25:44.920918 2628 sys_catalog.cc:455] T 00000000000000000000000000000000 P 55ccc811bdc54f7c9d30ad7dd73a4bc7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "55ccc811bdc54f7c9d30ad7dd73a4bc7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "55ccc811bdc54f7c9d30ad7dd73a4bc7" member_type: VOTER } }
I20240503 23:25:44.921154 2628 sys_catalog.cc:458] T 00000000000000000000000000000000 P 55ccc811bdc54f7c9d30ad7dd73a4bc7 [sys.catalog]: This master's current role is: LEADER
I20240503 23:25:44.921643 2633 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240503 23:25:44.923326 2633 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240503 23:25:44.927728 400 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20240503 23:25:44.927728 2633 catalog_manager.cc:1347] Generated new cluster ID: 9a562db22f1b4317b892cccce648630b
I20240503 23:25:44.927986 2633 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240503 23:25:44.947445 2633 catalog_manager.cc:1370] Generated new certificate authority record
I20240503 23:25:44.948509 2633 catalog_manager.cc:1504] Loading token signing keys...
I20240503 23:25:44.957113 2633 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 55ccc811bdc54f7c9d30ad7dd73a4bc7: Generated new TSK 0
I20240503 23:25:44.957711 2633 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240503 23:25:44.961483 400 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 23:25:44.965911 2645 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 23:25:44.966872 2646 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 23:25:44.967900 400 server_base.cc:1034] running on GCE node
W20240503 23:25:44.968793 2648 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 23:25:44.969256 400 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240503 23:25:44.969390 400 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240503 23:25:44.969444 400 hybrid_clock.cc:648] HybridClock initialized: now 1714778744969426 us; error 0 us; skew 500 ppm
I20240503 23:25:44.969604 400 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 23:25:44.971305 400 webserver.cc:460] Webserver started at http://127.0.100.1:40941/ using document root <none> and password file <none>
I20240503 23:25:44.971591 400 fs_manager.cc:359] Metadata directory not provided
I20240503 23:25:44.971685 400 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 23:25:44.971814 400 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 23:25:44.972637 400 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZhIiPz/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1714778688451250-400-0/minicluster-data/ts-0-root/instance:
uuid: "10a715f08a5b4c49a61ba47bcc1ca50c"
format_stamp: "Formatted at 2024-05-03 23:25:44 on dist-test-slave-jvbp"
I20240503 23:25:44.975765 400 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.003s
I20240503 23:25:44.977916 2653 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 23:25:44.978420 400 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20240503 23:25:44.978569 400 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZhIiPz/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1714778688451250-400-0/minicluster-data/ts-0-root
uuid: "10a715f08a5b4c49a61ba47bcc1ca50c"
format_stamp: "Formatted at 2024-05-03 23:25:44 on dist-test-slave-jvbp"
I20240503 23:25:44.978722 400 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZhIiPz/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1714778688451250-400-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskZhIiPz/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1714778688451250-400-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskZhIiPz/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1714778688451250-400-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 23:25:44.987946 400 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 23:25:44.988788 400 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 23:25:44.989490 400 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 23:25:44.990608 400 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 23:25:44.990697 400 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 23:25:44.990818 400 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 23:25:44.990888 400 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 23:25:45.007715 400 rpc_server.cc:307] RPC server started. Bound to: 127.0.100.1:40751
I20240503 23:25:45.007736 2715 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.100.1:40751 every 8 connection(s)
I20240503 23:25:45.009850 400 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 23:25:45.014348 2720 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 23:25:45.015240 2721 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 23:25:45.017035 400 server_base.cc:1034] running on GCE node
W20240503 23:25:45.017560 2723 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 23:25:45.018146 400 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240503 23:25:45.018250 400 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240503 23:25:45.018313 400 hybrid_clock.cc:648] HybridClock initialized: now 1714778745018284 us; error 0 us; skew 500 ppm
I20240503 23:25:45.018510 400 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 23:25:45.019001 2716 heartbeater.cc:347] Connected to a master server at 127.0.100.62:38247
I20240503 23:25:45.019186 2716 heartbeater.cc:464] Registering TS with master...
I20240503 23:25:45.019727 2716 heartbeater.cc:511] Master 127.0.100.62:38247 requested a full tablet report, sending...
I20240503 23:25:45.020200 400 webserver.cc:460] Webserver started at http://127.0.100.2:46749/ using document root <none> and password file <none>
I20240503 23:25:45.020481 400 fs_manager.cc:359] Metadata directory not provided
I20240503 23:25:45.020623 400 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 23:25:45.020797 400 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 23:25:45.021075 2589 ts_manager.cc:194] Registered new tserver with Master: 10a715f08a5b4c49a61ba47bcc1ca50c (127.0.100.1:40751)
I20240503 23:25:45.021735 400 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZhIiPz/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1714778688451250-400-0/minicluster-data/ts-1-root/instance:
uuid: "0b457d3401e248839ffa97d3c7ea1944"
format_stamp: "Formatted at 2024-05-03 23:25:45 on dist-test-slave-jvbp"
I20240503 23:25:45.022907 2589 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:36054
I20240503 23:25:45.025106 400 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.002s
I20240503 23:25:45.027041 2728 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 23:25:45.027626 400 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.001s
I20240503 23:25:45.027761 400 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZhIiPz/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1714778688451250-400-0/minicluster-data/ts-1-root
uuid: "0b457d3401e248839ffa97d3c7ea1944"
format_stamp: "Formatted at 2024-05-03 23:25:45 on dist-test-slave-jvbp"
I20240503 23:25:45.027915 400 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZhIiPz/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1714778688451250-400-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskZhIiPz/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1714778688451250-400-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskZhIiPz/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1714778688451250-400-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 23:25:45.041852 400 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 23:25:45.042634 400 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 23:25:45.043331 400 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 23:25:45.044353 400 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 23:25:45.044432 400 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 23:25:45.044523 400 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 23:25:45.044615 400 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 23:25:45.060884 2790 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.100.2:38347 every 8 connection(s)
I20240503 23:25:45.060884 400 rpc_server.cc:307] RPC server started. Bound to: 127.0.100.2:38347
I20240503 23:25:45.063136 400 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 23:25:45.068621 2795 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 23:25:45.069334 2794 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 23:25:45.070608 2797 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 23:25:45.071709 2791 heartbeater.cc:347] Connected to a master server at 127.0.100.62:38247
I20240503 23:25:45.071872 2791 heartbeater.cc:464] Registering TS with master...
I20240503 23:25:45.072331 2791 heartbeater.cc:511] Master 127.0.100.62:38247 requested a full tablet report, sending...
I20240503 23:25:45.073398 400 server_base.cc:1034] running on GCE node
I20240503 23:25:45.073731 2589 ts_manager.cc:194] Registered new tserver with Master: 0b457d3401e248839ffa97d3c7ea1944 (127.0.100.2:38347)
I20240503 23:25:45.073789 400 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240503 23:25:45.073869 400 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240503 23:25:45.073946 400 hybrid_clock.cc:648] HybridClock initialized: now 1714778745073910 us; error 0 us; skew 500 ppm
I20240503 23:25:45.074162 400 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 23:25:45.074983 2589 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:36062
I20240503 23:25:45.076226 400 webserver.cc:460] Webserver started at http://127.0.100.3:36549/ using document root <none> and password file <none>
I20240503 23:25:45.076516 400 fs_manager.cc:359] Metadata directory not provided
I20240503 23:25:45.076674 400 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 23:25:45.076844 400 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 23:25:45.077618 400 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZhIiPz/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1714778688451250-400-0/minicluster-data/ts-2-root/instance:
uuid: "3b04611734434f3bbc0c3ec76e174338"
format_stamp: "Formatted at 2024-05-03 23:25:45 on dist-test-slave-jvbp"
I20240503 23:25:45.080963 400 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.003s
I20240503 23:25:45.083375 2802 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 23:25:45.084041 400 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.001s
I20240503 23:25:45.084354 400 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZhIiPz/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1714778688451250-400-0/minicluster-data/ts-2-root
uuid: "3b04611734434f3bbc0c3ec76e174338"
format_stamp: "Formatted at 2024-05-03 23:25:45 on dist-test-slave-jvbp"
I20240503 23:25:45.084564 400 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZhIiPz/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1714778688451250-400-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-taskZhIiPz/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1714778688451250-400-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-taskZhIiPz/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1714778688451250-400-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 23:25:45.103401 400 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 23:25:45.104224 400 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 23:25:45.106225 400 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 23:25:45.116952 400 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 23:25:45.117077 400 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 23:25:45.117159 400 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 23:25:45.117208 400 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 23:25:45.133723 400 rpc_server.cc:307] RPC server started. Bound to: 127.0.100.3:40671
I20240503 23:25:45.133754 2864 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.100.3:40671 every 8 connection(s)
I20240503 23:25:45.142093 2865 heartbeater.cc:347] Connected to a master server at 127.0.100.62:38247
I20240503 23:25:45.142294 2865 heartbeater.cc:464] Registering TS with master...
I20240503 23:25:45.142762 2865 heartbeater.cc:511] Master 127.0.100.62:38247 requested a full tablet report, sending...
I20240503 23:25:45.143833 2589 ts_manager.cc:194] Registered new tserver with Master: 3b04611734434f3bbc0c3ec76e174338 (127.0.100.3:40671)
I20240503 23:25:45.144137 400 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.009347895s
I20240503 23:25:45.144837 2589 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:36072
I20240503 23:25:46.024718 2716 heartbeater.cc:503] Master 127.0.100.62:38247 was elected leader, sending a full tablet report...
I20240503 23:25:46.077129 2791 heartbeater.cc:503] Master 127.0.100.62:38247 was elected leader, sending a full tablet report...
I20240503 23:25:46.146457 2865 heartbeater.cc:503] Master 127.0.100.62:38247 was elected leader, sending a full tablet report...
I20240503 23:25:46.927623 2643 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240503 23:25:48.928321 2643 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240503 23:25:50.928941 2643 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240503 23:25:52.929553 2643 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240503 23:25:54.930152 2643 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240503 23:25:56.930763 2643 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240503 23:25:57.230002 400 tablet_server.cc:178] TabletServer@127.0.100.1:0 shutting down...
I20240503 23:25:57.246246 400 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240503 23:25:57.261754 400 tablet_server.cc:194] TabletServer@127.0.100.1:0 shutdown complete.
I20240503 23:25:57.265251 400 tablet_server.cc:178] TabletServer@127.0.100.2:0 shutting down...
I20240503 23:25:57.275136 400 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240503 23:25:57.288774 400 tablet_server.cc:194] TabletServer@127.0.100.2:0 shutdown complete.
I20240503 23:25:57.292160 400 tablet_server.cc:178] TabletServer@127.0.100.3:0 shutting down...
I20240503 23:25:57.301571 400 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240503 23:25:57.315259 400 tablet_server.cc:194] TabletServer@127.0.100.3:0 shutdown complete.
I20240503 23:25:57.318950 400 master.cc:537] Master@127.0.100.62:38247 shutting down...
I20240503 23:25:57.330273 400 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 55ccc811bdc54f7c9d30ad7dd73a4bc7 [term 1 LEADER]: Raft consensus shutting down.
I20240503 23:25:57.330612 400 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 55ccc811bdc54f7c9d30ad7dd73a4bc7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 23:25:57.330725 400 tablet_replica.cc:330] T 00000000000000000000000000000000 P 55ccc811bdc54f7c9d30ad7dd73a4bc7: stopping tablet replica
I20240503 23:25:57.345098 400 master.cc:552] Master@127.0.100.62:38247 shutdown complete.
[ OK ] AutoRebalancerTest.NoReplicaMovesIfNoTablets (12515 ms)
[ RUN ] AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne
I20240503 23:25:57.357029 400 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.0.100.62:37217
I20240503 23:25:57.357692 400 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 23:25:57.361738 2869 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 23:25:57.362634 2870 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 23:25:57.363247 2872 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 23:25:57.364122 400 server_base.cc:1034] running on GCE node
I20240503 23:25:57.364436 400 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240503 23:25:57.364512 400 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240503 23:25:57.364603 400 hybrid_clock.cc:648] HybridClock initialized: now 1714778757364580 us; error 0 us; skew 500 ppm
I20240503 23:25:57.364794 400 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 23:25:57.366650 400 webserver.cc:460] Webserver started at http://127.0.100.62:43489/ using document root <none> and password file <none>
I20240503 23:25:57.366886 400 fs_manager.cc:359] Metadata directory not provided
I20240503 23:25:57.366967 400 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 23:25:57.367100 400 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 23:25:57.367789 400 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZhIiPz/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1714778688451250-400-0/minicluster-data/master-0-root/instance:
uuid: "d4d0f1fd0e874f0e94701ca47953ac4e"
format_stamp: "Formatted at 2024-05-03 23:25:57 on dist-test-slave-jvbp"
I20240503 23:25:57.371541 400 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.004s
I20240503 23:25:57.374042 2877 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 23:25:57.374579 400 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20240503 23:25:57.374698 400 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZhIiPz/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1714778688451250-400-0/minicluster-data/master-0-root
uuid: "d4d0f1fd0e874f0e94701ca47953ac4e"
format_stamp: "Formatted at 2024-05-03 23:25:57 on dist-test-slave-jvbp"
I20240503 23:25:57.374855 400 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZhIiPz/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1714778688451250-400-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskZhIiPz/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1714778688451250-400-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskZhIiPz/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1714778688451250-400-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 23:25:57.399047 400 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 23:25:57.399780 400 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 23:25:57.416147 400 rpc_server.cc:307] RPC server started. Bound to: 127.0.100.62:37217
I20240503 23:25:57.416153 2928 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.100.62:37217 every 8 connection(s)
I20240503 23:25:57.417837 2929 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 23:25:57.422995 2929 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d4d0f1fd0e874f0e94701ca47953ac4e: Bootstrap starting.
I20240503 23:25:57.424705 2929 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d4d0f1fd0e874f0e94701ca47953ac4e: Neither blocks nor log segments found. Creating new log.
I20240503 23:25:57.426795 2929 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d4d0f1fd0e874f0e94701ca47953ac4e: No bootstrap required, opened a new log
I20240503 23:25:57.427598 2929 raft_consensus.cc:348] T 00000000000000000000000000000000 P d4d0f1fd0e874f0e94701ca47953ac4e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d4d0f1fd0e874f0e94701ca47953ac4e" member_type: VOTER }
I20240503 23:25:57.427791 2929 raft_consensus.cc:374] T 00000000000000000000000000000000 P d4d0f1fd0e874f0e94701ca47953ac4e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 23:25:57.427915 2929 raft_consensus.cc:729] T 00000000000000000000000000000000 P d4d0f1fd0e874f0e94701ca47953ac4e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d4d0f1fd0e874f0e94701ca47953ac4e, State: Initialized, Role: FOLLOWER
I20240503 23:25:57.428174 2929 consensus_queue.cc:260] T 00000000000000000000000000000000 P d4d0f1fd0e874f0e94701ca47953ac4e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d4d0f1fd0e874f0e94701ca47953ac4e" member_type: VOTER }
I20240503 23:25:57.428323 2929 raft_consensus.cc:388] T 00000000000000000000000000000000 P d4d0f1fd0e874f0e94701ca47953ac4e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 23:25:57.428407 2929 raft_consensus.cc:482] T 00000000000000000000000000000000 P d4d0f1fd0e874f0e94701ca47953ac4e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 23:25:57.428531 2929 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d4d0f1fd0e874f0e94701ca47953ac4e [term 0 FOLLOWER]: Advancing to term 1
I20240503 23:25:57.430783 2929 raft_consensus.cc:504] T 00000000000000000000000000000000 P d4d0f1fd0e874f0e94701ca47953ac4e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d4d0f1fd0e874f0e94701ca47953ac4e" member_type: VOTER }
I20240503 23:25:57.431085 2929 leader_election.cc:302] T 00000000000000000000000000000000 P d4d0f1fd0e874f0e94701ca47953ac4e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d4d0f1fd0e874f0e94701ca47953ac4e; no voters:
I20240503 23:25:57.431633 2929 leader_election.cc:288] T 00000000000000000000000000000000 P d4d0f1fd0e874f0e94701ca47953ac4e [CANDIDATE]: Term 1 election: Requested vote from peers
I20240503 23:25:57.431784 2932 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d4d0f1fd0e874f0e94701ca47953ac4e [term 1 FOLLOWER]: Leader election won for term 1
I20240503 23:25:57.432422 2932 raft_consensus.cc:686] T 00000000000000000000000000000000 P d4d0f1fd0e874f0e94701ca47953ac4e [term 1 LEADER]: Becoming Leader. State: Replica: d4d0f1fd0e874f0e94701ca47953ac4e, State: Running, Role: LEADER
I20240503 23:25:57.432693 2929 sys_catalog.cc:564] T 00000000000000000000000000000000 P d4d0f1fd0e874f0e94701ca47953ac4e [sys.catalog]: configured and running, proceeding with master startup.
I20240503 23:25:57.432762 2932 consensus_queue.cc:237] T 00000000000000000000000000000000 P d4d0f1fd0e874f0e94701ca47953ac4e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d4d0f1fd0e874f0e94701ca47953ac4e" member_type: VOTER }
I20240503 23:25:57.433898 2934 sys_catalog.cc:455] T 00000000000000000000000000000000 P d4d0f1fd0e874f0e94701ca47953ac4e [sys.catalog]: SysCatalogTable state changed. Reason: New leader d4d0f1fd0e874f0e94701ca47953ac4e. Latest consensus state: current_term: 1 leader_uuid: "d4d0f1fd0e874f0e94701ca47953ac4e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d4d0f1fd0e874f0e94701ca47953ac4e" member_type: VOTER } }
I20240503 23:25:57.433852 2933 sys_catalog.cc:455] T 00000000000000000000000000000000 P d4d0f1fd0e874f0e94701ca47953ac4e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d4d0f1fd0e874f0e94701ca47953ac4e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d4d0f1fd0e874f0e94701ca47953ac4e" member_type: VOTER } }
I20240503 23:25:57.434207 2933 sys_catalog.cc:458] T 00000000000000000000000000000000 P d4d0f1fd0e874f0e94701ca47953ac4e [sys.catalog]: This master's current role is: LEADER
I20240503 23:25:57.434206 2934 sys_catalog.cc:458] T 00000000000000000000000000000000 P d4d0f1fd0e874f0e94701ca47953ac4e [sys.catalog]: This master's current role is: LEADER
I20240503 23:25:57.435859 2937 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240503 23:25:57.437667 2937 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240503 23:25:57.439888 400 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20240503 23:25:57.441442 2937 catalog_manager.cc:1347] Generated new cluster ID: c61faa6726ba4b04b8fa2d2fc084b74f
I20240503 23:25:57.441586 2937 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240503 23:25:57.466841 2937 catalog_manager.cc:1370] Generated new certificate authority record
I20240503 23:25:57.467891 2937 catalog_manager.cc:1504] Loading token signing keys...
I20240503 23:25:57.477443 2937 catalog_manager.cc:5857] T 00000000000000000000000000000000 P d4d0f1fd0e874f0e94701ca47953ac4e: Generated new TSK 0
I20240503 23:25:57.477846 2937 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240503 23:25:57.505811 400 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 23:25:57.509907 2950 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 23:25:57.510752 2951 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 23:25:57.511654 2953 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 23:25:57.512531 400 server_base.cc:1034] running on GCE node
I20240503 23:25:57.512916 400 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240503 23:25:57.513015 400 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240503 23:25:57.513063 400 hybrid_clock.cc:648] HybridClock initialized: now 1714778757513046 us; error 0 us; skew 500 ppm
I20240503 23:25:57.513229 400 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 23:25:57.514905 400 webserver.cc:460] Webserver started at http://127.0.100.1:45111/ using document root <none> and password file <none>
I20240503 23:25:57.515177 400 fs_manager.cc:359] Metadata directory not provided
I20240503 23:25:57.515264 400 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 23:25:57.515390 400 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 23:25:57.516150 400 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZhIiPz/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1714778688451250-400-0/minicluster-data/ts-0-root/instance:
uuid: "e1476c60208e4a89819675f822041127"
format_stamp: "Formatted at 2024-05-03 23:25:57 on dist-test-slave-jvbp"
I20240503 23:25:57.519219 400 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20240503 23:25:57.521323 2958 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 23:25:57.521802 400 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.001s
I20240503 23:25:57.521953 400 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZhIiPz/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1714778688451250-400-0/minicluster-data/ts-0-root
uuid: "e1476c60208e4a89819675f822041127"
format_stamp: "Formatted at 2024-05-03 23:25:57 on dist-test-slave-jvbp"
I20240503 23:25:57.522151 400 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZhIiPz/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1714778688451250-400-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskZhIiPz/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1714778688451250-400-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskZhIiPz/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1714778688451250-400-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 23:25:57.562379 400 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 23:25:57.563194 400 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 23:25:57.563982 400 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 23:25:57.565433 400 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 23:25:57.565526 400 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 23:25:57.565654 400 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 23:25:57.565714 400 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 23:25:57.583472 400 rpc_server.cc:307] RPC server started. Bound to: 127.0.100.1:38213
I20240503 23:25:57.583508 3020 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.100.1:38213 every 8 connection(s)
I20240503 23:25:57.585669 400 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 23:25:57.590385 3025 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 23:25:57.591607 3026 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 23:25:57.594482 3021 heartbeater.cc:347] Connected to a master server at 127.0.100.62:37217
W20240503 23:25:57.594707 3028 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 23:25:57.594727 3021 heartbeater.cc:464] Registering TS with master...
I20240503 23:25:57.595283 3021 heartbeater.cc:511] Master 127.0.100.62:37217 requested a full tablet report, sending...
I20240503 23:25:57.595530 400 server_base.cc:1034] running on GCE node
I20240503 23:25:57.595875 400 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240503 23:25:57.596007 400 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240503 23:25:57.596077 400 hybrid_clock.cc:648] HybridClock initialized: now 1714778757596049 us; error 0 us; skew 500 ppm
I20240503 23:25:57.596293 400 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 23:25:57.596710 2894 ts_manager.cc:194] Registered new tserver with Master: e1476c60208e4a89819675f822041127 (127.0.100.1:38213)
I20240503 23:25:57.598594 400 webserver.cc:460] Webserver started at http://127.0.100.2:35537/ using document root <none> and password file <none>
I20240503 23:25:57.598832 2894 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:56252
I20240503 23:25:57.598969 400 fs_manager.cc:359] Metadata directory not provided
I20240503 23:25:57.599082 400 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 23:25:57.599274 400 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 23:25:57.600260 400 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZhIiPz/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1714778688451250-400-0/minicluster-data/ts-1-root/instance:
uuid: "6ebc119b32624bd5ba9eef9d5b007445"
format_stamp: "Formatted at 2024-05-03 23:25:57 on dist-test-slave-jvbp"
I20240503 23:25:57.603502 400 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.002s
I20240503 23:25:57.605690 3033 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 23:25:57.606401 400 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20240503 23:25:57.606621 400 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZhIiPz/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1714778688451250-400-0/minicluster-data/ts-1-root
uuid: "6ebc119b32624bd5ba9eef9d5b007445"
format_stamp: "Formatted at 2024-05-03 23:25:57 on dist-test-slave-jvbp"
I20240503 23:25:57.606774 400 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZhIiPz/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1714778688451250-400-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskZhIiPz/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1714778688451250-400-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskZhIiPz/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1714778688451250-400-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 23:25:57.636006 400 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 23:25:57.636910 400 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 23:25:57.637653 400 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 23:25:57.638739 400 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 23:25:57.638824 400 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 23:25:57.638959 400 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 23:25:57.639019 400 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 23:25:57.656097 400 rpc_server.cc:307] RPC server started. Bound to: 127.0.100.2:33559
I20240503 23:25:57.656127 3095 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.100.2:33559 every 8 connection(s)
I20240503 23:25:57.658069 400 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 23:25:57.663282 3099 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 23:25:57.663735 3100 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 23:25:57.664510 3102 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 23:25:57.665454 400 server_base.cc:1034] running on GCE node
I20240503 23:25:57.665866 400 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240503 23:25:57.665967 400 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240503 23:25:57.666034 400 hybrid_clock.cc:648] HybridClock initialized: now 1714778757666002 us; error 0 us; skew 500 ppm
I20240503 23:25:57.668310 3096 heartbeater.cc:347] Connected to a master server at 127.0.100.62:37217
I20240503 23:25:57.668464 3096 heartbeater.cc:464] Registering TS with master...
I20240503 23:25:57.668911 3096 heartbeater.cc:511] Master 127.0.100.62:37217 requested a full tablet report, sending...
I20240503 23:25:57.670142 2894 ts_manager.cc:194] Registered new tserver with Master: 6ebc119b32624bd5ba9eef9d5b007445 (127.0.100.2:33559)
I20240503 23:25:57.671124 2894 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:56268
I20240503 23:25:57.680274 400 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 23:25:57.682111 400 webserver.cc:460] Webserver started at http://127.0.100.3:40881/ using document root <none> and password file <none>
I20240503 23:25:57.682387 400 fs_manager.cc:359] Metadata directory not provided
I20240503 23:25:57.682467 400 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 23:25:57.682602 400 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 23:25:57.683293 400 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZhIiPz/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1714778688451250-400-0/minicluster-data/ts-2-root/instance:
uuid: "f061bab181e9448d9b7018f9f0698123"
format_stamp: "Formatted at 2024-05-03 23:25:57 on dist-test-slave-jvbp"
I20240503 23:25:57.686429 400 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20240503 23:25:57.688484 3107 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 23:25:57.688895 400 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.001s
I20240503 23:25:57.689010 400 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZhIiPz/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1714778688451250-400-0/minicluster-data/ts-2-root
uuid: "f061bab181e9448d9b7018f9f0698123"
format_stamp: "Formatted at 2024-05-03 23:25:57 on dist-test-slave-jvbp"
I20240503 23:25:57.689136 400 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZhIiPz/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1714778688451250-400-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-taskZhIiPz/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1714778688451250-400-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-taskZhIiPz/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1714778688451250-400-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 23:25:57.709183 400 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 23:25:57.709774 400 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 23:25:57.710371 400 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 23:25:57.711334 400 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 23:25:57.711395 400 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 23:25:57.711481 400 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 23:25:57.711519 400 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 23:25:57.725234 400 rpc_server.cc:307] RPC server started. Bound to: 127.0.100.3:37057
I20240503 23:25:57.725253 3169 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.100.3:37057 every 8 connection(s)
I20240503 23:25:57.727128 400 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 23:25:57.731155 3173 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 23:25:57.731966 3174 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 23:25:57.732781 3176 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 23:25:57.733618 400 server_base.cc:1034] running on GCE node
I20240503 23:25:57.734160 400 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240503 23:25:57.734243 400 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240503 23:25:57.734278 400 hybrid_clock.cc:648] HybridClock initialized: now 1714778757734273 us; error 0 us; skew 500 ppm
I20240503 23:25:57.734469 400 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 23:25:57.734575 3170 heartbeater.cc:347] Connected to a master server at 127.0.100.62:37217
I20240503 23:25:57.734737 3170 heartbeater.cc:464] Registering TS with master...
I20240503 23:25:57.735154 3170 heartbeater.cc:511] Master 127.0.100.62:37217 requested a full tablet report, sending...
I20240503 23:25:57.736346 2894 ts_manager.cc:194] Registered new tserver with Master: f061bab181e9448d9b7018f9f0698123 (127.0.100.3:37057)
I20240503 23:25:57.736467 400 webserver.cc:460] Webserver started at http://127.0.100.4:39433/ using document root <none> and password file <none>
I20240503 23:25:57.736744 400 fs_manager.cc:359] Metadata directory not provided
I20240503 23:25:57.736835 400 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 23:25:57.736981 400 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 23:25:57.737576 2894 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:56284
I20240503 23:25:57.737864 400 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZhIiPz/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1714778688451250-400-0/minicluster-data/ts-3-root/instance:
uuid: "a1e6f6a4b5c0437eb3fa2ba93f050964"
format_stamp: "Formatted at 2024-05-03 23:25:57 on dist-test-slave-jvbp"
I20240503 23:25:57.740988 400 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20240503 23:25:57.743043 3181 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 23:25:57.743528 400 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.001s
I20240503 23:25:57.743669 400 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZhIiPz/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1714778688451250-400-0/minicluster-data/ts-3-root
uuid: "a1e6f6a4b5c0437eb3fa2ba93f050964"
format_stamp: "Formatted at 2024-05-03 23:25:57 on dist-test-slave-jvbp"
I20240503 23:25:57.743798 400 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZhIiPz/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1714778688451250-400-0/minicluster-data/ts-3-root
metadata directory: /tmp/dist-test-taskZhIiPz/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1714778688451250-400-0/minicluster-data/ts-3-root
1 data directories: /tmp/dist-test-taskZhIiPz/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1714778688451250-400-0/minicluster-data/ts-3-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 23:25:57.758847 400 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 23:25:57.759552 400 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 23:25:57.760198 400 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 23:25:57.761221 400 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 23:25:57.761288 400 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 23:25:57.761387 400 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 23:25:57.761427 400 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 23:25:57.775972 400 rpc_server.cc:307] RPC server started. Bound to: 127.0.100.4:36287
I20240503 23:25:57.776041 3243 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.100.4:36287 every 8 connection(s)
I20240503 23:25:57.785107 3244 heartbeater.cc:347] Connected to a master server at 127.0.100.62:37217
I20240503 23:25:57.785317 3244 heartbeater.cc:464] Registering TS with master...
I20240503 23:25:57.785782 3244 heartbeater.cc:511] Master 127.0.100.62:37217 requested a full tablet report, sending...
I20240503 23:25:57.786868 2894 ts_manager.cc:194] Registered new tserver with Master: a1e6f6a4b5c0437eb3fa2ba93f050964 (127.0.100.4:36287)
I20240503 23:25:57.787720 400 internal_mini_cluster.cc:371] 4 TS(s) registered with all masters after 0.010662028s
I20240503 23:25:57.788049 2894 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:56288
I20240503 23:25:58.601207 3021 heartbeater.cc:503] Master 127.0.100.62:37217 was elected leader, sending a full tablet report...
I20240503 23:25:58.672772 3096 heartbeater.cc:503] Master 127.0.100.62:37217 was elected leader, sending a full tablet report...
I20240503 23:25:58.739365 3170 heartbeater.cc:503] Master 127.0.100.62:37217 was elected leader, sending a full tablet report...
I20240503 23:25:58.790037 3244 heartbeater.cc:503] Master 127.0.100.62:37217 was elected leader, sending a full tablet report...
I20240503 23:25:58.826387 400 test_util.cc:268] Using random seed: 180996471
I20240503 23:25:58.840823 2894 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56300:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240503 23:25:58.864832 3061 tablet_service.cc:1467] Processing CreateTablet for tablet b3363143c8e248ce9e13e4ba88cb2826 (DEFAULT_TABLE table=test-workload [id=a57b70bb8ef64ac3b29c7d45ef936ecf]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 23:25:58.865501 3061 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b3363143c8e248ce9e13e4ba88cb2826. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 23:25:58.868983 3135 tablet_service.cc:1467] Processing CreateTablet for tablet b3363143c8e248ce9e13e4ba88cb2826 (DEFAULT_TABLE table=test-workload [id=a57b70bb8ef64ac3b29c7d45ef936ecf]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 23:25:58.869485 3135 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b3363143c8e248ce9e13e4ba88cb2826. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 23:25:58.872762 3209 tablet_service.cc:1467] Processing CreateTablet for tablet b3363143c8e248ce9e13e4ba88cb2826 (DEFAULT_TABLE table=test-workload [id=a57b70bb8ef64ac3b29c7d45ef936ecf]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 23:25:58.873421 3209 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b3363143c8e248ce9e13e4ba88cb2826. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 23:25:58.874926 3265 tablet_bootstrap.cc:492] T b3363143c8e248ce9e13e4ba88cb2826 P 6ebc119b32624bd5ba9eef9d5b007445: Bootstrap starting.
I20240503 23:25:58.876006 3266 tablet_bootstrap.cc:492] T b3363143c8e248ce9e13e4ba88cb2826 P f061bab181e9448d9b7018f9f0698123: Bootstrap starting.
I20240503 23:25:58.877053 3265 tablet_bootstrap.cc:654] T b3363143c8e248ce9e13e4ba88cb2826 P 6ebc119b32624bd5ba9eef9d5b007445: Neither blocks nor log segments found. Creating new log.
I20240503 23:25:58.877907 3266 tablet_bootstrap.cc:654] T b3363143c8e248ce9e13e4ba88cb2826 P f061bab181e9448d9b7018f9f0698123: Neither blocks nor log segments found. Creating new log.
I20240503 23:25:58.879107 3265 tablet_bootstrap.cc:492] T b3363143c8e248ce9e13e4ba88cb2826 P 6ebc119b32624bd5ba9eef9d5b007445: No bootstrap required, opened a new log
I20240503 23:25:58.879320 3265 ts_tablet_manager.cc:1367] T b3363143c8e248ce9e13e4ba88cb2826 P 6ebc119b32624bd5ba9eef9d5b007445: Time spent bootstrapping tablet: real 0.004s user 0.004s sys 0.000s
I20240503 23:25:58.880194 3266 tablet_bootstrap.cc:492] T b3363143c8e248ce9e13e4ba88cb2826 P f061bab181e9448d9b7018f9f0698123: No bootstrap required, opened a new log
I20240503 23:25:58.880390 3266 ts_tablet_manager.cc:1367] T b3363143c8e248ce9e13e4ba88cb2826 P f061bab181e9448d9b7018f9f0698123: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.004s
I20240503 23:25:58.880281 3265 raft_consensus.cc:348] T b3363143c8e248ce9e13e4ba88cb2826 P 6ebc119b32624bd5ba9eef9d5b007445 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ebc119b32624bd5ba9eef9d5b007445" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 33559 } } peers { permanent_uuid: "f061bab181e9448d9b7018f9f0698123" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 37057 } } peers { permanent_uuid: "a1e6f6a4b5c0437eb3fa2ba93f050964" member_type: VOTER last_known_addr { host: "127.0.100.4" port: 36287 } }
I20240503 23:25:58.880767 3265 raft_consensus.cc:374] T b3363143c8e248ce9e13e4ba88cb2826 P 6ebc119b32624bd5ba9eef9d5b007445 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 23:25:58.880885 3265 raft_consensus.cc:729] T b3363143c8e248ce9e13e4ba88cb2826 P 6ebc119b32624bd5ba9eef9d5b007445 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6ebc119b32624bd5ba9eef9d5b007445, State: Initialized, Role: FOLLOWER
I20240503 23:25:58.881093 3269 tablet_bootstrap.cc:492] T b3363143c8e248ce9e13e4ba88cb2826 P a1e6f6a4b5c0437eb3fa2ba93f050964: Bootstrap starting.
I20240503 23:25:58.881163 3265 consensus_queue.cc:260] T b3363143c8e248ce9e13e4ba88cb2826 P 6ebc119b32624bd5ba9eef9d5b007445 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ebc119b32624bd5ba9eef9d5b007445" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 33559 } } peers { permanent_uuid: "f061bab181e9448d9b7018f9f0698123" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 37057 } } peers { permanent_uuid: "a1e6f6a4b5c0437eb3fa2ba93f050964" member_type: VOTER last_known_addr { host: "127.0.100.4" port: 36287 } }
I20240503 23:25:58.881705 3266 raft_consensus.cc:348] T b3363143c8e248ce9e13e4ba88cb2826 P f061bab181e9448d9b7018f9f0698123 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ebc119b32624bd5ba9eef9d5b007445" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 33559 } } peers { permanent_uuid: "f061bab181e9448d9b7018f9f0698123" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 37057 } } peers { permanent_uuid: "a1e6f6a4b5c0437eb3fa2ba93f050964" member_type: VOTER last_known_addr { host: "127.0.100.4" port: 36287 } }
I20240503 23:25:58.881951 3266 raft_consensus.cc:374] T b3363143c8e248ce9e13e4ba88cb2826 P f061bab181e9448d9b7018f9f0698123 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 23:25:58.882046 3265 ts_tablet_manager.cc:1398] T b3363143c8e248ce9e13e4ba88cb2826 P 6ebc119b32624bd5ba9eef9d5b007445: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240503 23:25:58.882066 3266 raft_consensus.cc:729] T b3363143c8e248ce9e13e4ba88cb2826 P f061bab181e9448d9b7018f9f0698123 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f061bab181e9448d9b7018f9f0698123, State: Initialized, Role: FOLLOWER
I20240503 23:25:58.882473 3266 consensus_queue.cc:260] T b3363143c8e248ce9e13e4ba88cb2826 P f061bab181e9448d9b7018f9f0698123 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ebc119b32624bd5ba9eef9d5b007445" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 33559 } } peers { permanent_uuid: "f061bab181e9448d9b7018f9f0698123" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 37057 } } peers { permanent_uuid: "a1e6f6a4b5c0437eb3fa2ba93f050964" member_type: VOTER last_known_addr { host: "127.0.100.4" port: 36287 } }
I20240503 23:25:58.882989 3269 tablet_bootstrap.cc:654] T b3363143c8e248ce9e13e4ba88cb2826 P a1e6f6a4b5c0437eb3fa2ba93f050964: Neither blocks nor log segments found. Creating new log.
I20240503 23:25:58.883865 3266 ts_tablet_manager.cc:1398] T b3363143c8e248ce9e13e4ba88cb2826 P f061bab181e9448d9b7018f9f0698123: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20240503 23:25:58.885651 3269 tablet_bootstrap.cc:492] T b3363143c8e248ce9e13e4ba88cb2826 P a1e6f6a4b5c0437eb3fa2ba93f050964: No bootstrap required, opened a new log
I20240503 23:25:58.885802 3269 ts_tablet_manager.cc:1367] T b3363143c8e248ce9e13e4ba88cb2826 P a1e6f6a4b5c0437eb3fa2ba93f050964: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20240503 23:25:58.886675 3269 raft_consensus.cc:348] T b3363143c8e248ce9e13e4ba88cb2826 P a1e6f6a4b5c0437eb3fa2ba93f050964 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ebc119b32624bd5ba9eef9d5b007445" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 33559 } } peers { permanent_uuid: "f061bab181e9448d9b7018f9f0698123" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 37057 } } peers { permanent_uuid: "a1e6f6a4b5c0437eb3fa2ba93f050964" member_type: VOTER last_known_addr { host: "127.0.100.4" port: 36287 } }
I20240503 23:25:58.886852 3269 raft_consensus.cc:374] T b3363143c8e248ce9e13e4ba88cb2826 P a1e6f6a4b5c0437eb3fa2ba93f050964 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 23:25:58.886900 3269 raft_consensus.cc:729] T b3363143c8e248ce9e13e4ba88cb2826 P a1e6f6a4b5c0437eb3fa2ba93f050964 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a1e6f6a4b5c0437eb3fa2ba93f050964, State: Initialized, Role: FOLLOWER
I20240503 23:25:58.887162 3269 consensus_queue.cc:260] T b3363143c8e248ce9e13e4ba88cb2826 P a1e6f6a4b5c0437eb3fa2ba93f050964 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ebc119b32624bd5ba9eef9d5b007445" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 33559 } } peers { permanent_uuid: "f061bab181e9448d9b7018f9f0698123" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 37057 } } peers { permanent_uuid: "a1e6f6a4b5c0437eb3fa2ba93f050964" member_type: VOTER last_known_addr { host: "127.0.100.4" port: 36287 } }
I20240503 23:25:58.888043 3269 ts_tablet_manager.cc:1398] T b3363143c8e248ce9e13e4ba88cb2826 P a1e6f6a4b5c0437eb3fa2ba93f050964: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
W20240503 23:25:58.908406 3097 tablet.cc:2369] T b3363143c8e248ce9e13e4ba88cb2826 P 6ebc119b32624bd5ba9eef9d5b007445: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240503 23:25:59.014410 3270 raft_consensus.cc:482] T b3363143c8e248ce9e13e4ba88cb2826 P 6ebc119b32624bd5ba9eef9d5b007445 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 23:25:59.014653 3270 raft_consensus.cc:504] T b3363143c8e248ce9e13e4ba88cb2826 P 6ebc119b32624bd5ba9eef9d5b007445 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ebc119b32624bd5ba9eef9d5b007445" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 33559 } } peers { permanent_uuid: "f061bab181e9448d9b7018f9f0698123" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 37057 } } peers { permanent_uuid: "a1e6f6a4b5c0437eb3fa2ba93f050964" member_type: VOTER last_known_addr { host: "127.0.100.4" port: 36287 } }
I20240503 23:25:59.015545 3270 leader_election.cc:288] T b3363143c8e248ce9e13e4ba88cb2826 P 6ebc119b32624bd5ba9eef9d5b007445 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f061bab181e9448d9b7018f9f0698123 (127.0.100.3:37057), a1e6f6a4b5c0437eb3fa2ba93f050964 (127.0.100.4:36287)
I20240503 23:25:59.022513 3145 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3363143c8e248ce9e13e4ba88cb2826" candidate_uuid: "6ebc119b32624bd5ba9eef9d5b007445" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f061bab181e9448d9b7018f9f0698123" is_pre_election: true
I20240503 23:25:59.022529 3219 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3363143c8e248ce9e13e4ba88cb2826" candidate_uuid: "6ebc119b32624bd5ba9eef9d5b007445" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a1e6f6a4b5c0437eb3fa2ba93f050964" is_pre_election: true
I20240503 23:25:59.022890 3145 raft_consensus.cc:2454] T b3363143c8e248ce9e13e4ba88cb2826 P f061bab181e9448d9b7018f9f0698123 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6ebc119b32624bd5ba9eef9d5b007445 in term 0.
I20240503 23:25:59.022891 3219 raft_consensus.cc:2454] T b3363143c8e248ce9e13e4ba88cb2826 P a1e6f6a4b5c0437eb3fa2ba93f050964 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6ebc119b32624bd5ba9eef9d5b007445 in term 0.
I20240503 23:25:59.023381 3035 leader_election.cc:302] T b3363143c8e248ce9e13e4ba88cb2826 P 6ebc119b32624bd5ba9eef9d5b007445 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6ebc119b32624bd5ba9eef9d5b007445, a1e6f6a4b5c0437eb3fa2ba93f050964; no voters:
I20240503 23:25:59.023669 3270 raft_consensus.cc:2781] T b3363143c8e248ce9e13e4ba88cb2826 P 6ebc119b32624bd5ba9eef9d5b007445 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 23:25:59.023754 3270 raft_consensus.cc:482] T b3363143c8e248ce9e13e4ba88cb2826 P 6ebc119b32624bd5ba9eef9d5b007445 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 23:25:59.023830 3270 raft_consensus.cc:3037] T b3363143c8e248ce9e13e4ba88cb2826 P 6ebc119b32624bd5ba9eef9d5b007445 [term 0 FOLLOWER]: Advancing to term 1
I20240503 23:25:59.025978 3270 raft_consensus.cc:504] T b3363143c8e248ce9e13e4ba88cb2826 P 6ebc119b32624bd5ba9eef9d5b007445 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ebc119b32624bd5ba9eef9d5b007445" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 33559 } } peers { permanent_uuid: "f061bab181e9448d9b7018f9f0698123" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 37057 } } peers { permanent_uuid: "a1e6f6a4b5c0437eb3fa2ba93f050964" member_type: VOTER last_known_addr { host: "127.0.100.4" port: 36287 } }
I20240503 23:25:59.026660 3270 leader_election.cc:288] T b3363143c8e248ce9e13e4ba88cb2826 P 6ebc119b32624bd5ba9eef9d5b007445 [CANDIDATE]: Term 1 election: Requested vote from peers f061bab181e9448d9b7018f9f0698123 (127.0.100.3:37057), a1e6f6a4b5c0437eb3fa2ba93f050964 (127.0.100.4:36287)
I20240503 23:25:59.027246 3145 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3363143c8e248ce9e13e4ba88cb2826" candidate_uuid: "6ebc119b32624bd5ba9eef9d5b007445" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f061bab181e9448d9b7018f9f0698123"
I20240503 23:25:59.027258 3219 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3363143c8e248ce9e13e4ba88cb2826" candidate_uuid: "6ebc119b32624bd5ba9eef9d5b007445" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a1e6f6a4b5c0437eb3fa2ba93f050964"
I20240503 23:25:59.027437 3145 raft_consensus.cc:3037] T b3363143c8e248ce9e13e4ba88cb2826 P f061bab181e9448d9b7018f9f0698123 [term 0 FOLLOWER]: Advancing to term 1
I20240503 23:25:59.027473 3219 raft_consensus.cc:3037] T b3363143c8e248ce9e13e4ba88cb2826 P a1e6f6a4b5c0437eb3fa2ba93f050964 [term 0 FOLLOWER]: Advancing to term 1
I20240503 23:25:59.029660 3219 raft_consensus.cc:2454] T b3363143c8e248ce9e13e4ba88cb2826 P a1e6f6a4b5c0437eb3fa2ba93f050964 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6ebc119b32624bd5ba9eef9d5b007445 in term 1.
I20240503 23:25:59.029660 3145 raft_consensus.cc:2454] T b3363143c8e248ce9e13e4ba88cb2826 P f061bab181e9448d9b7018f9f0698123 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6ebc119b32624bd5ba9eef9d5b007445 in term 1.
I20240503 23:25:59.030181 3035 leader_election.cc:302] T b3363143c8e248ce9e13e4ba88cb2826 P 6ebc119b32624bd5ba9eef9d5b007445 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6ebc119b32624bd5ba9eef9d5b007445, a1e6f6a4b5c0437eb3fa2ba93f050964; no voters:
I20240503 23:25:59.030453 3270 raft_consensus.cc:2781] T b3363143c8e248ce9e13e4ba88cb2826 P 6ebc119b32624bd5ba9eef9d5b007445 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 23:25:59.030913 3270 raft_consensus.cc:686] T b3363143c8e248ce9e13e4ba88cb2826 P 6ebc119b32624bd5ba9eef9d5b007445 [term 1 LEADER]: Becoming Leader. State: Replica: 6ebc119b32624bd5ba9eef9d5b007445, State: Running, Role: LEADER
I20240503 23:25:59.031198 3270 consensus_queue.cc:237] T b3363143c8e248ce9e13e4ba88cb2826 P 6ebc119b32624bd5ba9eef9d5b007445 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ebc119b32624bd5ba9eef9d5b007445" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 33559 } } peers { permanent_uuid: "f061bab181e9448d9b7018f9f0698123" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 37057 } } peers { permanent_uuid: "a1e6f6a4b5c0437eb3fa2ba93f050964" member_type: VOTER last_known_addr { host: "127.0.100.4" port: 36287 } }
I20240503 23:25:59.034137 2893 catalog_manager.cc:5484] T b3363143c8e248ce9e13e4ba88cb2826 P 6ebc119b32624bd5ba9eef9d5b007445 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6ebc119b32624bd5ba9eef9d5b007445 (127.0.100.2). New cstate: current_term: 1 leader_uuid: "6ebc119b32624bd5ba9eef9d5b007445" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ebc119b32624bd5ba9eef9d5b007445" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 33559 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f061bab181e9448d9b7018f9f0698123" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 37057 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a1e6f6a4b5c0437eb3fa2ba93f050964" member_type: VOTER last_known_addr { host: "127.0.100.4" port: 36287 } health_report { overall_health: UNKNOWN } } }
I20240503 23:25:59.439767 2947 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240503 23:25:59.440335 2947 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240503 23:25:59.440457 2947 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240503 23:25:59.542538 3278 consensus_queue.cc:1035] T b3363143c8e248ce9e13e4ba88cb2826 P 6ebc119b32624bd5ba9eef9d5b007445 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f061bab181e9448d9b7018f9f0698123" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 37057 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 23:25:59.548655 3283 consensus_queue.cc:1035] T b3363143c8e248ce9e13e4ba88cb2826 P 6ebc119b32624bd5ba9eef9d5b007445 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a1e6f6a4b5c0437eb3fa2ba93f050964" member_type: VOTER last_known_addr { host: "127.0.100.4" port: 36287 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 23:26:01.440973 2947 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240503 23:26:01.441573 2947 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240503 23:26:01.441690 2947 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240503 23:26:03.442229 2947 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240503 23:26:03.442828 2947 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240503 23:26:03.442943 2947 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240503 23:26:05.443527 2947 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240503 23:26:05.444149 2947 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240503 23:26:05.444286 2947 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240503 23:26:07.444808 2947 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240503 23:26:07.445567 2947 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240503 23:26:07.445678 2947 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240503 23:26:09.446117 2947 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240503 23:26:09.446705 2947 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240503 23:26:09.446807 2947 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240503 23:26:10.106809 400 tablet_server.cc:178] TabletServer@127.0.100.1:0 shutting down...
I20240503 23:26:10.120283 400 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240503 23:26:10.134871 400 tablet_server.cc:194] TabletServer@127.0.100.1:0 shutdown complete.
I20240503 23:26:10.138505 400 tablet_server.cc:178] TabletServer@127.0.100.2:0 shutting down...
I20240503 23:26:10.148799 400 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240503 23:26:10.149152 400 tablet_replica.cc:330] T b3363143c8e248ce9e13e4ba88cb2826 P 6ebc119b32624bd5ba9eef9d5b007445: stopping tablet replica
I20240503 23:26:10.149354 400 raft_consensus.cc:2229] T b3363143c8e248ce9e13e4ba88cb2826 P 6ebc119b32624bd5ba9eef9d5b007445 [term 1 LEADER]: Raft consensus shutting down.
I20240503 23:26:10.149761 400 raft_consensus.cc:2258] T b3363143c8e248ce9e13e4ba88cb2826 P 6ebc119b32624bd5ba9eef9d5b007445 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 23:26:10.164647 400 tablet_server.cc:194] TabletServer@127.0.100.2:0 shutdown complete.
I20240503 23:26:10.168203 400 tablet_server.cc:178] TabletServer@127.0.100.3:0 shutting down...
I20240503 23:26:10.177572 400 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240503 23:26:10.177881 400 tablet_replica.cc:330] T b3363143c8e248ce9e13e4ba88cb2826 P f061bab181e9448d9b7018f9f0698123: stopping tablet replica
I20240503 23:26:10.178108 400 raft_consensus.cc:2229] T b3363143c8e248ce9e13e4ba88cb2826 P f061bab181e9448d9b7018f9f0698123 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 23:26:10.178334 400 raft_consensus.cc:2258] T b3363143c8e248ce9e13e4ba88cb2826 P f061bab181e9448d9b7018f9f0698123 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 23:26:10.192828 400 tablet_server.cc:194] TabletServer@127.0.100.3:0 shutdown complete.
I20240503 23:26:10.196370 400 tablet_server.cc:178] TabletServer@127.0.100.4:0 shutting down...
I20240503 23:26:10.205973 400 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240503 23:26:10.206303 400 tablet_replica.cc:330] T b3363143c8e248ce9e13e4ba88cb2826 P a1e6f6a4b5c0437eb3fa2ba93f050964: stopping tablet replica
I20240503 23:26:10.206543 400 raft_consensus.cc:2229] T b3363143c8e248ce9e13e4ba88cb2826 P a1e6f6a4b5c0437eb3fa2ba93f050964 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 23:26:10.206784 400 raft_consensus.cc:2258] T b3363143c8e248ce9e13e4ba88cb2826 P a1e6f6a4b5c0437eb3fa2ba93f050964 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 23:26:10.233008 400 tablet_server.cc:194] TabletServer@127.0.100.4:0 shutdown complete.
I20240503 23:26:10.236882 400 master.cc:537] Master@127.0.100.62:37217 shutting down...
I20240503 23:26:10.246068 400 raft_consensus.cc:2229] T 00000000000000000000000000000000 P d4d0f1fd0e874f0e94701ca47953ac4e [term 1 LEADER]: Raft consensus shutting down.
I20240503 23:26:10.246403 400 raft_consensus.cc:2258] T 00000000000000000000000000000000 P d4d0f1fd0e874f0e94701ca47953ac4e [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 23:26:10.246510 400 tablet_replica.cc:330] T 00000000000000000000000000000000 P d4d0f1fd0e874f0e94701ca47953ac4e: stopping tablet replica
I20240503 23:26:10.261139 400 master.cc:552] Master@127.0.100.62:37217 shutdown complete.
[ OK ] AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne (12918 ms)
[ RUN ] AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy
I20240503 23:26:10.275277 400 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.0.100.62:39113
I20240503 23:26:10.276104 400 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 23:26:10.280249 3291 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 23:26:10.281154 3292 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 23:26:10.282049 3294 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 23:26:10.282795 400 server_base.cc:1034] running on GCE node
I20240503 23:26:10.283082 400 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240503 23:26:10.283138 400 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240503 23:26:10.283179 400 hybrid_clock.cc:648] HybridClock initialized: now 1714778770283173 us; error 0 us; skew 500 ppm
I20240503 23:26:10.283354 400 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 23:26:10.290076 400 webserver.cc:460] Webserver started at http://127.0.100.62:39349/ using document root <none> and password file <none>
I20240503 23:26:10.290416 400 fs_manager.cc:359] Metadata directory not provided
I20240503 23:26:10.290491 400 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 23:26:10.290652 400 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 23:26:10.291483 400 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZhIiPz/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1714778688451250-400-0/minicluster-data/master-0-root/instance:
uuid: "a9e74a3a3f174e3e96e71c9f0273165d"
format_stamp: "Formatted at 2024-05-03 23:26:10 on dist-test-slave-jvbp"
I20240503 23:26:10.294637 400 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20240503 23:26:10.296665 3299 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 23:26:10.297122 400 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.001s
I20240503 23:26:10.297217 400 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZhIiPz/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1714778688451250-400-0/minicluster-data/master-0-root
uuid: "a9e74a3a3f174e3e96e71c9f0273165d"
format_stamp: "Formatted at 2024-05-03 23:26:10 on dist-test-slave-jvbp"
I20240503 23:26:10.297336 400 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZhIiPz/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1714778688451250-400-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskZhIiPz/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1714778688451250-400-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskZhIiPz/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1714778688451250-400-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 23:26:10.308281 400 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 23:26:10.308987 400 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 23:26:10.321980 400 rpc_server.cc:307] RPC server started. Bound to: 127.0.100.62:39113
I20240503 23:26:10.321985 3350 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.100.62:39113 every 8 connection(s)
I20240503 23:26:10.323577 3351 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 23:26:10.328321 3351 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a9e74a3a3f174e3e96e71c9f0273165d: Bootstrap starting.
I20240503 23:26:10.329841 3351 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a9e74a3a3f174e3e96e71c9f0273165d: Neither blocks nor log segments found. Creating new log.
I20240503 23:26:10.331632 3351 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a9e74a3a3f174e3e96e71c9f0273165d: No bootstrap required, opened a new log
I20240503 23:26:10.332363 3351 raft_consensus.cc:348] T 00000000000000000000000000000000 P a9e74a3a3f174e3e96e71c9f0273165d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a9e74a3a3f174e3e96e71c9f0273165d" member_type: VOTER }
I20240503 23:26:10.332516 3351 raft_consensus.cc:374] T 00000000000000000000000000000000 P a9e74a3a3f174e3e96e71c9f0273165d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 23:26:10.332610 3351 raft_consensus.cc:729] T 00000000000000000000000000000000 P a9e74a3a3f174e3e96e71c9f0273165d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a9e74a3a3f174e3e96e71c9f0273165d, State: Initialized, Role: FOLLOWER
I20240503 23:26:10.332837 3351 consensus_queue.cc:260] T 00000000000000000000000000000000 P a9e74a3a3f174e3e96e71c9f0273165d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a9e74a3a3f174e3e96e71c9f0273165d" member_type: VOTER }
I20240503 23:26:10.332976 3351 raft_consensus.cc:388] T 00000000000000000000000000000000 P a9e74a3a3f174e3e96e71c9f0273165d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 23:26:10.333041 3351 raft_consensus.cc:482] T 00000000000000000000000000000000 P a9e74a3a3f174e3e96e71c9f0273165d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 23:26:10.333123 3351 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a9e74a3a3f174e3e96e71c9f0273165d [term 0 FOLLOWER]: Advancing to term 1
I20240503 23:26:10.335094 3351 raft_consensus.cc:504] T 00000000000000000000000000000000 P a9e74a3a3f174e3e96e71c9f0273165d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a9e74a3a3f174e3e96e71c9f0273165d" member_type: VOTER }
I20240503 23:26:10.335321 3351 leader_election.cc:302] T 00000000000000000000000000000000 P a9e74a3a3f174e3e96e71c9f0273165d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a9e74a3a3f174e3e96e71c9f0273165d; no voters:
I20240503 23:26:10.335851 3351 leader_election.cc:288] T 00000000000000000000000000000000 P a9e74a3a3f174e3e96e71c9f0273165d [CANDIDATE]: Term 1 election: Requested vote from peers
I20240503 23:26:10.336014 3354 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a9e74a3a3f174e3e96e71c9f0273165d [term 1 FOLLOWER]: Leader election won for term 1
I20240503 23:26:10.336525 3354 raft_consensus.cc:686] T 00000000000000000000000000000000 P a9e74a3a3f174e3e96e71c9f0273165d [term 1 LEADER]: Becoming Leader. State: Replica: a9e74a3a3f174e3e96e71c9f0273165d, State: Running, Role: LEADER
I20240503 23:26:10.336809 3354 consensus_queue.cc:237] T 00000000000000000000000000000000 P a9e74a3a3f174e3e96e71c9f0273165d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a9e74a3a3f174e3e96e71c9f0273165d" member_type: VOTER }
I20240503 23:26:10.336855 3351 sys_catalog.cc:564] T 00000000000000000000000000000000 P a9e74a3a3f174e3e96e71c9f0273165d [sys.catalog]: configured and running, proceeding with master startup.
I20240503 23:26:10.337798 3356 sys_catalog.cc:455] T 00000000000000000000000000000000 P a9e74a3a3f174e3e96e71c9f0273165d [sys.catalog]: SysCatalogTable state changed. Reason: New leader a9e74a3a3f174e3e96e71c9f0273165d. Latest consensus state: current_term: 1 leader_uuid: "a9e74a3a3f174e3e96e71c9f0273165d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a9e74a3a3f174e3e96e71c9f0273165d" member_type: VOTER } }
I20240503 23:26:10.337977 3356 sys_catalog.cc:458] T 00000000000000000000000000000000 P a9e74a3a3f174e3e96e71c9f0273165d [sys.catalog]: This master's current role is: LEADER
I20240503 23:26:10.337746 3355 sys_catalog.cc:455] T 00000000000000000000000000000000 P a9e74a3a3f174e3e96e71c9f0273165d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a9e74a3a3f174e3e96e71c9f0273165d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a9e74a3a3f174e3e96e71c9f0273165d" member_type: VOTER } }
I20240503 23:26:10.338136 3355 sys_catalog.cc:458] T 00000000000000000000000000000000 P a9e74a3a3f174e3e96e71c9f0273165d [sys.catalog]: This master's current role is: LEADER
I20240503 23:26:10.339501 3359 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240503 23:26:10.341884 3359 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240503 23:26:10.342970 400 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20240503 23:26:10.345567 3359 catalog_manager.cc:1347] Generated new cluster ID: 6a00755723b24fbda0b9f347c127290e
I20240503 23:26:10.345656 3359 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240503 23:26:10.355912 3359 catalog_manager.cc:1370] Generated new certificate authority record
I20240503 23:26:10.356848 3359 catalog_manager.cc:1504] Loading token signing keys...
I20240503 23:26:10.374985 3359 catalog_manager.cc:5857] T 00000000000000000000000000000000 P a9e74a3a3f174e3e96e71c9f0273165d: Generated new TSK 0
I20240503 23:26:10.375351 3359 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240503 23:26:10.408380 400 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 23:26:10.412400 3372 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 23:26:10.413324 3373 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 23:26:10.414062 3375 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 23:26:10.414844 400 server_base.cc:1034] running on GCE node
I20240503 23:26:10.415130 400 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240503 23:26:10.415199 400 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240503 23:26:10.415251 400 hybrid_clock.cc:648] HybridClock initialized: now 1714778770415245 us; error 0 us; skew 500 ppm
I20240503 23:26:10.415444 400 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 23:26:10.417028 400 webserver.cc:460] Webserver started at http://127.0.100.1:39655/ using document root <none> and password file <none>
I20240503 23:26:10.417287 400 fs_manager.cc:359] Metadata directory not provided
I20240503 23:26:10.417374 400 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 23:26:10.417520 400 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 23:26:10.418287 400 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZhIiPz/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1714778688451250-400-0/minicluster-data/ts-0-root/instance:
uuid: "01f756dedf2b40a8ab92c0c743c43d73"
format_stamp: "Formatted at 2024-05-03 23:26:10 on dist-test-slave-jvbp"
I20240503 23:26:10.421173 400 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.003s
I20240503 23:26:10.423038 3380 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 23:26:10.423435 400 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.001s
I20240503 23:26:10.423554 400 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZhIiPz/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1714778688451250-400-0/minicluster-data/ts-0-root
uuid: "01f756dedf2b40a8ab92c0c743c43d73"
format_stamp: "Formatted at 2024-05-03 23:26:10 on dist-test-slave-jvbp"
I20240503 23:26:10.423673 400 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZhIiPz/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1714778688451250-400-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskZhIiPz/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1714778688451250-400-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskZhIiPz/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1714778688451250-400-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 23:26:10.442296 400 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 23:26:10.443329 400 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 23:26:10.444015 400 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 23:26:10.445137 400 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 23:26:10.445204 400 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 23:26:10.445289 400 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 23:26:10.445346 400 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 23:26:10.460110 400 rpc_server.cc:307] RPC server started. Bound to: 127.0.100.1:39285
I20240503 23:26:10.460131 3442 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.100.1:39285 every 8 connection(s)
I20240503 23:26:10.461903 400 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 23:26:10.465644 3447 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 23:26:10.466363 3448 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 23:26:10.467185 3450 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 23:26:10.467917 400 server_base.cc:1034] running on GCE node
I20240503 23:26:10.468219 400 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240503 23:26:10.468286 400 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240503 23:26:10.468322 400 hybrid_clock.cc:648] HybridClock initialized: now 1714778770468315 us; error 0 us; skew 500 ppm
I20240503 23:26:10.468500 400 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 23:26:10.469220 3443 heartbeater.cc:347] Connected to a master server at 127.0.100.62:39113
I20240503 23:26:10.469394 3443 heartbeater.cc:464] Registering TS with master...
I20240503 23:26:10.469789 3443 heartbeater.cc:511] Master 127.0.100.62:39113 requested a full tablet report, sending...
I20240503 23:26:10.470144 400 webserver.cc:460] Webserver started at http://127.0.100.2:41615/ using document root <none> and password file <none>
I20240503 23:26:10.470427 400 fs_manager.cc:359] Metadata directory not provided
I20240503 23:26:10.470505 400 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 23:26:10.470645 400 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 23:26:10.470960 3316 ts_manager.cc:194] Registered new tserver with Master: 01f756dedf2b40a8ab92c0c743c43d73 (127.0.100.1:39285)
I20240503 23:26:10.471578 400 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZhIiPz/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1714778688451250-400-0/minicluster-data/ts-1-root/instance:
uuid: "5b7105e1f5ec453bba84e3e2dea2a441"
format_stamp: "Formatted at 2024-05-03 23:26:10 on dist-test-slave-jvbp"
I20240503 23:26:10.472626 3316 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:50030
I20240503 23:26:10.474709 400 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.002s
I20240503 23:26:10.477025 3455 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 23:26:10.477433 400 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.002s sys 0.000s
I20240503 23:26:10.477542 400 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZhIiPz/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1714778688451250-400-0/minicluster-data/ts-1-root
uuid: "5b7105e1f5ec453bba84e3e2dea2a441"
format_stamp: "Formatted at 2024-05-03 23:26:10 on dist-test-slave-jvbp"
I20240503 23:26:10.477658 400 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZhIiPz/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1714778688451250-400-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskZhIiPz/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1714778688451250-400-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskZhIiPz/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1714778688451250-400-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 23:26:10.492686 400 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 23:26:10.493343 400 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 23:26:10.493923 400 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 23:26:10.494855 400 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 23:26:10.494917 400 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 23:26:10.495007 400 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 23:26:10.495050 400 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 23:26:10.507750 400 rpc_server.cc:307] RPC server started. Bound to: 127.0.100.2:45981
I20240503 23:26:10.507768 3517 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.100.2:45981 every 8 connection(s)
I20240503 23:26:10.509413 400 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 23:26:10.513164 3521 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 23:26:10.513880 3522 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 23:26:10.514633 400 server_base.cc:1034] running on GCE node
W20240503 23:26:10.515296 3524 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 23:26:10.515827 400 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240503 23:26:10.515918 400 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240503 23:26:10.516003 400 hybrid_clock.cc:648] HybridClock initialized: now 1714778770515997 us; error 0 us; skew 500 ppm
I20240503 23:26:10.516202 400 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 23:26:10.516614 3518 heartbeater.cc:347] Connected to a master server at 127.0.100.62:39113
I20240503 23:26:10.516747 3518 heartbeater.cc:464] Registering TS with master...
I20240503 23:26:10.517284 3518 heartbeater.cc:511] Master 127.0.100.62:39113 requested a full tablet report, sending...
I20240503 23:26:10.517975 400 webserver.cc:460] Webserver started at http://127.0.100.3:45159/ using document root <none> and password file <none>
I20240503 23:26:10.518234 400 fs_manager.cc:359] Metadata directory not provided
I20240503 23:26:10.518308 400 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 23:26:10.518368 3316 ts_manager.cc:194] Registered new tserver with Master: 5b7105e1f5ec453bba84e3e2dea2a441 (127.0.100.2:45981)
I20240503 23:26:10.518442 400 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 23:26:10.519202 3316 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:50034
I20240503 23:26:10.519181 400 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZhIiPz/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1714778688451250-400-0/minicluster-data/ts-2-root/instance:
uuid: "9b3c810d66c349f09f6955e9796422b5"
format_stamp: "Formatted at 2024-05-03 23:26:10 on dist-test-slave-jvbp"
I20240503 23:26:10.522305 400 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20240503 23:26:10.524080 3529 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 23:26:10.524492 400 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.002s sys 0.000s
I20240503 23:26:10.524648 400 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZhIiPz/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1714778688451250-400-0/minicluster-data/ts-2-root
uuid: "9b3c810d66c349f09f6955e9796422b5"
format_stamp: "Formatted at 2024-05-03 23:26:10 on dist-test-slave-jvbp"
I20240503 23:26:10.524770 400 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZhIiPz/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1714778688451250-400-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-taskZhIiPz/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1714778688451250-400-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-taskZhIiPz/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1714778688451250-400-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 23:26:10.557907 400 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 23:26:10.558585 400 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 23:26:10.559175 400 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 23:26:10.560166 400 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 23:26:10.560230 400 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 23:26:10.560321 400 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 23:26:10.560359 400 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 23:26:10.573710 400 rpc_server.cc:307] RPC server started. Bound to: 127.0.100.3:35677
I20240503 23:26:10.573729 3591 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.100.3:35677 every 8 connection(s)
I20240503 23:26:10.581523 3592 heartbeater.cc:347] Connected to a master server at 127.0.100.62:39113
I20240503 23:26:10.581714 3592 heartbeater.cc:464] Registering TS with master...
I20240503 23:26:10.582116 3592 heartbeater.cc:511] Master 127.0.100.62:39113 requested a full tablet report, sending...
I20240503 23:26:10.583098 3316 ts_manager.cc:194] Registered new tserver with Master: 9b3c810d66c349f09f6955e9796422b5 (127.0.100.3:35677)
I20240503 23:26:10.583953 3316 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:50050
I20240503 23:26:10.583925 400 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.009258374s
I20240503 23:26:11.474200 3443 heartbeater.cc:503] Master 127.0.100.62:39113 was elected leader, sending a full tablet report...
I20240503 23:26:11.520876 3518 heartbeater.cc:503] Master 127.0.100.62:39113 was elected leader, sending a full tablet report...
I20240503 23:26:11.585762 3592 heartbeater.cc:503] Master 127.0.100.62:39113 was elected leader, sending a full tablet report...
I20240503 23:26:11.636282 400 test_util.cc:268] Using random seed: 193806359
I20240503 23:26:11.653362 3316 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50064:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 23:26:11.654160 3316 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-workload in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 23:26:11.687417 3557 tablet_service.cc:1467] Processing CreateTablet for tablet 876a97bab85246dcb091eb4298515fb4 (DEFAULT_TABLE table=test-workload [id=2ea15b85bc3540e29612e1357ca87114]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 23:26:11.688045 3557 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 876a97bab85246dcb091eb4298515fb4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 23:26:11.690088 3408 tablet_service.cc:1467] Processing CreateTablet for tablet 876a97bab85246dcb091eb4298515fb4 (DEFAULT_TABLE table=test-workload [id=2ea15b85bc3540e29612e1357ca87114]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 23:26:11.690569 3408 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 876a97bab85246dcb091eb4298515fb4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 23:26:11.700910 3613 tablet_bootstrap.cc:492] T 876a97bab85246dcb091eb4298515fb4 P 9b3c810d66c349f09f6955e9796422b5: Bootstrap starting.
I20240503 23:26:11.701390 3483 tablet_service.cc:1467] Processing CreateTablet for tablet 876a97bab85246dcb091eb4298515fb4 (DEFAULT_TABLE table=test-workload [id=2ea15b85bc3540e29612e1357ca87114]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 23:26:11.701941 3483 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 876a97bab85246dcb091eb4298515fb4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 23:26:11.704277 3613 tablet_bootstrap.cc:654] T 876a97bab85246dcb091eb4298515fb4 P 9b3c810d66c349f09f6955e9796422b5: Neither blocks nor log segments found. Creating new log.
I20240503 23:26:11.711405 3614 tablet_bootstrap.cc:492] T 876a97bab85246dcb091eb4298515fb4 P 01f756dedf2b40a8ab92c0c743c43d73: Bootstrap starting.
I20240503 23:26:11.714534 3614 tablet_bootstrap.cc:654] T 876a97bab85246dcb091eb4298515fb4 P 01f756dedf2b40a8ab92c0c743c43d73: Neither blocks nor log segments found. Creating new log.
I20240503 23:26:11.716864 3616 tablet_bootstrap.cc:492] T 876a97bab85246dcb091eb4298515fb4 P 5b7105e1f5ec453bba84e3e2dea2a441: Bootstrap starting.
I20240503 23:26:11.718349 3613 tablet_bootstrap.cc:492] T 876a97bab85246dcb091eb4298515fb4 P 9b3c810d66c349f09f6955e9796422b5: No bootstrap required, opened a new log
I20240503 23:26:11.718580 3613 ts_tablet_manager.cc:1367] T 876a97bab85246dcb091eb4298515fb4 P 9b3c810d66c349f09f6955e9796422b5: Time spent bootstrapping tablet: real 0.018s user 0.005s sys 0.009s
I20240503 23:26:11.719453 3613 raft_consensus.cc:348] T 876a97bab85246dcb091eb4298515fb4 P 9b3c810d66c349f09f6955e9796422b5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01f756dedf2b40a8ab92c0c743c43d73" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 39285 } } peers { permanent_uuid: "9b3c810d66c349f09f6955e9796422b5" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 35677 } } peers { permanent_uuid: "5b7105e1f5ec453bba84e3e2dea2a441" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 45981 } }
I20240503 23:26:11.719699 3613 raft_consensus.cc:374] T 876a97bab85246dcb091eb4298515fb4 P 9b3c810d66c349f09f6955e9796422b5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 23:26:11.719789 3613 raft_consensus.cc:729] T 876a97bab85246dcb091eb4298515fb4 P 9b3c810d66c349f09f6955e9796422b5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9b3c810d66c349f09f6955e9796422b5, State: Initialized, Role: FOLLOWER
I20240503 23:26:11.720175 3616 tablet_bootstrap.cc:654] T 876a97bab85246dcb091eb4298515fb4 P 5b7105e1f5ec453bba84e3e2dea2a441: Neither blocks nor log segments found. Creating new log.
I20240503 23:26:11.720954 3614 tablet_bootstrap.cc:492] T 876a97bab85246dcb091eb4298515fb4 P 01f756dedf2b40a8ab92c0c743c43d73: No bootstrap required, opened a new log
I20240503 23:26:11.721144 3614 ts_tablet_manager.cc:1367] T 876a97bab85246dcb091eb4298515fb4 P 01f756dedf2b40a8ab92c0c743c43d73: Time spent bootstrapping tablet: real 0.010s user 0.005s sys 0.000s
I20240503 23:26:11.720101 3613 consensus_queue.cc:260] T 876a97bab85246dcb091eb4298515fb4 P 9b3c810d66c349f09f6955e9796422b5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01f756dedf2b40a8ab92c0c743c43d73" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 39285 } } peers { permanent_uuid: "9b3c810d66c349f09f6955e9796422b5" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 35677 } } peers { permanent_uuid: "5b7105e1f5ec453bba84e3e2dea2a441" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 45981 } }
I20240503 23:26:11.722108 3614 raft_consensus.cc:348] T 876a97bab85246dcb091eb4298515fb4 P 01f756dedf2b40a8ab92c0c743c43d73 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01f756dedf2b40a8ab92c0c743c43d73" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 39285 } } peers { permanent_uuid: "9b3c810d66c349f09f6955e9796422b5" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 35677 } } peers { permanent_uuid: "5b7105e1f5ec453bba84e3e2dea2a441" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 45981 } }
I20240503 23:26:11.722404 3614 raft_consensus.cc:374] T 876a97bab85246dcb091eb4298515fb4 P 01f756dedf2b40a8ab92c0c743c43d73 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 23:26:11.722543 3614 raft_consensus.cc:729] T 876a97bab85246dcb091eb4298515fb4 P 01f756dedf2b40a8ab92c0c743c43d73 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 01f756dedf2b40a8ab92c0c743c43d73, State: Initialized, Role: FOLLOWER
I20240503 23:26:11.722925 3614 consensus_queue.cc:260] T 876a97bab85246dcb091eb4298515fb4 P 01f756dedf2b40a8ab92c0c743c43d73 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01f756dedf2b40a8ab92c0c743c43d73" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 39285 } } peers { permanent_uuid: "9b3c810d66c349f09f6955e9796422b5" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 35677 } } peers { permanent_uuid: "5b7105e1f5ec453bba84e3e2dea2a441" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 45981 } }
I20240503 23:26:11.725359 3616 tablet_bootstrap.cc:492] T 876a97bab85246dcb091eb4298515fb4 P 5b7105e1f5ec453bba84e3e2dea2a441: No bootstrap required, opened a new log
I20240503 23:26:11.725925 3613 ts_tablet_manager.cc:1398] T 876a97bab85246dcb091eb4298515fb4 P 9b3c810d66c349f09f6955e9796422b5: Time spent starting tablet: real 0.007s user 0.001s sys 0.005s
I20240503 23:26:11.726323 3616 ts_tablet_manager.cc:1367] T 876a97bab85246dcb091eb4298515fb4 P 5b7105e1f5ec453bba84e3e2dea2a441: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20240503 23:26:11.727290 3616 raft_consensus.cc:348] T 876a97bab85246dcb091eb4298515fb4 P 5b7105e1f5ec453bba84e3e2dea2a441 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01f756dedf2b40a8ab92c0c743c43d73" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 39285 } } peers { permanent_uuid: "9b3c810d66c349f09f6955e9796422b5" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 35677 } } peers { permanent_uuid: "5b7105e1f5ec453bba84e3e2dea2a441" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 45981 } }
I20240503 23:26:11.727572 3616 raft_consensus.cc:374] T 876a97bab85246dcb091eb4298515fb4 P 5b7105e1f5ec453bba84e3e2dea2a441 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 23:26:11.727713 3616 raft_consensus.cc:729] T 876a97bab85246dcb091eb4298515fb4 P 5b7105e1f5ec453bba84e3e2dea2a441 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5b7105e1f5ec453bba84e3e2dea2a441, State: Initialized, Role: FOLLOWER
I20240503 23:26:11.728058 3616 consensus_queue.cc:260] T 876a97bab85246dcb091eb4298515fb4 P 5b7105e1f5ec453bba84e3e2dea2a441 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01f756dedf2b40a8ab92c0c743c43d73" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 39285 } } peers { permanent_uuid: "9b3c810d66c349f09f6955e9796422b5" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 35677 } } peers { permanent_uuid: "5b7105e1f5ec453bba84e3e2dea2a441" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 45981 } }
I20240503 23:26:11.729179 3616 ts_tablet_manager.cc:1398] T 876a97bab85246dcb091eb4298515fb4 P 5b7105e1f5ec453bba84e3e2dea2a441: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240503 23:26:11.730175 3614 ts_tablet_manager.cc:1398] T 876a97bab85246dcb091eb4298515fb4 P 01f756dedf2b40a8ab92c0c743c43d73: Time spent starting tablet: real 0.009s user 0.005s sys 0.003s
I20240503 23:26:11.792646 3620 raft_consensus.cc:482] T 876a97bab85246dcb091eb4298515fb4 P 01f756dedf2b40a8ab92c0c743c43d73 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 23:26:11.792918 3620 raft_consensus.cc:504] T 876a97bab85246dcb091eb4298515fb4 P 01f756dedf2b40a8ab92c0c743c43d73 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01f756dedf2b40a8ab92c0c743c43d73" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 39285 } } peers { permanent_uuid: "9b3c810d66c349f09f6955e9796422b5" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 35677 } } peers { permanent_uuid: "5b7105e1f5ec453bba84e3e2dea2a441" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 45981 } }
I20240503 23:26:11.793991 3620 leader_election.cc:288] T 876a97bab85246dcb091eb4298515fb4 P 01f756dedf2b40a8ab92c0c743c43d73 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9b3c810d66c349f09f6955e9796422b5 (127.0.100.3:35677), 5b7105e1f5ec453bba84e3e2dea2a441 (127.0.100.2:45981)
I20240503 23:26:11.803556 3567 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "876a97bab85246dcb091eb4298515fb4" candidate_uuid: "01f756dedf2b40a8ab92c0c743c43d73" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9b3c810d66c349f09f6955e9796422b5" is_pre_election: true
I20240503 23:26:11.803556 3493 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "876a97bab85246dcb091eb4298515fb4" candidate_uuid: "01f756dedf2b40a8ab92c0c743c43d73" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b7105e1f5ec453bba84e3e2dea2a441" is_pre_election: true
I20240503 23:26:11.803984 3493 raft_consensus.cc:2454] T 876a97bab85246dcb091eb4298515fb4 P 5b7105e1f5ec453bba84e3e2dea2a441 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 01f756dedf2b40a8ab92c0c743c43d73 in term 0.
I20240503 23:26:11.803985 3567 raft_consensus.cc:2454] T 876a97bab85246dcb091eb4298515fb4 P 9b3c810d66c349f09f6955e9796422b5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 01f756dedf2b40a8ab92c0c743c43d73 in term 0.
I20240503 23:26:11.804536 3384 leader_election.cc:302] T 876a97bab85246dcb091eb4298515fb4 P 01f756dedf2b40a8ab92c0c743c43d73 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 01f756dedf2b40a8ab92c0c743c43d73, 9b3c810d66c349f09f6955e9796422b5; no voters:
I20240503 23:26:11.804863 3620 raft_consensus.cc:2781] T 876a97bab85246dcb091eb4298515fb4 P 01f756dedf2b40a8ab92c0c743c43d73 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 23:26:11.804983 3620 raft_consensus.cc:482] T 876a97bab85246dcb091eb4298515fb4 P 01f756dedf2b40a8ab92c0c743c43d73 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 23:26:11.805080 3620 raft_consensus.cc:3037] T 876a97bab85246dcb091eb4298515fb4 P 01f756dedf2b40a8ab92c0c743c43d73 [term 0 FOLLOWER]: Advancing to term 1
I20240503 23:26:11.807178 3620 raft_consensus.cc:504] T 876a97bab85246dcb091eb4298515fb4 P 01f756dedf2b40a8ab92c0c743c43d73 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01f756dedf2b40a8ab92c0c743c43d73" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 39285 } } peers { permanent_uuid: "9b3c810d66c349f09f6955e9796422b5" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 35677 } } peers { permanent_uuid: "5b7105e1f5ec453bba84e3e2dea2a441" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 45981 } }
I20240503 23:26:11.807868 3620 leader_election.cc:288] T 876a97bab85246dcb091eb4298515fb4 P 01f756dedf2b40a8ab92c0c743c43d73 [CANDIDATE]: Term 1 election: Requested vote from peers 9b3c810d66c349f09f6955e9796422b5 (127.0.100.3:35677), 5b7105e1f5ec453bba84e3e2dea2a441 (127.0.100.2:45981)
I20240503 23:26:11.808573 3567 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "876a97bab85246dcb091eb4298515fb4" candidate_uuid: "01f756dedf2b40a8ab92c0c743c43d73" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9b3c810d66c349f09f6955e9796422b5"
I20240503 23:26:11.808576 3493 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "876a97bab85246dcb091eb4298515fb4" candidate_uuid: "01f756dedf2b40a8ab92c0c743c43d73" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b7105e1f5ec453bba84e3e2dea2a441"
I20240503 23:26:11.808872 3567 raft_consensus.cc:3037] T 876a97bab85246dcb091eb4298515fb4 P 9b3c810d66c349f09f6955e9796422b5 [term 0 FOLLOWER]: Advancing to term 1
I20240503 23:26:11.808872 3493 raft_consensus.cc:3037] T 876a97bab85246dcb091eb4298515fb4 P 5b7105e1f5ec453bba84e3e2dea2a441 [term 0 FOLLOWER]: Advancing to term 1
I20240503 23:26:11.812053 3493 raft_consensus.cc:2454] T 876a97bab85246dcb091eb4298515fb4 P 5b7105e1f5ec453bba84e3e2dea2a441 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 01f756dedf2b40a8ab92c0c743c43d73 in term 1.
I20240503 23:26:11.812053 3567 raft_consensus.cc:2454] T 876a97bab85246dcb091eb4298515fb4 P 9b3c810d66c349f09f6955e9796422b5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 01f756dedf2b40a8ab92c0c743c43d73 in term 1.
I20240503 23:26:11.812721 3384 leader_election.cc:302] T 876a97bab85246dcb091eb4298515fb4 P 01f756dedf2b40a8ab92c0c743c43d73 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 01f756dedf2b40a8ab92c0c743c43d73, 9b3c810d66c349f09f6955e9796422b5; no voters:
I20240503 23:26:11.813069 3620 raft_consensus.cc:2781] T 876a97bab85246dcb091eb4298515fb4 P 01f756dedf2b40a8ab92c0c743c43d73 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 23:26:11.813660 3620 raft_consensus.cc:686] T 876a97bab85246dcb091eb4298515fb4 P 01f756dedf2b40a8ab92c0c743c43d73 [term 1 LEADER]: Becoming Leader. State: Replica: 01f756dedf2b40a8ab92c0c743c43d73, State: Running, Role: LEADER
I20240503 23:26:11.813993 3620 consensus_queue.cc:237] T 876a97bab85246dcb091eb4298515fb4 P 01f756dedf2b40a8ab92c0c743c43d73 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01f756dedf2b40a8ab92c0c743c43d73" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 39285 } } peers { permanent_uuid: "9b3c810d66c349f09f6955e9796422b5" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 35677 } } peers { permanent_uuid: "5b7105e1f5ec453bba84e3e2dea2a441" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 45981 } }
I20240503 23:26:11.817260 3315 catalog_manager.cc:5484] T 876a97bab85246dcb091eb4298515fb4 P 01f756dedf2b40a8ab92c0c743c43d73 reported cstate change: term changed from 0 to 1, leader changed from <none> to 01f756dedf2b40a8ab92c0c743c43d73 (127.0.100.1). New cstate: current_term: 1 leader_uuid: "01f756dedf2b40a8ab92c0c743c43d73" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01f756dedf2b40a8ab92c0c743c43d73" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 39285 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9b3c810d66c349f09f6955e9796422b5" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 35677 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5b7105e1f5ec453bba84e3e2dea2a441" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 45981 } health_report { overall_health: UNKNOWN } } }
I20240503 23:26:12.326105 3626 consensus_queue.cc:1035] T 876a97bab85246dcb091eb4298515fb4 P 01f756dedf2b40a8ab92c0c743c43d73 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9b3c810d66c349f09f6955e9796422b5" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 35677 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 23:26:12.332444 3629 consensus_queue.cc:1035] T 876a97bab85246dcb091eb4298515fb4 P 01f756dedf2b40a8ab92c0c743c43d73 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5b7105e1f5ec453bba84e3e2dea2a441" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 45981 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 23:26:12.342272 3364 placement_policy_util.cc:407] tablet 876a97bab85246dcb091eb4298515fb4: detected majority of replicas (2 of 3) at location L0
I20240503 23:26:12.342934 3369 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240503 23:26:12.343448 3369 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240503 23:26:12.343536 3369 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240503 23:26:13.345422 3364 placement_policy_util.cc:407] tablet 876a97bab85246dcb091eb4298515fb4: detected majority of replicas (2 of 3) at location L0
I20240503 23:26:14.343945 3369 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240503 23:26:14.344583 3369 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240503 23:26:14.344698 3369 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240503 23:26:14.348052 3364 placement_policy_util.cc:407] tablet 876a97bab85246dcb091eb4298515fb4: detected majority of replicas (2 of 3) at location L0
I20240503 23:26:15.350823 3364 placement_policy_util.cc:407] tablet 876a97bab85246dcb091eb4298515fb4: detected majority of replicas (2 of 3) at location L0
I20240503 23:26:16.345089 3369 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240503 23:26:16.345616 3369 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240503 23:26:16.345728 3369 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240503 23:26:16.353587 3364 placement_policy_util.cc:407] tablet 876a97bab85246dcb091eb4298515fb4: detected majority of replicas (2 of 3) at location L0
I20240503 23:26:17.356606 3364 placement_policy_util.cc:407] tablet 876a97bab85246dcb091eb4298515fb4: detected majority of replicas (2 of 3) at location L0
I20240503 23:26:18.346168 3369 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240503 23:26:18.346714 3369 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240503 23:26:18.346830 3369 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240503 23:26:18.359375 3364 placement_policy_util.cc:407] tablet 876a97bab85246dcb091eb4298515fb4: detected majority of replicas (2 of 3) at location L0
I20240503 23:26:19.362116 3364 placement_policy_util.cc:407] tablet 876a97bab85246dcb091eb4298515fb4: detected majority of replicas (2 of 3) at location L0
I20240503 23:26:20.347354 3369 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240503 23:26:20.347873 3369 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240503 23:26:20.347966 3369 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240503 23:26:20.364976 3364 placement_policy_util.cc:407] tablet 876a97bab85246dcb091eb4298515fb4: detected majority of replicas (2 of 3) at location L0
I20240503 23:26:21.367834 3364 placement_policy_util.cc:407] tablet 876a97bab85246dcb091eb4298515fb4: detected majority of replicas (2 of 3) at location L0
I20240503 23:26:22.348433 3369 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240503 23:26:22.348991 3369 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240503 23:26:22.349088 3369 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240503 23:26:22.370733 3364 placement_policy_util.cc:407] tablet 876a97bab85246dcb091eb4298515fb4: detected majority of replicas (2 of 3) at location L0
I20240503 23:26:22.889457 400 tablet_server.cc:178] TabletServer@127.0.100.1:0 shutting down...
I20240503 23:26:22.909902 400 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240503 23:26:22.910259 400 tablet_replica.cc:330] T 876a97bab85246dcb091eb4298515fb4 P 01f756dedf2b40a8ab92c0c743c43d73: stopping tablet replica
I20240503 23:26:22.910485 400 raft_consensus.cc:2229] T 876a97bab85246dcb091eb4298515fb4 P 01f756dedf2b40a8ab92c0c743c43d73 [term 1 LEADER]: Raft consensus shutting down.
I20240503 23:26:22.910894 400 raft_consensus.cc:2258] T 876a97bab85246dcb091eb4298515fb4 P 01f756dedf2b40a8ab92c0c743c43d73 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 23:26:22.926932 400 tablet_server.cc:194] TabletServer@127.0.100.1:0 shutdown complete.
I20240503 23:26:22.930703 400 tablet_server.cc:178] TabletServer@127.0.100.2:0 shutting down...
I20240503 23:26:22.941210 400 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240503 23:26:22.941565 400 tablet_replica.cc:330] T 876a97bab85246dcb091eb4298515fb4 P 5b7105e1f5ec453bba84e3e2dea2a441: stopping tablet replica
I20240503 23:26:22.941773 400 raft_consensus.cc:2229] T 876a97bab85246dcb091eb4298515fb4 P 5b7105e1f5ec453bba84e3e2dea2a441 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 23:26:22.942001 400 raft_consensus.cc:2258] T 876a97bab85246dcb091eb4298515fb4 P 5b7105e1f5ec453bba84e3e2dea2a441 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 23:26:22.956482 400 tablet_server.cc:194] TabletServer@127.0.100.2:0 shutdown complete.
I20240503 23:26:22.960206 400 tablet_server.cc:178] TabletServer@127.0.100.3:0 shutting down...
I20240503 23:26:22.971408 400 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240503 23:26:22.971773 400 tablet_replica.cc:330] T 876a97bab85246dcb091eb4298515fb4 P 9b3c810d66c349f09f6955e9796422b5: stopping tablet replica
I20240503 23:26:22.971994 400 raft_consensus.cc:2229] T 876a97bab85246dcb091eb4298515fb4 P 9b3c810d66c349f09f6955e9796422b5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 23:26:22.972231 400 raft_consensus.cc:2258] T 876a97bab85246dcb091eb4298515fb4 P 9b3c810d66c349f09f6955e9796422b5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 23:26:22.989221 400 tablet_server.cc:194] TabletServer@127.0.100.3:0 shutdown complete.
I20240503 23:26:22.993023 400 master.cc:537] Master@127.0.100.62:39113 shutting down...
I20240503 23:26:23.002314 400 raft_consensus.cc:2229] T 00000000000000000000000000000000 P a9e74a3a3f174e3e96e71c9f0273165d [term 1 LEADER]: Raft consensus shutting down.
I20240503 23:26:23.002642 400 raft_consensus.cc:2258] T 00000000000000000000000000000000 P a9e74a3a3f174e3e96e71c9f0273165d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 23:26:23.002784 400 tablet_replica.cc:330] T 00000000000000000000000000000000 P a9e74a3a3f174e3e96e71c9f0273165d: stopping tablet replica
I20240503 23:26:23.017091 400 master.cc:552] Master@127.0.100.62:39113 shutdown complete.
[ OK ] AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy (12754 ms)
[ RUN ] AutoRebalancerTest.TestMaxMovesPerServer
I20240503 23:26:23.030308 400 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.0.100.62:38775
I20240503 23:26:23.030961 400 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 23:26:23.034499 3637 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 23:26:23.035467 3638 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 23:26:23.036327 3640 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 23:26:23.037161 400 server_base.cc:1034] running on GCE node
I20240503 23:26:23.037456 400 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240503 23:26:23.037526 400 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240503 23:26:23.037568 400 hybrid_clock.cc:648] HybridClock initialized: now 1714778783037550 us; error 0 us; skew 500 ppm
I20240503 23:26:23.037729 400 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 23:26:23.039341 400 webserver.cc:460] Webserver started at http://127.0.100.62:43691/ using document root <none> and password file <none>
I20240503 23:26:23.039595 400 fs_manager.cc:359] Metadata directory not provided
I20240503 23:26:23.039669 400 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 23:26:23.039801 400 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 23:26:23.040611 400 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZhIiPz/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714778688451250-400-0/minicluster-data/master-0-root/instance:
uuid: "bd1b1f48d92849bcad7225c6235d0873"
format_stamp: "Formatted at 2024-05-03 23:26:23 on dist-test-slave-jvbp"
I20240503 23:26:23.043680 400 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20240503 23:26:23.045698 3645 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 23:26:23.046123 400 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20240503 23:26:23.046237 400 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZhIiPz/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714778688451250-400-0/minicluster-data/master-0-root
uuid: "bd1b1f48d92849bcad7225c6235d0873"
format_stamp: "Formatted at 2024-05-03 23:26:23 on dist-test-slave-jvbp"
I20240503 23:26:23.046404 400 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZhIiPz/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714778688451250-400-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskZhIiPz/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714778688451250-400-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskZhIiPz/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714778688451250-400-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 23:26:23.066417 400 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 23:26:23.067184 400 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 23:26:23.080235 3696 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.100.62:38775 every 8 connection(s)
I20240503 23:26:23.080228 400 rpc_server.cc:307] RPC server started. Bound to: 127.0.100.62:38775
I20240503 23:26:23.081945 3697 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 23:26:23.086829 3697 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bd1b1f48d92849bcad7225c6235d0873: Bootstrap starting.
I20240503 23:26:23.088299 3697 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bd1b1f48d92849bcad7225c6235d0873: Neither blocks nor log segments found. Creating new log.
I20240503 23:26:23.090444 3697 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bd1b1f48d92849bcad7225c6235d0873: No bootstrap required, opened a new log
I20240503 23:26:23.091259 3697 raft_consensus.cc:348] T 00000000000000000000000000000000 P bd1b1f48d92849bcad7225c6235d0873 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bd1b1f48d92849bcad7225c6235d0873" member_type: VOTER }
I20240503 23:26:23.091432 3697 raft_consensus.cc:374] T 00000000000000000000000000000000 P bd1b1f48d92849bcad7225c6235d0873 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 23:26:23.091488 3697 raft_consensus.cc:729] T 00000000000000000000000000000000 P bd1b1f48d92849bcad7225c6235d0873 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bd1b1f48d92849bcad7225c6235d0873, State: Initialized, Role: FOLLOWER
I20240503 23:26:23.091759 3697 consensus_queue.cc:260] T 00000000000000000000000000000000 P bd1b1f48d92849bcad7225c6235d0873 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bd1b1f48d92849bcad7225c6235d0873" member_type: VOTER }
I20240503 23:26:23.091920 3697 raft_consensus.cc:388] T 00000000000000000000000000000000 P bd1b1f48d92849bcad7225c6235d0873 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 23:26:23.091982 3697 raft_consensus.cc:482] T 00000000000000000000000000000000 P bd1b1f48d92849bcad7225c6235d0873 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 23:26:23.092085 3697 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bd1b1f48d92849bcad7225c6235d0873 [term 0 FOLLOWER]: Advancing to term 1
I20240503 23:26:23.094182 3697 raft_consensus.cc:504] T 00000000000000000000000000000000 P bd1b1f48d92849bcad7225c6235d0873 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bd1b1f48d92849bcad7225c6235d0873" member_type: VOTER }
I20240503 23:26:23.094453 3697 leader_election.cc:302] T 00000000000000000000000000000000 P bd1b1f48d92849bcad7225c6235d0873 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: bd1b1f48d92849bcad7225c6235d0873; no voters:
I20240503 23:26:23.094959 3697 leader_election.cc:288] T 00000000000000000000000000000000 P bd1b1f48d92849bcad7225c6235d0873 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240503 23:26:23.095150 3700 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bd1b1f48d92849bcad7225c6235d0873 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 23:26:23.095702 3700 raft_consensus.cc:686] T 00000000000000000000000000000000 P bd1b1f48d92849bcad7225c6235d0873 [term 1 LEADER]: Becoming Leader. State: Replica: bd1b1f48d92849bcad7225c6235d0873, State: Running, Role: LEADER
I20240503 23:26:23.096028 3697 sys_catalog.cc:564] T 00000000000000000000000000000000 P bd1b1f48d92849bcad7225c6235d0873 [sys.catalog]: configured and running, proceeding with master startup.
I20240503 23:26:23.096021 3700 consensus_queue.cc:237] T 00000000000000000000000000000000 P bd1b1f48d92849bcad7225c6235d0873 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bd1b1f48d92849bcad7225c6235d0873" member_type: VOTER }
I20240503 23:26:23.097314 3701 sys_catalog.cc:455] T 00000000000000000000000000000000 P bd1b1f48d92849bcad7225c6235d0873 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "bd1b1f48d92849bcad7225c6235d0873" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bd1b1f48d92849bcad7225c6235d0873" member_type: VOTER } }
I20240503 23:26:23.097384 3702 sys_catalog.cc:455] T 00000000000000000000000000000000 P bd1b1f48d92849bcad7225c6235d0873 [sys.catalog]: SysCatalogTable state changed. Reason: New leader bd1b1f48d92849bcad7225c6235d0873. Latest consensus state: current_term: 1 leader_uuid: "bd1b1f48d92849bcad7225c6235d0873" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bd1b1f48d92849bcad7225c6235d0873" member_type: VOTER } }
I20240503 23:26:23.097589 3701 sys_catalog.cc:458] T 00000000000000000000000000000000 P bd1b1f48d92849bcad7225c6235d0873 [sys.catalog]: This master's current role is: LEADER
I20240503 23:26:23.097643 3702 sys_catalog.cc:458] T 00000000000000000000000000000000 P bd1b1f48d92849bcad7225c6235d0873 [sys.catalog]: This master's current role is: LEADER
I20240503 23:26:23.098810 3705 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240503 23:26:23.100346 3705 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240503 23:26:23.102180 400 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20240503 23:26:23.104331 3705 catalog_manager.cc:1347] Generated new cluster ID: ee8c5d6b401b4f8d8ff8ec81f2e2bec2
I20240503 23:26:23.104454 3705 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240503 23:26:23.118463 3705 catalog_manager.cc:1370] Generated new certificate authority record
I20240503 23:26:23.119432 3705 catalog_manager.cc:1504] Loading token signing keys...
I20240503 23:26:23.129941 3705 catalog_manager.cc:5857] T 00000000000000000000000000000000 P bd1b1f48d92849bcad7225c6235d0873: Generated new TSK 0
I20240503 23:26:23.130328 3705 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240503 23:26:23.135641 400 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 23:26:23.139757 3718 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 23:26:23.140679 3719 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 23:26:23.141441 3721 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 23:26:23.142259 400 server_base.cc:1034] running on GCE node
I20240503 23:26:23.142580 400 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240503 23:26:23.142673 400 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240503 23:26:23.142724 400 hybrid_clock.cc:648] HybridClock initialized: now 1714778783142707 us; error 0 us; skew 500 ppm
I20240503 23:26:23.142894 400 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 23:26:23.144474 400 webserver.cc:460] Webserver started at http://127.0.100.1:37609/ using document root <none> and password file <none>
I20240503 23:26:23.144768 400 fs_manager.cc:359] Metadata directory not provided
I20240503 23:26:23.144851 400 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 23:26:23.144990 400 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 23:26:23.145720 400 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZhIiPz/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714778688451250-400-0/minicluster-data/ts-0-root/instance:
uuid: "83ffd7184ec94302bd1ee74b0fab3e17"
format_stamp: "Formatted at 2024-05-03 23:26:23 on dist-test-slave-jvbp"
I20240503 23:26:23.148767 400 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20240503 23:26:23.151139 3726 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 23:26:23.151638 400 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.002s sys 0.001s
I20240503 23:26:23.151773 400 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZhIiPz/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714778688451250-400-0/minicluster-data/ts-0-root
uuid: "83ffd7184ec94302bd1ee74b0fab3e17"
format_stamp: "Formatted at 2024-05-03 23:26:23 on dist-test-slave-jvbp"
I20240503 23:26:23.151961 400 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZhIiPz/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714778688451250-400-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskZhIiPz/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714778688451250-400-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskZhIiPz/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714778688451250-400-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 23:26:23.177810 400 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 23:26:23.178506 400 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 23:26:23.179237 400 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 23:26:23.180310 400 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 23:26:23.180377 400 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 23:26:23.180456 400 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 23:26:23.180510 400 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 23:26:23.195995 400 rpc_server.cc:307] RPC server started. Bound to: 127.0.100.1:32909
I20240503 23:26:23.196028 3788 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.100.1:32909 every 8 connection(s)
I20240503 23:26:23.198235 400 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 23:26:23.201913 3793 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 23:26:23.202852 3794 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 23:26:23.205682 3789 heartbeater.cc:347] Connected to a master server at 127.0.100.62:38775
I20240503 23:26:23.205940 3789 heartbeater.cc:464] Registering TS with master...
I20240503 23:26:23.206372 3789 heartbeater.cc:511] Master 127.0.100.62:38775 requested a full tablet report, sending...
I20240503 23:26:23.207408 3662 ts_manager.cc:194] Registered new tserver with Master: 83ffd7184ec94302bd1ee74b0fab3e17 (127.0.100.1:32909)
I20240503 23:26:23.208640 3662 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:57294
W20240503 23:26:23.209723 3796 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 23:26:23.210554 400 server_base.cc:1034] running on GCE node
I20240503 23:26:23.210810 400 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240503 23:26:23.210882 400 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240503 23:26:23.210929 400 hybrid_clock.cc:648] HybridClock initialized: now 1714778783210924 us; error 0 us; skew 500 ppm
I20240503 23:26:23.211082 400 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 23:26:23.212778 400 webserver.cc:460] Webserver started at http://127.0.100.2:39733/ using document root <none> and password file <none>
I20240503 23:26:23.213027 400 fs_manager.cc:359] Metadata directory not provided
I20240503 23:26:23.213105 400 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 23:26:23.213239 400 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 23:26:23.213932 400 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZhIiPz/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714778688451250-400-0/minicluster-data/ts-1-root/instance:
uuid: "a2f9bf2e5d9b4fc19f804797bca4a3ff"
format_stamp: "Formatted at 2024-05-03 23:26:23 on dist-test-slave-jvbp"
I20240503 23:26:23.216681 400 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.000s sys 0.003s
I20240503 23:26:23.218549 3801 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 23:26:23.218959 400 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.002s sys 0.000s
I20240503 23:26:23.219079 400 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZhIiPz/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714778688451250-400-0/minicluster-data/ts-1-root
uuid: "a2f9bf2e5d9b4fc19f804797bca4a3ff"
format_stamp: "Formatted at 2024-05-03 23:26:23 on dist-test-slave-jvbp"
I20240503 23:26:23.219210 400 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZhIiPz/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714778688451250-400-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskZhIiPz/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714778688451250-400-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskZhIiPz/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714778688451250-400-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 23:26:23.240983 400 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 23:26:23.241650 400 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 23:26:23.242293 400 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 23:26:23.243291 400 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 23:26:23.243363 400 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 23:26:23.243446 400 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 23:26:23.243485 400 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 23:26:23.258570 400 rpc_server.cc:307] RPC server started. Bound to: 127.0.100.2:46261
I20240503 23:26:23.258610 3863 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.100.2:46261 every 8 connection(s)
I20240503 23:26:23.260900 400 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 23:26:23.264905 3867 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 23:26:23.265803 3868 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 23:26:23.266616 400 server_base.cc:1034] running on GCE node
W20240503 23:26:23.267429 3870 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 23:26:23.267933 400 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240503 23:26:23.268009 400 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240503 23:26:23.268045 400 hybrid_clock.cc:648] HybridClock initialized: now 1714778783268040 us; error 0 us; skew 500 ppm
I20240503 23:26:23.268241 400 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 23:26:23.268465 3864 heartbeater.cc:347] Connected to a master server at 127.0.100.62:38775
I20240503 23:26:23.268658 3864 heartbeater.cc:464] Registering TS with master...
I20240503 23:26:23.269124 3864 heartbeater.cc:511] Master 127.0.100.62:38775 requested a full tablet report, sending...
I20240503 23:26:23.269922 400 webserver.cc:460] Webserver started at http://127.0.100.3:34107/ using document root <none> and password file <none>
I20240503 23:26:23.270159 400 fs_manager.cc:359] Metadata directory not provided
I20240503 23:26:23.270237 400 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 23:26:23.270272 3662 ts_manager.cc:194] Registered new tserver with Master: a2f9bf2e5d9b4fc19f804797bca4a3ff (127.0.100.2:46261)
I20240503 23:26:23.270375 400 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 23:26:23.271081 400 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZhIiPz/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714778688451250-400-0/minicluster-data/ts-2-root/instance:
uuid: "a3eecbaec43d4954b1096bc3121e824c"
format_stamp: "Formatted at 2024-05-03 23:26:23 on dist-test-slave-jvbp"
I20240503 23:26:23.271365 3662 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:57296
I20240503 23:26:23.274070 400 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.004s
I20240503 23:26:23.275827 3875 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 23:26:23.276314 400 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.001s
I20240503 23:26:23.276417 400 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZhIiPz/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714778688451250-400-0/minicluster-data/ts-2-root
uuid: "a3eecbaec43d4954b1096bc3121e824c"
format_stamp: "Formatted at 2024-05-03 23:26:23 on dist-test-slave-jvbp"
I20240503 23:26:23.276582 400 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZhIiPz/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714778688451250-400-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-taskZhIiPz/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714778688451250-400-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-taskZhIiPz/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714778688451250-400-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 23:26:23.300031 400 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 23:26:23.300803 400 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 23:26:23.301458 400 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 23:26:23.302410 400 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 23:26:23.302481 400 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 23:26:23.302579 400 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 23:26:23.302619 400 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 23:26:23.319216 400 rpc_server.cc:307] RPC server started. Bound to: 127.0.100.3:40649
I20240503 23:26:23.319258 3937 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.100.3:40649 every 8 connection(s)
I20240503 23:26:23.327623 3938 heartbeater.cc:347] Connected to a master server at 127.0.100.62:38775
I20240503 23:26:23.327812 3938 heartbeater.cc:464] Registering TS with master...
I20240503 23:26:23.328303 3938 heartbeater.cc:511] Master 127.0.100.62:38775 requested a full tablet report, sending...
I20240503 23:26:23.329329 3662 ts_manager.cc:194] Registered new tserver with Master: a3eecbaec43d4954b1096bc3121e824c (127.0.100.3:40649)
I20240503 23:26:23.329725 400 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.009355509s
I20240503 23:26:23.330471 3662 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:57304
I20240503 23:26:24.210340 3789 heartbeater.cc:503] Master 127.0.100.62:38775 was elected leader, sending a full tablet report...
I20240503 23:26:24.272904 3864 heartbeater.cc:503] Master 127.0.100.62:38775 was elected leader, sending a full tablet report...
I20240503 23:26:24.332418 3938 heartbeater.cc:503] Master 127.0.100.62:38775 was elected leader, sending a full tablet report...
I20240503 23:26:24.359431 400 test_util.cc:268] Using random seed: 206529533
I20240503 23:26:24.373575 3662 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57320:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "\004\001\000\252\252\252\n\004\001\000TUU\025\004\001\000\376\377\377\037\004\001\000\250\252\252*\004\001\000RUU5\004\001\000\374\377\377?\004\001\000\246\252\252J\004\001\000PUUU\004\001\000\372\377\377_\004\001\000\244\252\252j\004\001\000NUUu""\004\001\000\252\252\252\n\004\001\000TUU\025\004\001\000\376\377\377\037\004\001\000\250\252\252*\004\001\000RUU5\004\001\000\374\377\377?\004\001\000\246\252\252J\004\001\000PUUU\004\001\000\372\377\377_\004\001\000\244\252\252j\004\001\000NUUu"
indirect_data: """"
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 23:26:24.374634 3662 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-workload in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 23:26:24.416352 3903 tablet_service.cc:1467] Processing CreateTablet for tablet 6039230222ff4e57929bed49cd56caf5 (DEFAULT_TABLE table=test-workload [id=1345b2a336c84ec2b1de62853e7fa3f6]), partition=RANGE (key) PARTITION VALUES < 178956970
I20240503 23:26:24.417049 3903 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6039230222ff4e57929bed49cd56caf5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 23:26:24.417146 3829 tablet_service.cc:1467] Processing CreateTablet for tablet 6039230222ff4e57929bed49cd56caf5 (DEFAULT_TABLE table=test-workload [id=1345b2a336c84ec2b1de62853e7fa3f6]), partition=RANGE (key) PARTITION VALUES < 178956970
I20240503 23:26:24.417398 3753 tablet_service.cc:1467] Processing CreateTablet for tablet 487ac5b9bf2d4de2b80a1b13387a2a77 (DEFAULT_TABLE table=test-workload [id=1345b2a336c84ec2b1de62853e7fa3f6]), partition=RANGE (key) PARTITION 178956970 <= VALUES < 357913940
I20240503 23:26:24.417578 3829 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6039230222ff4e57929bed49cd56caf5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 23:26:24.417699 3900 tablet_service.cc:1467] Processing CreateTablet for tablet 7181e4e8c0324a8eb095dde23651a1ab (DEFAULT_TABLE table=test-workload [id=1345b2a336c84ec2b1de62853e7fa3f6]), partition=RANGE (key) PARTITION 536870910 <= VALUES < 715827880
I20240503 23:26:24.417867 3753 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 487ac5b9bf2d4de2b80a1b13387a2a77. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 23:26:24.418093 3900 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7181e4e8c0324a8eb095dde23651a1ab. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 23:26:24.421034 3752 tablet_service.cc:1467] Processing CreateTablet for tablet 481cd0f8c8d147adac217915a53ff6d9 (DEFAULT_TABLE table=test-workload [id=1345b2a336c84ec2b1de62853e7fa3f6]), partition=RANGE (key) PARTITION 357913940 <= VALUES < 536870910
I20240503 23:26:24.421355 3901 tablet_service.cc:1467] Processing CreateTablet for tablet 481cd0f8c8d147adac217915a53ff6d9 (DEFAULT_TABLE table=test-workload [id=1345b2a336c84ec2b1de62853e7fa3f6]), partition=RANGE (key) PARTITION 357913940 <= VALUES < 536870910
I20240503 23:26:24.421564 3752 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 481cd0f8c8d147adac217915a53ff6d9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 23:26:24.421670 3825 tablet_service.cc:1467] Processing CreateTablet for tablet 55cf664d5e874c0d9b9adf8bdc9c43ed (DEFAULT_TABLE table=test-workload [id=1345b2a336c84ec2b1de62853e7fa3f6]), partition=RANGE (key) PARTITION 715827880 <= VALUES < 894784850
I20240503 23:26:24.421816 3901 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 481cd0f8c8d147adac217915a53ff6d9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 23:26:24.422065 3825 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 55cf664d5e874c0d9b9adf8bdc9c43ed. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 23:26:24.425078 3751 tablet_service.cc:1467] Processing CreateTablet for tablet 7181e4e8c0324a8eb095dde23651a1ab (DEFAULT_TABLE table=test-workload [id=1345b2a336c84ec2b1de62853e7fa3f6]), partition=RANGE (key) PARTITION 536870910 <= VALUES < 715827880
I20240503 23:26:24.425449 3823 tablet_service.cc:1467] Processing CreateTablet for tablet 98c9500acb43423b86f54ce7a4d98a72 (DEFAULT_TABLE table=test-workload [id=1345b2a336c84ec2b1de62853e7fa3f6]), partition=RANGE (key) PARTITION 1073741820 <= VALUES < 1252698790
I20240503 23:26:24.425786 3751 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7181e4e8c0324a8eb095dde23651a1ab. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 23:26:24.425932 3823 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 98c9500acb43423b86f54ce7a4d98a72. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 23:26:24.429016 3822 tablet_service.cc:1467] Processing CreateTablet for tablet 1b9566be6c644cfca852ec6507fab893 (DEFAULT_TABLE table=test-workload [id=1345b2a336c84ec2b1de62853e7fa3f6]), partition=RANGE (key) PARTITION 1252698790 <= VALUES < 1431655760
I20240503 23:26:24.429086 3750 tablet_service.cc:1467] Processing CreateTablet for tablet 55cf664d5e874c0d9b9adf8bdc9c43ed (DEFAULT_TABLE table=test-workload [id=1345b2a336c84ec2b1de62853e7fa3f6]), partition=RANGE (key) PARTITION 715827880 <= VALUES < 894784850
I20240503 23:26:24.429538 3822 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1b9566be6c644cfca852ec6507fab893. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 23:26:24.429652 3750 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 55cf664d5e874c0d9b9adf8bdc9c43ed. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 23:26:24.433104 3821 tablet_service.cc:1467] Processing CreateTablet for tablet a14a12115b8d4fec947c6e223f1ef486 (DEFAULT_TABLE table=test-workload [id=1345b2a336c84ec2b1de62853e7fa3f6]), partition=RANGE (key) PARTITION 1431655760 <= VALUES < 1610612730
I20240503 23:26:24.433182 3749 tablet_service.cc:1467] Processing CreateTablet for tablet 60267cb4849d4823a0afb2380305905f (DEFAULT_TABLE table=test-workload [id=1345b2a336c84ec2b1de62853e7fa3f6]), partition=RANGE (key) PARTITION 894784850 <= VALUES < 1073741820
I20240503 23:26:24.433682 3821 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a14a12115b8d4fec947c6e223f1ef486. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 23:26:24.433770 3749 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 60267cb4849d4823a0afb2380305905f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 23:26:24.437062 3820 tablet_service.cc:1467] Processing CreateTablet for tablet a0e18d0975a54e3dbeb0a13cb226fefd (DEFAULT_TABLE table=test-workload [id=1345b2a336c84ec2b1de62853e7fa3f6]), partition=RANGE (key) PARTITION 1610612730 <= VALUES < 1789569700
I20240503 23:26:24.437482 3747 tablet_service.cc:1467] Processing CreateTablet for tablet 1b9566be6c644cfca852ec6507fab893 (DEFAULT_TABLE table=test-workload [id=1345b2a336c84ec2b1de62853e7fa3f6]), partition=RANGE (key) PARTITION 1252698790 <= VALUES < 1431655760
I20240503 23:26:24.437691 3820 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a0e18d0975a54e3dbeb0a13cb226fefd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 23:26:24.437811 3901 tablet_service.cc:1467] Processing CreateTablet for tablet d0c4430fc56644cb9f87deffe4fe9407 (DEFAULT_TABLE table=test-workload [id=1345b2a336c84ec2b1de62853e7fa3f6]), partition=RANGE (key) PARTITION 1789569700 <= VALUES < 1968526670
I20240503 23:26:24.438043 3747 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1b9566be6c644cfca852ec6507fab893. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 23:26:24.438141 3901 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d0c4430fc56644cb9f87deffe4fe9407. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 23:26:24.441125 3898 tablet_service.cc:1467] Processing CreateTablet for tablet 60267cb4849d4823a0afb2380305905f (DEFAULT_TABLE table=test-workload [id=1345b2a336c84ec2b1de62853e7fa3f6]), partition=RANGE (key) PARTITION 894784850 <= VALUES < 1073741820
I20240503 23:26:24.441737 3898 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 60267cb4849d4823a0afb2380305905f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 23:26:24.441797 3745 tablet_service.cc:1467] Processing CreateTablet for tablet a0e18d0975a54e3dbeb0a13cb226fefd (DEFAULT_TABLE table=test-workload [id=1345b2a336c84ec2b1de62853e7fa3f6]), partition=RANGE (key) PARTITION 1610612730 <= VALUES < 1789569700
I20240503 23:26:24.442478 3745 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a0e18d0975a54e3dbeb0a13cb226fefd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 23:26:24.421355 3827 tablet_service.cc:1467] Processing CreateTablet for tablet 481cd0f8c8d147adac217915a53ff6d9 (DEFAULT_TABLE table=test-workload [id=1345b2a336c84ec2b1de62853e7fa3f6]), partition=RANGE (key) PARTITION 357913940 <= VALUES < 536870910
I20240503 23:26:24.425791 3894 tablet_service.cc:1467] Processing CreateTablet for tablet a0e18d0975a54e3dbeb0a13cb226fefd (DEFAULT_TABLE table=test-workload [id=1345b2a336c84ec2b1de62853e7fa3f6]), partition=RANGE (key) PARTITION 1610612730 <= VALUES < 1789569700
I20240503 23:26:24.417398 3826 tablet_service.cc:1467] Processing CreateTablet for tablet 7181e4e8c0324a8eb095dde23651a1ab (DEFAULT_TABLE table=test-workload [id=1345b2a336c84ec2b1de62853e7fa3f6]), partition=RANGE (key) PARTITION 536870910 <= VALUES < 715827880
I20240503 23:26:24.417160 3754 tablet_service.cc:1467] Processing CreateTablet for tablet 6039230222ff4e57929bed49cd56caf5 (DEFAULT_TABLE table=test-workload [id=1345b2a336c84ec2b1de62853e7fa3f6]), partition=RANGE (key) PARTITION VALUES < 178956970
I20240503 23:26:24.437078 3748 tablet_service.cc:1467] Processing CreateTablet for tablet 98c9500acb43423b86f54ce7a4d98a72 (DEFAULT_TABLE table=test-workload [id=1345b2a336c84ec2b1de62853e7fa3f6]), partition=RANGE (key) PARTITION 1073741820 <= VALUES < 1252698790
I20240503 23:26:24.441165 3746 tablet_service.cc:1467] Processing CreateTablet for tablet a14a12115b8d4fec947c6e223f1ef486 (DEFAULT_TABLE table=test-workload [id=1345b2a336c84ec2b1de62853e7fa3f6]), partition=RANGE (key) PARTITION 1431655760 <= VALUES < 1610612730
I20240503 23:26:24.446822 3896 tablet_service.cc:1467] Processing CreateTablet for tablet 1b9566be6c644cfca852ec6507fab893 (DEFAULT_TABLE table=test-workload [id=1345b2a336c84ec2b1de62853e7fa3f6]), partition=RANGE (key) PARTITION 1252698790 <= VALUES < 1431655760
I20240503 23:26:24.447050 3746 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a14a12115b8d4fec947c6e223f1ef486. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 23:26:24.447305 3896 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1b9566be6c644cfca852ec6507fab893. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 23:26:24.421034 3902 tablet_service.cc:1467] Processing CreateTablet for tablet 487ac5b9bf2d4de2b80a1b13387a2a77 (DEFAULT_TABLE table=test-workload [id=1345b2a336c84ec2b1de62853e7fa3f6]), partition=RANGE (key) PARTITION 178956970 <= VALUES < 357913940
I20240503 23:26:24.449321 3902 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 487ac5b9bf2d4de2b80a1b13387a2a77. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 23:26:24.453773 3826 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7181e4e8c0324a8eb095dde23651a1ab. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 23:26:24.421034 3828 tablet_service.cc:1467] Processing CreateTablet for tablet 487ac5b9bf2d4de2b80a1b13387a2a77 (DEFAULT_TABLE table=test-workload [id=1345b2a336c84ec2b1de62853e7fa3f6]), partition=RANGE (key) PARTITION 178956970 <= VALUES < 357913940
I20240503 23:26:24.425078 3899 tablet_service.cc:1467] Processing CreateTablet for tablet 55cf664d5e874c0d9b9adf8bdc9c43ed (DEFAULT_TABLE table=test-workload [id=1345b2a336c84ec2b1de62853e7fa3f6]), partition=RANGE (key) PARTITION 715827880 <= VALUES < 894784850
I20240503 23:26:24.425079 3824 tablet_service.cc:1467] Processing CreateTablet for tablet 60267cb4849d4823a0afb2380305905f (DEFAULT_TABLE table=test-workload [id=1345b2a336c84ec2b1de62853e7fa3f6]), partition=RANGE (key) PARTITION 894784850 <= VALUES < 1073741820
I20240503 23:26:24.457192 3895 tablet_service.cc:1467] Processing CreateTablet for tablet a14a12115b8d4fec947c6e223f1ef486 (DEFAULT_TABLE table=test-workload [id=1345b2a336c84ec2b1de62853e7fa3f6]), partition=RANGE (key) PARTITION 1431655760 <= VALUES < 1610612730
I20240503 23:26:24.457827 3895 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a14a12115b8d4fec947c6e223f1ef486. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 23:26:24.459695 3894 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a0e18d0975a54e3dbeb0a13cb226fefd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 23:26:24.425449 3897 tablet_service.cc:1467] Processing CreateTablet for tablet 98c9500acb43423b86f54ce7a4d98a72 (DEFAULT_TABLE table=test-workload [id=1345b2a336c84ec2b1de62853e7fa3f6]), partition=RANGE (key) PARTITION 1073741820 <= VALUES < 1252698790
I20240503 23:26:24.465148 3897 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 98c9500acb43423b86f54ce7a4d98a72. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 23:26:24.468932 3824 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 60267cb4849d4823a0afb2380305905f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 23:26:24.470734 3825 tablet_service.cc:1467] Processing CreateTablet for tablet d0c4430fc56644cb9f87deffe4fe9407 (DEFAULT_TABLE table=test-workload [id=1345b2a336c84ec2b1de62853e7fa3f6]), partition=RANGE (key) PARTITION 1789569700 <= VALUES < 1968526670
I20240503 23:26:24.471241 3825 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d0c4430fc56644cb9f87deffe4fe9407. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 23:26:24.472970 3899 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 55cf664d5e874c0d9b9adf8bdc9c43ed. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 23:26:24.473037 3829 tablet_service.cc:1467] Processing CreateTablet for tablet 059f6233ea514fe69b8c6504efabeff0 (DEFAULT_TABLE table=test-workload [id=1345b2a336c84ec2b1de62853e7fa3f6]), partition=RANGE (key) PARTITION 1968526670 <= VALUES
I20240503 23:26:24.473467 3829 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 059f6233ea514fe69b8c6504efabeff0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 23:26:24.476773 3828 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 487ac5b9bf2d4de2b80a1b13387a2a77. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 23:26:24.477102 3903 tablet_service.cc:1467] Processing CreateTablet for tablet 059f6233ea514fe69b8c6504efabeff0 (DEFAULT_TABLE table=test-workload [id=1345b2a336c84ec2b1de62853e7fa3f6]), partition=RANGE (key) PARTITION 1968526670 <= VALUES
I20240503 23:26:24.477559 3903 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 059f6233ea514fe69b8c6504efabeff0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 23:26:24.481501 3827 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 481cd0f8c8d147adac217915a53ff6d9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 23:26:24.484902 3754 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6039230222ff4e57929bed49cd56caf5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 23:26:24.489017 3748 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 98c9500acb43423b86f54ce7a4d98a72. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 23:26:24.500595 3752 tablet_service.cc:1467] Processing CreateTablet for tablet d0c4430fc56644cb9f87deffe4fe9407 (DEFAULT_TABLE table=test-workload [id=1345b2a336c84ec2b1de62853e7fa3f6]), partition=RANGE (key) PARTITION 1789569700 <= VALUES < 1968526670
I20240503 23:26:24.501534 3752 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d0c4430fc56644cb9f87deffe4fe9407. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 23:26:24.507925 3747 tablet_service.cc:1467] Processing CreateTablet for tablet 059f6233ea514fe69b8c6504efabeff0 (DEFAULT_TABLE table=test-workload [id=1345b2a336c84ec2b1de62853e7fa3f6]), partition=RANGE (key) PARTITION 1968526670 <= VALUES
I20240503 23:26:24.508872 3747 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 059f6233ea514fe69b8c6504efabeff0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 23:26:24.539666 3959 tablet_bootstrap.cc:492] T 487ac5b9bf2d4de2b80a1b13387a2a77 P 83ffd7184ec94302bd1ee74b0fab3e17: Bootstrap starting.
I20240503 23:26:24.545424 3958 tablet_bootstrap.cc:492] T 7181e4e8c0324a8eb095dde23651a1ab P a3eecbaec43d4954b1096bc3121e824c: Bootstrap starting.
I20240503 23:26:24.555338 3960 tablet_bootstrap.cc:492] T 98c9500acb43423b86f54ce7a4d98a72 P a2f9bf2e5d9b4fc19f804797bca4a3ff: Bootstrap starting.
I20240503 23:26:24.556421 3959 tablet_bootstrap.cc:654] T 487ac5b9bf2d4de2b80a1b13387a2a77 P 83ffd7184ec94302bd1ee74b0fab3e17: Neither blocks nor log segments found. Creating new log.
I20240503 23:26:24.557338 3958 tablet_bootstrap.cc:654] T 7181e4e8c0324a8eb095dde23651a1ab P a3eecbaec43d4954b1096bc3121e824c: Neither blocks nor log segments found. Creating new log.
I20240503 23:26:24.559342 3959 tablet_bootstrap.cc:492] T 487ac5b9bf2d4de2b80a1b13387a2a77 P 83ffd7184ec94302bd1ee74b0fab3e17: No bootstrap required, opened a new log
I20240503 23:26:24.559576 3959 ts_tablet_manager.cc:1367] T 487ac5b9bf2d4de2b80a1b13387a2a77 P 83ffd7184ec94302bd1ee74b0fab3e17: Time spent bootstrapping tablet: real 0.020s user 0.002s sys 0.003s
I20240503 23:26:24.560832 3959 raft_consensus.cc:348] T 487ac5b9bf2d4de2b80a1b13387a2a77 P 83ffd7184ec94302bd1ee74b0fab3e17 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83ffd7184ec94302bd1ee74b0fab3e17" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 32909 } } peers { permanent_uuid: "a2f9bf2e5d9b4fc19f804797bca4a3ff" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 46261 } } peers { permanent_uuid: "a3eecbaec43d4954b1096bc3121e824c" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 40649 } }
I20240503 23:26:24.561120 3959 raft_consensus.cc:374] T 487ac5b9bf2d4de2b80a1b13387a2a77 P 83ffd7184ec94302bd1ee74b0fab3e17 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 23:26:24.561264 3959 raft_consensus.cc:729] T 487ac5b9bf2d4de2b80a1b13387a2a77 P 83ffd7184ec94302bd1ee74b0fab3e17 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 83ffd7184ec94302bd1ee74b0fab3e17, State: Initialized, Role: FOLLOWER
I20240503 23:26:24.561630 3959 consensus_queue.cc:260] T 487ac5b9bf2d4de2b80a1b13387a2a77 P 83ffd7184ec94302bd1ee74b0fab3e17 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83ffd7184ec94302bd1ee74b0fab3e17" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 32909 } } peers { permanent_uuid: "a2f9bf2e5d9b4fc19f804797bca4a3ff" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 46261 } } peers { permanent_uuid: "a3eecbaec43d4954b1096bc3121e824c" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 40649 } }
I20240503 23:26:24.562489 3960 tablet_bootstrap.cc:654] T 98c9500acb43423b86f54ce7a4d98a72 P a2f9bf2e5d9b4fc19f804797bca4a3ff: Neither blocks nor log segments found. Creating new log.
I20240503 23:26:24.562858 3959 ts_tablet_manager.cc:1398] T 487ac5b9bf2d4de2b80a1b13387a2a77 P 83ffd7184ec94302bd1ee74b0fab3e17: Time spent starting tablet: real 0.003s user 0.001s sys 0.002s
I20240503 23:26:24.563297 3959 tablet_bootstrap.cc:492] T 481cd0f8c8d147adac217915a53ff6d9 P 83ffd7184ec94302bd1ee74b0fab3e17: Bootstrap starting.
I20240503 23:26:24.565331 3959 tablet_bootstrap.cc:654] T 481cd0f8c8d147adac217915a53ff6d9 P 83ffd7184ec94302bd1ee74b0fab3e17: Neither blocks nor log segments found. Creating new log.
I20240503 23:26:24.566054 3960 tablet_bootstrap.cc:492] T 98c9500acb43423b86f54ce7a4d98a72 P a2f9bf2e5d9b4fc19f804797bca4a3ff: No bootstrap required, opened a new log
I20240503 23:26:24.566278 3960 ts_tablet_manager.cc:1367] T 98c9500acb43423b86f54ce7a4d98a72 P a2f9bf2e5d9b4fc19f804797bca4a3ff: Time spent bootstrapping tablet: real 0.011s user 0.000s sys 0.006s
I20240503 23:26:24.567239 3960 raft_consensus.cc:348] T 98c9500acb43423b86f54ce7a4d98a72 P a2f9bf2e5d9b4fc19f804797bca4a3ff [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83ffd7184ec94302bd1ee74b0fab3e17" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 32909 } } peers { permanent_uuid: "a2f9bf2e5d9b4fc19f804797bca4a3ff" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 46261 } } peers { permanent_uuid: "a3eecbaec43d4954b1096bc3121e824c" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 40649 } }
I20240503 23:26:24.567554 3960 raft_consensus.cc:374] T 98c9500acb43423b86f54ce7a4d98a72 P a2f9bf2e5d9b4fc19f804797bca4a3ff [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 23:26:24.567637 3960 raft_consensus.cc:729] T 98c9500acb43423b86f54ce7a4d98a72 P a2f9bf2e5d9b4fc19f804797bca4a3ff [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2f9bf2e5d9b4fc19f804797bca4a3ff, State: Initialized, Role: FOLLOWER
I20240503 23:26:24.567905 3960 consensus_queue.cc:260] T 98c9500acb43423b86f54ce7a4d98a72 P a2f9bf2e5d9b4fc19f804797bca4a3ff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83ffd7184ec94302bd1ee74b0fab3e17" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 32909 } } peers { permanent_uuid: "a2f9bf2e5d9b4fc19f804797bca4a3ff" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 46261 } } peers { permanent_uuid: "a3eecbaec43d4954b1096bc3121e824c" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 40649 } }
I20240503 23:26:24.568130 3958 tablet_bootstrap.cc:492] T 7181e4e8c0324a8eb095dde23651a1ab P a3eecbaec43d4954b1096bc3121e824c: No bootstrap required, opened a new log
I20240503 23:26:24.568447 3958 ts_tablet_manager.cc:1367] T 7181e4e8c0324a8eb095dde23651a1ab P a3eecbaec43d4954b1096bc3121e824c: Time spent bootstrapping tablet: real 0.024s user 0.002s sys 0.007s
I20240503 23:26:24.568866 3960 ts_tablet_manager.cc:1398] T 98c9500acb43423b86f54ce7a4d98a72 P a2f9bf2e5d9b4fc19f804797bca4a3ff: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240503 23:26:24.569278 3960 tablet_bootstrap.cc:492] T 6039230222ff4e57929bed49cd56caf5 P a2f9bf2e5d9b4fc19f804797bca4a3ff: Bootstrap starting.
I20240503 23:26:24.569252 3958 raft_consensus.cc:348] T 7181e4e8c0324a8eb095dde23651a1ab P a3eecbaec43d4954b1096bc3121e824c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83ffd7184ec94302bd1ee74b0fab3e17" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 32909 } } peers { permanent_uuid: "a2f9bf2e5d9b4fc19f804797bca4a3ff" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 46261 } } peers { permanent_uuid: "a3eecbaec43d4954b1096bc3121e824c" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 40649 } }
I20240503 23:26:24.569422 3958 raft_consensus.cc:374] T 7181e4e8c0324a8eb095dde23651a1ab P a3eecbaec43d4954b1096bc3121e824c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 23:26:24.569487 3958 raft_consensus.cc:729] T 7181e4e8c0324a8eb095dde23651a1ab P a3eecbaec43d4954b1096bc3121e824c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a3eecbaec43d4954b1096bc3121e824c, State: Initialized, Role: FOLLOWER
I20240503 23:26:24.569738 3958 consensus_queue.cc:260] T 7181e4e8c0324a8eb095dde23651a1ab P a3eecbaec43d4954b1096bc3121e824c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83ffd7184ec94302bd1ee74b0fab3e17" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 32909 } } peers { permanent_uuid: "a2f9bf2e5d9b4fc19f804797bca4a3ff" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 46261 } } peers { permanent_uuid: "a3eecbaec43d4954b1096bc3121e824c" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 40649 } }
I20240503 23:26:24.571045 3959 tablet_bootstrap.cc:492] T 481cd0f8c8d147adac217915a53ff6d9 P 83ffd7184ec94302bd1ee74b0fab3e17: No bootstrap required, opened a new log
I20240503 23:26:24.571218 3959 ts_tablet_manager.cc:1367] T 481cd0f8c8d147adac217915a53ff6d9 P 83ffd7184ec94302bd1ee74b0fab3e17: Time spent bootstrapping tablet: real 0.008s user 0.004s sys 0.000s
I20240503 23:26:24.571995 3959 raft_consensus.cc:348] T 481cd0f8c8d147adac217915a53ff6d9 P 83ffd7184ec94302bd1ee74b0fab3e17 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83ffd7184ec94302bd1ee74b0fab3e17" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 32909 } } peers { permanent_uuid: "a2f9bf2e5d9b4fc19f804797bca4a3ff" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 46261 } } peers { permanent_uuid: "a3eecbaec43d4954b1096bc3121e824c" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 40649 } }
I20240503 23:26:24.572198 3959 raft_consensus.cc:374] T 481cd0f8c8d147adac217915a53ff6d9 P 83ffd7184ec94302bd1ee74b0fab3e17 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 23:26:24.572261 3959 raft_consensus.cc:729] T 481cd0f8c8d147adac217915a53ff6d9 P 83ffd7184ec94302bd1ee74b0fab3e17 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 83ffd7184ec94302bd1ee74b0fab3e17, State: Initialized, Role: FOLLOWER
I20240503 23:26:24.572455 3960 tablet_bootstrap.cc:654] T 6039230222ff4e57929bed49cd56caf5 P a2f9bf2e5d9b4fc19f804797bca4a3ff: Neither blocks nor log segments found. Creating new log.
I20240503 23:26:24.572481 3959 consensus_queue.cc:260] T 481cd0f8c8d147adac217915a53ff6d9 P 83ffd7184ec94302bd1ee74b0fab3e17 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83ffd7184ec94302bd1ee74b0fab3e17" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 32909 } } peers { permanent_uuid: "a2f9bf2e5d9b4fc19f804797bca4a3ff" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 46261 } } peers { permanent_uuid: "a3eecbaec43d4954b1096bc3121e824c" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 40649 } }
I20240503 23:26:24.573104 3959 ts_tablet_manager.cc:1398] T 481cd0f8c8d147adac217915a53ff6d9 P 83ffd7184ec94302bd1ee74b0fab3e17: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 23:26:24.573421 3959 tablet_bootstrap.cc:492] T a14a12115b8d4fec947c6e223f1ef486 P 83ffd7184ec94302bd1ee74b0fab3e17: Bootstrap starting.
I20240503 23:26:24.574775 3958 ts_tablet_manager.cc:1398] T 7181e4e8c0324a8eb095dde23651a1ab P a3eecbaec43d4954b1096bc3121e824c: Time spent starting tablet: real 0.006s user 0.000s sys 0.005s
I20240503 23:26:24.575109 3958 tablet_bootstrap.cc:492] T 481cd0f8c8d147adac217915a53ff6d9 P a3eecbaec43d4954b1096bc3121e824c: Bootstrap starting.
I20240503 23:26:24.575271 3959 tablet_bootstrap.cc:654] T a14a12115b8d4fec947c6e223f1ef486 P 83ffd7184ec94302bd1ee74b0fab3e17: Neither blocks nor log segments found. Creating new log.
I20240503 23:26:24.575577 3960 tablet_bootstrap.cc:492] T 6039230222ff4e57929bed49cd56caf5 P a2f9bf2e5d9b4fc19f804797bca4a3ff: No bootstrap required, opened a new log
I20240503 23:26:24.575716 3960 ts_tablet_manager.cc:1367] T 6039230222ff4e57929bed49cd56caf5 P a2f9bf2e5d9b4fc19f804797bca4a3ff: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20240503 23:26:24.576915 3960 raft_consensus.cc:348] T 6039230222ff4e57929bed49cd56caf5 P a2f9bf2e5d9b4fc19f804797bca4a3ff [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3eecbaec43d4954b1096bc3121e824c" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 40649 } } peers { permanent_uuid: "a2f9bf2e5d9b4fc19f804797bca4a3ff" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 46261 } } peers { permanent_uuid: "83ffd7184ec94302bd1ee74b0fab3e17" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 32909 } }
I20240503 23:26:24.577204 3960 raft_consensus.cc:374] T 6039230222ff4e57929bed49cd56caf5 P a2f9bf2e5d9b4fc19f804797bca4a3ff [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 23:26:24.577294 3960 raft_consensus.cc:729] T 6039230222ff4e57929bed49cd56caf5 P a2f9bf2e5d9b4fc19f804797bca4a3ff [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2f9bf2e5d9b4fc19f804797bca4a3ff, State: Initialized, Role: FOLLOWER
I20240503 23:26:24.577412 3958 tablet_bootstrap.cc:654] T 481cd0f8c8d147adac217915a53ff6d9 P a3eecbaec43d4954b1096bc3121e824c: Neither blocks nor log segments found. Creating new log.
I20240503 23:26:24.577561 3960 consensus_queue.cc:260] T 6039230222ff4e57929bed49cd56caf5 P a2f9bf2e5d9b4fc19f804797bca4a3ff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3eecbaec43d4954b1096bc3121e824c" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 40649 } } peers { permanent_uuid: "a2f9bf2e5d9b4fc19f804797bca4a3ff" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 46261 } } peers { permanent_uuid: "83ffd7184ec94302bd1ee74b0fab3e17" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 32909 } }
I20240503 23:26:24.578955 3960 ts_tablet_manager.cc:1398] T 6039230222ff4e57929bed49cd56caf5 P a2f9bf2e5d9b4fc19f804797bca4a3ff: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20240503 23:26:24.579285 3960 tablet_bootstrap.cc:492] T 55cf664d5e874c0d9b9adf8bdc9c43ed P a2f9bf2e5d9b4fc19f804797bca4a3ff: Bootstrap starting.
I20240503 23:26:24.581398 3960 tablet_bootstrap.cc:654] T 55cf664d5e874c0d9b9adf8bdc9c43ed P a2f9bf2e5d9b4fc19f804797bca4a3ff: Neither blocks nor log segments found. Creating new log.
I20240503 23:26:24.582840 3959 tablet_bootstrap.cc:492] T a14a12115b8d4fec947c6e223f1ef486 P 83ffd7184ec94302bd1ee74b0fab3e17: No bootstrap required, opened a new log
I20240503 23:26:24.583135 3959 ts_tablet_manager.cc:1367] T a14a12115b8d4fec947c6e223f1ef486 P 83ffd7184ec94302bd1ee74b0fab3e17: Time spent bootstrapping tablet: real 0.010s user 0.003s sys 0.001s
I20240503 23:26:24.584347 3959 raft_consensus.cc:348] T a14a12115b8d4fec947c6e223f1ef486 P 83ffd7184ec94302bd1ee74b0fab3e17 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83ffd7184ec94302bd1ee74b0fab3e17" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 32909 } } peers { permanent_uuid: "a2f9bf2e5d9b4fc19f804797bca4a3ff" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 46261 } } peers { permanent_uuid: "a3eecbaec43d4954b1096bc3121e824c" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 40649 } }
I20240503 23:26:24.584825 3959 raft_consensus.cc:374] T a14a12115b8d4fec947c6e223f1ef486 P 83ffd7184ec94302bd1ee74b0fab3e17 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 23:26:24.584949 3959 raft_consensus.cc:729] T a14a12115b8d4fec947c6e223f1ef486 P 83ffd7184ec94302bd1ee74b0fab3e17 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 83ffd7184ec94302bd1ee74b0fab3e17, State: Initialized, Role: FOLLOWER
I20240503 23:26:24.585297 3960 tablet_bootstrap.cc:492] T 55cf664d5e874c0d9b9adf8bdc9c43ed P a2f9bf2e5d9b4fc19f804797bca4a3ff: No bootstrap required, opened a new log
I20240503 23:26:24.585268 3959 consensus_queue.cc:260] T a14a12115b8d4fec947c6e223f1ef486 P 83ffd7184ec94302bd1ee74b0fab3e17 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83ffd7184ec94302bd1ee74b0fab3e17" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 32909 } } peers { permanent_uuid: "a2f9bf2e5d9b4fc19f804797bca4a3ff" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 46261 } } peers { permanent_uuid: "a3eecbaec43d4954b1096bc3121e824c" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 40649 } }
I20240503 23:26:24.585475 3960 ts_tablet_manager.cc:1367] T 55cf664d5e874c0d9b9adf8bdc9c43ed P a2f9bf2e5d9b4fc19f804797bca4a3ff: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.001s
I20240503 23:26:24.585768 3958 tablet_bootstrap.cc:492] T 481cd0f8c8d147adac217915a53ff6d9 P a3eecbaec43d4954b1096bc3121e824c: No bootstrap required, opened a new log
I20240503 23:26:24.585989 3958 ts_tablet_manager.cc:1367] T 481cd0f8c8d147adac217915a53ff6d9 P a3eecbaec43d4954b1096bc3121e824c: Time spent bootstrapping tablet: real 0.011s user 0.001s sys 0.009s
I20240503 23:26:24.586046 3959 ts_tablet_manager.cc:1398] T a14a12115b8d4fec947c6e223f1ef486 P 83ffd7184ec94302bd1ee74b0fab3e17: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240503 23:26:24.586503 3959 tablet_bootstrap.cc:492] T 1b9566be6c644cfca852ec6507fab893 P 83ffd7184ec94302bd1ee74b0fab3e17: Bootstrap starting.
I20240503 23:26:24.586853 3958 raft_consensus.cc:348] T 481cd0f8c8d147adac217915a53ff6d9 P a3eecbaec43d4954b1096bc3121e824c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83ffd7184ec94302bd1ee74b0fab3e17" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 32909 } } peers { permanent_uuid: "a2f9bf2e5d9b4fc19f804797bca4a3ff" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 46261 } } peers { permanent_uuid: "a3eecbaec43d4954b1096bc3121e824c" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 40649 } }
I20240503 23:26:24.587119 3958 raft_consensus.cc:374] T 481cd0f8c8d147adac217915a53ff6d9 P a3eecbaec43d4954b1096bc3121e824c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 23:26:24.587260 3958 raft_consensus.cc:729] T 481cd0f8c8d147adac217915a53ff6d9 P a3eecbaec43d4954b1096bc3121e824c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a3eecbaec43d4954b1096bc3121e824c, State: Initialized, Role: FOLLOWER
I20240503 23:26:24.587615 3958 consensus_queue.cc:260] T 481cd0f8c8d147adac217915a53ff6d9 P a3eecbaec43d4954b1096bc3121e824c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83ffd7184ec94302bd1ee74b0fab3e17" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 32909 } } peers { permanent_uuid: "a2f9bf2e5d9b4fc19f804797bca4a3ff" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 46261 } } peers { permanent_uuid: "a3eecbaec43d4954b1096bc3121e824c" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 40649 } }
I20240503 23:26:24.588240 3958 ts_tablet_manager.cc:1398] T 481cd0f8c8d147adac217915a53ff6d9 P a3eecbaec43d4954b1096bc3121e824c: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240503 23:26:24.586360 3960 raft_consensus.cc:348] T 55cf664d5e874c0d9b9adf8bdc9c43ed P a2f9bf2e5d9b4fc19f804797bca4a3ff [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83ffd7184ec94302bd1ee74b0fab3e17" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 32909 } } peers { permanent_uuid: "a2f9bf2e5d9b4fc19f804797bca4a3ff" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 46261 } } peers { permanent_uuid: "a3eecbaec43d4954b1096bc3121e824c" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 40649 } }
I20240503 23:26:24.588671 3960 raft_consensus.cc:374] T 55cf664d5e874c0d9b9adf8bdc9c43ed P a2f9bf2e5d9b4fc19f804797bca4a3ff [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 23:26:24.588805 3958 tablet_bootstrap.cc:492] T 6039230222ff4e57929bed49cd56caf5 P a3eecbaec43d4954b1096bc3121e824c: Bootstrap starting.
I20240503 23:26:24.588980 3959 tablet_bootstrap.cc:654] T 1b9566be6c644cfca852ec6507fab893 P 83ffd7184ec94302bd1ee74b0fab3e17: Neither blocks nor log segments found. Creating new log.
I20240503 23:26:24.591665 3958 tablet_bootstrap.cc:654] T 6039230222ff4e57929bed49cd56caf5 P a3eecbaec43d4954b1096bc3121e824c: Neither blocks nor log segments found. Creating new log.
I20240503 23:26:24.588783 3960 raft_consensus.cc:729] T 55cf664d5e874c0d9b9adf8bdc9c43ed P a2f9bf2e5d9b4fc19f804797bca4a3ff [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2f9bf2e5d9b4fc19f804797bca4a3ff, State: Initialized, Role: FOLLOWER
I20240503 23:26:24.592598 3959 tablet_bootstrap.cc:492] T 1b9566be6c644cfca852ec6507fab893 P 83ffd7184ec94302bd1ee74b0fab3e17: No bootstrap required, opened a new log
I20240503 23:26:24.592978 3959 ts_tablet_manager.cc:1367] T 1b9566be6c644cfca852ec6507fab893 P 83ffd7184ec94302bd1ee74b0fab3e17: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.001s
I20240503 23:26:24.593036 3960 consensus_queue.cc:260] T 55cf664d5e874c0d9b9adf8bdc9c43ed P a2f9bf2e5d9b4fc19f804797bca4a3ff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83ffd7184ec94302bd1ee74b0fab3e17" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 32909 } } peers { permanent_uuid: "a2f9bf2e5d9b4fc19f804797bca4a3ff" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 46261 } } peers { permanent_uuid: "a3eecbaec43d4954b1096bc3121e824c" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 40649 } }
I20240503 23:26:24.593876 3960 ts_tablet_manager.cc:1398] T 55cf664d5e874c0d9b9adf8bdc9c43ed P a2f9bf2e5d9b4fc19f804797bca4a3ff: Time spent starting tablet: real 0.008s user 0.002s sys 0.000s
I20240503 23:26:24.594286 3960 tablet_bootstrap.cc:492] T 7181e4e8c0324a8eb095dde23651a1ab P a2f9bf2e5d9b4fc19f804797bca4a3ff: Bootstrap starting.
I20240503 23:26:24.594257 3959 raft_consensus.cc:348] T 1b9566be6c644cfca852ec6507fab893 P 83ffd7184ec94302bd1ee74b0fab3e17 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83ffd7184ec94302bd1ee74b0fab3e17" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 32909 } } peers { permanent_uuid: "a2f9bf2e5d9b4fc19f804797bca4a3ff" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 46261 } } peers { permanent_uuid: "a3eecbaec43d4954b1096bc3121e824c" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 40649 } }
I20240503 23:26:24.594753 3959 raft_consensus.cc:374] T 1b9566be6c644cfca852ec6507fab893 P 83ffd7184ec94302bd1ee74b0fab3e17 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 23:26:24.594923 3959 raft_consensus.cc:729] T 1b9566be6c644cfca852ec6507fab893 P 83ffd7184ec94302bd1ee74b0fab3e17 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 83ffd7184ec94302bd1ee74b0fab3e17, State: Initialized, Role: FOLLOWER
I20240503 23:26:24.595921 3959 consensus_queue.cc:260] T 1b9566be6c644cfca852ec6507fab893 P 83ffd7184ec94302bd1ee74b0fab3e17 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83ffd7184ec94302bd1ee74b0fab3e17" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 32909 } } peers { permanent_uuid: "a2f9bf2e5d9b4fc19f804797bca4a3ff" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 46261 } } peers { permanent_uuid: "a3eecbaec43d4954b1096bc3121e824c" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 40649 } }
I20240503 23:26:24.596462 3960 tablet_bootstrap.cc:654] T 7181e4e8c0324a8eb095dde23651a1ab P a2f9bf2e5d9b4fc19f804797bca4a3ff: Neither blocks nor log segments found. Creating new log.
I20240503 23:26:24.598517 3959 ts_tablet_manager.cc:1398] T 1b9566be6c644cfca852ec6507fab893 P 83ffd7184ec94302bd1ee74b0fab3e17: Time spent starting tablet: real 0.005s user 0.000s sys 0.002s
I20240503 23:26:24.599098 3959 tablet_bootstrap.cc:492] T 7181e4e8c0324a8eb095dde23651a1ab P 83ffd7184ec94302bd1ee74b0fab3e17: Bootstrap starting.
I20240503 23:26:24.602195 3960 tablet_bootstrap.cc:492] T 7181e4e8c0324a8eb095dde23651a1ab P a2f9bf2e5d9b4fc19f804797bca4a3ff: No bootstrap required, opened a new log
I20240503 23:26:24.602458 3960 ts_tablet_manager.cc:1367] T 7181e4e8c0324a8eb095dde23651a1ab P a2f9bf2e5d9b4fc19f804797bca4a3ff: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.006s
I20240503 23:26:24.603251 3960 raft_consensus.cc:348] T 7181e4e8c0324a8eb095dde23651a1ab P a2f9bf2e5d9b4fc19f804797bca4a3ff [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83ffd7184ec94302bd1ee74b0fab3e17" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 32909 } } peers { permanent_uuid: "a2f9bf2e5d9b4fc19f804797bca4a3ff" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 46261 } } peers { permanent_uuid: "a3eecbaec43d4954b1096bc3121e824c" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 40649 } }
I20240503 23:26:24.603466 3960 raft_consensus.cc:374] T 7181e4e8c0324a8eb095dde23651a1ab P a2f9bf2e5d9b4fc19f804797bca4a3ff [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 23:26:24.603586 3960 raft_consensus.cc:729] T 7181e4e8c0324a8eb095dde23651a1ab P a2f9bf2e5d9b4fc19f804797bca4a3ff [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2f9bf2e5d9b4fc19f804797bca4a3ff, State: Initialized, Role: FOLLOWER
I20240503 23:26:24.603837 3960 consensus_queue.cc:260] T 7181e4e8c0324a8eb095dde23651a1ab P a2f9bf2e5d9b4fc19f804797bca4a3ff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83ffd7184ec94302bd1ee74b0fab3e17" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 32909 } } peers { permanent_uuid: "a2f9bf2e5d9b4fc19f804797bca4a3ff" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 46261 } } peers { permanent_uuid: "a3eecbaec43d4954b1096bc3121e824c" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 40649 } }
I20240503 23:26:24.604307 3959 tablet_bootstrap.cc:654] T 7181e4e8c0324a8eb095dde23651a1ab P 83ffd7184ec94302bd1ee74b0fab3e17: Neither blocks nor log segments found. Creating new log.
I20240503 23:26:24.604379 3960 ts_tablet_manager.cc:1398] T 7181e4e8c0324a8eb095dde23651a1ab P a2f9bf2e5d9b4fc19f804797bca4a3ff: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240503 23:26:24.604825 3960 tablet_bootstrap.cc:492] T d0c4430fc56644cb9f87deffe4fe9407 P a2f9bf2e5d9b4fc19f804797bca4a3ff: Bootstrap starting.
I20240503 23:26:24.607944 3960 tablet_bootstrap.cc:654] T d0c4430fc56644cb9f87deffe4fe9407 P a2f9bf2e5d9b4fc19f804797bca4a3ff: Neither blocks nor log segments found. Creating new log.
I20240503 23:26:24.608711 3958 tablet_bootstrap.cc:492] T 6039230222ff4e57929bed49cd56caf5 P a3eecbaec43d4954b1096bc3121e824c: No bootstrap required, opened a new log
I20240503 23:26:24.609067 3958 ts_tablet_manager.cc:1367] T 6039230222ff4e57929bed49cd56caf5 P a3eecbaec43d4954b1096bc3121e824c: Time spent bootstrapping tablet: real 0.021s user 0.014s sys 0.006s
I20240503 23:26:24.610399 3958 raft_consensus.cc:348] T 6039230222ff4e57929bed49cd56caf5 P a3eecbaec43d4954b1096bc3121e824c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3eecbaec43d4954b1096bc3121e824c" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 40649 } } peers { permanent_uuid: "a2f9bf2e5d9b4fc19f804797bca4a3ff" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 46261 } } peers { permanent_uuid: "83ffd7184ec94302bd1ee74b0fab3e17" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 32909 } }
I20240503 23:26:24.610816 3958 raft_consensus.cc:374] T 6039230222ff4e57929bed49cd56caf5 P a3eecbaec43d4954b1096bc3121e824c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 23:26:24.610975 3958 raft_consensus.cc:729] T 6039230222ff4e57929bed49cd56caf5 P a3eecbaec43d4954b1096bc3121e824c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a3eecbaec43d4954b1096bc3121e824c, State: Initialized, Role: FOLLOWER
I20240503 23:26:24.611377 3958 consensus_queue.cc:260] T 6039230222ff4e57929bed49cd56caf5 P a3eecbaec43d4954b1096bc3121e824c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3eecbaec43d4954b1096bc3121e824c" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 40649 } } peers { permanent_uuid: "a2f9bf2e5d9b4fc19f804797bca4a3ff" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 46261 } } peers { permanent_uuid: "83ffd7184ec94302bd1ee74b0fab3e17" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 32909 } }
I20240503 23:26:24.612172 3958 ts_tablet_manager.cc:1398] T 6039230222ff4e57929bed49cd56caf5 P a3eecbaec43d4954b1096bc3121e824c: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240503 23:26:24.612236 3959 tablet_bootstrap.cc:492] T 7181e4e8c0324a8eb095dde23651a1ab P 83ffd7184ec94302bd1ee74b0fab3e17: No bootstrap required, opened a new log
I20240503 23:26:24.612457 3959 ts_tablet_manager.cc:1367] T 7181e4e8c0324a8eb095dde23651a1ab P 83ffd7184ec94302bd1ee74b0fab3e17: Time spent bootstrapping tablet: real 0.013s user 0.004s sys 0.004s
I20240503 23:26:24.612630 3958 tablet_bootstrap.cc:492] T 059f6233ea514fe69b8c6504efabeff0 P a3eecbaec43d4954b1096bc3121e824c: Bootstrap starting.
I20240503 23:26:24.613453 3959 raft_consensus.cc:348] T 7181e4e8c0324a8eb095dde23651a1ab P 83ffd7184ec94302bd1ee74b0fab3e17 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83ffd7184ec94302bd1ee74b0fab3e17" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 32909 } } peers { permanent_uuid: "a2f9bf2e5d9b4fc19f804797bca4a3ff" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 46261 } } peers { permanent_uuid: "a3eecbaec43d4954b1096bc3121e824c" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 40649 } }
I20240503 23:26:24.613751 3959 raft_consensus.cc:374] T 7181e4e8c0324a8eb095dde23651a1ab P 83ffd7184ec94302bd1ee74b0fab3e17 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 23:26:24.613860 3959 raft_consensus.cc:729] T 7181e4e8c0324a8eb095dde23651a1ab P 83ffd7184ec94302bd1ee74b0fab3e17 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 83ffd7184ec94302bd1ee74b0fab3e17, State: Initialized, Role: FOLLOWER
I20240503 23:26:24.614223 3959 consensus_queue.cc:260] T 7181e4e8c0324a8eb095dde23651a1ab P 83ffd7184ec94302bd1ee74b0fab3e17 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83ffd7184ec94302bd1ee74b0fab3e17" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 32909 } } peers { permanent_uuid: "a2f9bf2e5d9b4fc19f804797bca4a3ff" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 46261 } } peers { permanent_uuid: "a3eecbaec43d4954b1096bc3121e824c" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 40649 } }
I20240503 23:26:24.614874 3958 tablet_bootstrap.cc:654] T 059f6233ea514fe69b8c6504efabeff0 P a3eecbaec43d4954b1096bc3121e824c: Neither blocks nor log segments found. Creating new log.
I20240503 23:26:24.615010 3959 ts_tablet_manager.cc:1398] T 7181e4e8c0324a8eb095dde23651a1ab P 83ffd7184ec94302bd1ee74b0fab3e17: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 23:26:24.615415 3959 tablet_bootstrap.cc:492] T 55cf664d5e874c0d9b9adf8bdc9c43ed P 83ffd7184ec94302bd1ee74b0fab3e17: Bootstrap starting.
I20240503 23:26:24.618178 3959 tablet_bootstrap.cc:654] T 55cf664d5e874c0d9b9adf8bdc9c43ed P 83ffd7184ec94302bd1ee74b0fab3e17: Neither blocks nor log segments found. Creating new log.
I20240503 23:26:24.621301 3966 raft_consensus.cc:482] T 6039230222ff4e57929bed49cd56caf5 P a2f9bf2e5d9b4fc19f804797bca4a3ff [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 23:26:24.621706 3960 tablet_bootstrap.cc:492] T d0c4430fc56644cb9f87deffe4fe9407 P a2f9bf2e5d9b4fc19f804797bca4a3ff: No bootstrap required, opened a new log
I20240503 23:26:24.621547 3966 raft_consensus.cc:504] T 6039230222ff4e57929bed49cd56caf5 P a2f9bf2e5d9b4fc19f804797bca4a3ff [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3eecbaec43d4954b1096bc3121e824c" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 40649 } } peers { permanent_uuid: "a2f9bf2e5d9b4fc19f804797bca4a3ff" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 46261 } } peers { permanent_uuid: "83ffd7184ec94302bd1ee74b0fab3e17" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 32909 } }
I20240503 23:26:24.621907 3960 ts_tablet_manager.cc:1367] T d0c4430fc56644cb9f87deffe4fe9407 P a2f9bf2e5d9b4fc19f804797bca4a3ff: Time spent bootstrapping tablet: real 0.017s user 0.006s sys 0.001s
I20240503 23:26:24.622237 3958 tablet_bootstrap.cc:492] T 059f6233ea514fe69b8c6504efabeff0 P a3eecbaec43d4954b1096bc3121e824c: No bootstrap required, opened a new log
I20240503 23:26:24.622372 3958 ts_tablet_manager.cc:1367] T 059f6233ea514fe69b8c6504efabeff0 P a3eecbaec43d4954b1096bc3121e824c: Time spent bootstrapping tablet: real 0.010s user 0.009s sys 0.000s
I20240503 23:26:24.622812 3966 leader_election.cc:288] T 6039230222ff4e57929bed49cd56caf5 P a2f9bf2e5d9b4fc19f804797bca4a3ff [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a3eecbaec43d4954b1096bc3121e824c (127.0.100.3:40649), 83ffd7184ec94302bd1ee74b0fab3e17 (127.0.100.1:32909)
I20240503 23:26:24.622860 3960 raft_consensus.cc:348] T d0c4430fc56644cb9f87deffe4fe9407 P a2f9bf2e5d9b4fc19f804797bca4a3ff [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83ffd7184ec94302bd1ee74b0fab3e17" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 32909 } } peers { permanent_uuid: "a2f9bf2e5d9b4fc19f804797bca4a3ff" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 46261 } } peers { permanent_uuid: "a3eecbaec43d4954b1096bc3121e824c" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 40649 } }
I20240503 23:26:24.622959 3958 raft_consensus.cc:348] T 059f6233ea514fe69b8c6504efabeff0 P a3eecbaec43d4954b1096bc3121e824c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83ffd7184ec94302bd1ee74b0fab3e17" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 32909 } } peers { permanent_uuid: "a2f9bf2e5d9b4fc19f804797bca4a3ff" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 46261 } } peers { permanent_uuid: "a3eecbaec43d4954b1096bc3121e824c" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 40649 } }
I20240503 23:26:24.623114 3958 raft_consensus.cc:374] T 059f6233ea514fe69b8c6504efabeff0 P a3eecbaec43d4954b1096bc3121e824c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 23:26:24.623107 3960 raft_consensus.cc:374] T d0c4430fc56644cb9f87deffe4fe9407 P a2f9bf2e5d9b4fc19f804797bca4a3ff [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 23:26:24.623236 3958 raft_consensus.cc:729] T 059f6233ea514fe69b8c6504efabeff0 P a3eecbaec43d4954b1096bc3121e824c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a3eecbaec43d4954b1096bc3121e824c, State: Initialized, Role: FOLLOWER
I20240503 23:26:24.623250 3960 raft_consensus.cc:729] T d0c4430fc56644cb9f87deffe4fe9407 P a2f9bf2e5d9b4fc19f804797bca4a3ff [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2f9bf2e5d9b4fc19f804797bca4a3ff, State: Initialized, Role: FOLLOWER
I20240503 23:26:24.623504 3958 consensus_queue.cc:260] T 059f6233ea514fe69b8c6504efabeff0 P a3eecbaec43d4954b1096bc3121e824c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83ffd7184ec94302bd1ee74b0fab3e17" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 32909 } } peers { permanent_uuid: "a2f9bf2e5d9b4fc19f804797bca4a3ff" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 46261 } } peers { permanent_uuid: "a3eecbaec43d4954b1096bc3121e824c" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 40649 } }
I20240503 23:26:24.623500 3960 consensus_queue.cc:260] T d0c4430fc56644cb9f87deffe4fe9407 P a2f9bf2e5d9b4fc19f804797bca4a3ff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83ffd7184ec94302bd1ee74b0fab3e17" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 32909 } } peers { permanent_uuid: "a2f9bf2e5d9b4fc19f804797bca4a3ff" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 46261 } } peers { permanent_uuid: "a3eecbaec43d4954b1096bc3121e824c" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 40649 } }
I20240503 23:26:24.624110 3958 ts_tablet_manager.cc:1398] T 059f6233ea514fe69b8c6504efabeff0 P a3eecbaec43d4954b1096bc3121e824c: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 23:26:24.624156 3960 ts_tablet_manager.cc:1398] T d0c4430fc56644cb9f87deffe4fe9407 P a2f9bf2e5d9b4fc19f804797bca4a3ff: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240503 23:26:24.624521 3960 tablet_bootstrap.cc:492] T 059f6233ea514fe69b8c6504efabeff0 P a2f9bf2e5d9b4fc19f804797bca4a3ff: Bootstrap starting.
I20240503 23:26:24.626899 3960 tablet_bootstrap.cc:654] T 059f6233ea514fe69b8c6504efabeff0 P a2f9bf2e5d9b4fc19f804797bca4a3ff: Neither blocks nor log segments found. Creating new log.
I20240503 23:26:24.627647 3959 tablet_bootstrap.cc:492] T 55cf664d5e874c0d9b9adf8bdc9c43ed P 83ffd7184ec94302bd1ee74b0fab3e17: No bootstrap required, opened a new log
I20240503 23:26:24.627887 3959 ts_tablet_manager.cc:1367] T 55cf664d5e874c0d9b9adf8bdc9c43ed P 83ffd7184ec94302bd1ee74b0fab3e17: Time spent bootstrapping tablet: real 0.013s user 0.001s sys 0.004s
I20240503 23:26:24.629140 3959 raft_consensus.cc:348] T 55cf664d5e874c0d9b9adf8bdc9c43ed P 83ffd7184ec94302bd1ee74b0fab3e17 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83ffd7184ec94302bd1ee74b0fab3e17" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 32909 } } peers { permanent_uuid: "a2f9bf2e5d9b4fc19f804797bca4a3ff" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 46261 } } peers { permanent_uuid: "a3eecbaec43d4954b1096bc3121e824c" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 40649 } }
I20240503 23:26:24.629528 3959 raft_consensus.cc:374] T 55cf664d5e874c0d9b9adf8bdc9c43ed P 83ffd7184ec94302bd1ee74b0fab3e17 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 23:26:24.629654 3959 raft_consensus.cc:729] T 55cf664d5e874c0d9b9adf8bdc9c43ed P 83ffd7184ec94302bd1ee74b0fab3e17 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 83ffd7184ec94302bd1ee74b0fab3e17, State: Initialized, Role: FOLLOWER
I20240503 23:26:24.630018 3959 consensus_queue.cc:260] T 55cf664d5e874c0d9b9adf8bdc9c43ed P 83ffd7184ec94302bd1ee74b0fab3e17 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83ffd7184ec94302bd1ee74b0fab3e17" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 32909 } } peers { permanent_uuid: "a2f9bf2e5d9b4fc19f804797bca4a3ff" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 46261 } } peers { permanent_uuid: "a3eecbaec43d4954b1096bc3121e824c" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 40649 } }
I20240503 23:26:24.630574 3958 tablet_bootstrap.cc:492] T a14a12115b8d4fec947c6e223f1ef486 P a3eecbaec43d4954b1096bc3121e824c: Bootstrap starting.
I20240503 23:26:24.630873 3959 ts_tablet_manager.cc:1398] T 55cf664d5e874c0d9b9adf8bdc9c43ed P 83ffd7184ec94302bd1ee74b0fab3e17: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240503 23:26:24.631239 3959 tablet_bootstrap.cc:492] T 60267cb4849d4823a0afb2380305905f P 83ffd7184ec94302bd1ee74b0fab3e17: Bootstrap starting.
I20240503 23:26:24.633365 3959 tablet_bootstrap.cc:654] T 60267cb4849d4823a0afb2380305905f P 83ffd7184ec94302bd1ee74b0fab3e17: Neither blocks nor log segments found. Creating new log.
I20240503 23:26:24.633822 3958 tablet_bootstrap.cc:654] T a14a12115b8d4fec947c6e223f1ef486 P a3eecbaec43d4954b1096bc3121e824c: Neither blocks nor log segments found. Creating new log.
I20240503 23:26:24.637104 3967 raft_consensus.cc:482] T 6039230222ff4e57929bed49cd56caf5 P a3eecbaec43d4954b1096bc3121e824c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 23:26:24.637374 3967 raft_consensus.cc:504] T 6039230222ff4e57929bed49cd56caf5 P a3eecbaec43d4954b1096bc3121e824c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3eecbaec43d4954b1096bc3121e824c" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 40649 } } peers { permanent_uuid: "a2f9bf2e5d9b4fc19f804797bca4a3ff" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 46261 } } peers { permanent_uuid: "83ffd7184ec94302bd1ee74b0fab3e17" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 32909 } }
I20240503 23:26:24.638971 3967 leader_election.cc:288] T 6039230222ff4e57929bed49cd56caf5 P a3eecbaec43d4954b1096bc3121e824c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a2f9bf2e5d9b4fc19f804797bca4a3ff (127.0.100.2:46261), 83ffd7184ec94302bd1ee74b0fab3e17 (127.0.100.1:32909)
I20240503 23:26:24.642910 3960 tablet_bootstrap.cc:492] T 059f6233ea514fe69b8c6504efabeff0 P a2f9bf2e5d9b4fc19f804797bca4a3ff: No bootstrap required, opened a new log
I20240503 23:26:24.643189 3960 ts_tablet_manager.cc:1367] T 059f6233ea514fe69b8c6504efabeff0 P a2f9bf2e5d9b4fc19f804797bca4a3ff: Time spent bootstrapping tablet: real 0.019s user 0.003s sys 0.007s
I20240503 23:26:24.644338 3960 raft_consensus.cc:348] T 059f6233ea514fe69b8c6504efabeff0 P a2f9bf2e5d9b4fc19f804797bca4a3ff [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83ffd7184ec94302bd1ee74b0fab3e17" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 32909 } } peers { permanent_uuid: "a2f9bf2e5d9b4fc19f804797bca4a3ff" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 46261 } } peers { permanent_uuid: "a3eecbaec43d4954b1096bc3121e824c" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 40649 } }
I20240503 23:26:24.644623 3960 raft_consensus.cc:374] T 059f6233ea514fe69b8c6504efabeff0 P a2f9bf2e5d9b4fc19f804797bca4a3ff [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 23:26:24.644733 3960 raft_consensus.cc:729] T 059f6233ea514fe69b8c6504efabeff0 P a2f9bf2e5d9b4fc19f804797bca4a3ff [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2f9bf2e5d9b4fc19f804797bca4a3ff, State: Initialized, Role: FOLLOWER
I20240503 23:26:24.645015 3958 tablet_bootstrap.cc:492] T a14a12115b8d4fec947c6e223f1ef486 P a3eecbaec43d4954b1096bc3121e824c: No bootstrap required, opened a new log
I20240503 23:26:24.645062 3960 consensus_queue.cc:260] T 059f6233ea514fe69b8c6504efabeff0 P a2f9bf2e5d9b4fc19f804797bca4a3ff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83ffd7184ec94302bd1ee74b0fab3e17" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 32909 } } peers { permanent_uuid: "a2f9bf2e5d9b4fc19f804797bca4a3ff" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 46261 } } peers { permanent_uuid: "a3eecbaec43d4954b1096bc3121e824c" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 40649 } }
I20240503 23:26:24.645396 3958 ts_tablet_manager.cc:1367] T a14a12115b8d4fec947c6e223f1ef486 P a3eecbaec43d4954b1096bc3121e824c: Time spent bootstrapping tablet: real 0.015s user 0.011s sys 0.001s
I20240503 23:26:24.645857 3960 ts_tablet_manager.cc:1398] T 059f6233ea514fe69b8c6504efabeff0 P a2f9bf2e5d9b4fc19f804797bca4a3ff: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20240503 23:26:24.646320 3960 tablet_bootstrap.cc:492] T a0e18d0975a54e3dbeb0a13cb226fefd P a2f9bf2e5d9b4fc19f804797bca4a3ff: Bootstrap starting.
I20240503 23:26:24.646526 3958 raft_consensus.cc:348] T a14a12115b8d4fec947c6e223f1ef486 P a3eecbaec43d4954b1096bc3121e824c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83ffd7184ec94302bd1ee74b0fab3e17" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 32909 } } peers { permanent_uuid: "a2f9bf2e5d9b4fc19f804797bca4a3ff" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 46261 } } peers { permanent_uuid: "a3eecbaec43d4954b1096bc3121e824c" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 40649 } }
I20240503 23:26:24.646806 3958 raft_consensus.cc:374] T a14a12115b8d4fec947c6e223f1ef486 P a3eecbaec43d4954b1096bc3121e824c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 23:26:24.647006 3958 raft_consensus.cc:729] T a14a12115b8d4fec947c6e223f1ef486 P a3eecbaec43d4954b1096bc3121e824c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a3eecbaec43d4954b1096bc3121e824c, State: Initialized, Role: FOLLOWER
I20240503 23:26:24.647379 3958 consensus_queue.cc:260] T a14a12115b8d4fec947c6e223f1ef486 P a3eecbaec43d4954b1096bc3121e824c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83ffd7184ec94302bd1ee74b0fab3e17" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 32909 } } peers { permanent_uuid: "a2f9bf2e5d9b4fc19f804797bca4a3ff" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 46261 } } peers { permanent_uuid: "a3eecbaec43d4954b1096bc3121e824c" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 40649 } }
I20240503 23:26:24.648154 3958 ts_tablet_manager.cc:1398] T a14a12115b8d4fec947c6e223f1ef486 P a3eecbaec43d4954b1096bc3121e824c: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20240503 23:26:24.648588 3958 tablet_bootstrap.cc:492] T d0c4430fc56644cb9f87deffe4fe9407 P a3eecbaec43d4954b1096bc3121e824c: Bootstrap starting.
I20240503 23:26:24.650143 3960 tablet_bootstrap.cc:654] T a0e18d0975a54e3dbeb0a13cb226fefd P a2f9bf2e5d9b4fc19f804797bca4a3ff: Neither blocks nor log segments found. Creating new log.
I20240503 23:26:24.650560 3958 tablet_bootstrap.cc:654] T d0c4430fc56644cb9f87deffe4fe9407 P a3eecbaec43d4954b1096bc3121e824c: Neither blocks nor log segments found. Creating new log.
I20240503 23:26:24.663720 3958 tablet_bootstrap.cc:492] T d0c4430fc56644cb9f87deffe4fe9407 P a3eecbaec43d4954b1096bc3121e824c: No bootstrap required, opened a new log
I20240503 23:26:24.664245 3958 ts_tablet_manager.cc:1367] T d0c4430fc56644cb9f87deffe4fe9407 P a3eecbaec43d4954b1096bc3121e824c: Time spent bootstrapping tablet: real 0.016s user 0.000s sys 0.006s
I20240503 23:26:24.665777 3958 raft_consensus.cc:348] T d0c4430fc56644cb9f87deffe4fe9407 P a3eecbaec43d4954b1096bc3121e824c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83ffd7184ec94302bd1ee74b0fab3e17" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 32909 } } peers { permanent_uuid: "a2f9bf2e5d9b4fc19f804797bca4a3ff" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 46261 } } peers { permanent_uuid: "a3eecbaec43d4954b1096bc3121e824c" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 40649 } }
I20240503 23:26:24.666312 3958 raft_consensus.cc:374] T d0c4430fc56644cb9f87deffe4fe9407 P a3eecbaec43d4954b1096bc3121e824c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 23:26:24.666507 3958 raft_consensus.cc:729] T d0c4430fc56644cb9f87deffe4fe9407 P a3eecbaec43d4954b1096bc3121e824c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a3eecbaec43d4954b1096bc3121e824c, State: Initialized, Role: FOLLOWER
I20240503 23:26:24.667030 3967 raft_consensus.cc:482] T 481cd0f8c8d147adac217915a53ff6d9 P a3eecbaec43d4954b1096bc3121e824c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 23:26:24.667233 3967 raft_consensus.cc:504] T 481cd0f8c8d147adac217915a53ff6d9 P a3eecbaec43d4954b1096bc3121e824c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83ffd7184ec94302bd1ee74b0fab3e17" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 32909 } } peers { permanent_uuid: "a2f9bf2e5d9b4fc19f804797bca4a3ff" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 46261 } } peers { permanent_uuid: "a3eecbaec43d4954b1096bc3121e824c" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 40649 } }
I20240503 23:26:24.668025 3967 leader_election.cc:288] T 481cd0f8c8d147adac217915a53ff6d9 P a3eecbaec43d4954b1096bc3121e824c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 83ffd7184ec94302bd1ee74b0fab3e17 (127.0.100.1:32909), a2f9bf2e5d9b4fc19f804797bca4a3ff (127.0.100.2:46261)
I20240503 23:26:24.669277 3960 tablet_bootstrap.cc:492] T a0e18d0975a54e3dbeb0a13cb226fefd P a2f9bf2e5d9b4fc19f804797bca4a3ff: No bootstrap required, opened a new log
I20240503 23:26:24.669467 3960 ts_tablet_manager.cc:1367] T a0e18d0975a54e3dbeb0a13cb226fefd P a2f9bf2e5d9b4fc19f804797bca4a3ff: Time spent bootstrapping tablet: real 0.023s user 0.009s sys 0.010s
I20240503 23:26:24.670284 3960 raft_consensus.cc:348] T a0e18d0975a54e3dbeb0a13cb226fefd P a2f9bf2e5d9b4fc19f804797bca4a3ff [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83ffd7184ec94302bd1ee74b0fab3e17" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 32909 } } peers { permanent_uuid: "a2f9bf2e5d9b4fc19f804797bca4a3ff" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 46261 } } peers { permanent_uuid: "a3eecbaec43d4954b1096bc3121e824c" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 40649 } }
I20240503 23:26:24.670478 3960 raft_consensus.cc:374] T a0e18d0975a54e3dbeb0a13cb226fefd P a2f9bf2e5d9b4fc19f804797bca4a3ff [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 23:26:24.670608 3960 raft_consensus.cc:729] T a0e18d0975a54e3dbeb0a13cb226fefd P a2f9bf2e5d9b4fc19f804797bca4a3ff [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2f9bf2e5d9b4fc19f804797bca4a3ff, State: Initialized, Role: FOLLOWER
I20240503 23:26:24.670903 3960 consensus_queue.cc:260] T a0e18d0975a54e3dbeb0a13cb226fefd P a2f9bf2e5d9b4fc19f804797bca4a3ff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83ffd7184ec94302bd1ee74b0fab3e17" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 32909 } } peers { permanent_uuid: "a2f9bf2e5d9b4fc19f804797bca4a3ff" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 46261 } } peers { permanent_uuid: "a3eecbaec43d4954b1096bc3121e824c" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 40649 } }
I20240503 23:26:24.671602 3960 ts_tablet_manager.cc:1398] T a0e18d0975a54e3dbeb0a13cb226fefd P a2f9bf2e5d9b4fc19f804797bca4a3ff: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240503 23:26:24.671922 3960 tablet_bootstrap.cc:492] T 1b9566be6c644cfca852ec6507fab893 P a2f9bf2e5d9b4fc19f804797bca4a3ff: Bootstrap starting.
I20240503 23:26:24.673535 3839 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6039230222ff4e57929bed49cd56caf5" candidate_uuid: "a3eecbaec43d4954b1096bc3121e824c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2f9bf2e5d9b4fc19f804797bca4a3ff" is_pre_election: true
I20240503 23:26:24.673887 3839 raft_consensus.cc:2454] T 6039230222ff4e57929bed49cd56caf5 P a2f9bf2e5d9b4fc19f804797bca4a3ff [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a3eecbaec43d4954b1096bc3121e824c in term 0.
I20240503 23:26:24.674026 3838 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "481cd0f8c8d147adac217915a53ff6d9" candidate_uuid: "a3eecbaec43d4954b1096bc3121e824c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2f9bf2e5d9b4fc19f804797bca4a3ff" is_pre_election: true
I20240503 23:26:24.674923 3960 tablet_bootstrap.cc:654] T 1b9566be6c644cfca852ec6507fab893 P a2f9bf2e5d9b4fc19f804797bca4a3ff: Neither blocks nor log segments found. Creating new log.
I20240503 23:26:24.674772 3913 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6039230222ff4e57929bed49cd56caf5" candidate_uuid: "a2f9bf2e5d9b4fc19f804797bca4a3ff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a3eecbaec43d4954b1096bc3121e824c" is_pre_election: true
I20240503 23:26:24.675063 3913 raft_consensus.cc:2454] T 6039230222ff4e57929bed49cd56caf5 P a3eecbaec43d4954b1096bc3121e824c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2f9bf2e5d9b4fc19f804797bca4a3ff in term 0.
I20240503 23:26:24.667039 3958 consensus_queue.cc:260] T d0c4430fc56644cb9f87deffe4fe9407 P a3eecbaec43d4954b1096bc3121e824c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83ffd7184ec94302bd1ee74b0fab3e17" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 32909 } } peers { permanent_uuid: "a2f9bf2e5d9b4fc19f804797bca4a3ff" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 46261 } } peers { permanent_uuid: "a3eecbaec43d4954b1096bc3121e824c" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 40649 } }
I20240503 23:26:24.677356 3958 ts_tablet_manager.cc:1398] T d0c4430fc56644cb9f87deffe4fe9407 P a3eecbaec43d4954b1096bc3121e824c: Time spent starting tablet: real 0.013s user 0.003s sys 0.000s
I20240503 23:26:24.678881 3958 tablet_bootstrap.cc:492] T a0e18d0975a54e3dbeb0a13cb226fefd P a3eecbaec43d4954b1096bc3121e824c: Bootstrap starting.
I20240503 23:26:24.684875 3960 tablet_bootstrap.cc:492] T 1b9566be6c644cfca852ec6507fab893 P a2f9bf2e5d9b4fc19f804797bca4a3ff: No bootstrap required, opened a new log
I20240503 23:26:24.685245 3960 ts_tablet_manager.cc:1367] T 1b9566be6c644cfca852ec6507fab893 P a2f9bf2e5d9b4fc19f804797bca4a3ff: Time spent bootstrapping tablet: real 0.013s user 0.008s sys 0.003s
I20240503 23:26:24.685379 3879 leader_election.cc:302] T 6039230222ff4e57929bed49cd56caf5 P a3eecbaec43d4954b1096bc3121e824c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a2f9bf2e5d9b4fc19f804797bca4a3ff, a3eecbaec43d4954b1096bc3121e824c; no voters:
I20240503 23:26:24.685752 3967 raft_consensus.cc:2781] T 6039230222ff4e57929bed49cd56caf5 P a3eecbaec43d4954b1096bc3121e824c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 23:26:24.685855 3967 raft_consensus.cc:482] T 6039230222ff4e57929bed49cd56caf5 P a3eecbaec43d4954b1096bc3121e824c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 23:26:24.685961 3967 raft_consensus.cc:3037] T 6039230222ff4e57929bed49cd56caf5 P a3eecbaec43d4954b1096bc3121e824c [term 0 FOLLOWER]: Advancing to term 1
I20240503 23:26:24.686509 3960 raft_consensus.cc:348] T 1b9566be6c644cfca852ec6507fab893 P a2f9bf2e5d9b4fc19f804797bca4a3ff [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83ffd7184ec94302bd1ee74b0fab3e17" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 32909 } } peers { permanent_uuid: "a2f9bf2e5d9b4fc19f804797bca4a3ff" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 46261 } } peers { permanent_uuid: "a3eecbaec43d4954b1096bc3121e824c" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 40649 } }
I20240503 23:26:24.686779 3960 raft_consensus.cc:374] T 1b9566be6c644cfca852ec6507fab893 P a2f9bf2e5d9b4fc19f804797bca4a3ff [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 23:26:24.686861 3960 raft_consensus.cc:729] T 1b9566be6c644cfca852ec6507fab893 P a2f9bf2e5d9b4fc19f804797bca4a3ff [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2f9bf2e5d9b4fc19f804797bca4a3ff, State: Initialized, Role: FOLLOWER
I20240503 23:26:24.687165 3960 consensus_queue.cc:260] T 1b9566be6c644cfca852ec6507fab893 P a2f9bf2e5d9b4fc19f804797bca4a3ff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83ffd7184ec94302bd1ee74b0fab3e17" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 32909 } } peers { permanent_uuid: "a2f9bf2e5d9b4fc19f804797bca4a3ff" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 46261 } } peers { permanent_uuid: "a3eecbaec43d4954b1096bc3121e824c" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 40649 } }
I20240503 23:26:24.688017 3960 ts_tablet_manager.cc:1398] T 1b9566be6c644cfca852ec6507fab893 P a2f9bf2e5d9b4fc19f804797bca4a3ff: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240503 23:26:24.688409 3960 tablet_bootstrap.cc:492] T 487ac5b9bf2d4de2b80a1b13387a2a77 P a2f9bf2e5d9b4fc19f804797bca4a3ff: Bootstrap starting.
I20240503 23:26:24.690259 3967 raft_consensus.cc:504] T 6039230222ff4e57929bed49cd56caf5 P a3eecbaec43d4954b1096bc3121e824c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3eecbaec43d4954b1096bc3121e824c" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 40649 } } peers { permanent_uuid: "a2f9bf2e5d9b4fc19f804797bca4a3ff" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 46261 } } peers { permanent_uuid: "83ffd7184ec94302bd1ee74b0fab3e17" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 32909 } }
I20240503 23:26:24.691078 3960 tablet_bootstrap.cc:654] T 487ac5b9bf2d4de2b80a1b13387a2a77 P a2f9bf2e5d9b4fc19f804797bca4a3ff: Neither blocks nor log segments found. Creating new log.
I20240503 23:26:24.692301 3967 leader_election.cc:288] T 6039230222ff4e57929bed49cd56caf5 P a3eecbaec43d4954b1096bc3121e824c [CANDIDATE]: Term 1 election: Requested vote from peers a2f9bf2e5d9b4fc19f804797bca4a3ff (127.0.100.2:46261), 83ffd7184ec94302bd1ee74b0fab3e17 (127.0.100.1:32909)
W20240503 23:26:24.692901 3879 leader_election.cc:341] T 481cd0f8c8d147adac217915a53ff6d9 P a3eecbaec43d4954b1096bc3121e824c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a2f9bf2e5d9b4fc19f804797bca4a3ff (127.0.100.2:46261): Illegal state: must be running to vote when last-logged opid is not known
I20240503 23:26:24.693329 3802 leader_election.cc:302] T 6039230222ff4e57929bed49cd56caf5 P a2f9bf2e5d9b4fc19f804797bca4a3ff [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a2f9bf2e5d9b4fc19f804797bca4a3ff, a3eecbaec43d4954b1096bc3121e824c; no voters:
I20240503 23:26:24.693846 3966 raft_consensus.cc:2781] T 6039230222ff4e57929bed49cd56caf5 P a2f9bf2e5d9b4fc19f804797bca4a3ff [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 23:26:24.693989 3966 raft_consensus.cc:482] T 6039230222ff4e57929bed49cd56caf5 P a2f9bf2e5d9b4fc19f804797bca4a3ff [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 23:26:24.694110 3966 raft_consensus.cc:3037] T 6039230222ff4e57929bed49cd56caf5 P a2f9bf2e5d9b4fc19f804797bca4a3ff [term 0 FOLLOWER]: Advancing to term 1
I20240503 23:26:24.699452 3966 raft_consensus.cc:504] T 6039230222ff4e57929bed49cd56caf5 P a2f9bf2e5d9b4fc19f804797bca4a3ff [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3eecbaec43d4954b1096bc3121e824c" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 40649 } } peers { permanent_uuid: "a2f9bf2e5d9b4fc19f804797bca4a3ff" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 46261 } } peers { permanent_uuid: "83ffd7184ec94302bd1ee74b0fab3e17" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 32909 } }
I20240503 23:26:24.701160 3966 leader_election.cc:288] T 6039230222ff4e57929bed49cd56caf5 P a2f9bf2e5d9b4fc19f804797bca4a3ff [CANDIDATE]: Term 1 election: Requested vote from peers a3eecbaec43d4954b1096bc3121e824c (127.0.100.3:40649), 83ffd7184ec94302bd1ee74b0fab3e17 (127.0.100.1:32909)
I20240503 23:26:24.702461 3913 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6039230222ff4e57929bed49cd56caf5" candidate_uuid: "a2f9bf2e5d9b4fc19f804797bca4a3ff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a3eecbaec43d4954b1096bc3121e824c"
I20240503 23:26:24.702940 3913 raft_consensus.cc:2379] T 6039230222ff4e57929bed49cd56caf5 P a3eecbaec43d4954b1096bc3121e824c [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate a2f9bf2e5d9b4fc19f804797bca4a3ff in current term 1: Already voted for candidate a3eecbaec43d4954b1096bc3121e824c in this term.
I20240503 23:26:24.704313 3838 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6039230222ff4e57929bed49cd56caf5" candidate_uuid: "a3eecbaec43d4954b1096bc3121e824c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2f9bf2e5d9b4fc19f804797bca4a3ff"
I20240503 23:26:24.704823 3838 raft_consensus.cc:2379] T 6039230222ff4e57929bed49cd56caf5 P a2f9bf2e5d9b4fc19f804797bca4a3ff [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate a3eecbaec43d4954b1096bc3121e824c in current term 1: Already voted for candidate a2f9bf2e5d9b4fc19f804797bca4a3ff in this term.
I20240503 23:26:24.707937 3763 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6039230222ff4e57929bed49cd56caf5" candidate_uuid: "a2f9bf2e5d9b4fc19f804797bca4a3ff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83ffd7184ec94302bd1ee74b0fab3e17"
I20240503 23:26:24.710211 3764 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6039230222ff4e57929bed49cd56caf5" candidate_uuid: "a2f9bf2e5d9b4fc19f804797bca4a3ff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83ffd7184ec94302bd1ee74b0fab3e17" is_pre_election: true
W20240503 23:26:24.711522 3804 leader_election.cc:341] T 6039230222ff4e57929bed49cd56caf5 P a2f9bf2e5d9b4fc19f804797bca4a3ff [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 83ffd7184ec94302bd1ee74b0fab3e17 (127.0.100.1:32909): Illegal state: must be running to vote when last-logged opid is not known
W20240503 23:26:24.711859 3804 leader_election.cc:341] T 6039230222ff4e57929bed49cd56caf5 P a2f9bf2e5d9b4fc19f804797bca4a3ff [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 83ffd7184ec94302bd1ee74b0fab3e17 (127.0.100.1:32909): Illegal state: must be running to vote when last-logged opid is not known
I20240503 23:26:24.716826 3802 leader_election.cc:302] T 6039230222ff4e57929bed49cd56caf5 P a2f9bf2e5d9b4fc19f804797bca4a3ff [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2f9bf2e5d9b4fc19f804797bca4a3ff; no voters: 83ffd7184ec94302bd1ee74b0fab3e17, a3eecbaec43d4954b1096bc3121e824c
I20240503 23:26:24.718072 3966 raft_consensus.cc:2726] T 6039230222ff4e57929bed49cd56caf5 P a2f9bf2e5d9b4fc19f804797bca4a3ff [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20240503 23:26:24.719682 3958 tablet_bootstrap.cc:654] T a0e18d0975a54e3dbeb0a13cb226fefd P a3eecbaec43d4954b1096bc3121e824c: Neither blocks nor log segments found. Creating new log.
I20240503 23:26:24.722406 3959 tablet_bootstrap.cc:492] T 60267cb4849d4823a0afb2380305905f P 83ffd7184ec94302bd1ee74b0fab3e17: No bootstrap required, opened a new log
I20240503 23:26:24.722451 3764 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6039230222ff4e57929bed49cd56caf5" candidate_uuid: "a3eecbaec43d4954b1096bc3121e824c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83ffd7184ec94302bd1ee74b0fab3e17" is_pre_election: true
I20240503 23:26:24.722713 3959 ts_tablet_manager.cc:1367] T 60267cb4849d4823a0afb2380305905f P 83ffd7184ec94302bd1ee74b0fab3e17: Time spent bootstrapping tablet: real 0.092s user 0.025s sys 0.058s
I20240503 23:26:24.723140 3960 tablet_bootstrap.cc:492] T 487ac5b9bf2d4de2b80a1b13387a2a77 P a2f9bf2e5d9b4fc19f804797bca4a3ff: No bootstrap required, opened a new log
I20240503 23:26:24.723377 3960 ts_tablet_manager.cc:1367] T 487ac5b9bf2d4de2b80a1b13387a2a77 P a2f9bf2e5d9b4fc19f804797bca4a3ff: Time spent bootstrapping tablet: real 0.035s user 0.006s sys 0.026s
I20240503 23:26:24.723484 3764 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "481cd0f8c8d147adac217915a53ff6d9" candidate_uuid: "a3eecbaec43d4954b1096bc3121e824c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83ffd7184ec94302bd1ee74b0fab3e17" is_pre_election: true
I20240503 23:26:24.723750 3764 raft_consensus.cc:2454] T 481cd0f8c8d147adac217915a53ff6d9 P 83ffd7184ec94302bd1ee74b0fab3e17 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a3eecbaec43d4954b1096bc3121e824c in term 0.
I20240503 23:26:24.723754 3959 raft_consensus.cc:348] T 60267cb4849d4823a0afb2380305905f P 83ffd7184ec94302bd1ee74b0fab3e17 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83ffd7184ec94302bd1ee74b0fab3e17" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 32909 } } peers { permanent_uuid: "a2f9bf2e5d9b4fc19f804797bca4a3ff" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 46261 } } peers { permanent_uuid: "a3eecbaec43d4954b1096bc3121e824c" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 40649 } }
I20240503 23:26:24.724011 3959 raft_consensus.cc:374] T 60267cb4849d4823a0afb2380305905f P 83ffd7184ec94302bd1ee74b0fab3e17 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 23:26:24.724115 3959 raft_consensus.cc:729] T 60267cb4849d4823a0afb2380305905f P 83ffd7184ec94302bd1ee74b0fab3e17 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 83ffd7184ec94302bd1ee74b0fab3e17, State: Initialized, Role: FOLLOWER
I20240503 23:26:24.724222 3967 raft_consensus.cc:482] T 059f6233ea514fe69b8c6504efabeff0 P a3eecbaec43d4954b1096bc3121e824c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 23:26:24.724373 3967 raft_consensus.cc:504] T 059f6233ea514fe69b8c6504efabeff0 P a3eecbaec43d4954b1096bc3121e824c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83ffd7184ec94302bd1ee74b0fab3e17" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 32909 } } peers { permanent_uuid: "a2f9bf2e5d9b4fc19f804797bca4a3ff" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 46261 } } peers { permanent_uuid: "a3eecbaec43d4954b1096bc3121e824c" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 40649 } }
I20240503 23:26:24.724464 3959 consensus_queue.cc:260] T 60267cb4849d4823a0afb2380305905f P 83ffd7184ec94302bd1ee74b0fab3e17 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83ffd7184ec94302bd1ee74b0fab3e17" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 32909 } } peers { permanent_uuid: "a2f9bf2e5d9b4fc19f804797bca4a3ff" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 46261 } } peers { permanent_uuid: "a3eecbaec43d4954b1096bc3121e824c" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 40649 } }
I20240503 23:26:24.724481 3960 raft_consensus.cc:348] T 487ac5b9bf2d4de2b80a1b13387a2a77 P a2f9bf2e5d9b4fc19f804797bca4a3ff [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83ffd7184ec94302bd1ee74b0fab3e17" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 32909 } } peers { permanent_uuid: "a2f9bf2e5d9b4fc19f804797bca4a3ff" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 46261 } } peers { permanent_uuid: "a3eecbaec43d4954b1096bc3121e824c" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 40649 } }
I20240503 23:26:24.724815 3960 raft_consensus.cc:374] T 487ac5b9bf2d4de2b80a1b13387a2a77 P a2f9bf2e5d9b4fc19f804797bca4a3ff [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 23:26:24.724926 3960 raft_consensus.cc:729] T 487ac5b9bf2d4de2b80a1b13387a2a77 P a2f9bf2e5d9b4fc19f804797bca4a3ff [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2f9bf2e5d9b4fc19f804797bca4a3ff, State: Initialized, Role: FOLLOWER
I20240503 23:26:24.725128 3959 ts_tablet_manager.cc:1398] T 60267cb4849d4823a0afb2380305905f P 83ffd7184ec94302bd1ee74b0fab3e17: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 23:26:24.725219 3967 leader_election.cc:288] T 059f6233ea514fe69b8c6504efabeff0 P a3eecbaec43d4954b1096bc3121e824c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 83ffd7184ec94302bd1ee74b0fab3e17 (127.0.100.1:32909), a2f9bf2e5d9b4fc19f804797bca4a3ff (127.0.100.2:46261)
I20240503 23:26:24.725528 3959 tablet_bootstrap.cc:492] T 98c9500acb43423b86f54ce7a4d98a72 P 83ffd7184ec94302bd1ee74b0fab3e17: Bootstrap starting.
I20240503 23:26:24.726245 3838 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "059f6233ea514fe69b8c6504efabeff0" candidate_uuid: "a3eecbaec43d4954b1096bc3121e824c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2f9bf2e5d9b4fc19f804797bca4a3ff" is_pre_election: true
I20240503 23:26:24.726516 3838 raft_consensus.cc:2454] T 059f6233ea514fe69b8c6504efabeff0 P a2f9bf2e5d9b4fc19f804797bca4a3ff [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a3eecbaec43d4954b1096bc3121e824c in term 0.
I20240503 23:26:24.727409 3879 leader_election.cc:302] T 059f6233ea514fe69b8c6504efabeff0 P a3eecbaec43d4954b1096bc3121e824c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a2f9bf2e5d9b4fc19f804797bca4a3ff, a3eecbaec43d4954b1096bc3121e824c; no voters:
I20240503 23:26:24.727535 3966 raft_consensus.cc:482] T 55cf664d5e874c0d9b9adf8bdc9c43ed P a2f9bf2e5d9b4fc19f804797bca4a3ff [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 23:26:24.727670 3966 raft_consensus.cc:504] T 55cf664d5e874c0d9b9adf8bdc9c43ed P a2f9bf2e5d9b4fc19f804797bca4a3ff [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83ffd7184ec94302bd1ee74b0fab3e17" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 32909 } } peers { permanent_uuid: "a2f9bf2e5d9b4fc19f804797bca4a3ff" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 46261 } } peers { permanent_uuid: "a3eecbaec43d4954b1096bc3121e824c" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 40649 } }
I20240503 23:26:24.727815 3967 raft_consensus.cc:2781] T 059f6233ea514fe69b8c6504efabeff0 P a3eecbaec43d4954b1096bc3121e824c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 23:26:24.727902 3967 raft_consensus.cc:482] T 059f6233ea514fe69b8c6504efabeff0 P a3eecbaec43d4954b1096bc3121e824c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 23:26:24.728003 3967 raft_consensus.cc:3037] T 059f6233ea514fe69b8c6504efabeff0 P a3eecbaec43d4954b1096bc3121e824c [term 0 FOLLOWER]: Advancing to term 1
I20240503 23:26:24.728502 3966 leader_election.cc:288] T 55cf664d5e874c0d9b9adf8bdc9c43ed P a2f9bf2e5d9b4fc19f804797bca4a3ff [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 83ffd7184ec94302bd1ee74b0fab3e17 (127.0.100.1:32909), a3eecbaec43d4954b1096bc3121e824c (127.0.100.3:40649)
I20240503 23:26:24.729465 3959 tablet_bootstrap.cc:654] T 98c9500acb43423b86f54ce7a4d98a72 P 83ffd7184ec94302bd1ee74b0fab3e17: Neither blocks nor log segments found. Creating new log.
I20240503 23:26:24.731652 3960 consensus_queue.cc:260] T 487ac5b9bf2d4de2b80a1b13387a2a77 P a2f9bf2e5d9b4fc19f804797bca4a3ff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83ffd7184ec94302bd1ee74b0fab3e17" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 32909 } } peers { permanent_uuid: "a2f9bf2e5d9b4fc19f804797bca4a3ff" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 46261 } } peers { permanent_uuid: "a3eecbaec43d4954b1096bc3121e824c" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 40649 } }
I20240503 23:26:24.731905 3967 raft_consensus.cc:504] T 059f6233ea514fe69b8c6504efabeff0 P a3eecbaec43d4954b1096bc3121e824c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83ffd7184ec94302bd1ee74b0fab3e17" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 32909 } } peers { permanent_uuid: "a2f9bf2e5d9b4fc19f804797bca4a3ff" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 46261 } } peers { permanent_uuid: "a3eecbaec43d4954b1096bc3121e824c" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 40649 } }
I20240503 23:26:24.733633 3960 ts_tablet_manager.cc:1398] T 487ac5b9bf2d4de2b80a1b13387a2a77 P a2f9bf2e5d9b4fc19f804797bca4a3ff: Time spent starting tablet: real 0.010s user 0.003s sys 0.000s
I20240503 23:26:24.734181 3960 tablet_bootstrap.cc:492] T a14a12115b8d4fec947c6e223f1ef486 P a2f9bf2e5d9b4fc19f804797bca4a3ff: Bootstrap starting.
I20240503 23:26:24.734138 3913 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "55cf664d5e874c0d9b9adf8bdc9c43ed" candidate_uuid: "a2f9bf2e5d9b4fc19f804797bca4a3ff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a3eecbaec43d4954b1096bc3121e824c" is_pre_election: true
W20240503 23:26:24.735193 3802 leader_election.cc:341] T 55cf664d5e874c0d9b9adf8bdc9c43ed P a2f9bf2e5d9b4fc19f804797bca4a3ff [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a3eecbaec43d4954b1096bc3121e824c (127.0.100.3:40649): Illegal state: must be running to vote when last-logged opid is not known
I20240503 23:26:24.736033 3764 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "55cf664d5e874c0d9b9adf8bdc9c43ed" candidate_uuid: "a2f9bf2e5d9b4fc19f804797bca4a3ff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83ffd7184ec94302bd1ee74b0fab3e17" is_pre_election: true
I20240503 23:26:24.736522 3764 raft_consensus.cc:2454] T 55cf664d5e874c0d9b9adf8bdc9c43ed P 83ffd7184ec94302bd1ee74b0fab3e17 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2f9bf2e5d9b4fc19f804797bca4a3ff in term 0.
I20240503 23:26:24.738983 3804 leader_election.cc:302] T 55cf664d5e874c0d9b9adf8bdc9c43ed P a2f9bf2e5d9b4fc19f804797bca4a3ff [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 83ffd7184ec94302bd1ee74b0fab3e17, a2f9bf2e5d9b4fc19f804797bca4a3ff; no voters: a3eecbaec43d4954b1096bc3121e824c
I20240503 23:26:24.740784 3966 raft_consensus.cc:2781] T 55cf664d5e874c0d9b9adf8bdc9c43ed P a2f9bf2e5d9b4fc19f804797bca4a3ff [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 23:26:24.741029 3966 raft_consensus.cc:482] T 55cf664d5e874c0d9b9adf8bdc9c43ed P a2f9bf2e5d9b4fc19f804797bca4a3ff [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 23:26:24.741171 3966 raft_consensus.cc:3037] T 55cf664d5e874c0d9b9adf8bdc9c43ed P a2f9bf2e5d9b4fc19f804797bca4a3ff [term 0 FOLLOWER]: Advancing to term 1
I20240503 23:26:24.741267 3838 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "059f6233ea514fe69b8c6504efabeff0" candidate_uuid: "a3eecbaec43d4954b1096bc3121e824c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2f9bf2e5d9b4fc19f804797bca4a3ff"
I20240503 23:26:24.741657 3838 raft_consensus.cc:3037] T 059f6233ea514fe69b8c6504efabeff0 P a2f9bf2e5d9b4fc19f804797bca4a3ff [term 0 FOLLOWER]: Advancing to term 1
I20240503 23:26:24.744141 3967 leader_election.cc:288] T 059f6233ea514fe69b8c6504efabeff0 P a3eecbaec43d4954b1096bc3121e824c [CANDIDATE]: Term 1 election: Requested vote from peers 83ffd7184ec94302bd1ee74b0fab3e17 (127.0.100.1:32909), a2f9bf2e5d9b4fc19f804797bca4a3ff (127.0.100.2:46261)
I20240503 23:26:24.743474 3763 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "059f6233ea514fe69b8c6504efabeff0" candidate_uuid: "a3eecbaec43d4954b1096bc3121e824c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83ffd7184ec94302bd1ee74b0fab3e17" is_pre_election: true
W20240503 23:26:24.745842 3878 leader_election.cc:341] T 6039230222ff4e57929bed49cd56caf5 P a3eecbaec43d4954b1096bc3121e824c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 83ffd7184ec94302bd1ee74b0fab3e17 (127.0.100.1:32909): Illegal state: must be running to vote when last-logged opid is not known
I20240503 23:26:24.746474 3764 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6039230222ff4e57929bed49cd56caf5" candidate_uuid: "a3eecbaec43d4954b1096bc3121e824c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83ffd7184ec94302bd1ee74b0fab3e17"
I20240503 23:26:24.747484 3963 raft_consensus.cc:482] T 7181e4e8c0324a8eb095dde23651a1ab P 83ffd7184ec94302bd1ee74b0fab3e17 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 23:26:24.750512 3967 raft_consensus.cc:482] T 7181e4e8c0324a8eb095dde23651a1ab P a3eecbaec43d4954b1096bc3121e824c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 23:26:24.754153 3963 raft_consensus.cc:504] T 7181e4e8c0324a8eb095dde23651a1ab P 83ffd7184ec94302bd1ee74b0fab3e17 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83ffd7184ec94302bd1ee74b0fab3e17" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 32909 } } peers { permanent_uuid: "a2f9bf2e5d9b4fc19f804797bca4a3ff" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 46261 } } peers { permanent_uuid: "a3eecbaec43d4954b1096bc3121e824c" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 40649 } }
I20240503 23:26:24.760015 3838 raft_consensus.cc:2454] T 059f6233ea514fe69b8c6504efabeff0 P a2f9bf2e5d9b4fc19f804797bca4a3ff [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a3eecbaec43d4954b1096bc3121e824c in term 1.
I20240503 23:26:24.760607 3878 leader_election.cc:302] T 481cd0f8c8d147adac217915a53ff6d9 P a3eecbaec43d4954b1096bc3121e824c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 83ffd7184ec94302bd1ee74b0fab3e17, a3eecbaec43d4954b1096bc3121e824c; no voters: a2f9bf2e5d9b4fc19f804797bca4a3ff
I20240503 23:26:24.762070 3967 raft_consensus.cc:504] T 7181e4e8c0324a8eb095dde23651a1ab P a3eecbaec43d4954b1096bc3121e824c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83ffd7184ec94302bd1ee74b0fab3e17" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 32909 } } peers { permanent_uuid: "a2f9bf2e5d9b4fc19f804797bca4a3ff" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 46261 } } peers { permanent_uuid: "a3eecbaec43d4954b1096bc3121e824c" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 40649 } }
I20240503 23:26:24.765547 3960 tablet_bootstrap.cc:654] T a14a12115b8d4fec947c6e223f1ef486 P a2f9bf2e5d9b4fc19f804797bca4a3ff: Neither blocks nor log segments found. Creating new log.
I20240503 23:26:24.767527 3879 leader_election.cc:302] T 059f6233ea514fe69b8c6504efabeff0 P a3eecbaec43d4954b1096bc3121e824c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a2f9bf2e5d9b4fc19f804797bca4a3ff, a3eecbaec43d4954b1096bc3121e824c; no voters:
I20240503 23:26:24.761241 3762 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "059f6233ea514fe69b8c6504efabeff0" candidate_uuid: "a3eecbaec43d4954b1096bc3121e824c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83ffd7184ec94302bd1ee74b0fab3e17"
W20240503 23:26:24.769021 3878 leader_election.cc:341] T 059f6233ea514fe69b8c6504efabeff0 P a3eecbaec43d4954b1096bc3121e824c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 83ffd7184ec94302bd1ee74b0fab3e17 (127.0.100.1:32909): Illegal state: must be running to vote when last-logged opid is not known
I20240503 23:26:24.773034 3967 leader_election.cc:288] T 7181e4e8c0324a8eb095dde23651a1ab P a3eecbaec43d4954b1096bc3121e824c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 83ffd7184ec94302bd1ee74b0fab3e17 (127.0.100.1:32909), a2f9bf2e5d9b4fc19f804797bca4a3ff (127.0.100.2:46261)
W20240503 23:26:24.774324 3878 leader_election.cc:341] T 6039230222ff4e57929bed49cd56caf5 P a3eecbaec43d4954b1096bc3121e824c [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 83ffd7184ec94302bd1ee74b0fab3e17 (127.0.100.1:32909): Illegal state: must be running to vote when last-logged opid is not known
I20240503 23:26:24.778311 3967 raft_consensus.cc:2781] T 481cd0f8c8d147adac217915a53ff6d9 P a3eecbaec43d4954b1096bc3121e824c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 23:26:24.778622 3967 raft_consensus.cc:482] T 481cd0f8c8d147adac217915a53ff6d9 P a3eecbaec43d4954b1096bc3121e824c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 23:26:24.778899 3967 raft_consensus.cc:3037] T 481cd0f8c8d147adac217915a53ff6d9 P a3eecbaec43d4954b1096bc3121e824c [term 0 FOLLOWER]: Advancing to term 1
I20240503 23:26:24.779711 3963 leader_election.cc:288] T 7181e4e8c0324a8eb095dde23651a1ab P 83ffd7184ec94302bd1ee74b0fab3e17 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a2f9bf2e5d9b4fc19f804797bca4a3ff (127.0.100.2:46261), a3eecbaec43d4954b1096bc3121e824c (127.0.100.3:40649)
I20240503 23:26:24.779922 3878 leader_election.cc:302] T 6039230222ff4e57929bed49cd56caf5 P a3eecbaec43d4954b1096bc3121e824c [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a3eecbaec43d4954b1096bc3121e824c; no voters: 83ffd7184ec94302bd1ee74b0fab3e17, a2f9bf2e5d9b4fc19f804797bca4a3ff
I20240503 23:26:24.781673 3994 raft_consensus.cc:482] T 98c9500acb43423b86f54ce7a4d98a72 P a2f9bf2e5d9b4fc19f804797bca4a3ff [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 23:26:24.783638 3996 raft_consensus.cc:2781] T 059f6233ea514fe69b8c6504efabeff0 P a3eecbaec43d4954b1096bc3121e824c [term 1 FOLLOWER]: Leader election won for term 1
I20240503 23:26:24.785809 3762 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7181e4e8c0324a8eb095dde23651a1ab" candidate_uuid: "a3eecbaec43d4954b1096bc3121e824c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83ffd7184ec94302bd1ee74b0fab3e17" is_pre_election: true
I20240503 23:26:24.787153 3762 raft_consensus.cc:2454] T 7181e4e8c0324a8eb095dde23651a1ab P 83ffd7184ec94302bd1ee74b0fab3e17 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a3eecbaec43d4954b1096bc3121e824c in term 0.
W20240503 23:26:24.786303 3878 leader_election.cc:341] T 059f6233ea514fe69b8c6504efabeff0 P a3eecbaec43d4954b1096bc3121e824c [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 83ffd7184ec94302bd1ee74b0fab3e17 (127.0.100.1:32909): Illegal state: must be running to vote when last-logged opid is not known
I20240503 23:26:24.786418 3959 tablet_bootstrap.cc:492] T 98c9500acb43423b86f54ce7a4d98a72 P 83ffd7184ec94302bd1ee74b0fab3e17: No bootstrap required, opened a new log
I20240503 23:26:24.789332 3838 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7181e4e8c0324a8eb095dde23651a1ab" candidate_uuid: "a3eecbaec43d4954b1096bc3121e824c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2f9bf2e5d9b4fc19f804797bca4a3ff" is_pre_election: true
I20240503 23:26:24.789611 3838 raft_consensus.cc:2454] T 7181e4e8c0324a8eb095dde23651a1ab P a2f9bf2e5d9b4fc19f804797bca4a3ff [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a3eecbaec43d4954b1096bc3121e824c in term 0.
I20240503 23:26:24.790216 3958 tablet_bootstrap.cc:492] T a0e18d0975a54e3dbeb0a13cb226fefd P a3eecbaec43d4954b1096bc3121e824c: No bootstrap required, opened a new log
I20240503 23:26:24.790462 3958 ts_tablet_manager.cc:1367] T a0e18d0975a54e3dbeb0a13cb226fefd P a3eecbaec43d4954b1096bc3121e824c: Time spent bootstrapping tablet: real 0.113s user 0.019s sys 0.038s
I20240503 23:26:24.792779 3878 leader_election.cc:302] T 7181e4e8c0324a8eb095dde23651a1ab P a3eecbaec43d4954b1096bc3121e824c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 83ffd7184ec94302bd1ee74b0fab3e17, a3eecbaec43d4954b1096bc3121e824c; no voters:
I20240503 23:26:24.792377 3958 raft_consensus.cc:348] T a0e18d0975a54e3dbeb0a13cb226fefd P a3eecbaec43d4954b1096bc3121e824c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83ffd7184ec94302bd1ee74b0fab3e17" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 32909 } } peers { permanent_uuid: "a2f9bf2e5d9b4fc19f804797bca4a3ff" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 46261 } } peers { permanent_uuid: "a3eecbaec43d4954b1096bc3121e824c" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 40649 } }
I20240503 23:26:24.793831 3959 ts_tablet_manager.cc:1367] T 98c9500acb43423b86f54ce7a4d98a72 P 83ffd7184ec94302bd1ee74b0fab3e17: Time spent bootstrapping tablet: real 0.064s user 0.014s sys 0.002s
I20240503 23:26:24.792382 3966 raft_consensus.cc:504] T 55cf664d5e874c0d9b9adf8bdc9c43ed P a2f9bf2e5d9b4fc19f804797bca4a3ff [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83ffd7184ec94302bd1ee74b0fab3e17" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 32909 } } peers { permanent_uuid: "a2f9bf2e5d9b4fc19f804797bca4a3ff" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 46261 } } peers { permanent_uuid: "a3eecbaec43d4954b1096bc3121e824c" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 40649 } }
I20240503 23:26:24.795133 3958 raft_consensus.cc:374] T a0e18d0975a54e3dbeb0a13cb226fefd P a3eecbaec43d4954b1096bc3121e824c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 23:26:24.795377 3958 raft_consensus.cc:729] T a0e18d0975a54e3dbeb0a13cb226fefd P a3eecbaec43d4954b1096bc3121e824c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a3eecbaec43d4954b1096bc3121e824c, State: Initialized, Role: FOLLOWER
I20240503 23:26:24.786455 3994 raft_consensus.cc:504] T 98c9500acb43423b86f54ce7a4d98a72 P a2f9bf2e5d9b4fc19f804797bca4a3ff [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83ffd7184ec94302bd1ee74b0fab3e17" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 32909 } } peers { permanent_uuid: "a2f9bf2e5d9b4fc19f804797bca4a3ff" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 46261 } } peers { permanent_uuid: "a3eecbaec43d4954b1096bc3121e824c" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 40649 } }
I20240503 23:26:24.796232 3958 consensus_queue.cc:260] T a0e18d0975a54e3dbeb0a13cb226fefd P a3eecbaec43d4954b1096bc3121e824c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83ffd7184ec94302bd1ee74b0fab3e17" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 32909 } } peers { permanent_uuid: "a2f9bf2e5d9b4fc19f804797bca4a3ff" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 46261 } } peers { permanent_uuid: "a3eecbaec43d4954b1096bc3121e824c" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 40649 } }
I20240503 23:26:24.803193 3963 raft_consensus.cc:482] T a14a12115b8d4fec947c6e223f1ef486 P 83ffd7184ec94302bd1ee74b0fab3e17 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 23:26:24.804191 3992 raft_consensus.cc:2726] T 6039230222ff4e57929bed49cd56caf5 P a3eecbaec43d4954b1096bc3121e824c [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20240503 23:26:24.804684 3992 raft_consensus.cc:2781] T 7181e4e8c0324a8eb095dde23651a1ab P a3eecbaec43d4954b1096bc3121e824c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 23:26:24.804837 3992 raft_consensus.cc:482] T 7181e4e8c0324a8eb095dde23651a1ab P a3eecbaec43d4954b1096bc3121e824c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 23:26:24.804980 3992 raft_consensus.cc:3037] T 7181e4e8c0324a8eb095dde23651a1ab P a3eecbaec43d4954b1096bc3121e824c [term 0 FOLLOWER]: Advancing to term 1
I20240503 23:26:24.805320 3963 raft_consensus.cc:504] T a14a12115b8d4fec947c6e223f1ef486 P 83ffd7184ec94302bd1ee74b0fab3e17 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83ffd7184ec94302bd1ee74b0fab3e17" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 32909 } } peers { permanent_uuid: "a2f9bf2e5d9b4fc19f804797bca4a3ff" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 46261 } } peers { permanent_uuid: "a3eecbaec43d4954b1096bc3121e824c" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 40649 } }
I20240503 23:26:24.805881 3966 leader_election.cc:288] T 55cf664d5e874c0d9b9adf8bdc9c43ed P a2f9bf2e5d9b4fc19f804797bca4a3ff [CANDIDATE]: Term 1 election: Requested vote from peers 83ffd7184ec94302bd1ee74b0fab3e17 (127.0.100.1:32909), a3eecbaec43d4954b1096bc3121e824c (127.0.100.3:40649)
I20240503 23:26:24.808326 3994 leader_election.cc:288] T 98c9500acb43423b86f54ce7a4d98a72 P a2f9bf2e5d9b4fc19f804797bca4a3ff [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 83ffd7184ec94302bd1ee74b0fab3e17 (127.0.100.1:32909), a3eecbaec43d4954b1096bc3121e824c (127.0.100.3:40649)
I20240503 23:26:24.808310 3963 leader_election.cc:288] T a14a12115b8d4fec947c6e223f1ef486 P 83ffd7184ec94302bd1ee74b0fab3e17 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a2f9bf2e5d9b4fc19f804797bca4a3ff (127.0.100.2:46261), a3eecbaec43d4954b1096bc3121e824c (127.0.100.3:40649)
I20240503 23:26:24.808272 3913 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "55cf664d5e874c0d9b9adf8bdc9c43ed" candidate_uuid: "a2f9bf2e5d9b4fc19f804797bca4a3ff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a3eecbaec43d4954b1096bc3121e824c"
W20240503 23:26:24.809361 3802 leader_election.cc:341] T 55cf664d5e874c0d9b9adf8bdc9c43ed P a2f9bf2e5d9b4fc19f804797bca4a3ff [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer a3eecbaec43d4954b1096bc3121e824c (127.0.100.3:40649): Illegal state: must be running to vote when last-logged opid is not known
I20240503 23:26:24.810168 3966 raft_consensus.cc:482] T a0e18d0975a54e3dbeb0a13cb226fefd P a2f9bf2e5d9b4fc19f804797bca4a3ff [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 23:26:24.811937 3764 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "98c9500acb43423b86f54ce7a4d98a72" candidate_uuid: "a2f9bf2e5d9b4fc19f804797bca4a3ff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83ffd7184ec94302bd1ee74b0fab3e17" is_pre_election: true
I20240503 23:26:24.812426 3958 ts_tablet_manager.cc:1398] T a0e18d0975a54e3dbeb0a13cb226fefd P a3eecbaec43d4954b1096bc3121e824c: Time spent starting tablet: real 0.022s user 0.004s sys 0.001s
W20240503 23:26:24.812637 3804 leader_election.cc:341] T 98c9500acb43423b86f54ce7a4d98a72 P a2f9bf2e5d9b4fc19f804797bca4a3ff [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 83ffd7184ec94302bd1ee74b0fab3e17 (127.0.100.1:32909): Illegal state: must be running to vote when last-logged opid is not known
I20240503 23:26:24.812975 3958 tablet_bootstrap.cc:492] T 98c9500acb43423b86f54ce7a4d98a72 P a3eecbaec43d4954b1096bc3121e824c: Bootstrap starting.
I20240503 23:26:24.814711 3958 tablet_bootstrap.cc:654] T 98c9500acb43423b86f54ce7a4d98a72 P a3eecbaec43d4954b1096bc3121e824c: Neither blocks nor log segments found. Creating new log.
I20240503 23:26:24.815748 3762 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "55cf664d5e874c0d9b9adf8bdc9c43ed" candidate_uuid: "a2f9bf2e5d9b4fc19f804797bca4a3ff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83ffd7184ec94302bd1ee74b0fab3e17"
I20240503 23:26:24.816016 3762 raft_consensus.cc:3037] T 55cf664d5e874c0d9b9adf8bdc9c43ed P 83ffd7184ec94302bd1ee74b0fab3e17 [term 0 FOLLOWER]: Advancing to term 1
I20240503 23:26:24.819818 3762 raft_consensus.cc:2454] T 55cf664d5e874c0d9b9adf8bdc9c43ed P 83ffd7184ec94302bd1ee74b0fab3e17 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2f9bf2e5d9b4fc19f804797bca4a3ff in term 1.
I20240503 23:26:24.820536 3804 leader_election.cc:302] T 55cf664d5e874c0d9b9adf8bdc9c43ed P a2f9bf2e5d9b4fc19f804797bca4a3ff [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 83ffd7184ec94302bd1ee74b0fab3e17, a2f9bf2e5d9b4fc19f804797bca4a3ff; no voters: a3eecbaec43d4954b1096bc3121e824c
I20240503 23:26:24.808272 3912 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "98c9500acb43423b86f54ce7a4d98a72" candidate_uuid: "a2f9bf2e5d9b4fc19f804797bca4a3ff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a3eecbaec43d4954b1096bc3121e824c" is_pre_election: true
I20240503 23:26:24.829860 3960 tablet_bootstrap.cc:492] T a14a12115b8d4fec947c6e223f1ef486 P a2f9bf2e5d9b4fc19f804797bca4a3ff: No bootstrap required, opened a new log
I20240503 23:26:24.830134 3960 ts_tablet_manager.cc:1367] T a14a12115b8d4fec947c6e223f1ef486 P a2f9bf2e5d9b4fc19f804797bca4a3ff: Time spent bootstrapping tablet: real 0.096s user 0.016s sys 0.012s
W20240503 23:26:24.830345 3802 leader_election.cc:341] T 98c9500acb43423b86f54ce7a4d98a72 P a2f9bf2e5d9b4fc19f804797bca4a3ff [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a3eecbaec43d4954b1096bc3121e824c (127.0.100.3:40649): Illegal state: must be running to vote when last-logged opid is not known
I20240503 23:26:24.830282 3967 raft_consensus.cc:504] T 481cd0f8c8d147adac217915a53ff6d9 P a3eecbaec43d4954b1096bc3121e824c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83ffd7184ec94302bd1ee74b0fab3e17" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 32909 } } peers { permanent_uuid: "a2f9bf2e5d9b4fc19f804797bca4a3ff" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 46261 } } peers { permanent_uuid: "a3eecbaec43d4954b1096bc3121e824c" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 40649 } }
I20240503 23:26:24.830510 3802 leader_election.cc:302] T 98c9500acb43423b86f54ce7a4d98a72 P a2f9bf2e5d9b4fc19f804797bca4a3ff [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2f9bf2e5d9b4fc19f804797bca4a3ff; no voters: 83ffd7184ec94302bd1ee74b0fab3e17, a3eecbaec43d4954b1096bc3121e824c
I20240503 23:26:24.830873 3994 raft_consensus.cc:2726] T 98c9500acb43423b86f54ce7a4d98a72 P a2f9bf2e5d9b4fc19f804797bca4a3ff [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 23:26:24.830960 3966 raft_consensus.cc:504] T a0e18d0975a54e3dbeb0a13cb226fefd P a2f9bf2e5d9b4fc19f804797bca4a3ff [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83ffd7184ec94302bd1ee74b0fab3e17" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 32909 } } peers { permanent_uuid: "a2f9bf2e5d9b4fc19f804797bca4a3ff" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 46261 } } peers { permanent_uuid: "a3eecbaec43d4954b1096bc3121e824c" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 40649 } }
I20240503 23:26:24.831555 3960 raft_consensus.cc:348] T a14a12115b8d4fec947c6e223f1ef486 P a2f9bf2e5d9b4fc19f804797bca4a3ff [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83ffd7184ec94302bd1ee74b0fab3e17" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 32909 } } peers { permanent_uuid: "a2f9bf2e5d9b4fc19f804797bca4a3ff" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 46261 } } peers { permanent_uuid: "a3eecbaec43d4954b1096bc3121e824c" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 40649 } }
I20240503 23:26:24.831811 3959 raft_consensus.cc:348] T 98c9500acb43423b86f54ce7a4d98a72 P 83ffd7184ec94302bd1ee74b0fab3e17 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83ffd7184ec94302bd1ee74b0fab3e17" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 32909 } } peers { permanent_uuid: "a2f9bf2e5d9b4fc19f804797bca4a3ff" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 46261 } } peers { permanent_uuid: "a3eecbaec43d4954b1096bc3121e824c" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 40649 } }
I20240503 23:26:24.831992 3959 raft_consensus.cc:374] T 98c9500acb43423b86f54ce7a4d98a72 P 83ffd7184ec94302bd1ee74b0fab3e17 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 23:26:24.832039 3960 raft_consensus.cc:374] T a14a12115b8d4fec947c6e223f1ef486 P a2f9bf2e5d9b4fc19f804797bca4a3ff [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 23:26:24.832237 3960 raft_consensus.cc:729] T a14a12115b8d4fec947c6e223f1ef486 P a2f9bf2e5d9b4fc19f804797bca4a3ff [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2f9bf2e5d9b4fc19f804797bca4a3ff, State: Initialized, Role: FOLLOWER
I20240503 23:26:24.832595 3960 consensus_queue.cc:260] T a14a12115b8d4fec947c6e223f1ef486 P a2f9bf2e5d9b4fc19f804797bca4a3ff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83ffd7184ec94302bd1ee74b0fab3e17" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 32909 } } peers { permanent_uuid: "a2f9bf2e5d9b4fc19f804797bca4a3ff" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 46261 } } peers { permanent_uuid: "a3eecbaec43d4954b1096bc3121e824c" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 40649 } }
I20240503 23:26:24.832702 3912 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a0e18d0975a54e3dbeb0a13cb226fefd" candidate_uuid: "a2f9bf2e5d9b4fc19f804797bca4a3ff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a3eecbaec43d4954b1096bc3121e824c" is_pre_election: true
I20240503 23:26:24.833375 3912 raft_consensus.cc:2454] T a0e18d0975a54e3dbeb0a13cb226fefd P a3eecbaec43d4954b1096bc3121e824c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2f9bf2e5d9b4fc19f804797bca4a3ff in term 0.
I20240503 23:26:24.833621 3960 ts_tablet_manager.cc:1398] T a14a12115b8d4fec947c6e223f1ef486 P a2f9bf2e5d9b4fc19f804797bca4a3ff: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20240503 23:26:24.833837 3802 leader_election.cc:302] T a0e18d0975a54e3dbeb0a13cb226fefd P a2f9bf2e5d9b4fc19f804797bca4a3ff [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a2f9bf2e5d9b4fc19f804797bca4a3ff, a3eecbaec43d4954b1096bc3121e824c; no voters:
I20240503 23:26:24.831691 3966 leader_election.cc:288] T a0e18d0975a54e3dbeb0a13cb226fefd P a2f9bf2e5d9b4fc19f804797bca4a3ff [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 83ffd7184ec94302bd1ee74b0fab3e17 (127.0.100.1:32909), a3eecbaec43d4954b1096bc3121e824c (127.0.100.3:40649)
I20240503 23:26:24.834206 3966 raft_consensus.cc:2781] T a0e18d0975a54e3dbeb0a13cb226fefd P a2f9bf2e5d9b4fc19f804797bca4a3ff [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 23:26:24.834338 3966 raft_consensus.cc:482] T a0e18d0975a54e3dbeb0a13cb226fefd P a2f9bf2e5d9b4fc19f804797bca4a3ff [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 23:26:24.834451 3966 raft_consensus.cc:3037] T a0e18d0975a54e3dbeb0a13cb226fefd P a2f9bf2e5d9b4fc19f804797bca4a3ff [term 0 FOLLOWER]: Advancing to term 1
I20240503 23:26:24.834394 3762 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a0e18d0975a54e3dbeb0a13cb226fefd" candidate_uuid: "a2f9bf2e5d9b4fc19f804797bca4a3ff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83ffd7184ec94302bd1ee74b0fab3e17" is_pre_election: true
W20240503 23:26:24.834957 3804 leader_election.cc:341] T a0e18d0975a54e3dbeb0a13cb226fefd P a2f9bf2e5d9b4fc19f804797bca4a3ff [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 83ffd7184ec94302bd1ee74b0fab3e17 (127.0.100.1:32909): Illegal state: must be running to vote when last-logged opid is not known
I20240503 23:26:24.832895 3996 raft_consensus.cc:686] T 059f6233ea514fe69b8c6504efabeff0 P a3eecbaec43d4954b1096bc3121e824c [term 1 LEADER]: Becoming Leader. State: Replica: a3eecbaec43d4954b1096bc3121e824c, State: Running, Role: LEADER
I20240503 23:26:24.832090 3959 raft_consensus.cc:729] T 98c9500acb43423b86f54ce7a4d98a72 P 83ffd7184ec94302bd1ee74b0fab3e17 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 83ffd7184ec94302bd1ee74b0fab3e17, State: Initialized, Role: FOLLOWER
I20240503 23:26:24.834204 3960 tablet_bootstrap.cc:492] T 481cd0f8c8d147adac217915a53ff6d9 P a2f9bf2e5d9b4fc19f804797bca4a3ff: Bootstrap starting.
I20240503 23:26:24.838385 3995 raft_consensus.cc:2781] T 55cf664d5e874c0d9b9adf8bdc9c43ed P a2f9bf2e5d9b4fc19f804797bca4a3ff [term 1 FOLLOWER]: Leader election won for term 1
I20240503 23:26:24.839848 3966 raft_consensus.cc:504] T a0e18d0975a54e3dbeb0a13cb226fefd P a2f9bf2e5d9b4fc19f804797bca4a3ff [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83ffd7184ec94302bd1ee74b0fab3e17" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 32909 } } peers { permanent_uuid: "a2f9bf2e5d9b4fc19f804797bca4a3ff" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 46261 } } peers { permanent_uuid: "a3eecbaec43d4954b1096bc3121e824c" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 40649 } }
I20240503 23:26:24.839869 3959 consensus_queue.cc:260] T 98c9500acb43423b86f54ce7a4d98a72 P 83ffd7184ec94302bd1ee74b0fab3e17 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83ffd7184ec94302bd1ee74b0fab3e17" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 32909 } } peers { permanent_uuid: "a2f9bf2e5d9b4fc19f804797bca4a3ff" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 46261 } } peers { permanent_uuid: "a3eecbaec43d4954b1096bc3121e824c" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 40649 } }
I20240503 23:26:24.840833 4002 raft_consensus.cc:482] T d0c4430fc56644cb9f87deffe4fe9407 P a3eecbaec43d4954b1096bc3121e824c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 23:26:24.841814 3966 leader_election.cc:288] T a0e18d0975a54e3dbeb0a13cb226fefd P a2f9bf2e5d9b4fc19f804797bca4a3ff [CANDIDATE]: Term 1 election: Requested vote from peers 83ffd7184ec94302bd1ee74b0fab3e17 (127.0.100.1:32909), a3eecbaec43d4954b1096bc3121e824c (127.0.100.3:40649)
I20240503 23:26:24.841966 3838 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "481cd0f8c8d147adac217915a53ff6d9" candidate_uuid: "a3eecbaec43d4954b1096bc3121e824c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2f9bf2e5d9b4fc19f804797bca4a3ff"
I20240503 23:26:24.842432 3912 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a0e18d0975a54e3dbeb0a13cb226fefd" candidate_uuid: "a2f9bf2e5d9b4fc19f804797bca4a3ff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a3eecbaec43d4954b1096bc3121e824c"
I20240503 23:26:24.842636 3912 raft_consensus.cc:3037] T a0e18d0975a54e3dbeb0a13cb226fefd P a3eecbaec43d4954b1096bc3121e824c [term 0 FOLLOWER]: Advancing to term 1
I20240503 23:26:24.842679 3959 ts_tablet_manager.cc:1398] T 98c9500acb43423b86f54ce7a4d98a72 P 83ffd7184ec94302bd1ee74b0fab3e17: Time spent starting tablet: real 0.049s user 0.000s sys 0.003s
I20240503 23:26:24.846786 3959 tablet_bootstrap.cc:492] T a0e18d0975a54e3dbeb0a13cb226fefd P 83ffd7184ec94302bd1ee74b0fab3e17: Bootstrap starting.
I20240503 23:26:24.847337 3967 leader_election.cc:288] T 481cd0f8c8d147adac217915a53ff6d9 P a3eecbaec43d4954b1096bc3121e824c [CANDIDATE]: Term 1 election: Requested vote from peers 83ffd7184ec94302bd1ee74b0fab3e17 (127.0.100.1:32909), a2f9bf2e5d9b4fc19f804797bca4a3ff (127.0.100.2:46261)
W20240503 23:26:24.848922 3879 leader_election.cc:341] T 481cd0f8c8d147adac217915a53ff6d9 P a3eecbaec43d4954b1096bc3121e824c [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer a2f9bf2e5d9b4fc19f804797bca4a3ff (127.0.100.2:46261): Illegal state: must be running to vote when last-logged opid is not known
I20240503 23:26:24.851048 3762 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a0e18d0975a54e3dbeb0a13cb226fefd" candidate_uuid: "a2f9bf2e5d9b4fc19f804797bca4a3ff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83ffd7184ec94302bd1ee74b0fab3e17"
W20240503 23:26:24.852371 3804 leader_election.cc:341] T a0e18d0975a54e3dbeb0a13cb226fefd P a2f9bf2e5d9b4fc19f804797bca4a3ff [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 83ffd7184ec94302bd1ee74b0fab3e17 (127.0.100.1:32909): Illegal state: must be running to vote when last-logged opid is not known
I20240503 23:26:24.852860 3838 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7181e4e8c0324a8eb095dde23651a1ab" candidate_uuid: "83ffd7184ec94302bd1ee74b0fab3e17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2f9bf2e5d9b4fc19f804797bca4a3ff" is_pre_election: true
I20240503 23:26:24.853118 3838 raft_consensus.cc:2454] T 7181e4e8c0324a8eb095dde23651a1ab P a2f9bf2e5d9b4fc19f804797bca4a3ff [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83ffd7184ec94302bd1ee74b0fab3e17 in term 0.
I20240503 23:26:24.853730 3838 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a14a12115b8d4fec947c6e223f1ef486" candidate_uuid: "83ffd7184ec94302bd1ee74b0fab3e17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2f9bf2e5d9b4fc19f804797bca4a3ff" is_pre_election: true
I20240503 23:26:24.853953 3838 raft_consensus.cc:2454] T a14a12115b8d4fec947c6e223f1ef486 P a2f9bf2e5d9b4fc19f804797bca4a3ff [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83ffd7184ec94302bd1ee74b0fab3e17 in term 0.
I20240503 23:26:24.856745 3912 raft_consensus.cc:2454] T a0e18d0975a54e3dbeb0a13cb226fefd P a3eecbaec43d4954b1096bc3121e824c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2f9bf2e5d9b4fc19f804797bca4a3ff in term 1.
I20240503 23:26:24.857115 3967 raft_consensus.cc:482] T a14a12115b8d4fec947c6e223f1ef486 P a3eecbaec43d4954b1096bc3121e824c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 23:26:24.857229 3802 leader_election.cc:302] T a0e18d0975a54e3dbeb0a13cb226fefd P a2f9bf2e5d9b4fc19f804797bca4a3ff [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a2f9bf2e5d9b4fc19f804797bca4a3ff, a3eecbaec43d4954b1096bc3121e824c; no voters: 83ffd7184ec94302bd1ee74b0fab3e17
I20240503 23:26:24.857290 3967 raft_consensus.cc:504] T a14a12115b8d4fec947c6e223f1ef486 P a3eecbaec43d4954b1096bc3121e824c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83ffd7184ec94302bd1ee74b0fab3e17" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 32909 } } peers { permanent_uuid: "a2f9bf2e5d9b4fc19f804797bca4a3ff" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 46261 } } peers { permanent_uuid: "a3eecbaec43d4954b1096bc3121e824c" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 40649 } }
I20240503 23:26:24.857707 3966 raft_consensus.cc:2781] T a0e18d0975a54e3dbeb0a13cb226fefd P a2f9bf2e5d9b4fc19f804797bca4a3ff [term 1 FOLLOWER]: Leader election won for term 1
I20240503 23:26:24.857862 3966 raft_consensus.cc:686] T a0e18d0975a54e3dbeb0a13cb226fefd P a2f9bf2e5d9b4fc19f804797bca4a3ff [term 1 LEADER]: Becoming Leader. State: Replica: a2f9bf2e5d9b4fc19f804797bca4a3ff, State: Running, Role: LEADER
I20240503 23:26:24.858029 3967 leader_election.cc:288] T a14a12115b8d4fec947c6e223f1ef486 P a3eecbaec43d4954b1096bc3121e824c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 83ffd7184ec94302bd1ee74b0fab3e17 (127.0.100.1:32909), a2f9bf2e5d9b4fc19f804797bca4a3ff (127.0.100.2:46261)
I20240503 23:26:24.858176 3966 consensus_queue.cc:237] T a0e18d0975a54e3dbeb0a13cb226fefd P a2f9bf2e5d9b4fc19f804797bca4a3ff [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83ffd7184ec94302bd1ee74b0fab3e17" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 32909 } } peers { permanent_uuid: "a2f9bf2e5d9b4fc19f804797bca4a3ff" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 46261 } } peers { permanent_uuid: "a3eecbaec43d4954b1096bc3121e824c" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 40649 } }
I20240503 23:26:24.859330 3838 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a14a12115b8d4fec947c6e223f1ef486" candidate_uuid: "a3eecbaec43d4954b1096bc3121e824c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2f9bf2e5d9b4fc19f804797bca4a3ff" is_pre_election: true
I20240503 23:26:24.859570 3838 raft_consensus.cc:2454] T a14a12115b8d4fec947c6e223f1ef486 P a2f9bf2e5d9b4fc19f804797bca4a3ff [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a3eecbaec43d4954b1096bc3121e824c in term 0.
I20240503 23:26:24.848742 3995 raft_consensus.cc:686] T 55cf664d5e874c0d9b9adf8bdc9c43ed P a2f9bf2e5d9b4fc19f804797bca4a3ff [term 1 LEADER]: Becoming Leader. State: Replica: a2f9bf2e5d9b4fc19f804797bca4a3ff, State: Running, Role: LEADER
W20240503 23:26:24.860807 3878 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.100.1:32909, user_credentials={real_user=slave}} blocked reactor thread for 73090.7us
I20240503 23:26:24.862823 3764 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a14a12115b8d4fec947c6e223f1ef486" candidate_uuid: "a3eecbaec43d4954b1096bc3121e824c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83ffd7184ec94302bd1ee74b0fab3e17" is_pre_election: true
I20240503 23:26:24.863148 3764 raft_consensus.cc:2454] T a14a12115b8d4fec947c6e223f1ef486 P 83ffd7184ec94302bd1ee74b0fab3e17 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a3eecbaec43d4954b1096bc3121e824c in term 0.
I20240503 23:26:24.862854 3762 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "481cd0f8c8d147adac217915a53ff6d9" candidate_uuid: "a3eecbaec43d4954b1096bc3121e824c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83ffd7184ec94302bd1ee74b0fab3e17"
I20240503 23:26:24.863483 3762 raft_consensus.cc:3037] T 481cd0f8c8d147adac217915a53ff6d9 P 83ffd7184ec94302bd1ee74b0fab3e17 [term 0 FOLLOWER]: Advancing to term 1
I20240503 23:26:24.863729 3879 leader_election.cc:302] T a14a12115b8d4fec947c6e223f1ef486 P a3eecbaec43d4954b1096bc3121e824c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a2f9bf2e5d9b4fc19f804797bca4a3ff, a3eecbaec43d4954b1096bc3121e824c; no voters:
I20240503 23:26:24.864017 3730 leader_election.cc:302] T 7181e4e8c0324a8eb095dde23651a1ab P 83ffd7184ec94302bd1ee74b0fab3e17 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 83ffd7184ec94302bd1ee74b0fab3e17, a2f9bf2e5d9b4fc19f804797bca4a3ff; no voters:
I20240503 23:26:24.864753 3963 raft_consensus.cc:2781] T 7181e4e8c0324a8eb095dde23651a1ab P 83ffd7184ec94302bd1ee74b0fab3e17 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 23:26:24.864820 3730 leader_election.cc:302] T a14a12115b8d4fec947c6e223f1ef486 P 83ffd7184ec94302bd1ee74b0fab3e17 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 83ffd7184ec94302bd1ee74b0fab3e17, a2f9bf2e5d9b4fc19f804797bca4a3ff; no voters:
I20240503 23:26:24.864883 3963 raft_consensus.cc:482] T 7181e4e8c0324a8eb095dde23651a1ab P 83ffd7184ec94302bd1ee74b0fab3e17 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 23:26:24.864991 3963 raft_consensus.cc:3037] T 7181e4e8c0324a8eb095dde23651a1ab P 83ffd7184ec94302bd1ee74b0fab3e17 [term 0 FOLLOWER]: Advancing to term 1
I20240503 23:26:24.848834 4002 raft_consensus.cc:504] T d0c4430fc56644cb9f87deffe4fe9407 P a3eecbaec43d4954b1096bc3121e824c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83ffd7184ec94302bd1ee74b0fab3e17" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 32909 } } peers { permanent_uuid: "a2f9bf2e5d9b4fc19f804797bca4a3ff" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 46261 } } peers { permanent_uuid: "a3eecbaec43d4954b1096bc3121e824c" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 40649 } }
I20240503 23:26:24.864475 4001 raft_consensus.cc:2781] T a14a12115b8d4fec947c6e223f1ef486 P a3eecbaec43d4954b1096bc3121e824c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 23:26:24.863189 3995 consensus_queue.cc:237] T 55cf664d5e874c0d9b9adf8bdc9c43ed P a2f9bf2e5d9b4fc19f804797bca4a3ff [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83ffd7184ec94302bd1ee74b0fab3e17" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 32909 } } peers { permanent_uuid: "a2f9bf2e5d9b4fc19f804797bca4a3ff" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 46261 } } peers { permanent_uuid: "a3eecbaec43d4954b1096bc3121e824c" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 40649 } }
I20240503 23:26:24.849718 3996 consensus_queue.cc:237] T 059f6233ea514fe69b8c6504efabeff0 P a3eecbaec43d4954b1096bc3121e824c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83ffd7184ec94302bd1ee74b0fab3e17" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 32909 } } peers { permanent_uuid: "a2f9bf2e5d9b4fc19f804797bca4a3ff" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 46261 } } peers { permanent_uuid: "a3eecbaec43d4954b1096bc3121e824c" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 40649 } }
I20240503 23:26:24.866626 4001 raft_consensus.cc:482] T a14a12115b8d4fec947c6e223f1ef486 P a3eecbaec43d4954b1096bc3121e824c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 23:26:24.868464 3960 tablet_bootstrap.cc:654] T 481cd0f8c8d147adac217915a53ff6d9 P a2f9bf2e5d9b4fc19f804797bca4a3ff: Neither blocks nor log segments found. Creating new log.
I20240503 23:26:24.870378 4001 raft_consensus.cc:3037] T a14a12115b8d4fec947c6e223f1ef486 P a3eecbaec43d4954b1096bc3121e824c [term 0 FOLLOWER]: Advancing to term 1
I20240503 23:26:24.870532 3764 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0c4430fc56644cb9f87deffe4fe9407" candidate_uuid: "a3eecbaec43d4954b1096bc3121e824c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83ffd7184ec94302bd1ee74b0fab3e17" is_pre_election: true
I20240503 23:26:24.871647 4002 leader_election.cc:288] T d0c4430fc56644cb9f87deffe4fe9407 P a3eecbaec43d4954b1096bc3121e824c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 83ffd7184ec94302bd1ee74b0fab3e17 (127.0.100.1:32909), a2f9bf2e5d9b4fc19f804797bca4a3ff (127.0.100.2:46261)
I20240503 23:26:24.872112 3838 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0c4430fc56644cb9f87deffe4fe9407" candidate_uuid: "a3eecbaec43d4954b1096bc3121e824c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2f9bf2e5d9b4fc19f804797bca4a3ff" is_pre_election: true
I20240503 23:26:24.872812 3958 tablet_bootstrap.cc:492] T 98c9500acb43423b86f54ce7a4d98a72 P a3eecbaec43d4954b1096bc3121e824c: No bootstrap required, opened a new log
W20240503 23:26:24.873922 3878 leader_election.cc:341] T d0c4430fc56644cb9f87deffe4fe9407 P a3eecbaec43d4954b1096bc3121e824c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 83ffd7184ec94302bd1ee74b0fab3e17 (127.0.100.1:32909): Illegal state: must be running to vote when last-logged opid is not known
I20240503 23:26:24.874369 4005 raft_consensus.cc:2781] T a14a12115b8d4fec947c6e223f1ef486 P 83ffd7184ec94302bd1ee74b0fab3e17 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 23:26:24.874586 4005 raft_consensus.cc:482] T a14a12115b8d4fec947c6e223f1ef486 P 83ffd7184ec94302bd1ee74b0fab3e17 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 23:26:24.874742 4005 raft_consensus.cc:3037] T a14a12115b8d4fec947c6e223f1ef486 P 83ffd7184ec94302bd1ee74b0fab3e17 [term 0 FOLLOWER]: Advancing to term 1
I20240503 23:26:24.876084 3838 raft_consensus.cc:2454] T d0c4430fc56644cb9f87deffe4fe9407 P a2f9bf2e5d9b4fc19f804797bca4a3ff [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a3eecbaec43d4954b1096bc3121e824c in term 0.
I20240503 23:26:24.878460 3879 leader_election.cc:302] T d0c4430fc56644cb9f87deffe4fe9407 P a3eecbaec43d4954b1096bc3121e824c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a2f9bf2e5d9b4fc19f804797bca4a3ff, a3eecbaec43d4954b1096bc3121e824c; no voters: 83ffd7184ec94302bd1ee74b0fab3e17
I20240503 23:26:24.879025 4002 raft_consensus.cc:2781] T d0c4430fc56644cb9f87deffe4fe9407 P a3eecbaec43d4954b1096bc3121e824c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 23:26:24.879181 4002 raft_consensus.cc:482] T d0c4430fc56644cb9f87deffe4fe9407 P a3eecbaec43d4954b1096bc3121e824c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 23:26:24.879293 4002 raft_consensus.cc:3037] T d0c4430fc56644cb9f87deffe4fe9407 P a3eecbaec43d4954b1096bc3121e824c [term 0 FOLLOWER]: Advancing to term 1
I20240503 23:26:24.884168 3963 raft_consensus.cc:504] T 7181e4e8c0324a8eb095dde23651a1ab P 83ffd7184ec94302bd1ee74b0fab3e17 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83ffd7184ec94302bd1ee74b0fab3e17" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 32909 } } peers { permanent_uuid: "a2f9bf2e5d9b4fc19f804797bca4a3ff" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 46261 } } peers { permanent_uuid: "a3eecbaec43d4954b1096bc3121e824c" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 40649 } }
I20240503 23:26:24.884932 3959 tablet_bootstrap.cc:654] T a0e18d0975a54e3dbeb0a13cb226fefd P 83ffd7184ec94302bd1ee74b0fab3e17: Neither blocks nor log segments found. Creating new log.
I20240503 23:26:24.885102 3958 ts_tablet_manager.cc:1367] T 98c9500acb43423b86f54ce7a4d98a72 P a3eecbaec43d4954b1096bc3121e824c: Time spent bootstrapping tablet: real 0.061s user 0.007s sys 0.015s
I20240503 23:26:24.885279 4005 raft_consensus.cc:504] T a14a12115b8d4fec947c6e223f1ef486 P 83ffd7184ec94302bd1ee74b0fab3e17 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83ffd7184ec94302bd1ee74b0fab3e17" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 32909 } } peers { permanent_uuid: "a2f9bf2e5d9b4fc19f804797bca4a3ff" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 46261 } } peers { permanent_uuid: "a3eecbaec43d4954b1096bc3121e824c" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 40649 } }
I20240503 23:26:24.886843 3838 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7181e4e8c0324a8eb095dde23651a1ab" candidate_uuid: "83ffd7184ec94302bd1ee74b0fab3e17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2f9bf2e5d9b4fc19f804797bca4a3ff"
I20240503 23:26:24.887109 3838 raft_consensus.cc:3037] T 7181e4e8c0324a8eb095dde23651a1ab P a2f9bf2e5d9b4fc19f804797bca4a3ff [term 0 FOLLOWER]: Advancing to term 1
I20240503 23:26:24.887636 3839 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a14a12115b8d4fec947c6e223f1ef486" candidate_uuid: "83ffd7184ec94302bd1ee74b0fab3e17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2f9bf2e5d9b4fc19f804797bca4a3ff"
I20240503 23:26:24.884064 4002 raft_consensus.cc:504] T d0c4430fc56644cb9f87deffe4fe9407 P a3eecbaec43d4954b1096bc3121e824c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83ffd7184ec94302bd1ee74b0fab3e17" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 32909 } } peers { permanent_uuid: "a2f9bf2e5d9b4fc19f804797bca4a3ff" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 46261 } } peers { permanent_uuid: "a3eecbaec43d4954b1096bc3121e824c" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 40649 } }
I20240503 23:26:24.889189 3762 raft_consensus.cc:2454] T 481cd0f8c8d147adac217915a53ff6d9 P 83ffd7184ec94302bd1ee74b0fab3e17 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a3eecbaec43d4954b1096bc3121e824c in term 1.
I20240503 23:26:24.891736 3839 raft_consensus.cc:3037] T a14a12115b8d4fec947c6e223f1ef486 P a2f9bf2e5d9b4fc19f804797bca4a3ff [term 0 FOLLOWER]: Advancing to term 1
I20240503 23:26:24.891757 3958 raft_consensus.cc:348] T 98c9500acb43423b86f54ce7a4d98a72 P a3eecbaec43d4954b1096bc3121e824c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83ffd7184ec94302bd1ee74b0fab3e17" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 32909 } } peers { permanent_uuid: "a2f9bf2e5d9b4fc19f804797bca4a3ff" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 46261 } } peers { permanent_uuid: "a3eecbaec43d4954b1096bc3121e824c" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 40649 } }
I20240503 23:26:24.892860 4005 leader_election.cc:288] T a14a12115b8d4fec947c6e223f1ef486 P 83ffd7184ec94302bd1ee74b0fab3e17 [CANDIDATE]: Term 1 election: Requested vote from peers a2f9bf2e5d9b4fc19f804797bca4a3ff (127.0.100.2:46261), a3eecbaec43d4954b1096bc3121e824c (127.0.100.3:40649)
I20240503 23:26:24.893561 3992 raft_consensus.cc:504] T 7181e4e8c0324a8eb095dde23651a1ab P a3eecbaec43d4954b1096bc3121e824c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83ffd7184ec94302bd1ee74b0fab3e17" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 32909 } } peers { permanent_uuid: "a2f9bf2e5d9b4fc19f804797bca4a3ff" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 46261 } } peers { permanent_uuid: "a3eecbaec43d4954b1096bc3121e824c" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 40649 } }
I20240503 23:26:24.894448 3963 leader_election.cc:288] T 7181e4e8c0324a8eb095dde23651a1ab P 83ffd7184ec94302bd1ee74b0fab3e17 [CANDIDATE]: Term 1 election: Requested vote from peers a2f9bf2e5d9b4fc19f804797bca4a3ff (127.0.100.2:46261), a3eecbaec43d4954b1096bc3121e824c (127.0.100.3:40649)
I20240503 23:26:24.894572 4002 leader_election.cc:288] T d0c4430fc56644cb9f87deffe4fe9407 P a3eecbaec43d4954b1096bc3121e824c [CANDIDATE]: Term 1 election: Requested vote from peers 83ffd7184ec94302bd1ee74b0fab3e17 (127.0.100.1:32909), a2f9bf2e5d9b4fc19f804797bca4a3ff (127.0.100.2:46261)
I20240503 23:26:24.895808 3878 leader_election.cc:302] T 481cd0f8c8d147adac217915a53ff6d9 P a3eecbaec43d4954b1096bc3121e824c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 83ffd7184ec94302bd1ee74b0fab3e17, a3eecbaec43d4954b1096bc3121e824c; no voters: a2f9bf2e5d9b4fc19f804797bca4a3ff
I20240503 23:26:24.895908 3958 raft_consensus.cc:374] T 98c9500acb43423b86f54ce7a4d98a72 P a3eecbaec43d4954b1096bc3121e824c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 23:26:24.897173 3762 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0c4430fc56644cb9f87deffe4fe9407" candidate_uuid: "a3eecbaec43d4954b1096bc3121e824c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83ffd7184ec94302bd1ee74b0fab3e17"
I20240503 23:26:24.897738 3837 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0c4430fc56644cb9f87deffe4fe9407" candidate_uuid: "a3eecbaec43d4954b1096bc3121e824c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2f9bf2e5d9b4fc19f804797bca4a3ff"
I20240503 23:26:24.898028 3837 raft_consensus.cc:3037] T d0c4430fc56644cb9f87deffe4fe9407 P a2f9bf2e5d9b4fc19f804797bca4a3ff [term 0 FOLLOWER]: Advancing to term 1
I20240503 23:26:24.895671 3661 catalog_manager.cc:5484] T 059f6233ea514fe69b8c6504efabeff0 P a3eecbaec43d4954b1096bc3121e824c reported cstate change: term changed from 0 to 1, leader changed from <none> to a3eecbaec43d4954b1096bc3121e824c (127.0.100.3). New cstate: current_term: 1 leader_uuid: "a3eecbaec43d4954b1096bc3121e824c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83ffd7184ec94302bd1ee74b0fab3e17" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 32909 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a2f9bf2e5d9b4fc19f804797bca4a3ff" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 46261 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a3eecbaec43d4954b1096bc3121e824c" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 40649 } health_report { overall_health: HEALTHY } } }
I20240503 23:26:24.899436 3992 leader_election.cc:288] T 7181e4e8c0324a8eb095dde23651a1ab P a3eecbaec43d4954b1096bc3121e824c [CANDIDATE]: Term 1 election: Requested vote from peers 83ffd7184ec94302bd1ee74b0fab3e17 (127.0.100.1:32909), a2f9bf2e5d9b4fc19f804797bca4a3ff (127.0.100.2:46261)
I20240503 23:26:24.899464 4002 raft_consensus.cc:2781] T 481cd0f8c8d147adac217915a53ff6d9 P a3eecbaec43d4954b1096bc3121e824c [term 1 FOLLOWER]: Leader election won for term 1
I20240503 23:26:24.900205 3762 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7181e4e8c0324a8eb095dde23651a1ab" candidate_uuid: "a3eecbaec43d4954b1096bc3121e824c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83ffd7184ec94302bd1ee74b0fab3e17"
I20240503 23:26:24.899960 3958 raft_consensus.cc:729] T 98c9500acb43423b86f54ce7a4d98a72 P a3eecbaec43d4954b1096bc3121e824c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a3eecbaec43d4954b1096bc3121e824c, State: Initialized, Role: FOLLOWER
W20240503 23:26:24.901564 3878 leader_election.cc:341] T d0c4430fc56644cb9f87deffe4fe9407 P a3eecbaec43d4954b1096bc3121e824c [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 83ffd7184ec94302bd1ee74b0fab3e17 (127.0.100.1:32909): Illegal state: must be running to vote when last-logged opid is not known
I20240503 23:26:24.902805 4002 raft_consensus.cc:686] T 481cd0f8c8d147adac217915a53ff6d9 P a3eecbaec43d4954b1096bc3121e824c [term 1 LEADER]: Becoming Leader. State: Replica: a3eecbaec43d4954b1096bc3121e824c, State: Running, Role: LEADER
I20240503 23:26:24.903342 3762 raft_consensus.cc:2379] T 7181e4e8c0324a8eb095dde23651a1ab P 83ffd7184ec94302bd1ee74b0fab3e17 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate a3eecbaec43d4954b1096bc3121e824c in current term 1: Already voted for candidate 83ffd7184ec94302bd1ee74b0fab3e17 in this term.
I20240503 23:26:24.904664 3958 consensus_queue.cc:260] T 98c9500acb43423b86f54ce7a4d98a72 P a3eecbaec43d4954b1096bc3121e824c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83ffd7184ec94302bd1ee74b0fab3e17" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 32909 } } peers { permanent_uuid: "a2f9bf2e5d9b4fc19f804797bca4a3ff" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 46261 } } peers { permanent_uuid: "a3eecbaec43d4954b1096bc3121e824c" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 40649 } }
I20240503 23:26:24.905107 3838 raft_consensus.cc:2454] T 7181e4e8c0324a8eb095dde23651a1ab P a2f9bf2e5d9b4fc19f804797bca4a3ff [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83ffd7184ec94302bd1ee74b0fab3e17 in term 1.
I20240503 23:26:24.905400 3958 ts_tablet_manager.cc:1398] T 98c9500acb43423b86f54ce7a4d98a72 P a3eecbaec43d4954b1096bc3121e824c: Time spent starting tablet: real 0.020s user 0.004s sys 0.000s
I20240503 23:26:24.905794 3958 tablet_bootstrap.cc:492] T 1b9566be6c644cfca852ec6507fab893 P a3eecbaec43d4954b1096bc3121e824c: Bootstrap starting.
I20240503 23:26:24.905792 3730 leader_election.cc:302] T 7181e4e8c0324a8eb095dde23651a1ab P 83ffd7184ec94302bd1ee74b0fab3e17 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 83ffd7184ec94302bd1ee74b0fab3e17, a2f9bf2e5d9b4fc19f804797bca4a3ff; no voters:
I20240503 23:26:24.906201 3963 raft_consensus.cc:2781] T 7181e4e8c0324a8eb095dde23651a1ab P 83ffd7184ec94302bd1ee74b0fab3e17 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 23:26:24.906347 3963 raft_consensus.cc:686] T 7181e4e8c0324a8eb095dde23651a1ab P 83ffd7184ec94302bd1ee74b0fab3e17 [term 1 LEADER]: Becoming Leader. State: Replica: 83ffd7184ec94302bd1ee74b0fab3e17, State: Running, Role: LEADER
I20240503 23:26:24.912822 4002 consensus_queue.cc:237] T 481cd0f8c8d147adac217915a53ff6d9 P a3eecbaec43d4954b1096bc3121e824c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83ffd7184ec94302bd1ee74b0fab3e17" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 32909 } } peers { permanent_uuid: "a2f9bf2e5d9b4fc19f804797bca4a3ff" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 46261 } } peers { permanent_uuid: "a3eecbaec43d4954b1096bc3121e824c" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 40649 } }
I20240503 23:26:24.915107 3963 consensus_queue.cc:237] T 7181e4e8c0324a8eb095dde23651a1ab P 83ffd7184ec94302bd1ee74b0fab3e17 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83ffd7184ec94302bd1ee74b0fab3e17" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 32909 } } peers { permanent_uuid: "a2f9bf2e5d9b4fc19f804797bca4a3ff" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 46261 } } peers { permanent_uuid: "a3eecbaec43d4954b1096bc3121e824c" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 40649 } }
I20240503 23:26:24.898242 3836 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7181e4e8c0324a8eb095dde23651a1ab" candidate_uuid: "a3eecbaec43d4954b1096bc3121e824c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2f9bf2e5d9b4fc19f804797bca4a3ff"
I20240503 23:26:24.917474 3839 raft_consensus.cc:2454] T a14a12115b8d4fec947c6e223f1ef486 P a2f9bf2e5d9b4fc19f804797bca4a3ff [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83ffd7184ec94302bd1ee74b0fab3e17 in term 1.
I20240503 23:26:24.918241 3836 raft_consensus.cc:2379] T 7181e4e8c0324a8eb095dde23651a1ab P a2f9bf2e5d9b4fc19f804797bca4a3ff [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate a3eecbaec43d4954b1096bc3121e824c in current term 1: Already voted for candidate 83ffd7184ec94302bd1ee74b0fab3e17 in this term.
I20240503 23:26:24.918335 3730 leader_election.cc:302] T a14a12115b8d4fec947c6e223f1ef486 P 83ffd7184ec94302bd1ee74b0fab3e17 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 83ffd7184ec94302bd1ee74b0fab3e17, a2f9bf2e5d9b4fc19f804797bca4a3ff; no voters:
I20240503 23:26:24.918669 4005 raft_consensus.cc:2781] T a14a12115b8d4fec947c6e223f1ef486 P 83ffd7184ec94302bd1ee74b0fab3e17 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 23:26:24.918769 3879 leader_election.cc:302] T 7181e4e8c0324a8eb095dde23651a1ab P a3eecbaec43d4954b1096bc3121e824c [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a3eecbaec43d4954b1096bc3121e824c; no voters: 83ffd7184ec94302bd1ee74b0fab3e17, a2f9bf2e5d9b4fc19f804797bca4a3ff
I20240503 23:26:24.912847 3662 catalog_manager.cc:5484] T 55cf664d5e874c0d9b9adf8bdc9c43ed P a2f9bf2e5d9b4fc19f804797bca4a3ff reported cstate change: term changed from 0 to 1, leader changed from <none> to a2f9bf2e5d9b4fc19f804797bca4a3ff (127.0.100.2). New cstate: current_term: 1 leader_uuid: "a2f9bf2e5d9b4fc19f804797bca4a3ff" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83ffd7184ec94302bd1ee74b0fab3e17" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 32909 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a2f9bf2e5d9b4fc19f804797bca4a3ff" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 46261 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a3eecbaec43d4954b1096bc3121e824c" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 40649 } health_report { overall_health: UNKNOWN } } }
I20240503 23:26:24.924798 3660 catalog_manager.cc:5484] T 7181e4e8c0324a8eb095dde23651a1ab P 83ffd7184ec94302bd1ee74b0fab3e17 reported cstate change: term changed from 0 to 1, leader changed from <none> to 83ffd7184ec94302bd1ee74b0fab3e17 (127.0.100.1). New cstate: current_term: 1 leader_uuid: "83ffd7184ec94302bd1ee74b0fab3e17" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83ffd7184ec94302bd1ee74b0fab3e17" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 32909 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a2f9bf2e5d9b4fc19f804797bca4a3ff" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 46261 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a3eecbaec43d4954b1096bc3121e824c" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 40649 } health_report { overall_health: UNKNOWN } } }
I20240503 23:26:24.928764 4001 raft_consensus.cc:504] T a14a12115b8d4fec947c6e223f1ef486 P a3eecbaec43d4954b1096bc3121e824c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83ffd7184ec94302bd1ee74b0fab3e17" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 32909 } } peers { permanent_uuid: "a2f9bf2e5d9b4fc19f804797bca4a3ff" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 46261 } } peers { permanent_uuid: "a3eecbaec43d4954b1096bc3121e824c" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 40649 } }
I20240503 23:26:24.930325 3992 raft_consensus.cc:2726] T 7181e4e8c0324a8eb095dde23651a1ab P a3eecbaec43d4954b1096bc3121e824c [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20240503 23:26:24.932533 3762 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a14a12115b8d4fec947c6e223f1ef486" candidate_uuid: "a3eecbaec43d4954b1096bc3121e824c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83ffd7184ec94302bd1ee74b0fab3e17"
I20240503 23:26:24.934159 3836 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a14a12115b8d4fec947c6e223f1ef486" candidate_uuid: "a3eecbaec43d4954b1096bc3121e824c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2f9bf2e5d9b4fc19f804797bca4a3ff"
I20240503 23:26:24.934442 3837 raft_consensus.cc:2454] T d0c4430fc56644cb9f87deffe4fe9407 P a2f9bf2e5d9b4fc19f804797bca4a3ff [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a3eecbaec43d4954b1096bc3121e824c in term 1.
I20240503 23:26:24.935176 4001 leader_election.cc:288] T a14a12115b8d4fec947c6e223f1ef486 P a3eecbaec43d4954b1096bc3121e824c [CANDIDATE]: Term 1 election: Requested vote from peers 83ffd7184ec94302bd1ee74b0fab3e17 (127.0.100.1:32909), a2f9bf2e5d9b4fc19f804797bca4a3ff (127.0.100.2:46261)
I20240503 23:26:24.936399 3836 raft_consensus.cc:2379] T a14a12115b8d4fec947c6e223f1ef486 P a2f9bf2e5d9b4fc19f804797bca4a3ff [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate a3eecbaec43d4954b1096bc3121e824c in current term 1: Already voted for candidate 83ffd7184ec94302bd1ee74b0fab3e17 in this term.
I20240503 23:26:24.936383 3879 leader_election.cc:302] T d0c4430fc56644cb9f87deffe4fe9407 P a3eecbaec43d4954b1096bc3121e824c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a2f9bf2e5d9b4fc19f804797bca4a3ff, a3eecbaec43d4954b1096bc3121e824c; no voters: 83ffd7184ec94302bd1ee74b0fab3e17
I20240503 23:26:24.937179 4001 raft_consensus.cc:2781] T d0c4430fc56644cb9f87deffe4fe9407 P a3eecbaec43d4954b1096bc3121e824c [term 1 FOLLOWER]: Leader election won for term 1
I20240503 23:26:24.940883 4001 raft_consensus.cc:686] T d0c4430fc56644cb9f87deffe4fe9407 P a3eecbaec43d4954b1096bc3121e824c [term 1 LEADER]: Becoming Leader. State: Replica: a3eecbaec43d4954b1096bc3121e824c, State: Running, Role: LEADER
I20240503 23:26:24.941591 4001 consensus_queue.cc:237] T d0c4430fc56644cb9f87deffe4fe9407 P a3eecbaec43d4954b1096bc3121e824c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83ffd7184ec94302bd1ee74b0fab3e17" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 32909 } } peers { permanent_uuid: "a2f9bf2e5d9b4fc19f804797bca4a3ff" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 46261 } } peers { permanent_uuid: "a3eecbaec43d4954b1096bc3121e824c" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 40649 } }
I20240503 23:26:24.951802 3912 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7181e4e8c0324a8eb095dde23651a1ab" candidate_uuid: "83ffd7184ec94302bd1ee74b0fab3e17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a3eecbaec43d4954b1096bc3121e824c" is_pre_election: true
I20240503 23:26:24.952276 3912 raft_consensus.cc:2379] T 7181e4e8c0324a8eb095dde23651a1ab P a3eecbaec43d4954b1096bc3121e824c [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 83ffd7184ec94302bd1ee74b0fab3e17 in current term 1: Already voted for candidate a3eecbaec43d4954b1096bc3121e824c in this term.
I20240503 23:26:24.952291 3913 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a14a12115b8d4fec947c6e223f1ef486" candidate_uuid: "83ffd7184ec94302bd1ee74b0fab3e17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a3eecbaec43d4954b1096bc3121e824c" is_pre_election: true
I20240503 23:26:24.954015 3913 raft_consensus.cc:2379] T a14a12115b8d4fec947c6e223f1ef486 P a3eecbaec43d4954b1096bc3121e824c [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 83ffd7184ec94302bd1ee74b0fab3e17 in current term 1: Already voted for candidate a3eecbaec43d4954b1096bc3121e824c in this term.
I20240503 23:26:24.957688 3963 raft_consensus.cc:482] T 487ac5b9bf2d4de2b80a1b13387a2a77 P 83ffd7184ec94302bd1ee74b0fab3e17 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 23:26:24.957948 3963 raft_consensus.cc:504] T 487ac5b9bf2d4de2b80a1b13387a2a77 P 83ffd7184ec94302bd1ee74b0fab3e17 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83ffd7184ec94302bd1ee74b0fab3e17" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 32909 } } peers { permanent_uuid: "a2f9bf2e5d9b4fc19f804797bca4a3ff" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 46261 } } peers { permanent_uuid: "a3eecbaec43d4954b1096bc3121e824c" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 40649 } }
I20240503 23:26:24.958935 3960 tablet_bootstrap.cc:492] T 481cd0f8c8d147adac217915a53ff6d9 P a2f9bf2e5d9b4fc19f804797bca4a3ff: No bootstrap required, opened a new log
I20240503 23:26:24.959136 3960 ts_tablet_manager.cc:1367] T 481cd0f8c8d147adac217915a53ff6d9 P a2f9bf2e5d9b4fc19f804797bca4a3ff: Time spent bootstrapping tablet: real 0.125s user 0.011s sys 0.008s
I20240503 23:26:24.961916 3963 leader_election.cc:288] T 487ac5b9bf2d4de2b80a1b13387a2a77 P 83ffd7184ec94302bd1ee74b0fab3e17 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a2f9bf2e5d9b4fc19f804797bca4a3ff (127.0.100.2:46261), a3eecbaec43d4954b1096bc3121e824c (127.0.100.3:40649)
I20240503 23:26:24.957682 3911 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a14a12115b8d4fec947c6e223f1ef486" candidate_uuid: "83ffd7184ec94302bd1ee74b0fab3e17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a3eecbaec43d4954b1096bc3121e824c"
I20240503 23:26:24.962293 3958 tablet_bootstrap.cc:654] T 1b9566be6c644cfca852ec6507fab893 P a3eecbaec43d4954b1096bc3121e824c: Neither blocks nor log segments found. Creating new log.
I20240503 23:26:24.962361 3959 tablet_bootstrap.cc:492] T a0e18d0975a54e3dbeb0a13cb226fefd P 83ffd7184ec94302bd1ee74b0fab3e17: No bootstrap required, opened a new log
I20240503 23:26:24.962280 3836 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "487ac5b9bf2d4de2b80a1b13387a2a77" candidate_uuid: "83ffd7184ec94302bd1ee74b0fab3e17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2f9bf2e5d9b4fc19f804797bca4a3ff" is_pre_election: true
I20240503 23:26:24.956812 3910 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7181e4e8c0324a8eb095dde23651a1ab" candidate_uuid: "83ffd7184ec94302bd1ee74b0fab3e17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a3eecbaec43d4954b1096bc3121e824c"
I20240503 23:26:24.963003 3836 raft_consensus.cc:2454] T 487ac5b9bf2d4de2b80a1b13387a2a77 P a2f9bf2e5d9b4fc19f804797bca4a3ff [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83ffd7184ec94302bd1ee74b0fab3e17 in term 0.
I20240503 23:26:24.963027 3959 ts_tablet_manager.cc:1367] T a0e18d0975a54e3dbeb0a13cb226fefd P 83ffd7184ec94302bd1ee74b0fab3e17: Time spent bootstrapping tablet: real 0.120s user 0.008s sys 0.009s
I20240503 23:26:24.963011 3960 raft_consensus.cc:348] T 481cd0f8c8d147adac217915a53ff6d9 P a2f9bf2e5d9b4fc19f804797bca4a3ff [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83ffd7184ec94302bd1ee74b0fab3e17" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 32909 } } peers { permanent_uuid: "a2f9bf2e5d9b4fc19f804797bca4a3ff" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 46261 } } peers { permanent_uuid: "a3eecbaec43d4954b1096bc3121e824c" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 40649 } }
I20240503 23:26:24.964375 3910 raft_consensus.cc:2379] T 7181e4e8c0324a8eb095dde23651a1ab P a3eecbaec43d4954b1096bc3121e824c [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 83ffd7184ec94302bd1ee74b0fab3e17 in current term 1: Already voted for candidate a3eecbaec43d4954b1096bc3121e824c in this term.
I20240503 23:26:24.964241 3960 raft_consensus.cc:374] T 481cd0f8c8d147adac217915a53ff6d9 P a2f9bf2e5d9b4fc19f804797bca4a3ff [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 23:26:24.964300 3911 raft_consensus.cc:2379] T a14a12115b8d4fec947c6e223f1ef486 P a3eecbaec43d4954b1096bc3121e824c [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 83ffd7184ec94302bd1ee74b0fab3e17 in current term 1: Already voted for candidate a3eecbaec43d4954b1096bc3121e824c in this term.
I20240503 23:26:24.965377 4005 raft_consensus.cc:686] T a14a12115b8d4fec947c6e223f1ef486 P 83ffd7184ec94302bd1ee74b0fab3e17 [term 1 LEADER]: Becoming Leader. State: Replica: 83ffd7184ec94302bd1ee74b0fab3e17, State: Running, Role: LEADER
I20240503 23:26:24.965418 3913 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "487ac5b9bf2d4de2b80a1b13387a2a77" candidate_uuid: "83ffd7184ec94302bd1ee74b0fab3e17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a3eecbaec43d4954b1096bc3121e824c" is_pre_election: true
I20240503 23:26:24.966825 3730 leader_election.cc:302] T 487ac5b9bf2d4de2b80a1b13387a2a77 P 83ffd7184ec94302bd1ee74b0fab3e17 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 83ffd7184ec94302bd1ee74b0fab3e17, a2f9bf2e5d9b4fc19f804797bca4a3ff; no voters:
I20240503 23:26:24.966930 4005 consensus_queue.cc:237] T a14a12115b8d4fec947c6e223f1ef486 P 83ffd7184ec94302bd1ee74b0fab3e17 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83ffd7184ec94302bd1ee74b0fab3e17" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 32909 } } peers { permanent_uuid: "a2f9bf2e5d9b4fc19f804797bca4a3ff" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 46261 } } peers { permanent_uuid: "a3eecbaec43d4954b1096bc3121e824c" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 40649 } }
W20240503 23:26:24.967581 3727 leader_election.cc:341] T 487ac5b9bf2d4de2b80a1b13387a2a77 P 83ffd7184ec94302bd1ee74b0fab3e17 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a3eecbaec43d4954b1096bc3121e824c (127.0.100.3:40649): Illegal state: must be running to vote when last-logged opid is not known
I20240503 23:26:24.967980 3960 raft_consensus.cc:729] T 481cd0f8c8d147adac217915a53ff6d9 P a2f9bf2e5d9b4fc19f804797bca4a3ff [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2f9bf2e5d9b4fc19f804797bca4a3ff, State: Initialized, Role: FOLLOWER
I20240503 23:26:24.968436 3960 consensus_queue.cc:260] T 481cd0f8c8d147adac217915a53ff6d9 P a2f9bf2e5d9b4fc19f804797bca4a3ff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83ffd7184ec94302bd1ee74b0fab3e17" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 32909 } } peers { permanent_uuid: "a2f9bf2e5d9b4fc19f804797bca4a3ff" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 46261 } } peers { permanent_uuid: "a3eecbaec43d4954b1096bc3121e824c" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 40649 } }
I20240503 23:26:24.969900 3960 ts_tablet_manager.cc:1398] T 481cd0f8c8d147adac217915a53ff6d9 P a2f9bf2e5d9b4fc19f804797bca4a3ff: Time spent starting tablet: real 0.010s user 0.001s sys 0.001s
I20240503 23:26:24.970497 3960 tablet_bootstrap.cc:492] T 60267cb4849d4823a0afb2380305905f P a2f9bf2e5d9b4fc19f804797bca4a3ff: Bootstrap starting.
I20240503 23:26:24.973354 3959 raft_consensus.cc:348] T a0e18d0975a54e3dbeb0a13cb226fefd P 83ffd7184ec94302bd1ee74b0fab3e17 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83ffd7184ec94302bd1ee74b0fab3e17" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 32909 } } peers { permanent_uuid: "a2f9bf2e5d9b4fc19f804797bca4a3ff" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 46261 } } peers { permanent_uuid: "a3eecbaec43d4954b1096bc3121e824c" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 40649 } }
I20240503 23:26:24.973989 3959 raft_consensus.cc:374] T a0e18d0975a54e3dbeb0a13cb226fefd P 83ffd7184ec94302bd1ee74b0fab3e17 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 23:26:24.974205 3959 raft_consensus.cc:729] T a0e18d0975a54e3dbeb0a13cb226fefd P 83ffd7184ec94302bd1ee74b0fab3e17 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 83ffd7184ec94302bd1ee74b0fab3e17, State: Initialized, Role: FOLLOWER
I20240503 23:26:24.974691 3959 consensus_queue.cc:260] T a0e18d0975a54e3dbeb0a13cb226fefd P 83ffd7184ec94302bd1ee74b0fab3e17 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83ffd7184ec94302bd1ee74b0fab3e17" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 32909 } } peers { permanent_uuid: "a2f9bf2e5d9b4fc19f804797bca4a3ff" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 46261 } } peers { permanent_uuid: "a3eecbaec43d4954b1096bc3121e824c" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 40649 } }
I20240503 23:26:24.975301 3960 tablet_bootstrap.cc:654] T 60267cb4849d4823a0afb2380305905f P a2f9bf2e5d9b4fc19f804797bca4a3ff: Neither blocks nor log segments found. Creating new log.
I20240503 23:26:24.975858 3959 ts_tablet_manager.cc:1398] T a0e18d0975a54e3dbeb0a13cb226fefd P 83ffd7184ec94302bd1ee74b0fab3e17: Time spent starting tablet: real 0.013s user 0.002s sys 0.002s
I20240503 23:26:24.976454 3959 tablet_bootstrap.cc:492] T d0c4430fc56644cb9f87deffe4fe9407 P 83ffd7184ec94302bd1ee74b0fab3e17: Bootstrap starting.
I20240503 23:26:24.976675 4010 raft_consensus.cc:2781] T 487ac5b9bf2d4de2b80a1b13387a2a77 P 83ffd7184ec94302bd1ee74b0fab3e17 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 23:26:24.976862 4010 raft_consensus.cc:482] T 487ac5b9bf2d4de2b80a1b13387a2a77 P 83ffd7184ec94302bd1ee74b0fab3e17 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 23:26:24.977052 4010 raft_consensus.cc:3037] T 487ac5b9bf2d4de2b80a1b13387a2a77 P 83ffd7184ec94302bd1ee74b0fab3e17 [term 0 FOLLOWER]: Advancing to term 1
I20240503 23:26:24.979916 3959 tablet_bootstrap.cc:654] T d0c4430fc56644cb9f87deffe4fe9407 P 83ffd7184ec94302bd1ee74b0fab3e17: Neither blocks nor log segments found. Creating new log.
I20240503 23:26:24.984009 4010 raft_consensus.cc:504] T 487ac5b9bf2d4de2b80a1b13387a2a77 P 83ffd7184ec94302bd1ee74b0fab3e17 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83ffd7184ec94302bd1ee74b0fab3e17" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 32909 } } peers { permanent_uuid: "a2f9bf2e5d9b4fc19f804797bca4a3ff" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 46261 } } peers { permanent_uuid: "a3eecbaec43d4954b1096bc3121e824c" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 40649 } }
I20240503 23:26:24.987578 4010 leader_election.cc:288] T 487ac5b9bf2d4de2b80a1b13387a2a77 P 83ffd7184ec94302bd1ee74b0fab3e17 [CANDIDATE]: Term 1 election: Requested vote from peers a2f9bf2e5d9b4fc19f804797bca4a3ff (127.0.100.2:46261), a3eecbaec43d4954b1096bc3121e824c (127.0.100.3:40649)
I20240503 23:26:24.988273 3836 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "487ac5b9bf2d4de2b80a1b13387a2a77" candidate_uuid: "83ffd7184ec94302bd1ee74b0fab3e17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2f9bf2e5d9b4fc19f804797bca4a3ff"
I20240503 23:26:24.988791 3910 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "487ac5b9bf2d4de2b80a1b13387a2a77" candidate_uuid: "83ffd7184ec94302bd1ee74b0fab3e17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a3eecbaec43d4954b1096bc3121e824c"
I20240503 23:26:24.989113 3995 raft_consensus.cc:482] T 1b9566be6c644cfca852ec6507fab893 P a2f9bf2e5d9b4fc19f804797bca4a3ff [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 23:26:24.991663 3995 raft_consensus.cc:504] T 1b9566be6c644cfca852ec6507fab893 P a2f9bf2e5d9b4fc19f804797bca4a3ff [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83ffd7184ec94302bd1ee74b0fab3e17" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 32909 } } peers { permanent_uuid: "a2f9bf2e5d9b4fc19f804797bca4a3ff" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 46261 } } peers { permanent_uuid: "a3eecbaec43d4954b1096bc3121e824c" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 40649 } }
I20240503 23:26:24.992924 3836 raft_consensus.cc:3037] T 487ac5b9bf2d4de2b80a1b13387a2a77 P a2f9bf2e5d9b4fc19f804797bca4a3ff [term 0 FOLLOWER]: Advancing to term 1
I20240503 23:26:24.993139 3878 leader_election.cc:302] T a14a12115b8d4fec947c6e223f1ef486 P a3eecbaec43d4954b1096bc3121e824c [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a3eecbaec43d4954b1096bc3121e824c; no voters: 83ffd7184ec94302bd1ee74b0fab3e17, a2f9bf2e5d9b4fc19f804797bca4a3ff
I20240503 23:26:24.993738 4005 raft_consensus.cc:482] T a0e18d0975a54e3dbeb0a13cb226fefd P 83ffd7184ec94302bd1ee74b0fab3e17 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 23:26:24.993841 3995 leader_election.cc:288] T 1b9566be6c644cfca852ec6507fab893 P a2f9bf2e5d9b4fc19f804797bca4a3ff [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 83ffd7184ec94302bd1ee74b0fab3e17 (127.0.100.1:32909), a3eecbaec43d4954b1096bc3121e824c (127.0.100.3:40649)
I20240503 23:26:24.993959 4005 raft_consensus.cc:504] T a0e18d0975a54e3dbeb0a13cb226fefd P 83ffd7184ec94302bd1ee74b0fab3e17 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83ffd7184ec94302bd1ee74b0fab3e17" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 32909 } } peers { permanent_uuid: "a2f9bf2e5d9b4fc19f804797bca4a3ff" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 46261 } } peers { permanent_uuid: "a3eecbaec43d4954b1096bc3121e824c" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 40649 } }
I20240503 23:26:24.994679 3762 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b9566be6c644cfca852ec6507fab893" candidate_uuid: "a2f9bf2e5d9b4fc19f804797bca4a3ff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83ffd7184ec94302bd1ee74b0fab3e17" is_pre_election: true
I20240503 23:26:24.996709 3837 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a0e18d0975a54e3dbeb0a13cb226fefd" candidate_uuid: "83ffd7184ec94302bd1ee74b0fab3e17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2f9bf2e5d9b4fc19f804797bca4a3ff" is_pre_election: true
W20240503 23:26:24.998095 3727 leader_election.cc:341] T 487ac5b9bf2d4de2b80a1b13387a2a77 P 83ffd7184ec94302bd1ee74b0fab3e17 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer a3eecbaec43d4954b1096bc3121e824c (127.0.100.3:40649): Illegal state: must be running to vote when last-logged opid is not known
I20240503 23:26:24.998425 3911 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b9566be6c644cfca852ec6507fab893" candidate_uuid: "a2f9bf2e5d9b4fc19f804797bca4a3ff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a3eecbaec43d4954b1096bc3121e824c" is_pre_election: true
W20240503 23:26:24.999881 3802 leader_election.cc:341] T 1b9566be6c644cfca852ec6507fab893 P a2f9bf2e5d9b4fc19f804797bca4a3ff [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a3eecbaec43d4954b1096bc3121e824c (127.0.100.3:40649): Illegal state: must be running to vote when last-logged opid is not known
I20240503 23:26:25.000671 3911 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a0e18d0975a54e3dbeb0a13cb226fefd" candidate_uuid: "83ffd7184ec94302bd1ee74b0fab3e17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a3eecbaec43d4954b1096bc3121e824c" is_pre_election: true
I20240503 23:26:25.001127 3911 raft_consensus.cc:2379] T a0e18d0975a54e3dbeb0a13cb226fefd P a3eecbaec43d4954b1096bc3121e824c [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 83ffd7184ec94302bd1ee74b0fab3e17 in current term 1: Already voted for candidate a2f9bf2e5d9b4fc19f804797bca4a3ff in this term.
I20240503 23:26:25.001957 3727 leader_election.cc:302] T a0e18d0975a54e3dbeb0a13cb226fefd P 83ffd7184ec94302bd1ee74b0fab3e17 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83ffd7184ec94302bd1ee74b0fab3e17; no voters: a2f9bf2e5d9b4fc19f804797bca4a3ff, a3eecbaec43d4954b1096bc3121e824c
I20240503 23:26:25.002175 3836 raft_consensus.cc:2454] T 487ac5b9bf2d4de2b80a1b13387a2a77 P a2f9bf2e5d9b4fc19f804797bca4a3ff [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83ffd7184ec94302bd1ee74b0fab3e17 in term 1.
I20240503 23:26:25.002406 4010 raft_consensus.cc:3037] T a0e18d0975a54e3dbeb0a13cb226fefd P 83ffd7184ec94302bd1ee74b0fab3e17 [term 0 FOLLOWER]: Advancing to term 1
I20240503 23:26:25.002851 3730 leader_election.cc:302] T 487ac5b9bf2d4de2b80a1b13387a2a77 P 83ffd7184ec94302bd1ee74b0fab3e17 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 83ffd7184ec94302bd1ee74b0fab3e17, a2f9bf2e5d9b4fc19f804797bca4a3ff; no voters: a3eecbaec43d4954b1096bc3121e824c
I20240503 23:26:25.004735 3963 raft_consensus.cc:2781] T 487ac5b9bf2d4de2b80a1b13387a2a77 P 83ffd7184ec94302bd1ee74b0fab3e17 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 23:26:24.998142 3958 tablet_bootstrap.cc:492] T 1b9566be6c644cfca852ec6507fab893 P a3eecbaec43d4954b1096bc3121e824c: No bootstrap required, opened a new log
I20240503 23:26:25.005164 3958 ts_tablet_manager.cc:1367] T 1b9566be6c644cfca852ec6507fab893 P a3eecbaec43d4954b1096bc3121e824c: Time spent bootstrapping tablet: real 0.099s user 0.011s sys 0.014s
I20240503 23:26:25.006211 3958 raft_consensus.cc:348] T 1b9566be6c644cfca852ec6507fab893 P a3eecbaec43d4954b1096bc3121e824c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83ffd7184ec94302bd1ee74b0fab3e17" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 32909 } } peers { permanent_uuid: "a2f9bf2e5d9b4fc19f804797bca4a3ff" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 46261 } } peers { permanent_uuid: "a3eecbaec43d4954b1096bc3121e824c" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 40649 } }
I20240503 23:26:25.006448 3958 raft_consensus.cc:374] T 1b9566be6c644cfca852ec6507fab893 P a3eecbaec43d4954b1096bc3121e824c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 23:26:25.006544 3958 raft_consensus.cc:729] T 1b9566be6c644cfca852ec6507fab893 P a3eecbaec43d4954b1096bc3121e824c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a3eecbaec43d4954b1096bc3121e824c, State: Initialized, Role: FOLLOWER
I20240503 23:26:25.006914 3958 consensus_queue.cc:260] T 1b9566be6c644cfca852ec6507fab893 P a3eecbaec43d4954b1096bc3121e824c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83ffd7184ec94302bd1ee74b0fab3e17" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 32909 } } peers { permanent_uuid: "a2f9bf2e5d9b4fc19f804797bca4a3ff" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 46261 } } peers { permanent_uuid: "a3eecbaec43d4954b1096bc3121e824c" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 40649 } }
I20240503 23:26:25.007208 4010 raft_consensus.cc:2726] T a0e18d0975a54e3dbeb0a13cb226fefd P 83ffd7184ec94302bd1ee74b0fab3e17 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 23:26:25.007064 3661 catalog_manager.cc:5484] T 481cd0f8c8d147adac217915a53ff6d9 P a3eecbaec43d4954b1096bc3121e824c reported cstate change: term changed from 0 to 1, leader changed from <none> to a3eecbaec43d4954b1096bc3121e824c (127.0.100.3). New cstate: current_term: 1 leader_uuid: "a3eecbaec43d4954b1096bc3121e824c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83ffd7184ec94302bd1ee74b0fab3e17" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 32909 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a2f9bf2e5d9b4fc19f804797bca4a3ff" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 46261 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a3eecbaec43d4954b1096bc3121e824c" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 40649 } health_report { overall_health: HEALTHY } } }
I20240503 23:26:25.007774 3958 ts_tablet_manager.cc:1398] T 1b9566be6c644cfca852ec6507fab893 P a3eecbaec43d4954b1096bc3121e824c: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 23:26:25.008242 3958 tablet_bootstrap.cc:492] T 60267cb4849d4823a0afb2380305905f P a3eecbaec43d4954b1096bc3121e824c: Bootstrap starting.
I20240503 23:26:25.008687 4010 raft_consensus.cc:482] T 60267cb4849d4823a0afb2380305905f P 83ffd7184ec94302bd1ee74b0fab3e17 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 23:26:25.007937 3661 catalog_manager.cc:5484] T d0c4430fc56644cb9f87deffe4fe9407 P a3eecbaec43d4954b1096bc3121e824c reported cstate change: term changed from 0 to 1, leader changed from <none> to a3eecbaec43d4954b1096bc3121e824c (127.0.100.3). New cstate: current_term: 1 leader_uuid: "a3eecbaec43d4954b1096bc3121e824c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83ffd7184ec94302bd1ee74b0fab3e17" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 32909 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a2f9bf2e5d9b4fc19f804797bca4a3ff" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 46261 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a3eecbaec43d4954b1096bc3121e824c" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 40649 } health_report { overall_health: HEALTHY } } }
I20240503 23:26:25.010321 4010 raft_consensus.cc:504] T 60267cb4849d4823a0afb2380305905f P 83ffd7184ec94302bd1ee74b0fab3e17 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83ffd7184ec94302bd1ee74b0fab3e17" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 32909 } } peers { permanent_uuid: "a2f9bf2e5d9b4fc19f804797bca4a3ff" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 46261 } } peers { permanent_uuid: "a3eecbaec43d4954b1096bc3121e824c" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 40649 } }
I20240503 23:26:25.011229 4001 raft_consensus.cc:2726] T a14a12115b8d4fec947c6e223f1ef486 P a3eecbaec43d4954b1096bc3121e824c [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20240503 23:26:25.018352 3762 raft_consensus.cc:2454] T 1b9566be6c644cfca852ec6507fab893 P 83ffd7184ec94302bd1ee74b0fab3e17 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2f9bf2e5d9b4fc19f804797bca4a3ff in term 0.
I20240503 23:26:25.018945 3911 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "60267cb4849d4823a0afb2380305905f" candidate_uuid: "83ffd7184ec94302bd1ee74b0fab3e17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a3eecbaec43d4954b1096bc3121e824c" is_pre_election: true
I20240503 23:26:25.019599 4010 leader_election.cc:288] T 60267cb4849d4823a0afb2380305905f P 83ffd7184ec94302bd1ee74b0fab3e17 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a2f9bf2e5d9b4fc19f804797bca4a3ff (127.0.100.2:46261), a3eecbaec43d4954b1096bc3121e824c (127.0.100.3:40649)
I20240503 23:26:25.018419 3837 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "60267cb4849d4823a0afb2380305905f" candidate_uuid: "83ffd7184ec94302bd1ee74b0fab3e17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2f9bf2e5d9b4fc19f804797bca4a3ff" is_pre_election: true
I20240503 23:26:25.020968 4005 leader_election.cc:288] T a0e18d0975a54e3dbeb0a13cb226fefd P 83ffd7184ec94302bd1ee74b0fab3e17 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a2f9bf2e5d9b4fc19f804797bca4a3ff (127.0.100.2:46261), a3eecbaec43d4954b1096bc3121e824c (127.0.100.3:40649)
W20240503 23:26:25.021891 3730 leader_election.cc:341] T 60267cb4849d4823a0afb2380305905f P 83ffd7184ec94302bd1ee74b0fab3e17 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a2f9bf2e5d9b4fc19f804797bca4a3ff (127.0.100.2:46261): Illegal state: must be running to vote when last-logged opid is not known
W20240503 23:26:25.022161 3727 leader_election.cc:341] T 60267cb4849d4823a0afb2380305905f P 83ffd7184ec94302bd1ee74b0fab3e17 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a3eecbaec43d4954b1096bc3121e824c (127.0.100.3:40649): Illegal state: must be running to vote when last-logged opid is not known
I20240503 23:26:25.022413 3727 leader_election.cc:302] T 60267cb4849d4823a0afb2380305905f P 83ffd7184ec94302bd1ee74b0fab3e17 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83ffd7184ec94302bd1ee74b0fab3e17; no voters: a2f9bf2e5d9b4fc19f804797bca4a3ff, a3eecbaec43d4954b1096bc3121e824c
I20240503 23:26:25.022886 4005 raft_consensus.cc:2726] T 60267cb4849d4823a0afb2380305905f P 83ffd7184ec94302bd1ee74b0fab3e17 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 23:26:25.023182 3804 leader_election.cc:302] T 1b9566be6c644cfca852ec6507fab893 P a2f9bf2e5d9b4fc19f804797bca4a3ff [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 83ffd7184ec94302bd1ee74b0fab3e17, a2f9bf2e5d9b4fc19f804797bca4a3ff; no voters: a3eecbaec43d4954b1096bc3121e824c
I20240503 23:26:25.023591 3963 raft_consensus.cc:686] T 487ac5b9bf2d4de2b80a1b13387a2a77 P 83ffd7184ec94302bd1ee74b0fab3e17 [term 1 LEADER]: Becoming Leader. State: Replica: 83ffd7184ec94302bd1ee74b0fab3e17, State: Running, Role: LEADER
I20240503 23:26:25.023651 3995 raft_consensus.cc:2781] T 1b9566be6c644cfca852ec6507fab893 P a2f9bf2e5d9b4fc19f804797bca4a3ff [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 23:26:25.023787 3995 raft_consensus.cc:482] T 1b9566be6c644cfca852ec6507fab893 P a2f9bf2e5d9b4fc19f804797bca4a3ff [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 23:26:25.023929 3995 raft_consensus.cc:3037] T 1b9566be6c644cfca852ec6507fab893 P a2f9bf2e5d9b4fc19f804797bca4a3ff [term 0 FOLLOWER]: Advancing to term 1
I20240503 23:26:25.023990 3963 consensus_queue.cc:237] T 487ac5b9bf2d4de2b80a1b13387a2a77 P 83ffd7184ec94302bd1ee74b0fab3e17 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83ffd7184ec94302bd1ee74b0fab3e17" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 32909 } } peers { permanent_uuid: "a2f9bf2e5d9b4fc19f804797bca4a3ff" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 46261 } } peers { permanent_uuid: "a3eecbaec43d4954b1096bc3121e824c" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 40649 } }
I20240503 23:26:25.028646 3960 tablet_bootstrap.cc:492] T 60267cb4849d4823a0afb2380305905f P a2f9bf2e5d9b4fc19f804797bca4a3ff: No bootstrap required, opened a new log
I20240503 23:26:25.030609 3995 raft_consensus.cc:504] T 1b9566be6c644cfca852ec6507fab893 P a2f9bf2e5d9b4fc19f804797bca4a3ff [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83ffd7184ec94302bd1ee74b0fab3e17" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 32909 } } peers { permanent_uuid: "a2f9bf2e5d9b4fc19f804797bca4a3ff" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 46261 } } peers { permanent_uuid: "a3eecbaec43d4954b1096bc3121e824c" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 40649 } }
I20240503 23:26:25.031405 3960 ts_tablet_manager.cc:1367] T 60267cb4849d4823a0afb2380305905f P a2f9bf2e5d9b4fc19f804797bca4a3ff: Time spent bootstrapping tablet: real 0.059s user 0.026s sys 0.018s
I20240503 23:26:25.032096 3662 catalog_manager.cc:5484] T a0e18d0975a54e3dbeb0a13cb226fefd P a2f9bf2e5d9b4fc19f804797bca4a3ff reported cstate change: term changed from 0 to 1, leader changed from <none> to a2f9bf2e5d9b4fc19f804797bca4a3ff (127.0.100.2). New cstate: current_term: 1 leader_uuid: "a2f9bf2e5d9b4fc19f804797bca4a3ff" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83ffd7184ec94302bd1ee74b0fab3e17" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 32909 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a2f9bf2e5d9b4fc19f804797bca4a3ff" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 46261 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a3eecbaec43d4954b1096bc3121e824c" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 40649 } health_report { overall_health: UNKNOWN } } }
I20240503 23:26:25.034206 3960 raft_consensus.cc:348] T 60267cb4849d4823a0afb2380305905f P a2f9bf2e5d9b4fc19f804797bca4a3ff [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83ffd7184ec94302bd1ee74b0fab3e17" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 32909 } } peers { permanent_uuid: "a2f9bf2e5d9b4fc19f804797bca4a3ff" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 46261 } } peers { permanent_uuid: "a3eecbaec43d4954b1096bc3121e824c" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 40649 } }
I20240503 23:26:25.034663 3762 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b9566be6c644cfca852ec6507fab893" candidate_uuid: "a2f9bf2e5d9b4fc19f804797bca4a3ff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83ffd7184ec94302bd1ee74b0fab3e17"
I20240503 23:26:25.036357 3959 tablet_bootstrap.cc:492] T d0c4430fc56644cb9f87deffe4fe9407 P 83ffd7184ec94302bd1ee74b0fab3e17: No bootstrap required, opened a new log
I20240503 23:26:25.037596 3995 leader_election.cc:288] T 1b9566be6c644cfca852ec6507fab893 P a2f9bf2e5d9b4fc19f804797bca4a3ff [CANDIDATE]: Term 1 election: Requested vote from peers 83ffd7184ec94302bd1ee74b0fab3e17 (127.0.100.1:32909), a3eecbaec43d4954b1096bc3121e824c (127.0.100.3:40649)
I20240503 23:26:25.037886 3960 raft_consensus.cc:374] T 60267cb4849d4823a0afb2380305905f P a2f9bf2e5d9b4fc19f804797bca4a3ff [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 23:26:25.037922 3911 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b9566be6c644cfca852ec6507fab893" candidate_uuid: "a2f9bf2e5d9b4fc19f804797bca4a3ff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a3eecbaec43d4954b1096bc3121e824c"
I20240503 23:26:25.038013 3960 raft_consensus.cc:729] T 60267cb4849d4823a0afb2380305905f P a2f9bf2e5d9b4fc19f804797bca4a3ff [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2f9bf2e5d9b4fc19f804797bca4a3ff, State: Initialized, Role: FOLLOWER
I20240503 23:26:25.038178 3911 raft_consensus.cc:3037] T 1b9566be6c644cfca852ec6507fab893 P a3eecbaec43d4954b1096bc3121e824c [term 0 FOLLOWER]: Advancing to term 1
I20240503 23:26:25.038672 3762 raft_consensus.cc:3037] T 1b9566be6c644cfca852ec6507fab893 P 83ffd7184ec94302bd1ee74b0fab3e17 [term 0 FOLLOWER]: Advancing to term 1
I20240503 23:26:25.038578 3960 consensus_queue.cc:260] T 60267cb4849d4823a0afb2380305905f P a2f9bf2e5d9b4fc19f804797bca4a3ff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83ffd7184ec94302bd1ee74b0fab3e17" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 32909 } } peers { permanent_uuid: "a2f9bf2e5d9b4fc19f804797bca4a3ff" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 46261 } } peers { permanent_uuid: "a3eecbaec43d4954b1096bc3121e824c" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 40649 } }
I20240503 23:26:25.039378 3960 ts_tablet_manager.cc:1398] T 60267cb4849d4823a0afb2380305905f P a2f9bf2e5d9b4fc19f804797bca4a3ff: Time spent starting tablet: real 0.008s user 0.000s sys 0.002s
I20240503 23:26:25.039151 3660 catalog_manager.cc:5484] T a14a12115b8d4fec947c6e223f1ef486 P 83ffd7184ec94302bd1ee74b0fab3e17 reported cstate change: term changed from 0 to 1, leader changed from <none> to 83ffd7184ec94302bd1ee74b0fab3e17 (127.0.100.1). New cstate: current_term: 1 leader_uuid: "83ffd7184ec94302bd1ee74b0fab3e17" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83ffd7184ec94302bd1ee74b0fab3e17" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 32909 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a2f9bf2e5d9b4fc19f804797bca4a3ff" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 46261 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a3eecbaec43d4954b1096bc3121e824c" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 40649 } health_report { overall_health: UNKNOWN } } }
I20240503 23:26:25.041501 3911 raft_consensus.cc:2454] T 1b9566be6c644cfca852ec6507fab893 P a3eecbaec43d4954b1096bc3121e824c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2f9bf2e5d9b4fc19f804797bca4a3ff in term 1.
I20240503 23:26:25.041779 3762 raft_consensus.cc:2454] T 1b9566be6c644cfca852ec6507fab893 P 83ffd7184ec94302bd1ee74b0fab3e17 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2f9bf2e5d9b4fc19f804797bca4a3ff in term 1.
I20240503 23:26:25.042491 3802 leader_election.cc:302] T 1b9566be6c644cfca852ec6507fab893 P a2f9bf2e5d9b4fc19f804797bca4a3ff [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a2f9bf2e5d9b4fc19f804797bca4a3ff, a3eecbaec43d4954b1096bc3121e824c; no voters:
I20240503 23:26:25.038460 3958 tablet_bootstrap.cc:654] T 60267cb4849d4823a0afb2380305905f P a3eecbaec43d4954b1096bc3121e824c: Neither blocks nor log segments found. Creating new log.
I20240503 23:26:25.043265 3995 raft_consensus.cc:2781] T 1b9566be6c644cfca852ec6507fab893 P a2f9bf2e5d9b4fc19f804797bca4a3ff [term 1 FOLLOWER]: Leader election won for term 1
I20240503 23:26:25.041870 3959 ts_tablet_manager.cc:1367] T d0c4430fc56644cb9f87deffe4fe9407 P 83ffd7184ec94302bd1ee74b0fab3e17: Time spent bootstrapping tablet: real 0.060s user 0.014s sys 0.012s
I20240503 23:26:25.043488 3995 raft_consensus.cc:686] T 1b9566be6c644cfca852ec6507fab893 P a2f9bf2e5d9b4fc19f804797bca4a3ff [term 1 LEADER]: Becoming Leader. State: Replica: a2f9bf2e5d9b4fc19f804797bca4a3ff, State: Running, Role: LEADER
I20240503 23:26:25.043817 3995 consensus_queue.cc:237] T 1b9566be6c644cfca852ec6507fab893 P a2f9bf2e5d9b4fc19f804797bca4a3ff [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83ffd7184ec94302bd1ee74b0fab3e17" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 32909 } } peers { permanent_uuid: "a2f9bf2e5d9b4fc19f804797bca4a3ff" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 46261 } } peers { permanent_uuid: "a3eecbaec43d4954b1096bc3121e824c" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 40649 } }
I20240503 23:26:25.044265 3959 raft_consensus.cc:348] T d0c4430fc56644cb9f87deffe4fe9407 P 83ffd7184ec94302bd1ee74b0fab3e17 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83ffd7184ec94302bd1ee74b0fab3e17" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 32909 } } peers { permanent_uuid: "a2f9bf2e5d9b4fc19f804797bca4a3ff" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 46261 } } peers { permanent_uuid: "a3eecbaec43d4954b1096bc3121e824c" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 40649 } }
I20240503 23:26:25.044512 3959 raft_consensus.cc:374] T d0c4430fc56644cb9f87deffe4fe9407 P 83ffd7184ec94302bd1ee74b0fab3e17 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 23:26:25.044634 3959 raft_consensus.cc:729] T d0c4430fc56644cb9f87deffe4fe9407 P 83ffd7184ec94302bd1ee74b0fab3e17 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 83ffd7184ec94302bd1ee74b0fab3e17, State: Initialized, Role: FOLLOWER
I20240503 23:26:25.044870 3959 consensus_queue.cc:260] T d0c4430fc56644cb9f87deffe4fe9407 P 83ffd7184ec94302bd1ee74b0fab3e17 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83ffd7184ec94302bd1ee74b0fab3e17" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 32909 } } peers { permanent_uuid: "a2f9bf2e5d9b4fc19f804797bca4a3ff" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 46261 } } peers { permanent_uuid: "a3eecbaec43d4954b1096bc3121e824c" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 40649 } }
I20240503 23:26:25.045634 3959 ts_tablet_manager.cc:1398] T d0c4430fc56644cb9f87deffe4fe9407 P 83ffd7184ec94302bd1ee74b0fab3e17: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 23:26:25.045989 3959 tablet_bootstrap.cc:492] T 6039230222ff4e57929bed49cd56caf5 P 83ffd7184ec94302bd1ee74b0fab3e17: Bootstrap starting.
I20240503 23:26:25.048027 3660 catalog_manager.cc:5484] T 1b9566be6c644cfca852ec6507fab893 P a2f9bf2e5d9b4fc19f804797bca4a3ff reported cstate change: term changed from 0 to 1, leader changed from <none> to a2f9bf2e5d9b4fc19f804797bca4a3ff (127.0.100.2). New cstate: current_term: 1 leader_uuid: "a2f9bf2e5d9b4fc19f804797bca4a3ff" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83ffd7184ec94302bd1ee74b0fab3e17" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 32909 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a2f9bf2e5d9b4fc19f804797bca4a3ff" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 46261 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a3eecbaec43d4954b1096bc3121e824c" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 40649 } health_report { overall_health: UNKNOWN } } }
I20240503 23:26:25.048619 3959 tablet_bootstrap.cc:654] T 6039230222ff4e57929bed49cd56caf5 P 83ffd7184ec94302bd1ee74b0fab3e17: Neither blocks nor log segments found. Creating new log.
I20240503 23:26:25.048754 3662 catalog_manager.cc:5484] T 487ac5b9bf2d4de2b80a1b13387a2a77 P 83ffd7184ec94302bd1ee74b0fab3e17 reported cstate change: term changed from 0 to 1, leader changed from <none> to 83ffd7184ec94302bd1ee74b0fab3e17 (127.0.100.1). New cstate: current_term: 1 leader_uuid: "83ffd7184ec94302bd1ee74b0fab3e17" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83ffd7184ec94302bd1ee74b0fab3e17" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 32909 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a2f9bf2e5d9b4fc19f804797bca4a3ff" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 46261 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a3eecbaec43d4954b1096bc3121e824c" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 40649 } health_report { overall_health: UNKNOWN } } }
I20240503 23:26:25.051605 3959 tablet_bootstrap.cc:492] T 6039230222ff4e57929bed49cd56caf5 P 83ffd7184ec94302bd1ee74b0fab3e17: No bootstrap required, opened a new log
I20240503 23:26:25.051913 3959 ts_tablet_manager.cc:1367] T 6039230222ff4e57929bed49cd56caf5 P 83ffd7184ec94302bd1ee74b0fab3e17: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20240503 23:26:25.053292 3959 raft_consensus.cc:348] T 6039230222ff4e57929bed49cd56caf5 P 83ffd7184ec94302bd1ee74b0fab3e17 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3eecbaec43d4954b1096bc3121e824c" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 40649 } } peers { permanent_uuid: "a2f9bf2e5d9b4fc19f804797bca4a3ff" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 46261 } } peers { permanent_uuid: "83ffd7184ec94302bd1ee74b0fab3e17" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 32909 } }
I20240503 23:26:25.053560 3959 raft_consensus.cc:374] T 6039230222ff4e57929bed49cd56caf5 P 83ffd7184ec94302bd1ee74b0fab3e17 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 23:26:25.053670 3959 raft_consensus.cc:729] T 6039230222ff4e57929bed49cd56caf5 P 83ffd7184ec94302bd1ee74b0fab3e17 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 83ffd7184ec94302bd1ee74b0fab3e17, State: Initialized, Role: FOLLOWER
I20240503 23:26:25.053939 3959 consensus_queue.cc:260] T 6039230222ff4e57929bed49cd56caf5 P 83ffd7184ec94302bd1ee74b0fab3e17 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3eecbaec43d4954b1096bc3121e824c" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 40649 } } peers { permanent_uuid: "a2f9bf2e5d9b4fc19f804797bca4a3ff" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 46261 } } peers { permanent_uuid: "83ffd7184ec94302bd1ee74b0fab3e17" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 32909 } }
I20240503 23:26:25.055534 3958 tablet_bootstrap.cc:492] T 60267cb4849d4823a0afb2380305905f P a3eecbaec43d4954b1096bc3121e824c: No bootstrap required, opened a new log
I20240503 23:26:25.055536 3959 ts_tablet_manager.cc:1398] T 6039230222ff4e57929bed49cd56caf5 P 83ffd7184ec94302bd1ee74b0fab3e17: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240503 23:26:25.055888 3958 ts_tablet_manager.cc:1367] T 60267cb4849d4823a0afb2380305905f P a3eecbaec43d4954b1096bc3121e824c: Time spent bootstrapping tablet: real 0.048s user 0.006s sys 0.006s
I20240503 23:26:25.056025 3959 tablet_bootstrap.cc:492] T 059f6233ea514fe69b8c6504efabeff0 P 83ffd7184ec94302bd1ee74b0fab3e17: Bootstrap starting.
I20240503 23:26:25.056914 3958 raft_consensus.cc:348] T 60267cb4849d4823a0afb2380305905f P a3eecbaec43d4954b1096bc3121e824c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83ffd7184ec94302bd1ee74b0fab3e17" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 32909 } } peers { permanent_uuid: "a2f9bf2e5d9b4fc19f804797bca4a3ff" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 46261 } } peers { permanent_uuid: "a3eecbaec43d4954b1096bc3121e824c" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 40649 } }
I20240503 23:26:25.057233 3958 raft_consensus.cc:374] T 60267cb4849d4823a0afb2380305905f P a3eecbaec43d4954b1096bc3121e824c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 23:26:25.057346 3958 raft_consensus.cc:729] T 60267cb4849d4823a0afb2380305905f P a3eecbaec43d4954b1096bc3121e824c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a3eecbaec43d4954b1096bc3121e824c, State: Initialized, Role: FOLLOWER
I20240503 23:26:25.057665 3958 consensus_queue.cc:260] T 60267cb4849d4823a0afb2380305905f P a3eecbaec43d4954b1096bc3121e824c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83ffd7184ec94302bd1ee74b0fab3e17" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 32909 } } peers { permanent_uuid: "a2f9bf2e5d9b4fc19f804797bca4a3ff" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 46261 } } peers { permanent_uuid: "a3eecbaec43d4954b1096bc3121e824c" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 40649 } }
I20240503 23:26:25.058351 3958 ts_tablet_manager.cc:1398] T 60267cb4849d4823a0afb2380305905f P a3eecbaec43d4954b1096bc3121e824c: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 23:26:25.058451 3959 tablet_bootstrap.cc:654] T 059f6233ea514fe69b8c6504efabeff0 P 83ffd7184ec94302bd1ee74b0fab3e17: Neither blocks nor log segments found. Creating new log.
I20240503 23:26:25.058745 3958 tablet_bootstrap.cc:492] T 55cf664d5e874c0d9b9adf8bdc9c43ed P a3eecbaec43d4954b1096bc3121e824c: Bootstrap starting.
I20240503 23:26:25.060725 3958 tablet_bootstrap.cc:654] T 55cf664d5e874c0d9b9adf8bdc9c43ed P a3eecbaec43d4954b1096bc3121e824c: Neither blocks nor log segments found. Creating new log.
I20240503 23:26:25.060971 3959 tablet_bootstrap.cc:492] T 059f6233ea514fe69b8c6504efabeff0 P 83ffd7184ec94302bd1ee74b0fab3e17: No bootstrap required, opened a new log
I20240503 23:26:25.061208 3959 ts_tablet_manager.cc:1367] T 059f6233ea514fe69b8c6504efabeff0 P 83ffd7184ec94302bd1ee74b0fab3e17: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20240503 23:26:25.062203 3959 raft_consensus.cc:348] T 059f6233ea514fe69b8c6504efabeff0 P 83ffd7184ec94302bd1ee74b0fab3e17 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83ffd7184ec94302bd1ee74b0fab3e17" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 32909 } } peers { permanent_uuid: "a2f9bf2e5d9b4fc19f804797bca4a3ff" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 46261 } } peers { permanent_uuid: "a3eecbaec43d4954b1096bc3121e824c" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 40649 } }
I20240503 23:26:25.062438 3959 raft_consensus.cc:374] T 059f6233ea514fe69b8c6504efabeff0 P 83ffd7184ec94302bd1ee74b0fab3e17 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 23:26:25.062506 3959 raft_consensus.cc:729] T 059f6233ea514fe69b8c6504efabeff0 P 83ffd7184ec94302bd1ee74b0fab3e17 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 83ffd7184ec94302bd1ee74b0fab3e17, State: Initialized, Role: FOLLOWER
I20240503 23:26:25.062785 3959 consensus_queue.cc:260] T 059f6233ea514fe69b8c6504efabeff0 P 83ffd7184ec94302bd1ee74b0fab3e17 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83ffd7184ec94302bd1ee74b0fab3e17" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 32909 } } peers { permanent_uuid: "a2f9bf2e5d9b4fc19f804797bca4a3ff" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 46261 } } peers { permanent_uuid: "a3eecbaec43d4954b1096bc3121e824c" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 40649 } }
I20240503 23:26:25.063416 3959 ts_tablet_manager.cc:1398] T 059f6233ea514fe69b8c6504efabeff0 P 83ffd7184ec94302bd1ee74b0fab3e17: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240503 23:26:25.063417 3958 tablet_bootstrap.cc:492] T 55cf664d5e874c0d9b9adf8bdc9c43ed P a3eecbaec43d4954b1096bc3121e824c: No bootstrap required, opened a new log
I20240503 23:26:25.063668 3958 ts_tablet_manager.cc:1367] T 55cf664d5e874c0d9b9adf8bdc9c43ed P a3eecbaec43d4954b1096bc3121e824c: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.003s
I20240503 23:26:25.064421 3958 raft_consensus.cc:348] T 55cf664d5e874c0d9b9adf8bdc9c43ed P a3eecbaec43d4954b1096bc3121e824c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83ffd7184ec94302bd1ee74b0fab3e17" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 32909 } } peers { permanent_uuid: "a2f9bf2e5d9b4fc19f804797bca4a3ff" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 46261 } } peers { permanent_uuid: "a3eecbaec43d4954b1096bc3121e824c" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 40649 } }
I20240503 23:26:25.064656 3958 raft_consensus.cc:374] T 55cf664d5e874c0d9b9adf8bdc9c43ed P a3eecbaec43d4954b1096bc3121e824c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 23:26:25.064774 3958 raft_consensus.cc:729] T 55cf664d5e874c0d9b9adf8bdc9c43ed P a3eecbaec43d4954b1096bc3121e824c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a3eecbaec43d4954b1096bc3121e824c, State: Initialized, Role: FOLLOWER
I20240503 23:26:25.065013 3958 consensus_queue.cc:260] T 55cf664d5e874c0d9b9adf8bdc9c43ed P a3eecbaec43d4954b1096bc3121e824c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83ffd7184ec94302bd1ee74b0fab3e17" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 32909 } } peers { permanent_uuid: "a2f9bf2e5d9b4fc19f804797bca4a3ff" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 46261 } } peers { permanent_uuid: "a3eecbaec43d4954b1096bc3121e824c" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 40649 } }
I20240503 23:26:25.065567 3958 ts_tablet_manager.cc:1398] T 55cf664d5e874c0d9b9adf8bdc9c43ed P a3eecbaec43d4954b1096bc3121e824c: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240503 23:26:25.065845 3958 tablet_bootstrap.cc:492] T 487ac5b9bf2d4de2b80a1b13387a2a77 P a3eecbaec43d4954b1096bc3121e824c: Bootstrap starting.
I20240503 23:26:25.068042 3958 tablet_bootstrap.cc:654] T 487ac5b9bf2d4de2b80a1b13387a2a77 P a3eecbaec43d4954b1096bc3121e824c: Neither blocks nor log segments found. Creating new log.
I20240503 23:26:25.071422 3958 tablet_bootstrap.cc:492] T 487ac5b9bf2d4de2b80a1b13387a2a77 P a3eecbaec43d4954b1096bc3121e824c: No bootstrap required, opened a new log
I20240503 23:26:25.071635 3958 ts_tablet_manager.cc:1367] T 487ac5b9bf2d4de2b80a1b13387a2a77 P a3eecbaec43d4954b1096bc3121e824c: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.005s
I20240503 23:26:25.072412 3958 raft_consensus.cc:348] T 487ac5b9bf2d4de2b80a1b13387a2a77 P a3eecbaec43d4954b1096bc3121e824c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83ffd7184ec94302bd1ee74b0fab3e17" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 32909 } } peers { permanent_uuid: "a2f9bf2e5d9b4fc19f804797bca4a3ff" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 46261 } } peers { permanent_uuid: "a3eecbaec43d4954b1096bc3121e824c" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 40649 } }
I20240503 23:26:25.072706 3958 raft_consensus.cc:374] T 487ac5b9bf2d4de2b80a1b13387a2a77 P a3eecbaec43d4954b1096bc3121e824c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 23:26:25.072806 3958 raft_consensus.cc:729] T 487ac5b9bf2d4de2b80a1b13387a2a77 P a3eecbaec43d4954b1096bc3121e824c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a3eecbaec43d4954b1096bc3121e824c, State: Initialized, Role: FOLLOWER
I20240503 23:26:25.073069 3958 consensus_queue.cc:260] T 487ac5b9bf2d4de2b80a1b13387a2a77 P a3eecbaec43d4954b1096bc3121e824c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83ffd7184ec94302bd1ee74b0fab3e17" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 32909 } } peers { permanent_uuid: "a2f9bf2e5d9b4fc19f804797bca4a3ff" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 46261 } } peers { permanent_uuid: "a3eecbaec43d4954b1096bc3121e824c" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 40649 } }
I20240503 23:26:25.073581 3958 ts_tablet_manager.cc:1398] T 487ac5b9bf2d4de2b80a1b13387a2a77 P a3eecbaec43d4954b1096bc3121e824c: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
W20240503 23:26:25.101190 3710 auto_rebalancer.cc:227] Could not retrieve cluster info: Service unavailable: Tablet not running
I20240503 23:26:25.102144 3715 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240503 23:26:25.102551 3715 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240503 23:26:25.217278 3963 raft_consensus.cc:482] T 98c9500acb43423b86f54ce7a4d98a72 P 83ffd7184ec94302bd1ee74b0fab3e17 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 23:26:25.217533 3963 raft_consensus.cc:504] T 98c9500acb43423b86f54ce7a4d98a72 P 83ffd7184ec94302bd1ee74b0fab3e17 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83ffd7184ec94302bd1ee74b0fab3e17" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 32909 } } peers { permanent_uuid: "a2f9bf2e5d9b4fc19f804797bca4a3ff" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 46261 } } peers { permanent_uuid: "a3eecbaec43d4954b1096bc3121e824c" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 40649 } }
I20240503 23:26:25.218307 3963 leader_election.cc:288] T 98c9500acb43423b86f54ce7a4d98a72 P 83ffd7184ec94302bd1ee74b0fab3e17 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a2f9bf2e5d9b4fc19f804797bca4a3ff (127.0.100.2:46261), a3eecbaec43d4954b1096bc3121e824c (127.0.100.3:40649)
I20240503 23:26:25.218992 3837 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "98c9500acb43423b86f54ce7a4d98a72" candidate_uuid: "83ffd7184ec94302bd1ee74b0fab3e17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2f9bf2e5d9b4fc19f804797bca4a3ff" is_pre_election: true
I20240503 23:26:25.219000 3911 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "98c9500acb43423b86f54ce7a4d98a72" candidate_uuid: "83ffd7184ec94302bd1ee74b0fab3e17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a3eecbaec43d4954b1096bc3121e824c" is_pre_election: true
I20240503 23:26:25.219313 3837 raft_consensus.cc:2454] T 98c9500acb43423b86f54ce7a4d98a72 P a2f9bf2e5d9b4fc19f804797bca4a3ff [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83ffd7184ec94302bd1ee74b0fab3e17 in term 0.
I20240503 23:26:25.219364 3911 raft_consensus.cc:2454] T 98c9500acb43423b86f54ce7a4d98a72 P a3eecbaec43d4954b1096bc3121e824c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83ffd7184ec94302bd1ee74b0fab3e17 in term 0.
I20240503 23:26:25.219843 3730 leader_election.cc:302] T 98c9500acb43423b86f54ce7a4d98a72 P 83ffd7184ec94302bd1ee74b0fab3e17 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 83ffd7184ec94302bd1ee74b0fab3e17, a2f9bf2e5d9b4fc19f804797bca4a3ff; no voters:
I20240503 23:26:25.220252 3963 raft_consensus.cc:2781] T 98c9500acb43423b86f54ce7a4d98a72 P 83ffd7184ec94302bd1ee74b0fab3e17 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 23:26:25.220373 3963 raft_consensus.cc:482] T 98c9500acb43423b86f54ce7a4d98a72 P 83ffd7184ec94302bd1ee74b0fab3e17 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 23:26:25.220477 3963 raft_consensus.cc:3037] T 98c9500acb43423b86f54ce7a4d98a72 P 83ffd7184ec94302bd1ee74b0fab3e17 [term 0 FOLLOWER]: Advancing to term 1
I20240503 23:26:25.229820 3963 raft_consensus.cc:504] T 98c9500acb43423b86f54ce7a4d98a72 P 83ffd7184ec94302bd1ee74b0fab3e17 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83ffd7184ec94302bd1ee74b0fab3e17" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 32909 } } peers { permanent_uuid: "a2f9bf2e5d9b4fc19f804797bca4a3ff" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 46261 } } peers { permanent_uuid: "a3eecbaec43d4954b1096bc3121e824c" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 40649 } }
I20240503 23:26:25.230602 3963 leader_election.cc:288] T 98c9500acb43423b86f54ce7a4d98a72 P 83ffd7184ec94302bd1ee74b0fab3e17 [CANDIDATE]: Term 1 election: Requested vote from peers a2f9bf2e5d9b4fc19f804797bca4a3ff (127.0.100.2:46261), a3eecbaec43d4954b1096bc3121e824c (127.0.100.3:40649)
I20240503 23:26:25.231309 3837 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "98c9500acb43423b86f54ce7a4d98a72" candidate_uuid: "83ffd7184ec94302bd1ee74b0fab3e17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2f9bf2e5d9b4fc19f804797bca4a3ff"
I20240503 23:26:25.231315 3911 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "98c9500acb43423b86f54ce7a4d98a72" candidate_uuid: "83ffd7184ec94302bd1ee74b0fab3e17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a3eecbaec43d4954b1096bc3121e824c"
I20240503 23:26:25.231524 3837 raft_consensus.cc:3037] T 98c9500acb43423b86f54ce7a4d98a72 P a2f9bf2e5d9b4fc19f804797bca4a3ff [term 0 FOLLOWER]: Advancing to term 1
I20240503 23:26:25.231597 3911 raft_consensus.cc:3037] T 98c9500acb43423b86f54ce7a4d98a72 P a3eecbaec43d4954b1096bc3121e824c [term 0 FOLLOWER]: Advancing to term 1
I20240503 23:26:25.233794 3837 raft_consensus.cc:2454] T 98c9500acb43423b86f54ce7a4d98a72 P a2f9bf2e5d9b4fc19f804797bca4a3ff [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83ffd7184ec94302bd1ee74b0fab3e17 in term 1.
I20240503 23:26:25.233961 3911 raft_consensus.cc:2454] T 98c9500acb43423b86f54ce7a4d98a72 P a3eecbaec43d4954b1096bc3121e824c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83ffd7184ec94302bd1ee74b0fab3e17 in term 1.
I20240503 23:26:25.234439 3727 leader_election.cc:302] T 98c9500acb43423b86f54ce7a4d98a72 P 83ffd7184ec94302bd1ee74b0fab3e17 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 83ffd7184ec94302bd1ee74b0fab3e17, a2f9bf2e5d9b4fc19f804797bca4a3ff, a3eecbaec43d4954b1096bc3121e824c; no voters:
I20240503 23:26:25.234776 3963 raft_consensus.cc:2781] T 98c9500acb43423b86f54ce7a4d98a72 P 83ffd7184ec94302bd1ee74b0fab3e17 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 23:26:25.234937 3963 raft_consensus.cc:686] T 98c9500acb43423b86f54ce7a4d98a72 P 83ffd7184ec94302bd1ee74b0fab3e17 [term 1 LEADER]: Becoming Leader. State: Replica: 83ffd7184ec94302bd1ee74b0fab3e17, State: Running, Role: LEADER
I20240503 23:26:25.235205 3963 consensus_queue.cc:237] T 98c9500acb43423b86f54ce7a4d98a72 P 83ffd7184ec94302bd1ee74b0fab3e17 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83ffd7184ec94302bd1ee74b0fab3e17" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 32909 } } peers { permanent_uuid: "a2f9bf2e5d9b4fc19f804797bca4a3ff" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 46261 } } peers { permanent_uuid: "a3eecbaec43d4954b1096bc3121e824c" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 40649 } }
I20240503 23:26:25.238489 3660 catalog_manager.cc:5484] T 98c9500acb43423b86f54ce7a4d98a72 P 83ffd7184ec94302bd1ee74b0fab3e17 reported cstate change: term changed from 0 to 1, leader changed from <none> to 83ffd7184ec94302bd1ee74b0fab3e17 (127.0.100.1). New cstate: current_term: 1 leader_uuid: "83ffd7184ec94302bd1ee74b0fab3e17" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83ffd7184ec94302bd1ee74b0fab3e17" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 32909 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a2f9bf2e5d9b4fc19f804797bca4a3ff" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 46261 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a3eecbaec43d4954b1096bc3121e824c" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 40649 } health_report { overall_health: UNKNOWN } } }
I20240503 23:26:25.268350 4001 raft_consensus.cc:482] T 487ac5b9bf2d4de2b80a1b13387a2a77 P a3eecbaec43d4954b1096bc3121e824c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 23:26:25.268646 4001 raft_consensus.cc:504] T 487ac5b9bf2d4de2b80a1b13387a2a77 P a3eecbaec43d4954b1096bc3121e824c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83ffd7184ec94302bd1ee74b0fab3e17" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 32909 } } peers { permanent_uuid: "a2f9bf2e5d9b4fc19f804797bca4a3ff" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 46261 } } peers { permanent_uuid: "a3eecbaec43d4954b1096bc3121e824c" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 40649 } }
I20240503 23:26:25.269439 4001 leader_election.cc:288] T 487ac5b9bf2d4de2b80a1b13387a2a77 P a3eecbaec43d4954b1096bc3121e824c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 83ffd7184ec94302bd1ee74b0fab3e17 (127.0.100.1:32909), a2f9bf2e5d9b4fc19f804797bca4a3ff (127.0.100.2:46261)
I20240503 23:26:25.270056 3762 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "487ac5b9bf2d4de2b80a1b13387a2a77" candidate_uuid: "a3eecbaec43d4954b1096bc3121e824c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83ffd7184ec94302bd1ee74b0fab3e17" is_pre_election: true
I20240503 23:26:25.270068 3837 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "487ac5b9bf2d4de2b80a1b13387a2a77" candidate_uuid: "a3eecbaec43d4954b1096bc3121e824c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2f9bf2e5d9b4fc19f804797bca4a3ff" is_pre_election: true
I20240503 23:26:25.270434 3837 raft_consensus.cc:2379] T 487ac5b9bf2d4de2b80a1b13387a2a77 P a2f9bf2e5d9b4fc19f804797bca4a3ff [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a3eecbaec43d4954b1096bc3121e824c in current term 1: Already voted for candidate 83ffd7184ec94302bd1ee74b0fab3e17 in this term.
I20240503 23:26:25.270587 3911 raft_consensus.cc:3037] T 55cf664d5e874c0d9b9adf8bdc9c43ed P a3eecbaec43d4954b1096bc3121e824c [term 0 FOLLOWER]: Advancing to term 1
I20240503 23:26:25.270818 3879 leader_election.cc:302] T 487ac5b9bf2d4de2b80a1b13387a2a77 P a3eecbaec43d4954b1096bc3121e824c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a3eecbaec43d4954b1096bc3121e824c; no voters: 83ffd7184ec94302bd1ee74b0fab3e17, a2f9bf2e5d9b4fc19f804797bca4a3ff
I20240503 23:26:25.271077 4001 raft_consensus.cc:3037] T 487ac5b9bf2d4de2b80a1b13387a2a77 P a3eecbaec43d4954b1096bc3121e824c [term 0 FOLLOWER]: Advancing to term 1
I20240503 23:26:25.273288 4001 raft_consensus.cc:2726] T 487ac5b9bf2d4de2b80a1b13387a2a77 P a3eecbaec43d4954b1096bc3121e824c [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 23:26:25.274170 3995 consensus_queue.cc:1035] T 55cf664d5e874c0d9b9adf8bdc9c43ed P a2f9bf2e5d9b4fc19f804797bca4a3ff [LEADER]: Connected to new peer: Peer: permanent_uuid: "a3eecbaec43d4954b1096bc3121e824c" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 40649 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 23:26:25.279502 3995 consensus_queue.cc:1035] T 55cf664d5e874c0d9b9adf8bdc9c43ed P a2f9bf2e5d9b4fc19f804797bca4a3ff [LEADER]: Connected to new peer: Peer: permanent_uuid: "83ffd7184ec94302bd1ee74b0fab3e17" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 32909 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 23:26:25.283442 3963 raft_consensus.cc:482] T 6039230222ff4e57929bed49cd56caf5 P 83ffd7184ec94302bd1ee74b0fab3e17 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 23:26:25.283623 3963 raft_consensus.cc:504] T 6039230222ff4e57929bed49cd56caf5 P 83ffd7184ec94302bd1ee74b0fab3e17 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3eecbaec43d4954b1096bc3121e824c" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 40649 } } peers { permanent_uuid: "a2f9bf2e5d9b4fc19f804797bca4a3ff" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 46261 } } peers { permanent_uuid: "83ffd7184ec94302bd1ee74b0fab3e17" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 32909 } }
I20240503 23:26:25.284191 3963 leader_election.cc:288] T 6039230222ff4e57929bed49cd56caf5 P 83ffd7184ec94302bd1ee74b0fab3e17 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a3eecbaec43d4954b1096bc3121e824c (127.0.100.3:40649), a2f9bf2e5d9b4fc19f804797bca4a3ff (127.0.100.2:46261)
I20240503 23:26:25.284659 3837 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6039230222ff4e57929bed49cd56caf5" candidate_uuid: "83ffd7184ec94302bd1ee74b0fab3e17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2f9bf2e5d9b4fc19f804797bca4a3ff" is_pre_election: true
I20240503 23:26:25.284659 3911 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6039230222ff4e57929bed49cd56caf5" candidate_uuid: "83ffd7184ec94302bd1ee74b0fab3e17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a3eecbaec43d4954b1096bc3121e824c" is_pre_election: true
I20240503 23:26:25.284994 3837 raft_consensus.cc:2379] T 6039230222ff4e57929bed49cd56caf5 P a2f9bf2e5d9b4fc19f804797bca4a3ff [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 83ffd7184ec94302bd1ee74b0fab3e17 in current term 1: Already voted for candidate a2f9bf2e5d9b4fc19f804797bca4a3ff in this term.
I20240503 23:26:25.285007 3911 raft_consensus.cc:2379] T 6039230222ff4e57929bed49cd56caf5 P a3eecbaec43d4954b1096bc3121e824c [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 83ffd7184ec94302bd1ee74b0fab3e17 in current term 1: Already voted for candidate a3eecbaec43d4954b1096bc3121e824c in this term.
I20240503 23:26:25.285601 3727 leader_election.cc:302] T 6039230222ff4e57929bed49cd56caf5 P 83ffd7184ec94302bd1ee74b0fab3e17 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83ffd7184ec94302bd1ee74b0fab3e17; no voters: a2f9bf2e5d9b4fc19f804797bca4a3ff, a3eecbaec43d4954b1096bc3121e824c
I20240503 23:26:25.285904 3963 raft_consensus.cc:3037] T 6039230222ff4e57929bed49cd56caf5 P 83ffd7184ec94302bd1ee74b0fab3e17 [term 0 FOLLOWER]: Advancing to term 1
I20240503 23:26:25.288762 3963 raft_consensus.cc:2726] T 6039230222ff4e57929bed49cd56caf5 P 83ffd7184ec94302bd1ee74b0fab3e17 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 23:26:25.312465 3963 raft_consensus.cc:482] T d0c4430fc56644cb9f87deffe4fe9407 P 83ffd7184ec94302bd1ee74b0fab3e17 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 23:26:25.312776 3963 raft_consensus.cc:504] T d0c4430fc56644cb9f87deffe4fe9407 P 83ffd7184ec94302bd1ee74b0fab3e17 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83ffd7184ec94302bd1ee74b0fab3e17" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 32909 } } peers { permanent_uuid: "a2f9bf2e5d9b4fc19f804797bca4a3ff" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 46261 } } peers { permanent_uuid: "a3eecbaec43d4954b1096bc3121e824c" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 40649 } }
I20240503 23:26:25.313553 3963 leader_election.cc:288] T d0c4430fc56644cb9f87deffe4fe9407 P 83ffd7184ec94302bd1ee74b0fab3e17 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a2f9bf2e5d9b4fc19f804797bca4a3ff (127.0.100.2:46261), a3eecbaec43d4954b1096bc3121e824c (127.0.100.3:40649)
I20240503 23:26:25.314298 3837 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0c4430fc56644cb9f87deffe4fe9407" candidate_uuid: "83ffd7184ec94302bd1ee74b0fab3e17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2f9bf2e5d9b4fc19f804797bca4a3ff" is_pre_election: true
I20240503 23:26:25.314298 3911 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0c4430fc56644cb9f87deffe4fe9407" candidate_uuid: "83ffd7184ec94302bd1ee74b0fab3e17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a3eecbaec43d4954b1096bc3121e824c" is_pre_election: true
I20240503 23:26:25.314630 3837 raft_consensus.cc:2379] T d0c4430fc56644cb9f87deffe4fe9407 P a2f9bf2e5d9b4fc19f804797bca4a3ff [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 83ffd7184ec94302bd1ee74b0fab3e17 in current term 1: Already voted for candidate a3eecbaec43d4954b1096bc3121e824c in this term.
I20240503 23:26:25.315061 3730 leader_election.cc:302] T d0c4430fc56644cb9f87deffe4fe9407 P 83ffd7184ec94302bd1ee74b0fab3e17 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83ffd7184ec94302bd1ee74b0fab3e17; no voters: a2f9bf2e5d9b4fc19f804797bca4a3ff, a3eecbaec43d4954b1096bc3121e824c
I20240503 23:26:25.315368 3963 raft_consensus.cc:3037] T d0c4430fc56644cb9f87deffe4fe9407 P 83ffd7184ec94302bd1ee74b0fab3e17 [term 0 FOLLOWER]: Advancing to term 1
I20240503 23:26:25.317596 3963 raft_consensus.cc:2726] T d0c4430fc56644cb9f87deffe4fe9407 P 83ffd7184ec94302bd1ee74b0fab3e17 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 23:26:25.318830 3995 consensus_queue.cc:1035] T a0e18d0975a54e3dbeb0a13cb226fefd P a2f9bf2e5d9b4fc19f804797bca4a3ff [LEADER]: Connected to new peer: Peer: permanent_uuid: "83ffd7184ec94302bd1ee74b0fab3e17" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 32909 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 23:26:25.324457 3994 consensus_queue.cc:1035] T a0e18d0975a54e3dbeb0a13cb226fefd P a2f9bf2e5d9b4fc19f804797bca4a3ff [LEADER]: Connected to new peer: Peer: permanent_uuid: "a3eecbaec43d4954b1096bc3121e824c" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 40649 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 23:26:25.328814 3963 consensus_queue.cc:1035] T 7181e4e8c0324a8eb095dde23651a1ab P 83ffd7184ec94302bd1ee74b0fab3e17 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2f9bf2e5d9b4fc19f804797bca4a3ff" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 46261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 23:26:25.335304 3963 consensus_queue.cc:1035] T 7181e4e8c0324a8eb095dde23651a1ab P 83ffd7184ec94302bd1ee74b0fab3e17 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a3eecbaec43d4954b1096bc3121e824c" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 40649 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 23:26:25.355136 4001 consensus_queue.cc:1035] T 059f6233ea514fe69b8c6504efabeff0 P a3eecbaec43d4954b1096bc3121e824c [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2f9bf2e5d9b4fc19f804797bca4a3ff" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 46261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 23:26:25.359660 3762 raft_consensus.cc:3037] T 059f6233ea514fe69b8c6504efabeff0 P 83ffd7184ec94302bd1ee74b0fab3e17 [term 0 FOLLOWER]: Advancing to term 1
I20240503 23:26:25.363487 4001 consensus_queue.cc:1035] T 059f6233ea514fe69b8c6504efabeff0 P a3eecbaec43d4954b1096bc3121e824c [LEADER]: Connected to new peer: Peer: permanent_uuid: "83ffd7184ec94302bd1ee74b0fab3e17" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 32909 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 23:26:25.402240 4001 raft_consensus.cc:482] T 60267cb4849d4823a0afb2380305905f P a3eecbaec43d4954b1096bc3121e824c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 23:26:25.402479 4001 raft_consensus.cc:504] T 60267cb4849d4823a0afb2380305905f P a3eecbaec43d4954b1096bc3121e824c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83ffd7184ec94302bd1ee74b0fab3e17" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 32909 } } peers { permanent_uuid: "a2f9bf2e5d9b4fc19f804797bca4a3ff" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 46261 } } peers { permanent_uuid: "a3eecbaec43d4954b1096bc3121e824c" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 40649 } }
I20240503 23:26:25.403236 4001 leader_election.cc:288] T 60267cb4849d4823a0afb2380305905f P a3eecbaec43d4954b1096bc3121e824c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 83ffd7184ec94302bd1ee74b0fab3e17 (127.0.100.1:32909), a2f9bf2e5d9b4fc19f804797bca4a3ff (127.0.100.2:46261)
I20240503 23:26:25.403921 3762 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "60267cb4849d4823a0afb2380305905f" candidate_uuid: "a3eecbaec43d4954b1096bc3121e824c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83ffd7184ec94302bd1ee74b0fab3e17" is_pre_election: true
I20240503 23:26:25.403931 3837 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "60267cb4849d4823a0afb2380305905f" candidate_uuid: "a3eecbaec43d4954b1096bc3121e824c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2f9bf2e5d9b4fc19f804797bca4a3ff" is_pre_election: true
I20240503 23:26:25.404145 3837 raft_consensus.cc:2454] T 60267cb4849d4823a0afb2380305905f P a2f9bf2e5d9b4fc19f804797bca4a3ff [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a3eecbaec43d4954b1096bc3121e824c in term 0.
I20240503 23:26:25.404143 3762 raft_consensus.cc:2454] T 60267cb4849d4823a0afb2380305905f P 83ffd7184ec94302bd1ee74b0fab3e17 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a3eecbaec43d4954b1096bc3121e824c in term 0.
I20240503 23:26:25.404661 3879 leader_election.cc:302] T 60267cb4849d4823a0afb2380305905f P a3eecbaec43d4954b1096bc3121e824c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a2f9bf2e5d9b4fc19f804797bca4a3ff, a3eecbaec43d4954b1096bc3121e824c; no voters:
I20240503 23:26:25.404965 4001 raft_consensus.cc:2781] T 60267cb4849d4823a0afb2380305905f P a3eecbaec43d4954b1096bc3121e824c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 23:26:25.405050 4001 raft_consensus.cc:482] T 60267cb4849d4823a0afb2380305905f P a3eecbaec43d4954b1096bc3121e824c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 23:26:25.405126 4001 raft_consensus.cc:3037] T 60267cb4849d4823a0afb2380305905f P a3eecbaec43d4954b1096bc3121e824c [term 0 FOLLOWER]: Advancing to term 1
I20240503 23:26:25.405733 3994 raft_consensus.cc:482] T 481cd0f8c8d147adac217915a53ff6d9 P a2f9bf2e5d9b4fc19f804797bca4a3ff [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 23:26:25.405851 3994 raft_consensus.cc:504] T 481cd0f8c8d147adac217915a53ff6d9 P a2f9bf2e5d9b4fc19f804797bca4a3ff [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83ffd7184ec94302bd1ee74b0fab3e17" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 32909 } } peers { permanent_uuid: "a2f9bf2e5d9b4fc19f804797bca4a3ff" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 46261 } } peers { permanent_uuid: "a3eecbaec43d4954b1096bc3121e824c" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 40649 } }
I20240503 23:26:25.406458 3994 leader_election.cc:288] T 481cd0f8c8d147adac217915a53ff6d9 P a2f9bf2e5d9b4fc19f804797bca4a3ff [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 83ffd7184ec94302bd1ee74b0fab3e17 (127.0.100.1:32909), a3eecbaec43d4954b1096bc3121e824c (127.0.100.3:40649)
I20240503 23:26:25.406971 3762 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "481cd0f8c8d147adac217915a53ff6d9" candidate_uuid: "a2f9bf2e5d9b4fc19f804797bca4a3ff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83ffd7184ec94302bd1ee74b0fab3e17" is_pre_election: true
I20240503 23:26:25.407092 3911 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "481cd0f8c8d147adac217915a53ff6d9" candidate_uuid: "a2f9bf2e5d9b4fc19f804797bca4a3ff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a3eecbaec43d4954b1096bc3121e824c" is_pre_election: true
I20240503 23:26:25.407253 3762 raft_consensus.cc:2379] T 481cd0f8c8d147adac217915a53ff6d9 P 83ffd7184ec94302bd1ee74b0fab3e17 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a2f9bf2e5d9b4fc19f804797bca4a3ff in current term 1: Already voted for candidate a3eecbaec43d4954b1096bc3121e824c in this term.
I20240503 23:26:25.407672 3804 leader_election.cc:302] T 481cd0f8c8d147adac217915a53ff6d9 P a2f9bf2e5d9b4fc19f804797bca4a3ff [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2f9bf2e5d9b4fc19f804797bca4a3ff; no voters: 83ffd7184ec94302bd1ee74b0fab3e17, a3eecbaec43d4954b1096bc3121e824c
I20240503 23:26:25.407899 3994 raft_consensus.cc:3037] T 481cd0f8c8d147adac217915a53ff6d9 P a2f9bf2e5d9b4fc19f804797bca4a3ff [term 0 FOLLOWER]: Advancing to term 1
I20240503 23:26:25.407876 4001 raft_consensus.cc:504] T 60267cb4849d4823a0afb2380305905f P a3eecbaec43d4954b1096bc3121e824c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83ffd7184ec94302bd1ee74b0fab3e17" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 32909 } } peers { permanent_uuid: "a2f9bf2e5d9b4fc19f804797bca4a3ff" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 46261 } } peers { permanent_uuid: "a3eecbaec43d4954b1096bc3121e824c" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 40649 } }
I20240503 23:26:25.408583 4001 leader_election.cc:288] T 60267cb4849d4823a0afb2380305905f P a3eecbaec43d4954b1096bc3121e824c [CANDIDATE]: Term 1 election: Requested vote from peers 83ffd7184ec94302bd1ee74b0fab3e17 (127.0.100.1:32909), a2f9bf2e5d9b4fc19f804797bca4a3ff (127.0.100.2:46261)
I20240503 23:26:25.408986 3762 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "60267cb4849d4823a0afb2380305905f" candidate_uuid: "a3eecbaec43d4954b1096bc3121e824c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83ffd7184ec94302bd1ee74b0fab3e17"
I20240503 23:26:25.409085 3837 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "60267cb4849d4823a0afb2380305905f" candidate_uuid: "a3eecbaec43d4954b1096bc3121e824c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2f9bf2e5d9b4fc19f804797bca4a3ff"
I20240503 23:26:25.409302 3837 raft_consensus.cc:3037] T 60267cb4849d4823a0afb2380305905f P a2f9bf2e5d9b4fc19f804797bca4a3ff [term 0 FOLLOWER]: Advancing to term 1
I20240503 23:26:25.409366 3762 raft_consensus.cc:3037] T 60267cb4849d4823a0afb2380305905f P 83ffd7184ec94302bd1ee74b0fab3e17 [term 0 FOLLOWER]: Advancing to term 1
I20240503 23:26:25.410827 3994 raft_consensus.cc:2726] T 481cd0f8c8d147adac217915a53ff6d9 P a2f9bf2e5d9b4fc19f804797bca4a3ff [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 23:26:25.411293 3837 raft_consensus.cc:2454] T 60267cb4849d4823a0afb2380305905f P a2f9bf2e5d9b4fc19f804797bca4a3ff [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a3eecbaec43d4954b1096bc3121e824c in term 1.
I20240503 23:26:25.411787 3879 leader_election.cc:302] T 60267cb4849d4823a0afb2380305905f P a3eecbaec43d4954b1096bc3121e824c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a2f9bf2e5d9b4fc19f804797bca4a3ff, a3eecbaec43d4954b1096bc3121e824c; no voters:
I20240503 23:26:25.411825 3762 raft_consensus.cc:2454] T 60267cb4849d4823a0afb2380305905f P 83ffd7184ec94302bd1ee74b0fab3e17 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a3eecbaec43d4954b1096bc3121e824c in term 1.
I20240503 23:26:25.412039 4001 raft_consensus.cc:2781] T 60267cb4849d4823a0afb2380305905f P a3eecbaec43d4954b1096bc3121e824c [term 1 FOLLOWER]: Leader election won for term 1
I20240503 23:26:25.412150 4001 raft_consensus.cc:686] T 60267cb4849d4823a0afb2380305905f P a3eecbaec43d4954b1096bc3121e824c [term 1 LEADER]: Becoming Leader. State: Replica: a3eecbaec43d4954b1096bc3121e824c, State: Running, Role: LEADER
I20240503 23:26:25.412393 4001 consensus_queue.cc:237] T 60267cb4849d4823a0afb2380305905f P a3eecbaec43d4954b1096bc3121e824c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83ffd7184ec94302bd1ee74b0fab3e17" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 32909 } } peers { permanent_uuid: "a2f9bf2e5d9b4fc19f804797bca4a3ff" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 46261 } } peers { permanent_uuid: "a3eecbaec43d4954b1096bc3121e824c" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 40649 } }
I20240503 23:26:25.415243 3660 catalog_manager.cc:5484] T 60267cb4849d4823a0afb2380305905f P a3eecbaec43d4954b1096bc3121e824c reported cstate change: term changed from 0 to 1, leader changed from <none> to a3eecbaec43d4954b1096bc3121e824c (127.0.100.3). New cstate: current_term: 1 leader_uuid: "a3eecbaec43d4954b1096bc3121e824c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83ffd7184ec94302bd1ee74b0fab3e17" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 32909 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a2f9bf2e5d9b4fc19f804797bca4a3ff" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 46261 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a3eecbaec43d4954b1096bc3121e824c" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 40649 } health_report { overall_health: HEALTHY } } }
I20240503 23:26:25.428514 3994 consensus_queue.cc:1035] T 1b9566be6c644cfca852ec6507fab893 P a2f9bf2e5d9b4fc19f804797bca4a3ff [LEADER]: Connected to new peer: Peer: permanent_uuid: "83ffd7184ec94302bd1ee74b0fab3e17" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 32909 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 23:26:25.433576 3966 consensus_queue.cc:1035] T 1b9566be6c644cfca852ec6507fab893 P a2f9bf2e5d9b4fc19f804797bca4a3ff [LEADER]: Connected to new peer: Peer: permanent_uuid: "a3eecbaec43d4954b1096bc3121e824c" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 40649 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 23:26:25.436787 4001 consensus_queue.cc:1035] T d0c4430fc56644cb9f87deffe4fe9407 P a3eecbaec43d4954b1096bc3121e824c [LEADER]: Connected to new peer: Peer: permanent_uuid: "83ffd7184ec94302bd1ee74b0fab3e17" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 32909 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 23:26:25.442243 3992 consensus_queue.cc:1035] T d0c4430fc56644cb9f87deffe4fe9407 P a3eecbaec43d4954b1096bc3121e824c [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2f9bf2e5d9b4fc19f804797bca4a3ff" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 46261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 23:26:25.446358 3992 consensus_queue.cc:1035] T 481cd0f8c8d147adac217915a53ff6d9 P a3eecbaec43d4954b1096bc3121e824c [LEADER]: Connected to new peer: Peer: permanent_uuid: "83ffd7184ec94302bd1ee74b0fab3e17" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 32909 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 23:26:25.450603 3996 consensus_queue.cc:1035] T 481cd0f8c8d147adac217915a53ff6d9 P a3eecbaec43d4954b1096bc3121e824c [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2f9bf2e5d9b4fc19f804797bca4a3ff" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 46261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 23:26:25.588654 3963 consensus_queue.cc:1035] T a14a12115b8d4fec947c6e223f1ef486 P 83ffd7184ec94302bd1ee74b0fab3e17 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2f9bf2e5d9b4fc19f804797bca4a3ff" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 46261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 23:26:25.594066 3963 consensus_queue.cc:1035] T a14a12115b8d4fec947c6e223f1ef486 P 83ffd7184ec94302bd1ee74b0fab3e17 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a3eecbaec43d4954b1096bc3121e824c" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 40649 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 23:26:25.630496 4005 consensus_queue.cc:1035] T 487ac5b9bf2d4de2b80a1b13387a2a77 P 83ffd7184ec94302bd1ee74b0fab3e17 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a3eecbaec43d4954b1096bc3121e824c" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 40649 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 23:26:25.635578 4023 consensus_queue.cc:1035] T 487ac5b9bf2d4de2b80a1b13387a2a77 P 83ffd7184ec94302bd1ee74b0fab3e17 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2f9bf2e5d9b4fc19f804797bca4a3ff" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 46261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 23:26:25.785140 4053 consensus_queue.cc:1035] T 98c9500acb43423b86f54ce7a4d98a72 P 83ffd7184ec94302bd1ee74b0fab3e17 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a3eecbaec43d4954b1096bc3121e824c" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 40649 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 23:26:25.790506 4053 consensus_queue.cc:1035] T 98c9500acb43423b86f54ce7a4d98a72 P 83ffd7184ec94302bd1ee74b0fab3e17 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2f9bf2e5d9b4fc19f804797bca4a3ff" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 46261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 23:26:25.982578 4001 consensus_queue.cc:1035] T 60267cb4849d4823a0afb2380305905f P a3eecbaec43d4954b1096bc3121e824c [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2f9bf2e5d9b4fc19f804797bca4a3ff" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 46261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 23:26:25.986892 4001 consensus_queue.cc:1035] T 60267cb4849d4823a0afb2380305905f P a3eecbaec43d4954b1096bc3121e824c [LEADER]: Connected to new peer: Peer: permanent_uuid: "83ffd7184ec94302bd1ee74b0fab3e17" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 32909 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240503 23:26:26.102080 3710 auto_rebalancer.cc:227] Could not retrieve cluster info: Service unavailable: Tablet not running
I20240503 23:26:26.698097 4001 raft_consensus.cc:482] T 6039230222ff4e57929bed49cd56caf5 P a3eecbaec43d4954b1096bc3121e824c [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 23:26:26.698349 4001 raft_consensus.cc:504] T 6039230222ff4e57929bed49cd56caf5 P a3eecbaec43d4954b1096bc3121e824c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3eecbaec43d4954b1096bc3121e824c" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 40649 } } peers { permanent_uuid: "a2f9bf2e5d9b4fc19f804797bca4a3ff" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 46261 } } peers { permanent_uuid: "83ffd7184ec94302bd1ee74b0fab3e17" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 32909 } }
I20240503 23:26:26.699434 4001 leader_election.cc:288] T 6039230222ff4e57929bed49cd56caf5 P a3eecbaec43d4954b1096bc3121e824c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a2f9bf2e5d9b4fc19f804797bca4a3ff (127.0.100.2:46261), 83ffd7184ec94302bd1ee74b0fab3e17 (127.0.100.1:32909)
I20240503 23:26:26.700054 3837 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6039230222ff4e57929bed49cd56caf5" candidate_uuid: "a3eecbaec43d4954b1096bc3121e824c" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2f9bf2e5d9b4fc19f804797bca4a3ff" is_pre_election: true
I20240503 23:26:26.700083 3764 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6039230222ff4e57929bed49cd56caf5" candidate_uuid: "a3eecbaec43d4954b1096bc3121e824c" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83ffd7184ec94302bd1ee74b0fab3e17" is_pre_election: true
I20240503 23:26:26.700325 3837 raft_consensus.cc:2454] T 6039230222ff4e57929bed49cd56caf5 P a2f9bf2e5d9b4fc19f804797bca4a3ff [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a3eecbaec43d4954b1096bc3121e824c in term 1.
I20240503 23:26:26.700352 3764 raft_consensus.cc:2454] T 6039230222ff4e57929bed49cd56caf5 P 83ffd7184ec94302bd1ee74b0fab3e17 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a3eecbaec43d4954b1096bc3121e824c in term 1.
I20240503 23:26:26.700842 3879 leader_election.cc:302] T 6039230222ff4e57929bed49cd56caf5 P a3eecbaec43d4954b1096bc3121e824c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 83ffd7184ec94302bd1ee74b0fab3e17, a2f9bf2e5d9b4fc19f804797bca4a3ff, a3eecbaec43d4954b1096bc3121e824c; no voters:
I20240503 23:26:26.701118 4001 raft_consensus.cc:2781] T 6039230222ff4e57929bed49cd56caf5 P a3eecbaec43d4954b1096bc3121e824c [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240503 23:26:26.701191 4001 raft_consensus.cc:482] T 6039230222ff4e57929bed49cd56caf5 P a3eecbaec43d4954b1096bc3121e824c [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 23:26:26.701257 4001 raft_consensus.cc:3037] T 6039230222ff4e57929bed49cd56caf5 P a3eecbaec43d4954b1096bc3121e824c [term 1 FOLLOWER]: Advancing to term 2
I20240503 23:26:26.703444 4001 raft_consensus.cc:504] T 6039230222ff4e57929bed49cd56caf5 P a3eecbaec43d4954b1096bc3121e824c [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3eecbaec43d4954b1096bc3121e824c" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 40649 } } peers { permanent_uuid: "a2f9bf2e5d9b4fc19f804797bca4a3ff" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 46261 } } peers { permanent_uuid: "83ffd7184ec94302bd1ee74b0fab3e17" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 32909 } }
I20240503 23:26:26.704129 4001 leader_election.cc:288] T 6039230222ff4e57929bed49cd56caf5 P a3eecbaec43d4954b1096bc3121e824c [CANDIDATE]: Term 2 election: Requested vote from peers a2f9bf2e5d9b4fc19f804797bca4a3ff (127.0.100.2:46261), 83ffd7184ec94302bd1ee74b0fab3e17 (127.0.100.1:32909)
I20240503 23:26:26.704784 3764 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6039230222ff4e57929bed49cd56caf5" candidate_uuid: "a3eecbaec43d4954b1096bc3121e824c" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83ffd7184ec94302bd1ee74b0fab3e17"
I20240503 23:26:26.704782 3837 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6039230222ff4e57929bed49cd56caf5" candidate_uuid: "a3eecbaec43d4954b1096bc3121e824c" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2f9bf2e5d9b4fc19f804797bca4a3ff"
I20240503 23:26:26.705010 3764 raft_consensus.cc:3037] T 6039230222ff4e57929bed49cd56caf5 P 83ffd7184ec94302bd1ee74b0fab3e17 [term 1 FOLLOWER]: Advancing to term 2
I20240503 23:26:26.705024 3837 raft_consensus.cc:3037] T 6039230222ff4e57929bed49cd56caf5 P a2f9bf2e5d9b4fc19f804797bca4a3ff [term 1 FOLLOWER]: Advancing to term 2
I20240503 23:26:26.707191 3837 raft_consensus.cc:2454] T 6039230222ff4e57929bed49cd56caf5 P a2f9bf2e5d9b4fc19f804797bca4a3ff [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a3eecbaec43d4954b1096bc3121e824c in term 2.
I20240503 23:26:26.707576 3764 raft_consensus.cc:2454] T 6039230222ff4e57929bed49cd56caf5 P 83ffd7184ec94302bd1ee74b0fab3e17 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a3eecbaec43d4954b1096bc3121e824c in term 2.
I20240503 23:26:26.708205 3878 leader_election.cc:302] T 6039230222ff4e57929bed49cd56caf5 P a3eecbaec43d4954b1096bc3121e824c [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 83ffd7184ec94302bd1ee74b0fab3e17, a3eecbaec43d4954b1096bc3121e824c; no voters:
I20240503 23:26:26.708631 4001 raft_consensus.cc:2781] T 6039230222ff4e57929bed49cd56caf5 P a3eecbaec43d4954b1096bc3121e824c [term 2 FOLLOWER]: Leader election won for term 2
I20240503 23:26:26.708815 4001 raft_consensus.cc:686] T 6039230222ff4e57929bed49cd56caf5 P a3eecbaec43d4954b1096bc3121e824c [term 2 LEADER]: Becoming Leader. State: Replica: a3eecbaec43d4954b1096bc3121e824c, State: Running, Role: LEADER
I20240503 23:26:26.709089 4001 consensus_queue.cc:237] T 6039230222ff4e57929bed49cd56caf5 P a3eecbaec43d4954b1096bc3121e824c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3eecbaec43d4954b1096bc3121e824c" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 40649 } } peers { permanent_uuid: "a2f9bf2e5d9b4fc19f804797bca4a3ff" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 46261 } } peers { permanent_uuid: "83ffd7184ec94302bd1ee74b0fab3e17" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 32909 } }
I20240503 23:26:26.712658 3662 catalog_manager.cc:5484] T 6039230222ff4e57929bed49cd56caf5 P a3eecbaec43d4954b1096bc3121e824c reported cstate change: term changed from 0 to 2, leader changed from <none> to a3eecbaec43d4954b1096bc3121e824c (127.0.100.3). New cstate: current_term: 2 leader_uuid: "a3eecbaec43d4954b1096bc3121e824c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3eecbaec43d4954b1096bc3121e824c" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 40649 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a2f9bf2e5d9b4fc19f804797bca4a3ff" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 46261 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "83ffd7184ec94302bd1ee74b0fab3e17" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 32909 } health_report { overall_health: UNKNOWN } } }
I20240503 23:26:27.104854 3715 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240503 23:26:27.112944 3764 raft_consensus.cc:1261] T 6039230222ff4e57929bed49cd56caf5 P 83ffd7184ec94302bd1ee74b0fab3e17 [term 2 FOLLOWER]: Refusing update from remote peer a3eecbaec43d4954b1096bc3121e824c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 2 index: 1. (index mismatch)
I20240503 23:26:27.118643 4001 consensus_queue.cc:1035] T 6039230222ff4e57929bed49cd56caf5 P a3eecbaec43d4954b1096bc3121e824c [LEADER]: Connected to new peer: Peer: permanent_uuid: "83ffd7184ec94302bd1ee74b0fab3e17" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 32909 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 23:26:27.183599 3911 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "059f6233ea514fe69b8c6504efabeff0"
dest_uuid: "a3eecbaec43d4954b1096bc3121e824c"
mode: GRACEFUL
new_leader_uuid: "a2f9bf2e5d9b4fc19f804797bca4a3ff"
from {username='slave'} at 127.0.0.1:40650
I20240503 23:26:27.183889 3911 raft_consensus.cc:595] T 059f6233ea514fe69b8c6504efabeff0 P a3eecbaec43d4954b1096bc3121e824c [term 1 LEADER]: Received request to transfer leadership to TS a2f9bf2e5d9b4fc19f804797bca4a3ff
I20240503 23:26:27.184664 3715 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 1
I20240503 23:26:27.185041 3715 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240503 23:26:27.194411 3837 raft_consensus.cc:1261] T 6039230222ff4e57929bed49cd56caf5 P a2f9bf2e5d9b4fc19f804797bca4a3ff [term 2 FOLLOWER]: Refusing update from remote peer a3eecbaec43d4954b1096bc3121e824c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 2 index: 1. (index mismatch)
I20240503 23:26:27.195804 4074 consensus_queue.cc:1035] T 6039230222ff4e57929bed49cd56caf5 P a3eecbaec43d4954b1096bc3121e824c [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2f9bf2e5d9b4fc19f804797bca4a3ff" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 46261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240503 23:26:27.269335 3892 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:40666: Service unavailable: leader transfer in progress
W20240503 23:26:27.295619 3890 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:40666: Service unavailable: leader transfer in progress
W20240503 23:26:27.298784 3890 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:40666: Service unavailable: leader transfer in progress
W20240503 23:26:27.311319 3891 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:40666: Service unavailable: leader transfer in progress
W20240503 23:26:27.320010 3889 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:40666: Service unavailable: leader transfer in progress
W20240503 23:26:27.320858 3891 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:40666: Service unavailable: leader transfer in progress
W20240503 23:26:27.330152 3889 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:40666: Service unavailable: leader transfer in progress
W20240503 23:26:27.333678 3891 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:40666: Service unavailable: leader transfer in progress
W20240503 23:26:27.337260 3889 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:40666: Service unavailable: leader transfer in progress
W20240503 23:26:27.352098 3889 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:40666: Service unavailable: leader transfer in progress
W20240503 23:26:27.358180 3892 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:40666: Service unavailable: leader transfer in progress
W20240503 23:26:27.379566 3891 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:40666: Service unavailable: leader transfer in progress
W20240503 23:26:27.382119 3891 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:40666: Service unavailable: leader transfer in progress
W20240503 23:26:27.388331 3892 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:40666: Service unavailable: leader transfer in progress
W20240503 23:26:27.389076 3889 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:40666: Service unavailable: leader transfer in progress
W20240503 23:26:27.427171 3893 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:40666: Service unavailable: leader transfer in progress
W20240503 23:26:27.436584 3893 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:40666: Service unavailable: leader transfer in progress
W20240503 23:26:27.437341 3892 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:40666: Service unavailable: leader transfer in progress
W20240503 23:26:27.443341 3889 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:40666: Service unavailable: leader transfer in progress
W20240503 23:26:27.449276 3891 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:40666: Service unavailable: leader transfer in progress
W20240503 23:26:27.455171 3891 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:40666: Service unavailable: leader transfer in progress
W20240503 23:26:27.456943 3889 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:40666: Service unavailable: leader transfer in progress
W20240503 23:26:27.480412 4092 log.cc:924] Time spent T 481cd0f8c8d147adac217915a53ff6d9 P a3eecbaec43d4954b1096bc3121e824c: Append to log took a long time: real 0.061s user 0.000s sys 0.000s
W20240503 23:26:27.500973 3889 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:40666: Service unavailable: leader transfer in progress
W20240503 23:26:27.530757 3889 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:40666: Service unavailable: leader transfer in progress
W20240503 23:26:27.543576 3891 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:40666: Service unavailable: leader transfer in progress
W20240503 23:26:27.551016 4102 log.cc:924] Time spent T a0e18d0975a54e3dbeb0a13cb226fefd P a3eecbaec43d4954b1096bc3121e824c: Append to log took a long time: real 0.056s user 0.001s sys 0.000s
W20240503 23:26:27.571445 4113 log.cc:924] Time spent T 487ac5b9bf2d4de2b80a1b13387a2a77 P 83ffd7184ec94302bd1ee74b0fab3e17: Append to log took a long time: real 0.074s user 0.000s sys 0.000s
W20240503 23:26:27.573025 3889 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:40666: Service unavailable: leader transfer in progress
W20240503 23:26:27.583282 4106 log.cc:924] Time spent T a0e18d0975a54e3dbeb0a13cb226fefd P 83ffd7184ec94302bd1ee74b0fab3e17: Append to log took a long time: real 0.065s user 0.000s sys 0.001s
W20240503 23:26:27.594724 3889 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:40666: Service unavailable: leader transfer in progress
W20240503 23:26:27.595700 3879 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.100.2:46261, user_credentials={real_user=slave}} blocked reactor thread for 123824us
W20240503 23:26:27.608492 3889 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:40666: Service unavailable: leader transfer in progress
W20240503 23:26:27.609469 3878 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.100.1:32909, user_credentials={real_user=slave}} blocked reactor thread for 137895us
W20240503 23:26:27.615981 3891 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:40666: Service unavailable: leader transfer in progress
W20240503 23:26:27.646378 3802 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.100.3:40649, user_credentials={real_user=slave}} blocked reactor thread for 48432.1us
W20240503 23:26:27.645085 3892 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:40666: Service unavailable: leader transfer in progress
W20240503 23:26:27.641364 3891 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:40666: Service unavailable: leader transfer in progress
W20240503 23:26:27.662660 3891 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:40666: Service unavailable: leader transfer in progress
W20240503 23:26:27.669003 3889 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:40666: Service unavailable: leader transfer in progress
W20240503 23:26:27.674832 3891 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:40666: Service unavailable: leader transfer in progress
W20240503 23:26:27.712783 3889 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:40666: Service unavailable: leader transfer in progress
W20240503 23:26:27.742916 3804 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.100.1:32909, user_credentials={real_user=slave}} blocked reactor thread for 145389us
I20240503 23:26:27.747527 4001 raft_consensus.cc:979] T 059f6233ea514fe69b8c6504efabeff0 P a3eecbaec43d4954b1096bc3121e824c: : Instructing follower a2f9bf2e5d9b4fc19f804797bca4a3ff to start an election
I20240503 23:26:27.747746 4074 raft_consensus.cc:1067] T 059f6233ea514fe69b8c6504efabeff0 P a3eecbaec43d4954b1096bc3121e824c [term 1 LEADER]: Signalling peer a2f9bf2e5d9b4fc19f804797bca4a3ff to start an election
W20240503 23:26:27.757956 3889 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:40666: Service unavailable: leader transfer in progress
W20240503 23:26:27.766206 3892 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:40666: Service unavailable: leader transfer in progress
W20240503 23:26:27.796533 3891 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:40666: Service unavailable: leader transfer in progress
W20240503 23:26:27.809082 4128 log.cc:924] Time spent T 7181e4e8c0324a8eb095dde23651a1ab P a2f9bf2e5d9b4fc19f804797bca4a3ff: Append to log took a long time: real 0.089s user 0.000s sys 0.001s
I20240503 23:26:27.806846 3835 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "059f6233ea514fe69b8c6504efabeff0"
dest_uuid: "a2f9bf2e5d9b4fc19f804797bca4a3ff"
from {username='slave'} at 127.0.0.1:45358
I20240503 23:26:27.832334 3835 raft_consensus.cc:482] T 059f6233ea514fe69b8c6504efabeff0 P a2f9bf2e5d9b4fc19f804797bca4a3ff [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
W20240503 23:26:27.837136 3892 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:40666: Service unavailable: leader transfer in progress
W20240503 23:26:27.839741 3891 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:40666: Service unavailable: leader transfer in progress
I20240503 23:26:27.843678 3835 raft_consensus.cc:3037] T 059f6233ea514fe69b8c6504efabeff0 P a2f9bf2e5d9b4fc19f804797bca4a3ff [term 1 FOLLOWER]: Advancing to term 2
W20240503 23:26:27.846968 3893 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:40666: Service unavailable: leader transfer in progress
I20240503 23:26:27.848343 3835 raft_consensus.cc:504] T 059f6233ea514fe69b8c6504efabeff0 P a2f9bf2e5d9b4fc19f804797bca4a3ff [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83ffd7184ec94302bd1ee74b0fab3e17" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 32909 } } peers { permanent_uuid: "a2f9bf2e5d9b4fc19f804797bca4a3ff" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 46261 } } peers { permanent_uuid: "a3eecbaec43d4954b1096bc3121e824c" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 40649 } }
W20240503 23:26:27.861631 3889 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:40666: Service unavailable: leader transfer in progress
W20240503 23:26:27.863700 3893 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:40666: Service unavailable: leader transfer in progress
I20240503 23:26:27.863890 3910 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "059f6233ea514fe69b8c6504efabeff0" candidate_uuid: "a2f9bf2e5d9b4fc19f804797bca4a3ff" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "a3eecbaec43d4954b1096bc3121e824c"
I20240503 23:26:27.864418 3762 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "059f6233ea514fe69b8c6504efabeff0" candidate_uuid: "a2f9bf2e5d9b4fc19f804797bca4a3ff" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "83ffd7184ec94302bd1ee74b0fab3e17"
I20240503 23:26:27.865696 3835 leader_election.cc:288] T 059f6233ea514fe69b8c6504efabeff0 P a2f9bf2e5d9b4fc19f804797bca4a3ff [CANDIDATE]: Term 2 election: Requested vote from peers 83ffd7184ec94302bd1ee74b0fab3e17 (127.0.100.1:32909), a3eecbaec43d4954b1096bc3121e824c (127.0.100.3:40649)
W20240503 23:26:27.866288 3889 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:40666: Service unavailable: leader transfer in progress
I20240503 23:26:27.866905 3910 raft_consensus.cc:3032] T 059f6233ea514fe69b8c6504efabeff0 P a3eecbaec43d4954b1096bc3121e824c [term 1 LEADER]: Stepping down as leader of term 1
I20240503 23:26:27.867275 3762 raft_consensus.cc:3037] T 059f6233ea514fe69b8c6504efabeff0 P 83ffd7184ec94302bd1ee74b0fab3e17 [term 1 FOLLOWER]: Advancing to term 2
I20240503 23:26:27.868242 3910 raft_consensus.cc:729] T 059f6233ea514fe69b8c6504efabeff0 P a3eecbaec43d4954b1096bc3121e824c [term 1 LEADER]: Becoming Follower/Learner. State: Replica: a3eecbaec43d4954b1096bc3121e824c, State: Running, Role: LEADER
I20240503 23:26:27.869726 3910 consensus_queue.cc:260] T 059f6233ea514fe69b8c6504efabeff0 P a3eecbaec43d4954b1096bc3121e824c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83ffd7184ec94302bd1ee74b0fab3e17" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 32909 } } peers { permanent_uuid: "a2f9bf2e5d9b4fc19f804797bca4a3ff" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 46261 } } peers { permanent_uuid: "a3eecbaec43d4954b1096bc3121e824c" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 40649 } }
I20240503 23:26:27.870713 3910 raft_consensus.cc:3037] T 059f6233ea514fe69b8c6504efabeff0 P a3eecbaec43d4954b1096bc3121e824c [term 1 FOLLOWER]: Advancing to term 2
I20240503 23:26:27.872747 3762 raft_consensus.cc:2454] T 059f6233ea514fe69b8c6504efabeff0 P 83ffd7184ec94302bd1ee74b0fab3e17 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2f9bf2e5d9b4fc19f804797bca4a3ff in term 2.
I20240503 23:26:27.873788 3910 raft_consensus.cc:2454] T 059f6233ea514fe69b8c6504efabeff0 P a3eecbaec43d4954b1096bc3121e824c [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2f9bf2e5d9b4fc19f804797bca4a3ff in term 2.
I20240503 23:26:27.874137 3804 leader_election.cc:302] T 059f6233ea514fe69b8c6504efabeff0 P a2f9bf2e5d9b4fc19f804797bca4a3ff [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 83ffd7184ec94302bd1ee74b0fab3e17, a2f9bf2e5d9b4fc19f804797bca4a3ff; no voters:
I20240503 23:26:27.874715 4108 raft_consensus.cc:2781] T 059f6233ea514fe69b8c6504efabeff0 P a2f9bf2e5d9b4fc19f804797bca4a3ff [term 2 FOLLOWER]: Leader election won for term 2
I20240503 23:26:27.874902 4108 raft_consensus.cc:686] T 059f6233ea514fe69b8c6504efabeff0 P a2f9bf2e5d9b4fc19f804797bca4a3ff [term 2 LEADER]: Becoming Leader. State: Replica: a2f9bf2e5d9b4fc19f804797bca4a3ff, State: Running, Role: LEADER
I20240503 23:26:27.875344 4108 consensus_queue.cc:237] T 059f6233ea514fe69b8c6504efabeff0 P a2f9bf2e5d9b4fc19f804797bca4a3ff [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83ffd7184ec94302bd1ee74b0fab3e17" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 32909 } } peers { permanent_uuid: "a2f9bf2e5d9b4fc19f804797bca4a3ff" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 46261 } } peers { permanent_uuid: "a3eecbaec43d4954b1096bc3121e824c" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 40649 } }
W20240503 23:26:27.876446 3889 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:40666: Illegal state: replica a3eecbaec43d4954b1096bc3121e824c is not leader of this config: current role FOLLOWER
W20240503 23:26:27.876471 3893 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:40666: Illegal state: replica a3eecbaec43d4954b1096bc3121e824c is not leader of this config: current role FOLLOWER
I20240503 23:26:27.879141 3662 catalog_manager.cc:5484] T 059f6233ea514fe69b8c6504efabeff0 P a2f9bf2e5d9b4fc19f804797bca4a3ff reported cstate change: term changed from 1 to 2, leader changed from a3eecbaec43d4954b1096bc3121e824c (127.0.100.3) to a2f9bf2e5d9b4fc19f804797bca4a3ff (127.0.100.2). New cstate: current_term: 2 leader_uuid: "a2f9bf2e5d9b4fc19f804797bca4a3ff" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83ffd7184ec94302bd1ee74b0fab3e17" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 32909 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a2f9bf2e5d9b4fc19f804797bca4a3ff" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 46261 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a3eecbaec43d4954b1096bc3121e824c" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 40649 } health_report { overall_health: UNKNOWN } } }
W20240503 23:26:27.880221 3893 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:40666: Illegal state: replica a3eecbaec43d4954b1096bc3121e824c is not leader of this config: current role FOLLOWER
W20240503 23:26:27.886569 3893 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:40666: Illegal state: replica a3eecbaec43d4954b1096bc3121e824c is not leader of this config: current role FOLLOWER
W20240503 23:26:27.892820 3740 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:35600: Illegal state: replica 83ffd7184ec94302bd1ee74b0fab3e17 is not leader of this config: current role FOLLOWER
W20240503 23:26:27.893654 3739 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:35600: Illegal state: replica 83ffd7184ec94302bd1ee74b0fab3e17 is not leader of this config: current role FOLLOWER
W20240503 23:26:27.897552 3739 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:35600: Illegal state: replica 83ffd7184ec94302bd1ee74b0fab3e17 is not leader of this config: current role FOLLOWER
W20240503 23:26:27.901757 3739 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:35600: Illegal state: replica 83ffd7184ec94302bd1ee74b0fab3e17 is not leader of this config: current role FOLLOWER
I20240503 23:26:27.910701 3912 raft_consensus.cc:1261] T 059f6233ea514fe69b8c6504efabeff0 P a3eecbaec43d4954b1096bc3121e824c [term 2 FOLLOWER]: Refusing update from remote peer a2f9bf2e5d9b4fc19f804797bca4a3ff: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20240503 23:26:27.910753 3762 raft_consensus.cc:1261] T 059f6233ea514fe69b8c6504efabeff0 P 83ffd7184ec94302bd1ee74b0fab3e17 [term 2 FOLLOWER]: Refusing update from remote peer a2f9bf2e5d9b4fc19f804797bca4a3ff: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20240503 23:26:27.911398 4137 consensus_queue.cc:1035] T 059f6233ea514fe69b8c6504efabeff0 P a2f9bf2e5d9b4fc19f804797bca4a3ff [LEADER]: Connected to new peer: Peer: permanent_uuid: "a3eecbaec43d4954b1096bc3121e824c" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 40649 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 23:26:27.911707 4108 consensus_queue.cc:1035] T 059f6233ea514fe69b8c6504efabeff0 P a2f9bf2e5d9b4fc19f804797bca4a3ff [LEADER]: Connected to new peer: Peer: permanent_uuid: "83ffd7184ec94302bd1ee74b0fab3e17" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 32909 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20240503 23:26:28.083232 3730 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.100.2:46261, user_credentials={real_user=slave}} blocked reactor thread for 110082us
W20240503 23:26:28.251794 3727 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.100.3:40649, user_credentials={real_user=slave}} blocked reactor thread for 86532.2us
W20240503 23:26:28.854396 3878 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.100.1:32909, user_credentials={real_user=slave}} blocked reactor thread for 69459.6us
I20240503 23:26:29.185482 3715 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240503 23:26:29.191534 3715 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240503 23:26:29.191917 3715 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240503 23:26:29.229501 400 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 23:26:29.233969 4172 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 23:26:29.234851 4173 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 23:26:29.235730 4175 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 23:26:29.236630 400 server_base.cc:1034] running on GCE node
I20240503 23:26:29.236979 400 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240503 23:26:29.237067 400 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240503 23:26:29.237128 400 hybrid_clock.cc:648] HybridClock initialized: now 1714778789237122 us; error 0 us; skew 500 ppm
I20240503 23:26:29.237270 400 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 23:26:29.239166 400 webserver.cc:460] Webserver started at http://127.0.100.4:41627/ using document root <none> and password file <none>
I20240503 23:26:29.239456 400 fs_manager.cc:359] Metadata directory not provided
I20240503 23:26:29.239539 400 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 23:26:29.239889 400 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 23:26:29.240896 400 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZhIiPz/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714778688451250-400-0/minicluster-data/ts-3-root/instance:
uuid: "e4086c099d7f4447b2b4733b6ca364df"
format_stamp: "Formatted at 2024-05-03 23:26:29 on dist-test-slave-jvbp"
I20240503 23:26:29.244338 400 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20240503 23:26:29.246600 4180 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 23:26:29.247119 400 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.002s sys 0.000s
I20240503 23:26:29.247249 400 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZhIiPz/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714778688451250-400-0/minicluster-data/ts-3-root
uuid: "e4086c099d7f4447b2b4733b6ca364df"
format_stamp: "Formatted at 2024-05-03 23:26:29 on dist-test-slave-jvbp"
I20240503 23:26:29.247393 400 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZhIiPz/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714778688451250-400-0/minicluster-data/ts-3-root
metadata directory: /tmp/dist-test-taskZhIiPz/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714778688451250-400-0/minicluster-data/ts-3-root
1 data directories: /tmp/dist-test-taskZhIiPz/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714778688451250-400-0/minicluster-data/ts-3-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 23:26:29.254797 400 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 23:26:29.255434 400 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 23:26:29.256016 400 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 23:26:29.257189 400 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 23:26:29.257272 400 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 23:26:29.257355 400 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 23:26:29.257393 400 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 23:26:29.278201 400 rpc_server.cc:307] RPC server started. Bound to: 127.0.100.4:35011
I20240503 23:26:29.278218 4242 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.100.4:35011 every 8 connection(s)
I20240503 23:26:29.280395 400 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 23:26:29.285728 4247 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 23:26:29.285909 4248 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 23:26:29.286686 4250 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 23:26:29.287483 400 server_base.cc:1034] running on GCE node
I20240503 23:26:29.287849 400 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240503 23:26:29.287914 400 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240503 23:26:29.287955 400 hybrid_clock.cc:648] HybridClock initialized: now 1714778789287949 us; error 0 us; skew 500 ppm
I20240503 23:26:29.288147 400 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 23:26:29.290083 400 webserver.cc:460] Webserver started at http://127.0.100.5:40655/ using document root <none> and password file <none>
I20240503 23:26:29.290221 4243 heartbeater.cc:347] Connected to a master server at 127.0.100.62:38775
I20240503 23:26:29.290382 4243 heartbeater.cc:464] Registering TS with master...
I20240503 23:26:29.290391 400 fs_manager.cc:359] Metadata directory not provided
I20240503 23:26:29.290498 400 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 23:26:29.290633 400 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 23:26:29.290886 4243 heartbeater.cc:511] Master 127.0.100.62:38775 requested a full tablet report, sending...
I20240503 23:26:29.291435 400 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZhIiPz/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714778688451250-400-0/minicluster-data/ts-4-root/instance:
uuid: "692da0319a2d4a81ad81f5d8f9f4c86b"
format_stamp: "Formatted at 2024-05-03 23:26:29 on dist-test-slave-jvbp"
I20240503 23:26:29.292093 3655 ts_manager.cc:194] Registered new tserver with Master: e4086c099d7f4447b2b4733b6ca364df (127.0.100.4:35011)
I20240503 23:26:29.293519 3655 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:57328
I20240503 23:26:29.295215 400 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20240503 23:26:29.297091 4255 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 23:26:29.297506 400 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.002s sys 0.000s
I20240503 23:26:29.297616 400 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZhIiPz/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714778688451250-400-0/minicluster-data/ts-4-root
uuid: "692da0319a2d4a81ad81f5d8f9f4c86b"
format_stamp: "Formatted at 2024-05-03 23:26:29 on dist-test-slave-jvbp"
I20240503 23:26:29.297747 400 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZhIiPz/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714778688451250-400-0/minicluster-data/ts-4-root
metadata directory: /tmp/dist-test-taskZhIiPz/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714778688451250-400-0/minicluster-data/ts-4-root
1 data directories: /tmp/dist-test-taskZhIiPz/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714778688451250-400-0/minicluster-data/ts-4-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 23:26:29.310845 400 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 23:26:29.311501 400 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 23:26:29.312148 400 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 23:26:29.313237 400 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 23:26:29.313305 400 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 23:26:29.313395 400 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 23:26:29.313434 400 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 23:26:29.337445 4317 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.100.5:38561 every 8 connection(s)
I20240503 23:26:29.337445 400 rpc_server.cc:307] RPC server started. Bound to: 127.0.100.5:38561
I20240503 23:26:29.339447 400 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 23:26:29.344839 4321 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 23:26:29.345803 4322 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 23:26:29.346508 4324 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 23:26:29.347437 400 server_base.cc:1034] running on GCE node
I20240503 23:26:29.347856 400 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240503 23:26:29.347930 400 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240503 23:26:29.347975 400 hybrid_clock.cc:648] HybridClock initialized: now 1714778789347969 us; error 0 us; skew 500 ppm
I20240503 23:26:29.348140 400 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 23:26:29.348766 4318 heartbeater.cc:347] Connected to a master server at 127.0.100.62:38775
I20240503 23:26:29.348908 4318 heartbeater.cc:464] Registering TS with master...
I20240503 23:26:29.349411 4318 heartbeater.cc:511] Master 127.0.100.62:38775 requested a full tablet report, sending...
I20240503 23:26:29.349927 400 webserver.cc:460] Webserver started at http://127.0.100.6:44669/ using document root <none> and password file <none>
I20240503 23:26:29.350216 400 fs_manager.cc:359] Metadata directory not provided
I20240503 23:26:29.350304 400 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 23:26:29.350404 400 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 23:26:29.350582 3655 ts_manager.cc:194] Registered new tserver with Master: 692da0319a2d4a81ad81f5d8f9f4c86b (127.0.100.5:38561)
I20240503 23:26:29.351064 400 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZhIiPz/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714778688451250-400-0/minicluster-data/ts-5-root/instance:
uuid: "4d1d4e261584467fbbce4c72fa18045a"
format_stamp: "Formatted at 2024-05-03 23:26:29 on dist-test-slave-jvbp"
I20240503 23:26:29.351445 3655 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:57338
I20240503 23:26:29.354151 400 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20240503 23:26:29.356525 4329 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 23:26:29.357048 400 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20240503 23:26:29.357180 400 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZhIiPz/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714778688451250-400-0/minicluster-data/ts-5-root
uuid: "4d1d4e261584467fbbce4c72fa18045a"
format_stamp: "Formatted at 2024-05-03 23:26:29 on dist-test-slave-jvbp"
I20240503 23:26:29.357337 400 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZhIiPz/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714778688451250-400-0/minicluster-data/ts-5-root
metadata directory: /tmp/dist-test-taskZhIiPz/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714778688451250-400-0/minicluster-data/ts-5-root
1 data directories: /tmp/dist-test-taskZhIiPz/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714778688451250-400-0/minicluster-data/ts-5-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 23:26:29.364087 400 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 23:26:29.364884 400 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 23:26:29.365576 400 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 23:26:29.366546 400 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 23:26:29.366611 400 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 23:26:29.366700 400 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 23:26:29.366739 400 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 23:26:29.389286 400 rpc_server.cc:307] RPC server started. Bound to: 127.0.100.6:35349
I20240503 23:26:29.389379 4391 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.100.6:35349 every 8 connection(s)
I20240503 23:26:29.397617 4392 heartbeater.cc:347] Connected to a master server at 127.0.100.62:38775
I20240503 23:26:29.397805 4392 heartbeater.cc:464] Registering TS with master...
I20240503 23:26:29.398367 4392 heartbeater.cc:511] Master 127.0.100.62:38775 requested a full tablet report, sending...
I20240503 23:26:29.399524 3655 ts_manager.cc:194] Registered new tserver with Master: 4d1d4e261584467fbbce4c72fa18045a (127.0.100.6:35349)
I20240503 23:26:29.400424 3655 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:57344
I20240503 23:26:30.232764 3761 consensus_queue.cc:237] T a14a12115b8d4fec947c6e223f1ef486 P 83ffd7184ec94302bd1ee74b0fab3e17 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 26, Committed index: 26, Last appended: 1.26, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "83ffd7184ec94302bd1ee74b0fab3e17" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 32909 } } peers { permanent_uuid: "a2f9bf2e5d9b4fc19f804797bca4a3ff" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 46261 } attrs { replace: true } } peers { permanent_uuid: "a3eecbaec43d4954b1096bc3121e824c" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 40649 } } peers { permanent_uuid: "4d1d4e261584467fbbce4c72fa18045a" member_type: NON_VOTER last_known_addr { host: "127.0.100.6" port: 35349 } attrs { promote: true } }
I20240503 23:26:30.236207 3833 raft_consensus.cc:1261] T a14a12115b8d4fec947c6e223f1ef486 P a2f9bf2e5d9b4fc19f804797bca4a3ff [term 1 FOLLOWER]: Refusing update from remote peer 83ffd7184ec94302bd1ee74b0fab3e17: Log matching property violated. Preceding OpId in replica: term: 1 index: 26. Preceding OpId from leader: term: 1 index: 27. (index mismatch)
I20240503 23:26:30.236207 3907 raft_consensus.cc:1261] T a14a12115b8d4fec947c6e223f1ef486 P a3eecbaec43d4954b1096bc3121e824c [term 1 FOLLOWER]: Refusing update from remote peer 83ffd7184ec94302bd1ee74b0fab3e17: Log matching property violated. Preceding OpId in replica: term: 1 index: 26. Preceding OpId from leader: term: 1 index: 27. (index mismatch)
I20240503 23:26:30.236963 4153 consensus_queue.cc:1035] T a14a12115b8d4fec947c6e223f1ef486 P 83ffd7184ec94302bd1ee74b0fab3e17 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2f9bf2e5d9b4fc19f804797bca4a3ff" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 46261 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 27, Last known committed idx: 26, Time since last communication: 0.000s
I20240503 23:26:30.237939 4153 consensus_queue.cc:1035] T a14a12115b8d4fec947c6e223f1ef486 P 83ffd7184ec94302bd1ee74b0fab3e17 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a3eecbaec43d4954b1096bc3121e824c" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 40649 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 27, Last known committed idx: 26, Time since last communication: 0.000s
I20240503 23:26:30.240656 4398 raft_consensus.cc:2932] T a14a12115b8d4fec947c6e223f1ef486 P 83ffd7184ec94302bd1ee74b0fab3e17 [term 1 LEADER]: Committing config change with OpId 1.27: config changed from index -1 to 27, NON_VOTER 4d1d4e261584467fbbce4c72fa18045a (127.0.100.6) added. New config: { opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "83ffd7184ec94302bd1ee74b0fab3e17" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 32909 } } peers { permanent_uuid: "a2f9bf2e5d9b4fc19f804797bca4a3ff" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 46261 } attrs { replace: true } } peers { permanent_uuid: "a3eecbaec43d4954b1096bc3121e824c" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 40649 } } peers { permanent_uuid: "4d1d4e261584467fbbce4c72fa18045a" member_type: NON_VOTER last_known_addr { host: "127.0.100.6" port: 35349 } attrs { promote: true } } }
I20240503 23:26:30.241473 3833 raft_consensus.cc:2932] T a14a12115b8d4fec947c6e223f1ef486 P a2f9bf2e5d9b4fc19f804797bca4a3ff [term 1 FOLLOWER]: Committing config change with OpId 1.27: config changed from index -1 to 27, NON_VOTER 4d1d4e261584467fbbce4c72fa18045a (127.0.100.6) added. New config: { opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "83ffd7184ec94302bd1ee74b0fab3e17" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 32909 } } peers { permanent_uuid: "a2f9bf2e5d9b4fc19f804797bca4a3ff" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 46261 } attrs { replace: true } } peers { permanent_uuid: "a3eecbaec43d4954b1096bc3121e824c" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 40649 } } peers { permanent_uuid: "4d1d4e261584467fbbce4c72fa18045a" member_type: NON_VOTER last_known_addr { host: "127.0.100.6" port: 35349 } attrs { promote: true } } }
I20240503 23:26:30.243832 3907 raft_consensus.cc:2932] T a14a12115b8d4fec947c6e223f1ef486 P a3eecbaec43d4954b1096bc3121e824c [term 1 FOLLOWER]: Committing config change with OpId 1.27: config changed from index -1 to 27, NON_VOTER 4d1d4e261584467fbbce4c72fa18045a (127.0.100.6) added. New config: { opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "83ffd7184ec94302bd1ee74b0fab3e17" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 32909 } } peers { permanent_uuid: "a2f9bf2e5d9b4fc19f804797bca4a3ff" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 46261 } attrs { replace: true } } peers { permanent_uuid: "a3eecbaec43d4954b1096bc3121e824c" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 40649 } } peers { permanent_uuid: "4d1d4e261584467fbbce4c72fa18045a" member_type: NON_VOTER last_known_addr { host: "127.0.100.6" port: 35349 } attrs { promote: true } } }
I20240503 23:26:30.247326 3659 catalog_manager.cc:5484] T a14a12115b8d4fec947c6e223f1ef486 P a2f9bf2e5d9b4fc19f804797bca4a3ff reported cstate change: config changed from index -1 to 27, NON_VOTER 4d1d4e261584467fbbce4c72fa18045a (127.0.100.6) added. New cstate: current_term: 1 leader_uuid: "83ffd7184ec94302bd1ee74b0fab3e17" committed_config { opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "83ffd7184ec94302bd1ee74b0fab3e17" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 32909 } } peers { permanent_uuid: "a2f9bf2e5d9b4fc19f804797bca4a3ff" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 46261 } attrs { replace: true } } peers { permanent_uuid: "a3eecbaec43d4954b1096bc3121e824c" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 40649 } } peers { permanent_uuid: "4d1d4e261584467fbbce4c72fa18045a" member_type: NON_VOTER last_known_addr { host: "127.0.100.6" port: 35349 } attrs { promote: true } } }
W20240503 23:26:30.248417 3728 consensus_peers.cc:485] T a14a12115b8d4fec947c6e223f1ef486 P 83ffd7184ec94302bd1ee74b0fab3e17 -> Peer 4d1d4e261584467fbbce4c72fa18045a (127.0.100.6:35349): Couldn't send request to peer 4d1d4e261584467fbbce4c72fa18045a. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: a14a12115b8d4fec947c6e223f1ef486. This is attempt 1: this message will repeat every 5th retry.
I20240503 23:26:30.257789 3833 consensus_queue.cc:237] T 55cf664d5e874c0d9b9adf8bdc9c43ed P a2f9bf2e5d9b4fc19f804797bca4a3ff [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 26, Committed index: 26, Last appended: 1.26, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "83ffd7184ec94302bd1ee74b0fab3e17" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 32909 } } peers { permanent_uuid: "a2f9bf2e5d9b4fc19f804797bca4a3ff" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 46261 } } peers { permanent_uuid: "a3eecbaec43d4954b1096bc3121e824c" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 40649 } attrs { replace: true } } peers { permanent_uuid: "e4086c099d7f4447b2b4733b6ca364df" member_type: NON_VOTER last_known_addr { host: "127.0.100.4" port: 35011 } attrs { promote: true } }
I20240503 23:26:30.260934 3761 raft_consensus.cc:1261] T 55cf664d5e874c0d9b9adf8bdc9c43ed P 83ffd7184ec94302bd1ee74b0fab3e17 [term 1 FOLLOWER]: Refusing update from remote peer a2f9bf2e5d9b4fc19f804797bca4a3ff: Log matching property violated. Preceding OpId in replica: term: 1 index: 26. Preceding OpId from leader: term: 1 index: 27. (index mismatch)
I20240503 23:26:30.261715 4410 consensus_queue.cc:1035] T 55cf664d5e874c0d9b9adf8bdc9c43ed P a2f9bf2e5d9b4fc19f804797bca4a3ff [LEADER]: Connected to new peer: Peer: permanent_uuid: "83ffd7184ec94302bd1ee74b0fab3e17" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 32909 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 27, Last known committed idx: 26, Time since last communication: 0.000s
I20240503 23:26:30.262614 3907 raft_consensus.cc:1261] T 55cf664d5e874c0d9b9adf8bdc9c43ed P a3eecbaec43d4954b1096bc3121e824c [term 1 FOLLOWER]: Refusing update from remote peer a2f9bf2e5d9b4fc19f804797bca4a3ff: Log matching property violated. Preceding OpId in replica: term: 1 index: 26. Preceding OpId from leader: term: 1 index: 27. (index mismatch)
I20240503 23:26:30.263094 4410 consensus_queue.cc:1035] T 55cf664d5e874c0d9b9adf8bdc9c43ed P a2f9bf2e5d9b4fc19f804797bca4a3ff [LEADER]: Connected to new peer: Peer: permanent_uuid: "a3eecbaec43d4954b1096bc3121e824c" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 40649 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 27, Last known committed idx: 26, Time since last communication: 0.000s
I20240503 23:26:30.266463 3907 raft_consensus.cc:2932] T 55cf664d5e874c0d9b9adf8bdc9c43ed P a3eecbaec43d4954b1096bc3121e824c [term 1 FOLLOWER]: Committing config change with OpId 1.27: config changed from index -1 to 27, NON_VOTER e4086c099d7f4447b2b4733b6ca364df (127.0.100.4) added. New config: { opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "83ffd7184ec94302bd1ee74b0fab3e17" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 32909 } } peers { permanent_uuid: "a2f9bf2e5d9b4fc19f804797bca4a3ff" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 46261 } } peers { permanent_uuid: "a3eecbaec43d4954b1096bc3121e824c" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 40649 } attrs { replace: true } } peers { permanent_uuid: "e4086c099d7f4447b2b4733b6ca364df" member_type: NON_VOTER last_known_addr { host: "127.0.100.4" port: 35011 } attrs { promote: true } } }
I20240503 23:26:30.265803 4411 raft_consensus.cc:2932] T 55cf664d5e874c0d9b9adf8bdc9c43ed P a2f9bf2e5d9b4fc19f804797bca4a3ff [term 1 LEADER]: Committing config change with OpId 1.27: config changed from index -1 to 27, NON_VOTER e4086c099d7f4447b2b4733b6ca364df (127.0.100.4) added. New config: { opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "83ffd7184ec94302bd1ee74b0fab3e17" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 32909 } } peers { permanent_uuid: "a2f9bf2e5d9b4fc19f804797bca4a3ff" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 46261 } } peers { permanent_uuid: "a3eecbaec43d4954b1096bc3121e824c" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 40649 } attrs { replace: true } } peers { permanent_uuid: "e4086c099d7f4447b2b4733b6ca364df" member_type: NON_VOTER last_known_addr { host: "127.0.100.4" port: 35011 } attrs { promote: true } } }
I20240503 23:26:30.272212 3662 catalog_manager.cc:5484] T 55cf664d5e874c0d9b9adf8bdc9c43ed P a3eecbaec43d4954b1096bc3121e824c reported cstate change: config changed from index -1 to 27, NON_VOTER e4086c099d7f4447b2b4733b6ca364df (127.0.100.4) added. New cstate: current_term: 1 leader_uuid: "a2f9bf2e5d9b4fc19f804797bca4a3ff" committed_config { opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "83ffd7184ec94302bd1ee74b0fab3e17" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 32909 } } peers { permanent_uuid: "a2f9bf2e5d9b4fc19f804797bca4a3ff" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 46261 } } peers { permanent_uuid: "a3eecbaec43d4954b1096bc3121e824c" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 40649 } attrs { replace: true } } peers { permanent_uuid: "e4086c099d7f4447b2b4733b6ca364df" member_type: NON_VOTER last_known_addr { host: "127.0.100.4" port: 35011 } attrs { promote: true } } }
I20240503 23:26:30.276530 3833 consensus_queue.cc:237] T 1b9566be6c644cfca852ec6507fab893 P a2f9bf2e5d9b4fc19f804797bca4a3ff [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 26, Committed index: 26, Last appended: 1.26, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "83ffd7184ec94302bd1ee74b0fab3e17" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 32909 } attrs { replace: true } } peers { permanent_uuid: "a2f9bf2e5d9b4fc19f804797bca4a3ff" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 46261 } } peers { permanent_uuid: "a3eecbaec43d4954b1096bc3121e824c" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 40649 } } peers { permanent_uuid: "692da0319a2d4a81ad81f5d8f9f4c86b" member_type: NON_VOTER last_known_addr { host: "127.0.100.5" port: 38561 } attrs { promote: true } }
I20240503 23:26:30.278905 3761 raft_consensus.cc:2932] T 55cf664d5e874c0d9b9adf8bdc9c43ed P 83ffd7184ec94302bd1ee74b0fab3e17 [term 1 FOLLOWER]: Committing config change with OpId 1.27: config changed from index -1 to 27, NON_VOTER e4086c099d7f4447b2b4733b6ca364df (127.0.100.4) added. New config: { opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "83ffd7184ec94302bd1ee74b0fab3e17" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 32909 } } peers { permanent_uuid: "a2f9bf2e5d9b4fc19f804797bca4a3ff" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 46261 } } peers { permanent_uuid: "a3eecbaec43d4954b1096bc3121e824c" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 40649 } attrs { replace: true } } peers { permanent_uuid: "e4086c099d7f4447b2b4733b6ca364df" member_type: NON_VOTER last_known_addr { host: "127.0.100.4" port: 35011 } attrs { promote: true } } }
I20240503 23:26:30.281592 3760 raft_consensus.cc:1261] T 1b9566be6c644cfca852ec6507fab893 P 83ffd7184ec94302bd1ee74b0fab3e17 [term 1 FOLLOWER]: Refusing update from remote peer a2f9bf2e5d9b4fc19f804797bca4a3ff: Log matching property violated. Preceding OpId in replica: term: 1 index: 26. Preceding OpId from leader: term: 1 index: 27. (index mismatch)
I20240503 23:26:30.282351 3966 consensus_queue.cc:1035] T 1b9566be6c644cfca852ec6507fab893 P a2f9bf2e5d9b4fc19f804797bca4a3ff [LEADER]: Connected to new peer: Peer: permanent_uuid: "83ffd7184ec94302bd1ee74b0fab3e17" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 32909 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 27, Last known committed idx: 26, Time since last communication: 0.000s
I20240503 23:26:30.289309 3907 raft_consensus.cc:1261] T 1b9566be6c644cfca852ec6507fab893 P a3eecbaec43d4954b1096bc3121e824c [term 1 FOLLOWER]: Refusing update from remote peer a2f9bf2e5d9b4fc19f804797bca4a3ff: Log matching property violated. Preceding OpId in replica: term: 1 index: 26. Preceding OpId from leader: term: 1 index: 27. (index mismatch)
I20240503 23:26:30.289779 4411 raft_consensus.cc:2932] T 1b9566be6c644cfca852ec6507fab893 P a2f9bf2e5d9b4fc19f804797bca4a3ff [term 1 LEADER]: Committing config change with OpId 1.27: config changed from index -1 to 27, NON_VOTER 692da0319a2d4a81ad81f5d8f9f4c86b (127.0.100.5) added. New config: { opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "83ffd7184ec94302bd1ee74b0fab3e17" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 32909 } attrs { replace: true } } peers { permanent_uuid: "a2f9bf2e5d9b4fc19f804797bca4a3ff" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 46261 } } peers { permanent_uuid: "a3eecbaec43d4954b1096bc3121e824c" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 40649 } } peers { permanent_uuid: "692da0319a2d4a81ad81f5d8f9f4c86b" member_type: NON_VOTER last_known_addr { host: "127.0.100.5" port: 38561 } attrs { promote: true } } }
I20240503 23:26:30.290349 3760 raft_consensus.cc:2932] T 1b9566be6c644cfca852ec6507fab893 P 83ffd7184ec94302bd1ee74b0fab3e17 [term 1 FOLLOWER]: Committing config change with OpId 1.27: config changed from index -1 to 27, NON_VOTER 692da0319a2d4a81ad81f5d8f9f4c86b (127.0.100.5) added. New config: { opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "83ffd7184ec94302bd1ee74b0fab3e17" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 32909 } attrs { replace: true } } peers { permanent_uuid: "a2f9bf2e5d9b4fc19f804797bca4a3ff" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 46261 } } peers { permanent_uuid: "a3eecbaec43d4954b1096bc3121e824c" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 40649 } } peers { permanent_uuid: "692da0319a2d4a81ad81f5d8f9f4c86b" member_type: NON_VOTER last_known_addr { host: "127.0.100.5" port: 38561 } attrs { promote: true } } }
I20240503 23:26:30.292385 4410 consensus_queue.cc:1035] T 1b9566be6c644cfca852ec6507fab893 P a2f9bf2e5d9b4fc19f804797bca4a3ff [LEADER]: Connected to new peer: Peer: permanent_uuid: "a3eecbaec43d4954b1096bc3121e824c" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 40649 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 27, Last known committed idx: 26, Time since last communication: 0.000s
I20240503 23:26:30.295292 4243 heartbeater.cc:503] Master 127.0.100.62:38775 was elected leader, sending a full tablet report...
I20240503 23:26:30.296357 3655 catalog_manager.cc:5484] T 1b9566be6c644cfca852ec6507fab893 P a2f9bf2e5d9b4fc19f804797bca4a3ff reported cstate change: config changed from index -1 to 27, NON_VOTER 692da0319a2d4a81ad81f5d8f9f4c86b (127.0.100.5) added. New cstate: current_term: 1 leader_uuid: "a2f9bf2e5d9b4fc19f804797bca4a3ff" committed_config { opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "83ffd7184ec94302bd1ee74b0fab3e17" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 32909 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a2f9bf2e5d9b4fc19f804797bca4a3ff" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 46261 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a3eecbaec43d4954b1096bc3121e824c" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 40649 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "692da0319a2d4a81ad81f5d8f9f4c86b" member_type: NON_VOTER last_known_addr { host: "127.0.100.5" port: 38561 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 23:26:30.297448 3761 consensus_queue.cc:237] T 7181e4e8c0324a8eb095dde23651a1ab P 83ffd7184ec94302bd1ee74b0fab3e17 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 25, Committed index: 25, Last appended: 1.25, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 26 OBSOLETE_local: false peers { permanent_uuid: "83ffd7184ec94302bd1ee74b0fab3e17" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 32909 } } peers { permanent_uuid: "a2f9bf2e5d9b4fc19f804797bca4a3ff" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 46261 } attrs { replace: true } } peers { permanent_uuid: "a3eecbaec43d4954b1096bc3121e824c" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 40649 } } peers { permanent_uuid: "e4086c099d7f4447b2b4733b6ca364df" member_type: NON_VOTER last_known_addr { host: "127.0.100.4" port: 35011 } attrs { promote: true } }
W20240503 23:26:30.298072 3802 consensus_peers.cc:485] T 55cf664d5e874c0d9b9adf8bdc9c43ed P a2f9bf2e5d9b4fc19f804797bca4a3ff -> Peer e4086c099d7f4447b2b4733b6ca364df (127.0.100.4:35011): Couldn't send request to peer e4086c099d7f4447b2b4733b6ca364df. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 55cf664d5e874c0d9b9adf8bdc9c43ed. This is attempt 1: this message will repeat every 5th retry.
I20240503 23:26:30.300776 3907 raft_consensus.cc:2932] T 1b9566be6c644cfca852ec6507fab893 P a3eecbaec43d4954b1096bc3121e824c [term 1 FOLLOWER]: Committing config change with OpId 1.27: config changed from index -1 to 27, NON_VOTER 692da0319a2d4a81ad81f5d8f9f4c86b (127.0.100.5) added. New config: { opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "83ffd7184ec94302bd1ee74b0fab3e17" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 32909 } attrs { replace: true } } peers { permanent_uuid: "a2f9bf2e5d9b4fc19f804797bca4a3ff" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 46261 } } peers { permanent_uuid: "a3eecbaec43d4954b1096bc3121e824c" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 40649 } } peers { permanent_uuid: "692da0319a2d4a81ad81f5d8f9f4c86b" member_type: NON_VOTER last_known_addr { host: "127.0.100.5" port: 38561 } attrs { promote: true } } }
W20240503 23:26:30.301582 3802 consensus_peers.cc:485] T 1b9566be6c644cfca852ec6507fab893 P a2f9bf2e5d9b4fc19f804797bca4a3ff -> Peer 692da0319a2d4a81ad81f5d8f9f4c86b (127.0.100.5:38561): Couldn't send request to peer 692da0319a2d4a81ad81f5d8f9f4c86b. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 1b9566be6c644cfca852ec6507fab893. This is attempt 1: this message will repeat every 5th retry.
I20240503 23:26:30.306936 3833 raft_consensus.cc:1261] T 7181e4e8c0324a8eb095dde23651a1ab P a2f9bf2e5d9b4fc19f804797bca4a3ff [term 1 FOLLOWER]: Refusing update from remote peer 83ffd7184ec94302bd1ee74b0fab3e17: Log matching property violated. Preceding OpId in replica: term: 1 index: 25. Preceding OpId from leader: term: 1 index: 26. (index mismatch)
I20240503 23:26:30.306941 3908 raft_consensus.cc:1261] T 7181e4e8c0324a8eb095dde23651a1ab P a3eecbaec43d4954b1096bc3121e824c [term 1 FOLLOWER]: Refusing update from remote peer 83ffd7184ec94302bd1ee74b0fab3e17: Log matching property violated. Preceding OpId in replica: term: 1 index: 25. Preceding OpId from leader: term: 1 index: 26. (index mismatch)
I20240503 23:26:30.307487 4398 consensus_queue.cc:1035] T 7181e4e8c0324a8eb095dde23651a1ab P 83ffd7184ec94302bd1ee74b0fab3e17 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2f9bf2e5d9b4fc19f804797bca4a3ff" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 46261 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 26, Last known committed idx: 25, Time since last communication: 0.000s
I20240503 23:26:30.307837 4403 consensus_queue.cc:1035] T 7181e4e8c0324a8eb095dde23651a1ab P 83ffd7184ec94302bd1ee74b0fab3e17 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a3eecbaec43d4954b1096bc3121e824c" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 40649 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 26, Last known committed idx: 25, Time since last communication: 0.000s
I20240503 23:26:30.311347 4403 raft_consensus.cc:2932] T 7181e4e8c0324a8eb095dde23651a1ab P 83ffd7184ec94302bd1ee74b0fab3e17 [term 1 LEADER]: Committing config change with OpId 1.26: config changed from index -1 to 26, NON_VOTER e4086c099d7f4447b2b4733b6ca364df (127.0.100.4) added. New config: { opid_index: 26 OBSOLETE_local: false peers { permanent_uuid: "83ffd7184ec94302bd1ee74b0fab3e17" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 32909 } } peers { permanent_uuid: "a2f9bf2e5d9b4fc19f804797bca4a3ff" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 46261 } attrs { replace: true } } peers { permanent_uuid: "a3eecbaec43d4954b1096bc3121e824c" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 40649 } } peers { permanent_uuid: "e4086c099d7f4447b2b4733b6ca364df" member_type: NON_VOTER last_known_addr { host: "127.0.100.4" port: 35011 } attrs { promote: true } } }
I20240503 23:26:30.312083 3838 raft_consensus.cc:2932] T 7181e4e8c0324a8eb095dde23651a1ab P a2f9bf2e5d9b4fc19f804797bca4a3ff [term 1 FOLLOWER]: Committing config change with OpId 1.26: config changed from index -1 to 26, NON_VOTER e4086c099d7f4447b2b4733b6ca364df (127.0.100.4) added. New config: { opid_index: 26 OBSOLETE_local: false peers { permanent_uuid: "83ffd7184ec94302bd1ee74b0fab3e17" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 32909 } } peers { permanent_uuid: "a2f9bf2e5d9b4fc19f804797bca4a3ff" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 46261 } attrs { replace: true } } peers { permanent_uuid: "a3eecbaec43d4954b1096bc3121e824c" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 40649 } } peers { permanent_uuid: "e4086c099d7f4447b2b4733b6ca364df" member_type: NON_VOTER last_known_addr { host: "127.0.100.4" port: 35011 } attrs { promote: true } } }
W20240503 23:26:30.317232 3727 consensus_peers.cc:485] T 7181e4e8c0324a8eb095dde23651a1ab P 83ffd7184ec94302bd1ee74b0fab3e17 -> Peer e4086c099d7f4447b2b4733b6ca364df (127.0.100.4:35011): Couldn't send request to peer e4086c099d7f4447b2b4733b6ca364df. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 7181e4e8c0324a8eb095dde23651a1ab. This is attempt 1: this message will repeat every 5th retry.
I20240503 23:26:30.317476 3655 catalog_manager.cc:5484] T 7181e4e8c0324a8eb095dde23651a1ab P a2f9bf2e5d9b4fc19f804797bca4a3ff reported cstate change: config changed from index -1 to 26, NON_VOTER e4086c099d7f4447b2b4733b6ca364df (127.0.100.4) added. New cstate: current_term: 1 leader_uuid: "83ffd7184ec94302bd1ee74b0fab3e17" committed_config { opid_index: 26 OBSOLETE_local: false peers { permanent_uuid: "83ffd7184ec94302bd1ee74b0fab3e17" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 32909 } } peers { permanent_uuid: "a2f9bf2e5d9b4fc19f804797bca4a3ff" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 46261 } attrs { replace: true } } peers { permanent_uuid: "a3eecbaec43d4954b1096bc3121e824c" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 40649 } } peers { permanent_uuid: "e4086c099d7f4447b2b4733b6ca364df" member_type: NON_VOTER last_known_addr { host: "127.0.100.4" port: 35011 } attrs { promote: true } } }
I20240503 23:26:30.318670 3908 raft_consensus.cc:2932] T 7181e4e8c0324a8eb095dde23651a1ab P a3eecbaec43d4954b1096bc3121e824c [term 1 FOLLOWER]: Committing config change with OpId 1.26: config changed from index -1 to 26, NON_VOTER e4086c099d7f4447b2b4733b6ca364df (127.0.100.4) added. New config: { opid_index: 26 OBSOLETE_local: false peers { permanent_uuid: "83ffd7184ec94302bd1ee74b0fab3e17" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 32909 } } peers { permanent_uuid: "a2f9bf2e5d9b4fc19f804797bca4a3ff" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 46261 } attrs { replace: true } } peers { permanent_uuid: "a3eecbaec43d4954b1096bc3121e824c" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 40649 } } peers { permanent_uuid: "e4086c099d7f4447b2b4733b6ca364df" member_type: NON_VOTER last_known_addr { host: "127.0.100.4" port: 35011 } attrs { promote: true } } }
I20240503 23:26:30.331748 3908 consensus_queue.cc:237] T 6039230222ff4e57929bed49cd56caf5 P a3eecbaec43d4954b1096bc3121e824c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 26, Committed index: 26, Last appended: 2.26, Last appended by leader: 0, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "a3eecbaec43d4954b1096bc3121e824c" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 40649 } } peers { permanent_uuid: "a2f9bf2e5d9b4fc19f804797bca4a3ff" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 46261 } } peers { permanent_uuid: "83ffd7184ec94302bd1ee74b0fab3e17" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 32909 } attrs { replace: true } } peers { permanent_uuid: "692da0319a2d4a81ad81f5d8f9f4c86b" member_type: NON_VOTER last_known_addr { host: "127.0.100.5" port: 38561 } attrs { promote: true } }
I20240503 23:26:30.334796 3838 raft_consensus.cc:1261] T 6039230222ff4e57929bed49cd56caf5 P a2f9bf2e5d9b4fc19f804797bca4a3ff [term 2 FOLLOWER]: Refusing update from remote peer a3eecbaec43d4954b1096bc3121e824c: Log matching property violated. Preceding OpId in replica: term: 2 index: 26. Preceding OpId from leader: term: 2 index: 27. (index mismatch)
I20240503 23:26:30.334817 3761 raft_consensus.cc:1261] T 6039230222ff4e57929bed49cd56caf5 P 83ffd7184ec94302bd1ee74b0fab3e17 [term 2 FOLLOWER]: Refusing update from remote peer a3eecbaec43d4954b1096bc3121e824c: Log matching property violated. Preceding OpId in replica: term: 2 index: 26. Preceding OpId from leader: term: 2 index: 27. (index mismatch)
I20240503 23:26:30.335707 4426 consensus_queue.cc:1035] T 6039230222ff4e57929bed49cd56caf5 P a3eecbaec43d4954b1096bc3121e824c [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2f9bf2e5d9b4fc19f804797bca4a3ff" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 46261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 27, Last known committed idx: 26, Time since last communication: 0.000s
I20240503 23:26:30.336284 4001 consensus_queue.cc:1035] T 6039230222ff4e57929bed49cd56caf5 P a3eecbaec43d4954b1096bc3121e824c [LEADER]: Connected to new peer: Peer: permanent_uuid: "83ffd7184ec94302bd1ee74b0fab3e17" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 32909 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 27, Last known committed idx: 26, Time since last communication: 0.000s
I20240503 23:26:30.340989 3761 raft_consensus.cc:2932] T 6039230222ff4e57929bed49cd56caf5 P 83ffd7184ec94302bd1ee74b0fab3e17 [term 2 FOLLOWER]: Committing config change with OpId 2.27: config changed from index -1 to 27, NON_VOTER 692da0319a2d4a81ad81f5d8f9f4c86b (127.0.100.5) added. New config: { opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "a3eecbaec43d4954b1096bc3121e824c" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 40649 } } peers { permanent_uuid: "a2f9bf2e5d9b4fc19f804797bca4a3ff" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 46261 } } peers { permanent_uuid: "83ffd7184ec94302bd1ee74b0fab3e17" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 32909 } attrs { replace: true } } peers { permanent_uuid: "692da0319a2d4a81ad81f5d8f9f4c86b" member_type: NON_VOTER last_known_addr { host: "127.0.100.5" port: 38561 } attrs { promote: true } } }
I20240503 23:26:30.340873 3838 raft_consensus.cc:2932] T 6039230222ff4e57929bed49cd56caf5 P a2f9bf2e5d9b4fc19f804797bca4a3ff [term 2 FOLLOWER]: Committing config change with OpId 2.27: config changed from index -1 to 27, NON_VOTER 692da0319a2d4a81ad81f5d8f9f4c86b (127.0.100.5) added. New config: { opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "a3eecbaec43d4954b1096bc3121e824c" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 40649 } } peers { permanent_uuid: "a2f9bf2e5d9b4fc19f804797bca4a3ff" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 46261 } } peers { permanent_uuid: "83ffd7184ec94302bd1ee74b0fab3e17" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 32909 } attrs { replace: true } } peers { permanent_uuid: "692da0319a2d4a81ad81f5d8f9f4c86b" member_type: NON_VOTER last_known_addr { host: "127.0.100.5" port: 38561 } attrs { promote: true } } }
W20240503 23:26:30.345629 3876 consensus_peers.cc:485] T 6039230222ff4e57929bed49cd56caf5 P a3eecbaec43d4954b1096bc3121e824c -> Peer 692da0319a2d4a81ad81f5d8f9f4c86b (127.0.100.5:38561): Couldn't send request to peer 692da0319a2d4a81ad81f5d8f9f4c86b. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 6039230222ff4e57929bed49cd56caf5. This is attempt 1: this message will repeat every 5th retry.
I20240503 23:26:30.340086 4427 raft_consensus.cc:2932] T 6039230222ff4e57929bed49cd56caf5 P a3eecbaec43d4954b1096bc3121e824c [term 2 LEADER]: Committing config change with OpId 2.27: config changed from index -1 to 27, NON_VOTER 692da0319a2d4a81ad81f5d8f9f4c86b (127.0.100.5) added. New config: { opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "a3eecbaec43d4954b1096bc3121e824c" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 40649 } } peers { permanent_uuid: "a2f9bf2e5d9b4fc19f804797bca4a3ff" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 46261 } } peers { permanent_uuid: "83ffd7184ec94302bd1ee74b0fab3e17" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 32909 } attrs { replace: true } } peers { permanent_uuid: "692da0319a2d4a81ad81f5d8f9f4c86b" member_type: NON_VOTER last_known_addr { host: "127.0.100.5" port: 38561 } attrs { promote: true } } }
I20240503 23:26:30.346943 3662 catalog_manager.cc:5484] T 6039230222ff4e57929bed49cd56caf5 P a2f9bf2e5d9b4fc19f804797bca4a3ff reported cstate change: config changed from index -1 to 27, NON_VOTER 692da0319a2d4a81ad81f5d8f9f4c86b (127.0.100.5) added. New cstate: current_term: 2 leader_uuid: "a3eecbaec43d4954b1096bc3121e824c" committed_config { opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "a3eecbaec43d4954b1096bc3121e824c" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 40649 } } peers { permanent_uuid: "a2f9bf2e5d9b4fc19f804797bca4a3ff" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 46261 } } peers { permanent_uuid: "83ffd7184ec94302bd1ee74b0fab3e17" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 32909 } attrs { replace: true } } peers { permanent_uuid: "692da0319a2d4a81ad81f5d8f9f4c86b" member_type: NON_VOTER last_known_addr { host: "127.0.100.5" port: 38561 } attrs { promote: true } } }
I20240503 23:26:30.352011 3761 consensus_queue.cc:237] T 98c9500acb43423b86f54ce7a4d98a72 P 83ffd7184ec94302bd1ee74b0fab3e17 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 25, Committed index: 25, Last appended: 1.25, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 26 OBSOLETE_local: false peers { permanent_uuid: "83ffd7184ec94302bd1ee74b0fab3e17" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 32909 } } peers { permanent_uuid: "a2f9bf2e5d9b4fc19f804797bca4a3ff" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 46261 } } peers { permanent_uuid: "a3eecbaec43d4954b1096bc3121e824c" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 40649 } attrs { replace: true } } peers { permanent_uuid: "4d1d4e261584467fbbce4c72fa18045a" member_type: NON_VOTER last_known_addr { host: "127.0.100.6" port: 35349 } attrs { promote: true } }
I20240503 23:26:30.352960 4318 heartbeater.cc:503] Master 127.0.100.62:38775 was elected leader, sending a full tablet report...
I20240503 23:26:30.356258 3908 raft_consensus.cc:1261] T 98c9500acb43423b86f54ce7a4d98a72 P a3eecbaec43d4954b1096bc3121e824c [term 1 FOLLOWER]: Refusing update from remote peer 83ffd7184ec94302bd1ee74b0fab3e17: Log matching property violated. Preceding OpId in replica: term: 1 index: 25. Preceding OpId from leader: term: 1 index: 26. (index mismatch)
I20240503 23:26:30.356273 3838 raft_consensus.cc:1261] T 98c9500acb43423b86f54ce7a4d98a72 P a2f9bf2e5d9b4fc19f804797bca4a3ff [term 1 FOLLOWER]: Refusing update from remote peer 83ffd7184ec94302bd1ee74b0fab3e17: Log matching property violated. Preceding OpId in replica: term: 1 index: 25. Preceding OpId from leader: term: 1 index: 26. (index mismatch)
W20240503 23:26:30.356431 3728 consensus_peers.cc:485] T 98c9500acb43423b86f54ce7a4d98a72 P 83ffd7184ec94302bd1ee74b0fab3e17 -> Peer 4d1d4e261584467fbbce4c72fa18045a (127.0.100.6:35349): Couldn't send request to peer 4d1d4e261584467fbbce4c72fa18045a. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 98c9500acb43423b86f54ce7a4d98a72. This is attempt 1: this message will repeat every 5th retry.
I20240503 23:26:30.356858 4403 consensus_queue.cc:1035] T 98c9500acb43423b86f54ce7a4d98a72 P 83ffd7184ec94302bd1ee74b0fab3e17 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a3eecbaec43d4954b1096bc3121e824c" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 40649 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 26, Last known committed idx: 25, Time since last communication: 0.000s
I20240503 23:26:30.357204 4398 consensus_queue.cc:1035] T 98c9500acb43423b86f54ce7a4d98a72 P 83ffd7184ec94302bd1ee74b0fab3e17 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2f9bf2e5d9b4fc19f804797bca4a3ff" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 46261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 26, Last known committed idx: 25, Time since last communication: 0.000s
I20240503 23:26:30.360234 4403 raft_consensus.cc:2932] T 98c9500acb43423b86f54ce7a4d98a72 P 83ffd7184ec94302bd1ee74b0fab3e17 [term 1 LEADER]: Committing config change with OpId 1.26: config changed from index -1 to 26, NON_VOTER 4d1d4e261584467fbbce4c72fa18045a (127.0.100.6) added. New config: { opid_index: 26 OBSOLETE_local: false peers { permanent_uuid: "83ffd7184ec94302bd1ee74b0fab3e17" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 32909 } } peers { permanent_uuid: "a2f9bf2e5d9b4fc19f804797bca4a3ff" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 46261 } } peers { permanent_uuid: "a3eecbaec43d4954b1096bc3121e824c" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 40649 } attrs { replace: true } } peers { permanent_uuid: "4d1d4e261584467fbbce4c72fa18045a" member_type: NON_VOTER last_known_addr { host: "127.0.100.6" port: 35349 } attrs { promote: true } } }
I20240503 23:26:30.361013 3838 raft_consensus.cc:2932] T 98c9500acb43423b86f54ce7a4d98a72 P a2f9bf2e5d9b4fc19f804797bca4a3ff [term 1 FOLLOWER]: Committing config change with OpId 1.26: config changed from index -1 to 26, NON_VOTER 4d1d4e261584467fbbce4c72fa18045a (127.0.100.6) added. New config: { opid_index: 26 OBSOLETE_local: false peers { permanent_uuid: "83ffd7184ec94302bd1ee74b0fab3e17" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 32909 } } peers { permanent_uuid: "a2f9bf2e5d9b4fc19f804797bca4a3ff" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 46261 } } peers { permanent_uuid: "a3eecbaec43d4954b1096bc3121e824c" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 40649 } attrs { replace: true } } peers { permanent_uuid: "4d1d4e261584467fbbce4c72fa18045a" member_type: NON_VOTER last_known_addr { host: "127.0.100.6" port: 35349 } attrs { promote: true } } }
I20240503 23:26:30.364490 3908 raft_consensus.cc:2932] T 98c9500acb43423b86f54ce7a4d98a72 P a3eecbaec43d4954b1096bc3121e824c [term 1 FOLLOWER]: Committing config change with OpId 1.26: config changed from index -1 to 26, NON_VOTER 4d1d4e261584467fbbce4c72fa18045a (127.0.100.6) added. New config: { opid_index: 26 OBSOLETE_local: false peers { permanent_uuid: "83ffd7184ec94302bd1ee74b0fab3e17" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 32909 } } peers { permanent_uuid: "a2f9bf2e5d9b4fc19f804797bca4a3ff" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 46261 } } peers { permanent_uuid: "a3eecbaec43d4954b1096bc3121e824c" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 40649 } attrs { replace: true } } peers { permanent_uuid: "4d1d4e261584467fbbce4c72fa18045a" member_type: NON_VOTER last_known_addr { host: "127.0.100.6" port: 35349 } attrs { promote: true } } }
I20240503 23:26:30.365072 3662 catalog_manager.cc:5484] T 98c9500acb43423b86f54ce7a4d98a72 P 83ffd7184ec94302bd1ee74b0fab3e17 reported cstate change: config changed from index -1 to 26, NON_VOTER 4d1d4e261584467fbbce4c72fa18045a (127.0.100.6) added. New cstate: current_term: 1 leader_uuid: "83ffd7184ec94302bd1ee74b0fab3e17" committed_config { opid_index: 26 OBSOLETE_local: false peers { permanent_uuid: "83ffd7184ec94302bd1ee74b0fab3e17" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 32909 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a2f9bf2e5d9b4fc19f804797bca4a3ff" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 46261 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a3eecbaec43d4954b1096bc3121e824c" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 40649 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4d1d4e261584467fbbce4c72fa18045a" member_type: NON_VOTER last_known_addr { host: "127.0.100.6" port: 35349 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 23:26:30.402098 4392 heartbeater.cc:503] Master 127.0.100.62:38775 was elected leader, sending a full tablet report...
I20240503 23:26:30.692823 4434 ts_tablet_manager.cc:902] T 55cf664d5e874c0d9b9adf8bdc9c43ed P e4086c099d7f4447b2b4733b6ca364df: Initiating tablet copy from peer a2f9bf2e5d9b4fc19f804797bca4a3ff (127.0.100.2:46261)
I20240503 23:26:30.694587 4434 tablet_copy_client.cc:316] T 55cf664d5e874c0d9b9adf8bdc9c43ed P e4086c099d7f4447b2b4733b6ca364df: tablet copy: Beginning tablet copy session from remote peer at address 127.0.100.2:46261
I20240503 23:26:30.695389 4436 ts_tablet_manager.cc:902] T 1b9566be6c644cfca852ec6507fab893 P 692da0319a2d4a81ad81f5d8f9f4c86b: Initiating tablet copy from peer a2f9bf2e5d9b4fc19f804797bca4a3ff (127.0.100.2:46261)
I20240503 23:26:30.697012 4436 tablet_copy_client.cc:316] T 1b9566be6c644cfca852ec6507fab893 P 692da0319a2d4a81ad81f5d8f9f4c86b: tablet copy: Beginning tablet copy session from remote peer at address 127.0.100.2:46261
I20240503 23:26:30.702584 3849 tablet_copy_service.cc:140] P a2f9bf2e5d9b4fc19f804797bca4a3ff: Received BeginTabletCopySession request for tablet 55cf664d5e874c0d9b9adf8bdc9c43ed from peer e4086c099d7f4447b2b4733b6ca364df ({username='slave'} at 127.0.0.1:55448)
I20240503 23:26:30.702854 3849 tablet_copy_service.cc:161] P a2f9bf2e5d9b4fc19f804797bca4a3ff: Beginning new tablet copy session on tablet 55cf664d5e874c0d9b9adf8bdc9c43ed from peer e4086c099d7f4447b2b4733b6ca364df at {username='slave'} at 127.0.0.1:55448: session id = e4086c099d7f4447b2b4733b6ca364df-55cf664d5e874c0d9b9adf8bdc9c43ed
I20240503 23:26:30.706312 3849 tablet_copy_source_session.cc:205] T 55cf664d5e874c0d9b9adf8bdc9c43ed P a2f9bf2e5d9b4fc19f804797bca4a3ff: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 23:26:30.706889 3848 tablet_copy_service.cc:140] P a2f9bf2e5d9b4fc19f804797bca4a3ff: Received BeginTabletCopySession request for tablet 1b9566be6c644cfca852ec6507fab893 from peer 692da0319a2d4a81ad81f5d8f9f4c86b ({username='slave'} at 127.0.0.1:55456)
I20240503 23:26:30.707062 3848 tablet_copy_service.cc:161] P a2f9bf2e5d9b4fc19f804797bca4a3ff: Beginning new tablet copy session on tablet 1b9566be6c644cfca852ec6507fab893 from peer 692da0319a2d4a81ad81f5d8f9f4c86b at {username='slave'} at 127.0.0.1:55456: session id = 692da0319a2d4a81ad81f5d8f9f4c86b-1b9566be6c644cfca852ec6507fab893
I20240503 23:26:30.707895 4434 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 55cf664d5e874c0d9b9adf8bdc9c43ed. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 23:26:30.710263 3848 tablet_copy_source_session.cc:205] T 1b9566be6c644cfca852ec6507fab893 P a2f9bf2e5d9b4fc19f804797bca4a3ff: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 23:26:30.711678 4436 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1b9566be6c644cfca852ec6507fab893. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 23:26:30.715049 4434 tablet_copy_client.cc:791] T 55cf664d5e874c0d9b9adf8bdc9c43ed P e4086c099d7f4447b2b4733b6ca364df: tablet copy: Starting download of 0 data blocks...
I20240503 23:26:30.715382 4434 tablet_copy_client.cc:655] T 55cf664d5e874c0d9b9adf8bdc9c43ed P e4086c099d7f4447b2b4733b6ca364df: tablet copy: Starting download of 1 WAL segments...
I20240503 23:26:30.717976 4435 tablet_copy_client.cc:1010] T 55cf664d5e874c0d9b9adf8bdc9c43ed P e4086c099d7f4447b2b4733b6ca364df: tablet copy: Injecting latency into file download: 2000
I20240503 23:26:30.718942 4436 tablet_copy_client.cc:791] T 1b9566be6c644cfca852ec6507fab893 P 692da0319a2d4a81ad81f5d8f9f4c86b: tablet copy: Starting download of 0 data blocks...
I20240503 23:26:30.719200 4436 tablet_copy_client.cc:655] T 1b9566be6c644cfca852ec6507fab893 P 692da0319a2d4a81ad81f5d8f9f4c86b: tablet copy: Starting download of 1 WAL segments...
I20240503 23:26:30.722122 4441 ts_tablet_manager.cc:902] T 7181e4e8c0324a8eb095dde23651a1ab P e4086c099d7f4447b2b4733b6ca364df: Initiating tablet copy from peer 83ffd7184ec94302bd1ee74b0fab3e17 (127.0.100.1:32909)
I20240503 23:26:30.722931 4441 tablet_copy_client.cc:316] T 7181e4e8c0324a8eb095dde23651a1ab P e4086c099d7f4447b2b4733b6ca364df: tablet copy: Beginning tablet copy session from remote peer at address 127.0.100.1:32909
I20240503 23:26:30.722163 4438 tablet_copy_client.cc:1010] T 1b9566be6c644cfca852ec6507fab893 P 692da0319a2d4a81ad81f5d8f9f4c86b: tablet copy: Injecting latency into file download: 2000
I20240503 23:26:30.729691 3774 tablet_copy_service.cc:140] P 83ffd7184ec94302bd1ee74b0fab3e17: Received BeginTabletCopySession request for tablet 7181e4e8c0324a8eb095dde23651a1ab from peer e4086c099d7f4447b2b4733b6ca364df ({username='slave'} at 127.0.0.1:57918)
I20240503 23:26:30.729970 3774 tablet_copy_service.cc:161] P 83ffd7184ec94302bd1ee74b0fab3e17: Beginning new tablet copy session on tablet 7181e4e8c0324a8eb095dde23651a1ab from peer e4086c099d7f4447b2b4733b6ca364df at {username='slave'} at 127.0.0.1:57918: session id = e4086c099d7f4447b2b4733b6ca364df-7181e4e8c0324a8eb095dde23651a1ab
I20240503 23:26:30.732863 3774 tablet_copy_source_session.cc:205] T 7181e4e8c0324a8eb095dde23651a1ab P 83ffd7184ec94302bd1ee74b0fab3e17: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 23:26:30.734534 4441 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7181e4e8c0324a8eb095dde23651a1ab. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 23:26:30.740268 4441 tablet_copy_client.cc:791] T 7181e4e8c0324a8eb095dde23651a1ab P e4086c099d7f4447b2b4733b6ca364df: tablet copy: Starting download of 0 data blocks...
I20240503 23:26:30.740587 4441 tablet_copy_client.cc:655] T 7181e4e8c0324a8eb095dde23651a1ab P e4086c099d7f4447b2b4733b6ca364df: tablet copy: Starting download of 1 WAL segments...
I20240503 23:26:30.743086 4442 tablet_copy_client.cc:1010] T 7181e4e8c0324a8eb095dde23651a1ab P e4086c099d7f4447b2b4733b6ca364df: tablet copy: Injecting latency into file download: 2000
I20240503 23:26:30.777978 4443 ts_tablet_manager.cc:902] T 98c9500acb43423b86f54ce7a4d98a72 P 4d1d4e261584467fbbce4c72fa18045a: Initiating tablet copy from peer 83ffd7184ec94302bd1ee74b0fab3e17 (127.0.100.1:32909)
I20240503 23:26:30.778748 4443 tablet_copy_client.cc:316] T 98c9500acb43423b86f54ce7a4d98a72 P 4d1d4e261584467fbbce4c72fa18045a: tablet copy: Beginning tablet copy session from remote peer at address 127.0.100.1:32909
I20240503 23:26:30.799847 3774 tablet_copy_service.cc:140] P 83ffd7184ec94302bd1ee74b0fab3e17: Received BeginTabletCopySession request for tablet 98c9500acb43423b86f54ce7a4d98a72 from peer 4d1d4e261584467fbbce4c72fa18045a ({username='slave'} at 127.0.0.1:57922)
I20240503 23:26:30.800061 3774 tablet_copy_service.cc:161] P 83ffd7184ec94302bd1ee74b0fab3e17: Beginning new tablet copy session on tablet 98c9500acb43423b86f54ce7a4d98a72 from peer 4d1d4e261584467fbbce4c72fa18045a at {username='slave'} at 127.0.0.1:57922: session id = 4d1d4e261584467fbbce4c72fa18045a-98c9500acb43423b86f54ce7a4d98a72
I20240503 23:26:30.802771 3774 tablet_copy_source_session.cc:205] T 98c9500acb43423b86f54ce7a4d98a72 P 83ffd7184ec94302bd1ee74b0fab3e17: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 23:26:30.804100 4443 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 98c9500acb43423b86f54ce7a4d98a72. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 23:26:30.810081 4443 tablet_copy_client.cc:791] T 98c9500acb43423b86f54ce7a4d98a72 P 4d1d4e261584467fbbce4c72fa18045a: tablet copy: Starting download of 0 data blocks...
I20240503 23:26:30.810424 4443 tablet_copy_client.cc:655] T 98c9500acb43423b86f54ce7a4d98a72 P 4d1d4e261584467fbbce4c72fa18045a: tablet copy: Starting download of 1 WAL segments...
I20240503 23:26:30.812933 4444 tablet_copy_client.cc:1010] T 98c9500acb43423b86f54ce7a4d98a72 P 4d1d4e261584467fbbce4c72fa18045a: tablet copy: Injecting latency into file download: 2000
I20240503 23:26:30.833653 4446 ts_tablet_manager.cc:902] T a14a12115b8d4fec947c6e223f1ef486 P 4d1d4e261584467fbbce4c72fa18045a: Initiating tablet copy from peer 83ffd7184ec94302bd1ee74b0fab3e17 (127.0.100.1:32909)
I20240503 23:26:30.834434 4446 tablet_copy_client.cc:316] T a14a12115b8d4fec947c6e223f1ef486 P 4d1d4e261584467fbbce4c72fa18045a: tablet copy: Beginning tablet copy session from remote peer at address 127.0.100.1:32909
I20240503 23:26:30.834966 3774 tablet_copy_service.cc:140] P 83ffd7184ec94302bd1ee74b0fab3e17: Received BeginTabletCopySession request for tablet a14a12115b8d4fec947c6e223f1ef486 from peer 4d1d4e261584467fbbce4c72fa18045a ({username='slave'} at 127.0.0.1:57922)
I20240503 23:26:30.835091 3774 tablet_copy_service.cc:161] P 83ffd7184ec94302bd1ee74b0fab3e17: Beginning new tablet copy session on tablet a14a12115b8d4fec947c6e223f1ef486 from peer 4d1d4e261584467fbbce4c72fa18045a at {username='slave'} at 127.0.0.1:57922: session id = 4d1d4e261584467fbbce4c72fa18045a-a14a12115b8d4fec947c6e223f1ef486
I20240503 23:26:30.837458 3774 tablet_copy_source_session.cc:205] T a14a12115b8d4fec947c6e223f1ef486 P 83ffd7184ec94302bd1ee74b0fab3e17: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 23:26:30.838761 4446 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a14a12115b8d4fec947c6e223f1ef486. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 23:26:30.845224 4446 tablet_copy_client.cc:791] T a14a12115b8d4fec947c6e223f1ef486 P 4d1d4e261584467fbbce4c72fa18045a: tablet copy: Starting download of 0 data blocks...
I20240503 23:26:30.845561 4446 tablet_copy_client.cc:655] T a14a12115b8d4fec947c6e223f1ef486 P 4d1d4e261584467fbbce4c72fa18045a: tablet copy: Starting download of 1 WAL segments...
I20240503 23:26:30.847064 4447 tablet_copy_client.cc:1010] T a14a12115b8d4fec947c6e223f1ef486 P 4d1d4e261584467fbbce4c72fa18045a: tablet copy: Injecting latency into file download: 2000
I20240503 23:26:30.945063 4450 ts_tablet_manager.cc:902] T 6039230222ff4e57929bed49cd56caf5 P 692da0319a2d4a81ad81f5d8f9f4c86b: Initiating tablet copy from peer a3eecbaec43d4954b1096bc3121e824c (127.0.100.3:40649)
I20240503 23:26:30.945866 4450 tablet_copy_client.cc:316] T 6039230222ff4e57929bed49cd56caf5 P 692da0319a2d4a81ad81f5d8f9f4c86b: tablet copy: Beginning tablet copy session from remote peer at address 127.0.100.3:40649
I20240503 23:26:30.954414 3923 tablet_copy_service.cc:140] P a3eecbaec43d4954b1096bc3121e824c: Received BeginTabletCopySession request for tablet 6039230222ff4e57929bed49cd56caf5 from peer 692da0319a2d4a81ad81f5d8f9f4c86b ({username='slave'} at 127.0.0.1:39444)
I20240503 23:26:30.954630 3923 tablet_copy_service.cc:161] P a3eecbaec43d4954b1096bc3121e824c: Beginning new tablet copy session on tablet 6039230222ff4e57929bed49cd56caf5 from peer 692da0319a2d4a81ad81f5d8f9f4c86b at {username='slave'} at 127.0.0.1:39444: session id = 692da0319a2d4a81ad81f5d8f9f4c86b-6039230222ff4e57929bed49cd56caf5
I20240503 23:26:30.957314 3923 tablet_copy_source_session.cc:205] T 6039230222ff4e57929bed49cd56caf5 P a3eecbaec43d4954b1096bc3121e824c: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 23:26:30.958663 4450 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6039230222ff4e57929bed49cd56caf5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 23:26:30.964754 4450 tablet_copy_client.cc:791] T 6039230222ff4e57929bed49cd56caf5 P 692da0319a2d4a81ad81f5d8f9f4c86b: tablet copy: Starting download of 0 data blocks...
I20240503 23:26:30.965104 4450 tablet_copy_client.cc:655] T 6039230222ff4e57929bed49cd56caf5 P 692da0319a2d4a81ad81f5d8f9f4c86b: tablet copy: Starting download of 1 WAL segments...
I20240503 23:26:30.966792 4451 tablet_copy_client.cc:1010] T 6039230222ff4e57929bed49cd56caf5 P 692da0319a2d4a81ad81f5d8f9f4c86b: tablet copy: Injecting latency into file download: 2000
I20240503 23:26:31.192404 3715 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240503 23:26:31.194664 3715 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240503 23:26:31.194928 3715 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240503 23:26:32.718416 4434 tablet_copy_client.cc:531] T 55cf664d5e874c0d9b9adf8bdc9c43ed P e4086c099d7f4447b2b4733b6ca364df: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 23:26:32.722860 4434 tablet_bootstrap.cc:492] T 55cf664d5e874c0d9b9adf8bdc9c43ed P e4086c099d7f4447b2b4733b6ca364df: Bootstrap starting.
I20240503 23:26:32.723557 4436 tablet_copy_client.cc:531] T 1b9566be6c644cfca852ec6507fab893 P 692da0319a2d4a81ad81f5d8f9f4c86b: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 23:26:32.727252 4436 tablet_bootstrap.cc:492] T 1b9566be6c644cfca852ec6507fab893 P 692da0319a2d4a81ad81f5d8f9f4c86b: Bootstrap starting.
I20240503 23:26:32.743554 4441 tablet_copy_client.cc:531] T 7181e4e8c0324a8eb095dde23651a1ab P e4086c099d7f4447b2b4733b6ca364df: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 23:26:32.747431 4441 tablet_bootstrap.cc:492] T 7181e4e8c0324a8eb095dde23651a1ab P e4086c099d7f4447b2b4733b6ca364df: Bootstrap starting.
I20240503 23:26:32.768721 4436 tablet_bootstrap.cc:492] T 1b9566be6c644cfca852ec6507fab893 P 692da0319a2d4a81ad81f5d8f9f4c86b: Bootstrap replayed 1/1 log segments. Stats: ops{read=27 overwritten=0 applied=27 ignored=0} inserts{seen=90 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 23:26:32.769295 4436 tablet_bootstrap.cc:492] T 1b9566be6c644cfca852ec6507fab893 P 692da0319a2d4a81ad81f5d8f9f4c86b: Bootstrap complete.
I20240503 23:26:32.769727 4436 ts_tablet_manager.cc:1367] T 1b9566be6c644cfca852ec6507fab893 P 692da0319a2d4a81ad81f5d8f9f4c86b: Time spent bootstrapping tablet: real 0.043s user 0.042s sys 0.002s
I20240503 23:26:32.771468 4436 raft_consensus.cc:348] T 1b9566be6c644cfca852ec6507fab893 P 692da0319a2d4a81ad81f5d8f9f4c86b [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "83ffd7184ec94302bd1ee74b0fab3e17" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 32909 } attrs { replace: true } } peers { permanent_uuid: "a2f9bf2e5d9b4fc19f804797bca4a3ff" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 46261 } } peers { permanent_uuid: "a3eecbaec43d4954b1096bc3121e824c" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 40649 } } peers { permanent_uuid: "692da0319a2d4a81ad81f5d8f9f4c86b" member_type: NON_VOTER last_known_addr { host: "127.0.100.5" port: 38561 } attrs { promote: true } }
I20240503 23:26:32.771772 4436 raft_consensus.cc:729] T 1b9566be6c644cfca852ec6507fab893 P 692da0319a2d4a81ad81f5d8f9f4c86b [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 692da0319a2d4a81ad81f5d8f9f4c86b, State: Initialized, Role: LEARNER
I20240503 23:26:32.772001 4436 consensus_queue.cc:260] T 1b9566be6c644cfca852ec6507fab893 P 692da0319a2d4a81ad81f5d8f9f4c86b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 27, Last appended: 1.27, Last appended by leader: 27, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "83ffd7184ec94302bd1ee74b0fab3e17" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 32909 } attrs { replace: true } } peers { permanent_uuid: "a2f9bf2e5d9b4fc19f804797bca4a3ff" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 46261 } } peers { permanent_uuid: "a3eecbaec43d4954b1096bc3121e824c" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 40649 } } peers { permanent_uuid: "692da0319a2d4a81ad81f5d8f9f4c86b" member_type: NON_VOTER last_known_addr { host: "127.0.100.5" port: 38561 } attrs { promote: true } }
I20240503 23:26:32.773505 4436 ts_tablet_manager.cc:1398] T 1b9566be6c644cfca852ec6507fab893 P 692da0319a2d4a81ad81f5d8f9f4c86b: Time spent starting tablet: real 0.004s user 0.002s sys 0.001s
I20240503 23:26:32.774454 3848 tablet_copy_service.cc:342] P a2f9bf2e5d9b4fc19f804797bca4a3ff: Request end of tablet copy session 692da0319a2d4a81ad81f5d8f9f4c86b-1b9566be6c644cfca852ec6507fab893 received from {username='slave'} at 127.0.0.1:55456
I20240503 23:26:32.774577 3848 tablet_copy_service.cc:433] P a2f9bf2e5d9b4fc19f804797bca4a3ff: ending tablet copy session 692da0319a2d4a81ad81f5d8f9f4c86b-1b9566be6c644cfca852ec6507fab893 on tablet 1b9566be6c644cfca852ec6507fab893 with peer 692da0319a2d4a81ad81f5d8f9f4c86b
I20240503 23:26:32.787015 4434 tablet_bootstrap.cc:492] T 55cf664d5e874c0d9b9adf8bdc9c43ed P e4086c099d7f4447b2b4733b6ca364df: Bootstrap replayed 1/1 log segments. Stats: ops{read=27 overwritten=0 applied=27 ignored=0} inserts{seen=116 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 23:26:32.787480 4434 tablet_bootstrap.cc:492] T 55cf664d5e874c0d9b9adf8bdc9c43ed P e4086c099d7f4447b2b4733b6ca364df: Bootstrap complete.
I20240503 23:26:32.787757 4434 ts_tablet_manager.cc:1367] T 55cf664d5e874c0d9b9adf8bdc9c43ed P e4086c099d7f4447b2b4733b6ca364df: Time spent bootstrapping tablet: real 0.065s user 0.036s sys 0.005s
I20240503 23:26:32.788522 4434 raft_consensus.cc:348] T 55cf664d5e874c0d9b9adf8bdc9c43ed P e4086c099d7f4447b2b4733b6ca364df [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "83ffd7184ec94302bd1ee74b0fab3e17" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 32909 } } peers { permanent_uuid: "a2f9bf2e5d9b4fc19f804797bca4a3ff" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 46261 } } peers { permanent_uuid: "a3eecbaec43d4954b1096bc3121e824c" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 40649 } attrs { replace: true } } peers { permanent_uuid: "e4086c099d7f4447b2b4733b6ca364df" member_type: NON_VOTER last_known_addr { host: "127.0.100.4" port: 35011 } attrs { promote: true } }
I20240503 23:26:32.788769 4434 raft_consensus.cc:729] T 55cf664d5e874c0d9b9adf8bdc9c43ed P e4086c099d7f4447b2b4733b6ca364df [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: e4086c099d7f4447b2b4733b6ca364df, State: Initialized, Role: LEARNER
I20240503 23:26:32.788972 4434 consensus_queue.cc:260] T 55cf664d5e874c0d9b9adf8bdc9c43ed P e4086c099d7f4447b2b4733b6ca364df [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 27, Last appended: 1.27, Last appended by leader: 27, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "83ffd7184ec94302bd1ee74b0fab3e17" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 32909 } } peers { permanent_uuid: "a2f9bf2e5d9b4fc19f804797bca4a3ff" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 46261 } } peers { permanent_uuid: "a3eecbaec43d4954b1096bc3121e824c" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 40649 } attrs { replace: true } } peers { permanent_uuid: "e4086c099d7f4447b2b4733b6ca364df" member_type: NON_VOTER last_known_addr { host: "127.0.100.4" port: 35011 } attrs { promote: true } }
I20240503 23:26:32.789861 4434 ts_tablet_manager.cc:1398] T 55cf664d5e874c0d9b9adf8bdc9c43ed P e4086c099d7f4447b2b4733b6ca364df: Time spent starting tablet: real 0.002s user 0.001s sys 0.004s
I20240503 23:26:32.790880 3848 tablet_copy_service.cc:342] P a2f9bf2e5d9b4fc19f804797bca4a3ff: Request end of tablet copy session e4086c099d7f4447b2b4733b6ca364df-55cf664d5e874c0d9b9adf8bdc9c43ed received from {username='slave'} at 127.0.0.1:55448
I20240503 23:26:32.791009 3848 tablet_copy_service.cc:433] P a2f9bf2e5d9b4fc19f804797bca4a3ff: ending tablet copy session e4086c099d7f4447b2b4733b6ca364df-55cf664d5e874c0d9b9adf8bdc9c43ed on tablet 55cf664d5e874c0d9b9adf8bdc9c43ed with peer e4086c099d7f4447b2b4733b6ca364df
I20240503 23:26:32.800218 4441 tablet_bootstrap.cc:492] T 7181e4e8c0324a8eb095dde23651a1ab P e4086c099d7f4447b2b4733b6ca364df: Bootstrap replayed 1/1 log segments. Stats: ops{read=26 overwritten=0 applied=26 ignored=0} inserts{seen=96 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 23:26:32.800751 4441 tablet_bootstrap.cc:492] T 7181e4e8c0324a8eb095dde23651a1ab P e4086c099d7f4447b2b4733b6ca364df: Bootstrap complete.
I20240503 23:26:32.801041 4441 ts_tablet_manager.cc:1367] T 7181e4e8c0324a8eb095dde23651a1ab P e4086c099d7f4447b2b4733b6ca364df: Time spent bootstrapping tablet: real 0.054s user 0.033s sys 0.009s
I20240503 23:26:32.801759 4441 raft_consensus.cc:348] T 7181e4e8c0324a8eb095dde23651a1ab P e4086c099d7f4447b2b4733b6ca364df [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 26 OBSOLETE_local: false peers { permanent_uuid: "83ffd7184ec94302bd1ee74b0fab3e17" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 32909 } } peers { permanent_uuid: "a2f9bf2e5d9b4fc19f804797bca4a3ff" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 46261 } attrs { replace: true } } peers { permanent_uuid: "a3eecbaec43d4954b1096bc3121e824c" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 40649 } } peers { permanent_uuid: "e4086c099d7f4447b2b4733b6ca364df" member_type: NON_VOTER last_known_addr { host: "127.0.100.4" port: 35011 } attrs { promote: true } }
I20240503 23:26:32.802026 4441 raft_consensus.cc:729] T 7181e4e8c0324a8eb095dde23651a1ab P e4086c099d7f4447b2b4733b6ca364df [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: e4086c099d7f4447b2b4733b6ca364df, State: Initialized, Role: LEARNER
I20240503 23:26:32.802251 4441 consensus_queue.cc:260] T 7181e4e8c0324a8eb095dde23651a1ab P e4086c099d7f4447b2b4733b6ca364df [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 26, Last appended: 1.26, Last appended by leader: 26, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 26 OBSOLETE_local: false peers { permanent_uuid: "83ffd7184ec94302bd1ee74b0fab3e17" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 32909 } } peers { permanent_uuid: "a2f9bf2e5d9b4fc19f804797bca4a3ff" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 46261 } attrs { replace: true } } peers { permanent_uuid: "a3eecbaec43d4954b1096bc3121e824c" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 40649 } } peers { permanent_uuid: "e4086c099d7f4447b2b4733b6ca364df" member_type: NON_VOTER last_known_addr { host: "127.0.100.4" port: 35011 } attrs { promote: true } }
I20240503 23:26:32.802860 4441 ts_tablet_manager.cc:1398] T 7181e4e8c0324a8eb095dde23651a1ab P e4086c099d7f4447b2b4733b6ca364df: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240503 23:26:32.803552 3774 tablet_copy_service.cc:342] P 83ffd7184ec94302bd1ee74b0fab3e17: Request end of tablet copy session e4086c099d7f4447b2b4733b6ca364df-7181e4e8c0324a8eb095dde23651a1ab received from {username='slave'} at 127.0.0.1:57918
I20240503 23:26:32.803678 3774 tablet_copy_service.cc:433] P 83ffd7184ec94302bd1ee74b0fab3e17: ending tablet copy session e4086c099d7f4447b2b4733b6ca364df-7181e4e8c0324a8eb095dde23651a1ab on tablet 7181e4e8c0324a8eb095dde23651a1ab with peer e4086c099d7f4447b2b4733b6ca364df
I20240503 23:26:32.813345 4443 tablet_copy_client.cc:531] T 98c9500acb43423b86f54ce7a4d98a72 P 4d1d4e261584467fbbce4c72fa18045a: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 23:26:32.817651 4443 tablet_bootstrap.cc:492] T 98c9500acb43423b86f54ce7a4d98a72 P 4d1d4e261584467fbbce4c72fa18045a: Bootstrap starting.
I20240503 23:26:32.847471 4446 tablet_copy_client.cc:531] T a14a12115b8d4fec947c6e223f1ef486 P 4d1d4e261584467fbbce4c72fa18045a: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 23:26:32.851220 4446 tablet_bootstrap.cc:492] T a14a12115b8d4fec947c6e223f1ef486 P 4d1d4e261584467fbbce4c72fa18045a: Bootstrap starting.
I20240503 23:26:32.855558 4443 tablet_bootstrap.cc:492] T 98c9500acb43423b86f54ce7a4d98a72 P 4d1d4e261584467fbbce4c72fa18045a: Bootstrap replayed 1/1 log segments. Stats: ops{read=26 overwritten=0 applied=26 ignored=0} inserts{seen=102 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 23:26:32.856022 4443 tablet_bootstrap.cc:492] T 98c9500acb43423b86f54ce7a4d98a72 P 4d1d4e261584467fbbce4c72fa18045a: Bootstrap complete.
I20240503 23:26:32.856297 4443 ts_tablet_manager.cc:1367] T 98c9500acb43423b86f54ce7a4d98a72 P 4d1d4e261584467fbbce4c72fa18045a: Time spent bootstrapping tablet: real 0.039s user 0.035s sys 0.000s
I20240503 23:26:32.857136 4443 raft_consensus.cc:348] T 98c9500acb43423b86f54ce7a4d98a72 P 4d1d4e261584467fbbce4c72fa18045a [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 26 OBSOLETE_local: false peers { permanent_uuid: "83ffd7184ec94302bd1ee74b0fab3e17" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 32909 } } peers { permanent_uuid: "a2f9bf2e5d9b4fc19f804797bca4a3ff" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 46261 } } peers { permanent_uuid: "a3eecbaec43d4954b1096bc3121e824c" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 40649 } attrs { replace: true } } peers { permanent_uuid: "4d1d4e261584467fbbce4c72fa18045a" member_type: NON_VOTER last_known_addr { host: "127.0.100.6" port: 35349 } attrs { promote: true } }
I20240503 23:26:32.857450 4443 raft_consensus.cc:729] T 98c9500acb43423b86f54ce7a4d98a72 P 4d1d4e261584467fbbce4c72fa18045a [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 4d1d4e261584467fbbce4c72fa18045a, State: Initialized, Role: LEARNER
I20240503 23:26:32.857726 4443 consensus_queue.cc:260] T 98c9500acb43423b86f54ce7a4d98a72 P 4d1d4e261584467fbbce4c72fa18045a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 26, Last appended: 1.26, Last appended by leader: 26, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 26 OBSOLETE_local: false peers { permanent_uuid: "83ffd7184ec94302bd1ee74b0fab3e17" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 32909 } } peers { permanent_uuid: "a2f9bf2e5d9b4fc19f804797bca4a3ff" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 46261 } } peers { permanent_uuid: "a3eecbaec43d4954b1096bc3121e824c" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 40649 } attrs { replace: true } } peers { permanent_uuid: "4d1d4e261584467fbbce4c72fa18045a" member_type: NON_VOTER last_known_addr { host: "127.0.100.6" port: 35349 } attrs { promote: true } }
I20240503 23:26:32.859511 4443 ts_tablet_manager.cc:1398] T 98c9500acb43423b86f54ce7a4d98a72 P 4d1d4e261584467fbbce4c72fa18045a: Time spent starting tablet: real 0.003s user 0.006s sys 0.000s
I20240503 23:26:32.860240 3774 tablet_copy_service.cc:342] P 83ffd7184ec94302bd1ee74b0fab3e17: Request end of tablet copy session 4d1d4e261584467fbbce4c72fa18045a-98c9500acb43423b86f54ce7a4d98a72 received from {username='slave'} at 127.0.0.1:57922
I20240503 23:26:32.860364 3774 tablet_copy_service.cc:433] P 83ffd7184ec94302bd1ee74b0fab3e17: ending tablet copy session 4d1d4e261584467fbbce4c72fa18045a-98c9500acb43423b86f54ce7a4d98a72 on tablet 98c9500acb43423b86f54ce7a4d98a72 with peer 4d1d4e261584467fbbce4c72fa18045a
I20240503 23:26:32.900583 4446 tablet_bootstrap.cc:492] T a14a12115b8d4fec947c6e223f1ef486 P 4d1d4e261584467fbbce4c72fa18045a: Bootstrap replayed 1/1 log segments. Stats: ops{read=27 overwritten=0 applied=27 ignored=0} inserts{seen=120 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 23:26:32.901051 4446 tablet_bootstrap.cc:492] T a14a12115b8d4fec947c6e223f1ef486 P 4d1d4e261584467fbbce4c72fa18045a: Bootstrap complete.
I20240503 23:26:32.901341 4446 ts_tablet_manager.cc:1367] T a14a12115b8d4fec947c6e223f1ef486 P 4d1d4e261584467fbbce4c72fa18045a: Time spent bootstrapping tablet: real 0.050s user 0.042s sys 0.007s
I20240503 23:26:32.902133 4446 raft_consensus.cc:348] T a14a12115b8d4fec947c6e223f1ef486 P 4d1d4e261584467fbbce4c72fa18045a [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "83ffd7184ec94302bd1ee74b0fab3e17" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 32909 } } peers { permanent_uuid: "a2f9bf2e5d9b4fc19f804797bca4a3ff" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 46261 } attrs { replace: true } } peers { permanent_uuid: "a3eecbaec43d4954b1096bc3121e824c" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 40649 } } peers { permanent_uuid: "4d1d4e261584467fbbce4c72fa18045a" member_type: NON_VOTER last_known_addr { host: "127.0.100.6" port: 35349 } attrs { promote: true } }
I20240503 23:26:32.902336 4446 raft_consensus.cc:729] T a14a12115b8d4fec947c6e223f1ef486 P 4d1d4e261584467fbbce4c72fa18045a [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 4d1d4e261584467fbbce4c72fa18045a, State: Initialized, Role: LEARNER
I20240503 23:26:32.902535 4446 consensus_queue.cc:260] T a14a12115b8d4fec947c6e223f1ef486 P 4d1d4e261584467fbbce4c72fa18045a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 27, Last appended: 1.27, Last appended by leader: 27, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "83ffd7184ec94302bd1ee74b0fab3e17" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 32909 } } peers { permanent_uuid: "a2f9bf2e5d9b4fc19f804797bca4a3ff" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 46261 } attrs { replace: true } } peers { permanent_uuid: "a3eecbaec43d4954b1096bc3121e824c" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 40649 } } peers { permanent_uuid: "4d1d4e261584467fbbce4c72fa18045a" member_type: NON_VOTER last_known_addr { host: "127.0.100.6" port: 35349 } attrs { promote: true } }
I20240503 23:26:32.903079 4446 ts_tablet_manager.cc:1398] T a14a12115b8d4fec947c6e223f1ef486 P 4d1d4e261584467fbbce4c72fa18045a: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240503 23:26:32.903822 3774 tablet_copy_service.cc:342] P 83ffd7184ec94302bd1ee74b0fab3e17: Request end of tablet copy session 4d1d4e261584467fbbce4c72fa18045a-a14a12115b8d4fec947c6e223f1ef486 received from {username='slave'} at 127.0.0.1:57922
I20240503 23:26:32.903945 3774 tablet_copy_service.cc:433] P 83ffd7184ec94302bd1ee74b0fab3e17: ending tablet copy session 4d1d4e261584467fbbce4c72fa18045a-a14a12115b8d4fec947c6e223f1ef486 on tablet a14a12115b8d4fec947c6e223f1ef486 with peer 4d1d4e261584467fbbce4c72fa18045a
I20240503 23:26:32.964432 4218 raft_consensus.cc:1203] T 55cf664d5e874c0d9b9adf8bdc9c43ed P e4086c099d7f4447b2b4733b6ca364df [term 1 LEARNER]: Deduplicated request from leader. Original: 1.26->[1.27-1.27] Dedup: 1.27->[]
I20240503 23:26:32.967875 4450 tablet_copy_client.cc:531] T 6039230222ff4e57929bed49cd56caf5 P 692da0319a2d4a81ad81f5d8f9f4c86b: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 23:26:32.971622 4450 tablet_bootstrap.cc:492] T 6039230222ff4e57929bed49cd56caf5 P 692da0319a2d4a81ad81f5d8f9f4c86b: Bootstrap starting.
I20240503 23:26:33.001756 4367 raft_consensus.cc:1203] T 98c9500acb43423b86f54ce7a4d98a72 P 4d1d4e261584467fbbce4c72fa18045a [term 1 LEARNER]: Deduplicated request from leader. Original: 1.25->[1.26-1.26] Dedup: 1.26->[]
I20240503 23:26:33.010419 4450 tablet_bootstrap.cc:492] T 6039230222ff4e57929bed49cd56caf5 P 692da0319a2d4a81ad81f5d8f9f4c86b: Bootstrap replayed 1/1 log segments. Stats: ops{read=27 overwritten=0 applied=27 ignored=0} inserts{seen=104 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 23:26:33.010847 4450 tablet_bootstrap.cc:492] T 6039230222ff4e57929bed49cd56caf5 P 692da0319a2d4a81ad81f5d8f9f4c86b: Bootstrap complete.
I20240503 23:26:33.011102 4450 ts_tablet_manager.cc:1367] T 6039230222ff4e57929bed49cd56caf5 P 692da0319a2d4a81ad81f5d8f9f4c86b: Time spent bootstrapping tablet: real 0.040s user 0.040s sys 0.000s
I20240503 23:26:33.011834 4450 raft_consensus.cc:348] T 6039230222ff4e57929bed49cd56caf5 P 692da0319a2d4a81ad81f5d8f9f4c86b [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "a3eecbaec43d4954b1096bc3121e824c" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 40649 } } peers { permanent_uuid: "a2f9bf2e5d9b4fc19f804797bca4a3ff" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 46261 } } peers { permanent_uuid: "83ffd7184ec94302bd1ee74b0fab3e17" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 32909 } attrs { replace: true } } peers { permanent_uuid: "692da0319a2d4a81ad81f5d8f9f4c86b" member_type: NON_VOTER last_known_addr { host: "127.0.100.5" port: 38561 } attrs { promote: true } }
I20240503 23:26:33.012091 4450 raft_consensus.cc:729] T 6039230222ff4e57929bed49cd56caf5 P 692da0319a2d4a81ad81f5d8f9f4c86b [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: 692da0319a2d4a81ad81f5d8f9f4c86b, State: Initialized, Role: LEARNER
I20240503 23:26:33.012306 4450 consensus_queue.cc:260] T 6039230222ff4e57929bed49cd56caf5 P 692da0319a2d4a81ad81f5d8f9f4c86b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 27, Last appended: 2.27, Last appended by leader: 27, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "a3eecbaec43d4954b1096bc3121e824c" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 40649 } } peers { permanent_uuid: "a2f9bf2e5d9b4fc19f804797bca4a3ff" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 46261 } } peers { permanent_uuid: "83ffd7184ec94302bd1ee74b0fab3e17" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 32909 } attrs { replace: true } } peers { permanent_uuid: "692da0319a2d4a81ad81f5d8f9f4c86b" member_type: NON_VOTER last_known_addr { host: "127.0.100.5" port: 38561 } attrs { promote: true } }
I20240503 23:26:33.012876 4450 ts_tablet_manager.cc:1398] T 6039230222ff4e57929bed49cd56caf5 P 692da0319a2d4a81ad81f5d8f9f4c86b: Time spent starting tablet: real 0.002s user 0.000s sys 0.004s
I20240503 23:26:33.013617 3923 tablet_copy_service.cc:342] P a3eecbaec43d4954b1096bc3121e824c: Request end of tablet copy session 692da0319a2d4a81ad81f5d8f9f4c86b-6039230222ff4e57929bed49cd56caf5 received from {username='slave'} at 127.0.0.1:39444
I20240503 23:26:33.013733 3923 tablet_copy_service.cc:433] P a3eecbaec43d4954b1096bc3121e824c: ending tablet copy session 692da0319a2d4a81ad81f5d8f9f4c86b-6039230222ff4e57929bed49cd56caf5 on tablet 6039230222ff4e57929bed49cd56caf5 with peer 692da0319a2d4a81ad81f5d8f9f4c86b
I20240503 23:26:33.167418 4293 raft_consensus.cc:1203] T 6039230222ff4e57929bed49cd56caf5 P 692da0319a2d4a81ad81f5d8f9f4c86b [term 2 LEARNER]: Deduplicated request from leader. Original: 2.26->[2.27-2.27] Dedup: 2.27->[]
I20240503 23:26:33.195257 3715 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240503 23:26:33.198745 3715 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240503 23:26:33.199141 3715 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240503 23:26:33.199887 4293 raft_consensus.cc:1203] T 1b9566be6c644cfca852ec6507fab893 P 692da0319a2d4a81ad81f5d8f9f4c86b [term 1 LEARNER]: Deduplicated request from leader. Original: 1.26->[1.27-1.27] Dedup: 1.27->[]
I20240503 23:26:33.260133 4367 raft_consensus.cc:1203] T a14a12115b8d4fec947c6e223f1ef486 P 4d1d4e261584467fbbce4c72fa18045a [term 1 LEARNER]: Deduplicated request from leader. Original: 1.26->[1.27-1.27] Dedup: 1.27->[]
I20240503 23:26:33.355398 4467 raft_consensus.cc:1050] T 55cf664d5e874c0d9b9adf8bdc9c43ed P a2f9bf2e5d9b4fc19f804797bca4a3ff: attempting to promote NON_VOTER e4086c099d7f4447b2b4733b6ca364df to VOTER
I20240503 23:26:33.356453 4467 consensus_queue.cc:237] T 55cf664d5e874c0d9b9adf8bdc9c43ed P a2f9bf2e5d9b4fc19f804797bca4a3ff [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 27, Committed index: 27, Last appended: 1.27, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "83ffd7184ec94302bd1ee74b0fab3e17" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 32909 } } peers { permanent_uuid: "a2f9bf2e5d9b4fc19f804797bca4a3ff" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 46261 } } peers { permanent_uuid: "a3eecbaec43d4954b1096bc3121e824c" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 40649 } attrs { replace: true } } peers { permanent_uuid: "e4086c099d7f4447b2b4733b6ca364df" member_type: VOTER last_known_addr { host: "127.0.100.4" port: 35011 } attrs { promote: false } }
I20240503 23:26:33.357996 4218 raft_consensus.cc:1203] T 7181e4e8c0324a8eb095dde23651a1ab P e4086c099d7f4447b2b4733b6ca364df [term 1 LEARNER]: Deduplicated request from leader. Original: 1.25->[1.26-1.26] Dedup: 1.26->[]
I20240503 23:26:33.360158 3760 raft_consensus.cc:1261] T 55cf664d5e874c0d9b9adf8bdc9c43ed P 83ffd7184ec94302bd1ee74b0fab3e17 [term 1 FOLLOWER]: Refusing update from remote peer a2f9bf2e5d9b4fc19f804797bca4a3ff: Log matching property violated. Preceding OpId in replica: term: 1 index: 27. Preceding OpId from leader: term: 1 index: 28. (index mismatch)
I20240503 23:26:33.360616 3908 raft_consensus.cc:1261] T 55cf664d5e874c0d9b9adf8bdc9c43ed P a3eecbaec43d4954b1096bc3121e824c [term 1 FOLLOWER]: Refusing update from remote peer a2f9bf2e5d9b4fc19f804797bca4a3ff: Log matching property violated. Preceding OpId in replica: term: 1 index: 27. Preceding OpId from leader: term: 1 index: 28. (index mismatch)
I20240503 23:26:33.360802 4218 raft_consensus.cc:1261] T 55cf664d5e874c0d9b9adf8bdc9c43ed P e4086c099d7f4447b2b4733b6ca364df [term 1 LEARNER]: Refusing update from remote peer a2f9bf2e5d9b4fc19f804797bca4a3ff: Log matching property violated. Preceding OpId in replica: term: 1 index: 27. Preceding OpId from leader: term: 1 index: 28. (index mismatch)
I20240503 23:26:33.360811 4466 consensus_queue.cc:1035] T 55cf664d5e874c0d9b9adf8bdc9c43ed P a2f9bf2e5d9b4fc19f804797bca4a3ff [LEADER]: Connected to new peer: Peer: permanent_uuid: "83ffd7184ec94302bd1ee74b0fab3e17" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 32909 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 28, Last known committed idx: 27, Time since last communication: 0.000s
I20240503 23:26:33.361467 3966 consensus_queue.cc:1035] T 55cf664d5e874c0d9b9adf8bdc9c43ed P a2f9bf2e5d9b4fc19f804797bca4a3ff [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4086c099d7f4447b2b4733b6ca364df" member_type: VOTER last_known_addr { host: "127.0.100.4" port: 35011 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 28, Last known committed idx: 27, Time since last communication: 0.000s
I20240503 23:26:33.364168 4467 consensus_queue.cc:1035] T 55cf664d5e874c0d9b9adf8bdc9c43ed P a2f9bf2e5d9b4fc19f804797bca4a3ff [LEADER]: Connected to new peer: Peer: permanent_uuid: "a3eecbaec43d4954b1096bc3121e824c" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 40649 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 28, Last known committed idx: 27, Time since last communication: 0.000s
I20240503 23:26:33.368103 4466 raft_consensus.cc:2932] T 55cf664d5e874c0d9b9adf8bdc9c43ed P a2f9bf2e5d9b4fc19f804797bca4a3ff [term 1 LEADER]: Committing config change with OpId 1.28: config changed from index 27 to 28, e4086c099d7f4447b2b4733b6ca364df (127.0.100.4) changed from NON_VOTER to VOTER. New config: { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "83ffd7184ec94302bd1ee74b0fab3e17" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 32909 } } peers { permanent_uuid: "a2f9bf2e5d9b4fc19f804797bca4a3ff" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 46261 } } peers { permanent_uuid: "a3eecbaec43d4954b1096bc3121e824c" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 40649 } attrs { replace: true } } peers { permanent_uuid: "e4086c099d7f4447b2b4733b6ca364df" member_type: VOTER last_known_addr { host: "127.0.100.4" port: 35011 } attrs { promote: false } } }
I20240503 23:26:33.368870 3908 raft_consensus.cc:2932] T 55cf664d5e874c0d9b9adf8bdc9c43ed P a3eecbaec43d4954b1096bc3121e824c [term 1 FOLLOWER]: Committing config change with OpId 1.28: config changed from index 27 to 28, e4086c099d7f4447b2b4733b6ca364df (127.0.100.4) changed from NON_VOTER to VOTER. New config: { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "83ffd7184ec94302bd1ee74b0fab3e17" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 32909 } } peers { permanent_uuid: "a2f9bf2e5d9b4fc19f804797bca4a3ff" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 46261 } } peers { permanent_uuid: "a3eecbaec43d4954b1096bc3121e824c" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 40649 } attrs { replace: true } } peers { permanent_uuid: "e4086c099d7f4447b2b4733b6ca364df" member_type: VOTER last_known_addr { host: "127.0.100.4" port: 35011 } attrs { promote: false } } }
I20240503 23:26:33.374878 3760 raft_consensus.cc:2932] T 55cf664d5e874c0d9b9adf8bdc9c43ed P 83ffd7184ec94302bd1ee74b0fab3e17 [term 1 FOLLOWER]: Committing config change with OpId 1.28: config changed from index 27 to 28, e4086c099d7f4447b2b4733b6ca364df (127.0.100.4) changed from NON_VOTER to VOTER. New config: { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "83ffd7184ec94302bd1ee74b0fab3e17" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 32909 } } peers { permanent_uuid: "a2f9bf2e5d9b4fc19f804797bca4a3ff" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 46261 } } peers { permanent_uuid: "a3eecbaec43d4954b1096bc3121e824c" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 40649 } attrs { replace: true } } peers { permanent_uuid: "e4086c099d7f4447b2b4733b6ca364df" member_type: VOTER last_known_addr { host: "127.0.100.4" port: 35011 } attrs { promote: false } } }
I20240503 23:26:33.374998 3662 catalog_manager.cc:5484] T 55cf664d5e874c0d9b9adf8bdc9c43ed P a2f9bf2e5d9b4fc19f804797bca4a3ff reported cstate change: config changed from index 27 to 28, e4086c099d7f4447b2b4733b6ca364df (127.0.100.4) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "a2f9bf2e5d9b4fc19f804797bca4a3ff" committed_config { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "83ffd7184ec94302bd1ee74b0fab3e17" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 32909 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a2f9bf2e5d9b4fc19f804797bca4a3ff" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 46261 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a3eecbaec43d4954b1096bc3121e824c" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 40649 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e4086c099d7f4447b2b4733b6ca364df" member_type: VOTER last_known_addr { host: "127.0.100.4" port: 35011 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240503 23:26:33.380168 4218 raft_consensus.cc:2932] T 55cf664d5e874c0d9b9adf8bdc9c43ed P e4086c099d7f4447b2b4733b6ca364df [term 1 FOLLOWER]: Committing config change with OpId 1.28: config changed from index 27 to 28, e4086c099d7f4447b2b4733b6ca364df (127.0.100.4) changed from NON_VOTER to VOTER. New config: { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "83ffd7184ec94302bd1ee74b0fab3e17" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 32909 } } peers { permanent_uuid: "a2f9bf2e5d9b4fc19f804797bca4a3ff" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 46261 } } peers { permanent_uuid: "a3eecbaec43d4954b1096bc3121e824c" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 40649 } attrs { replace: true } } peers { permanent_uuid: "e4086c099d7f4447b2b4733b6ca364df" member_type: VOTER last_known_addr { host: "127.0.100.4" port: 35011 } attrs { promote: false } } }
I20240503 23:26:33.390237 3838 consensus_queue.cc:237] T 55cf664d5e874c0d9b9adf8bdc9c43ed P a2f9bf2e5d9b4fc19f804797bca4a3ff [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 28, Committed index: 28, Last appended: 1.28, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "83ffd7184ec94302bd1ee74b0fab3e17" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 32909 } } peers { permanent_uuid: "a2f9bf2e5d9b4fc19f804797bca4a3ff" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 46261 } } peers { permanent_uuid: "e4086c099d7f4447b2b4733b6ca364df" member_type: VOTER last_known_addr { host: "127.0.100.4" port: 35011 } attrs { promote: false } }
I20240503 23:26:33.392323 4218 raft_consensus.cc:1261] T 55cf664d5e874c0d9b9adf8bdc9c43ed P e4086c099d7f4447b2b4733b6ca364df [term 1 FOLLOWER]: Refusing update from remote peer a2f9bf2e5d9b4fc19f804797bca4a3ff: Log matching property violated. Preceding OpId in replica: term: 1 index: 28. Preceding OpId from leader: term: 1 index: 29. (index mismatch)
I20240503 23:26:33.392601 3760 raft_consensus.cc:1261] T 55cf664d5e874c0d9b9adf8bdc9c43ed P 83ffd7184ec94302bd1ee74b0fab3e17 [term 1 FOLLOWER]: Refusing update from remote peer a2f9bf2e5d9b4fc19f804797bca4a3ff: Log matching property violated. Preceding OpId in replica: term: 1 index: 28. Preceding OpId from leader: term: 1 index: 29. (index mismatch)
I20240503 23:26:33.393126 4466 consensus_queue.cc:1035] T 55cf664d5e874c0d9b9adf8bdc9c43ed P a2f9bf2e5d9b4fc19f804797bca4a3ff [LEADER]: Connected to new peer: Peer: permanent_uuid: "83ffd7184ec94302bd1ee74b0fab3e17" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 32909 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.000s
I20240503 23:26:33.393456 4467 consensus_queue.cc:1035] T 55cf664d5e874c0d9b9adf8bdc9c43ed P a2f9bf2e5d9b4fc19f804797bca4a3ff [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4086c099d7f4447b2b4733b6ca364df" member_type: VOTER last_known_addr { host: "127.0.100.4" port: 35011 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.000s
I20240503 23:26:33.395799 3966 raft_consensus.cc:2932] T 55cf664d5e874c0d9b9adf8bdc9c43ed P a2f9bf2e5d9b4fc19f804797bca4a3ff [term 1 LEADER]: Committing config change with OpId 1.29: config changed from index 28 to 29, VOTER a3eecbaec43d4954b1096bc3121e824c (127.0.100.3) evicted. New config: { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "83ffd7184ec94302bd1ee74b0fab3e17" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 32909 } } peers { permanent_uuid: "a2f9bf2e5d9b4fc19f804797bca4a3ff" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 46261 } } peers { permanent_uuid: "e4086c099d7f4447b2b4733b6ca364df" member_type: VOTER last_known_addr { host: "127.0.100.4" port: 35011 } attrs { promote: false } } }
I20240503 23:26:33.396502 4218 raft_consensus.cc:2932] T 55cf664d5e874c0d9b9adf8bdc9c43ed P e4086c099d7f4447b2b4733b6ca364df [term 1 FOLLOWER]: Committing config change with OpId 1.29: config changed from index 28 to 29, VOTER a3eecbaec43d4954b1096bc3121e824c (127.0.100.3) evicted. New config: { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "83ffd7184ec94302bd1ee74b0fab3e17" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 32909 } } peers { permanent_uuid: "a2f9bf2e5d9b4fc19f804797bca4a3ff" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 46261 } } peers { permanent_uuid: "e4086c099d7f4447b2b4733b6ca364df" member_type: VOTER last_known_addr { host: "127.0.100.4" port: 35011 } attrs { promote: false } } }
I20240503 23:26:33.397171 3761 raft_consensus.cc:2932] T 55cf664d5e874c0d9b9adf8bdc9c43ed P 83ffd7184ec94302bd1ee74b0fab3e17 [term 1 FOLLOWER]: Committing config change with OpId 1.29: config changed from index 28 to 29, VOTER a3eecbaec43d4954b1096bc3121e824c (127.0.100.3) evicted. New config: { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "83ffd7184ec94302bd1ee74b0fab3e17" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 32909 } } peers { permanent_uuid: "a2f9bf2e5d9b4fc19f804797bca4a3ff" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 46261 } } peers { permanent_uuid: "e4086c099d7f4447b2b4733b6ca364df" member_type: VOTER last_known_addr { host: "127.0.100.4" port: 35011 } attrs { promote: false } } }
I20240503 23:26:33.400810 3649 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 55cf664d5e874c0d9b9adf8bdc9c43ed with cas_config_opid_index 28: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 23:26:33.402251 3655 catalog_manager.cc:5484] T 55cf664d5e874c0d9b9adf8bdc9c43ed P e4086c099d7f4447b2b4733b6ca364df reported cstate change: config changed from index 28 to 29, VOTER a3eecbaec43d4954b1096bc3121e824c (127.0.100.3) evicted. New cstate: current_term: 1 leader_uuid: "a2f9bf2e5d9b4fc19f804797bca4a3ff" committed_config { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "83ffd7184ec94302bd1ee74b0fab3e17" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 32909 } } peers { permanent_uuid: "a2f9bf2e5d9b4fc19f804797bca4a3ff" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 46261 } } peers { permanent_uuid: "e4086c099d7f4447b2b4733b6ca364df" member_type: VOTER last_known_addr { host: "127.0.100.4" port: 35011 } attrs { promote: false } } }
I20240503 23:26:33.407390 3902 tablet_service.cc:1514] Processing DeleteTablet for tablet 55cf664d5e874c0d9b9adf8bdc9c43ed with delete_type TABLET_DATA_TOMBSTONED (TS a3eecbaec43d4954b1096bc3121e824c not found in new config with opid_index 29) from {username='slave'} at 127.0.0.1:40632
I20240503 23:26:33.408390 4476 tablet_replica.cc:330] T 55cf664d5e874c0d9b9adf8bdc9c43ed P a3eecbaec43d4954b1096bc3121e824c: stopping tablet replica
I20240503 23:26:33.408811 4476 raft_consensus.cc:2229] T 55cf664d5e874c0d9b9adf8bdc9c43ed P a3eecbaec43d4954b1096bc3121e824c [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 23:26:33.409168 4476 raft_consensus.cc:2258] T 55cf664d5e874c0d9b9adf8bdc9c43ed P a3eecbaec43d4954b1096bc3121e824c [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 23:26:33.410109 4476 ts_tablet_manager.cc:1875] T 55cf664d5e874c0d9b9adf8bdc9c43ed P a3eecbaec43d4954b1096bc3121e824c: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 23:26:33.416363 4476 ts_tablet_manager.cc:1888] T 55cf664d5e874c0d9b9adf8bdc9c43ed P a3eecbaec43d4954b1096bc3121e824c: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.28
I20240503 23:26:33.416621 4476 log.cc:1196] T 55cf664d5e874c0d9b9adf8bdc9c43ed P a3eecbaec43d4954b1096bc3121e824c: Deleting WAL directory at /tmp/dist-test-taskZhIiPz/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714778688451250-400-0/minicluster-data/ts-2-root/wals/55cf664d5e874c0d9b9adf8bdc9c43ed
I20240503 23:26:33.417338 3646 catalog_manager.cc:4830] TS a3eecbaec43d4954b1096bc3121e824c (127.0.100.3:40649): tablet 55cf664d5e874c0d9b9adf8bdc9c43ed (table test-workload [id=1345b2a336c84ec2b1de62853e7fa3f6]) successfully deleted
I20240503 23:26:33.462868 4478 raft_consensus.cc:1050] T 98c9500acb43423b86f54ce7a4d98a72 P 83ffd7184ec94302bd1ee74b0fab3e17: attempting to promote NON_VOTER 4d1d4e261584467fbbce4c72fa18045a to VOTER
I20240503 23:26:33.463888 4478 consensus_queue.cc:237] T 98c9500acb43423b86f54ce7a4d98a72 P 83ffd7184ec94302bd1ee74b0fab3e17 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 26, Committed index: 26, Last appended: 1.26, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "83ffd7184ec94302bd1ee74b0fab3e17" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 32909 } } peers { permanent_uuid: "a2f9bf2e5d9b4fc19f804797bca4a3ff" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 46261 } } peers { permanent_uuid: "a3eecbaec43d4954b1096bc3121e824c" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 40649 } attrs { replace: true } } peers { permanent_uuid: "4d1d4e261584467fbbce4c72fa18045a" member_type: VOTER last_known_addr { host: "127.0.100.6" port: 35349 } attrs { promote: false } }
I20240503 23:26:33.466827 3838 raft_consensus.cc:1261] T 98c9500acb43423b86f54ce7a4d98a72 P a2f9bf2e5d9b4fc19f804797bca4a3ff [term 1 FOLLOWER]: Refusing update from remote peer 83ffd7184ec94302bd1ee74b0fab3e17: Log matching property violated. Preceding OpId in replica: term: 1 index: 26. Preceding OpId from leader: term: 1 index: 27. (index mismatch)
I20240503 23:26:33.466828 4367 raft_consensus.cc:1261] T 98c9500acb43423b86f54ce7a4d98a72 P 4d1d4e261584467fbbce4c72fa18045a [term 1 LEARNER]: Refusing update from remote peer 83ffd7184ec94302bd1ee74b0fab3e17: Log matching property violated. Preceding OpId in replica: term: 1 index: 26. Preceding OpId from leader: term: 1 index: 27. (index mismatch)
I20240503 23:26:33.467442 4477 consensus_queue.cc:1035] T 98c9500acb43423b86f54ce7a4d98a72 P 83ffd7184ec94302bd1ee74b0fab3e17 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2f9bf2e5d9b4fc19f804797bca4a3ff" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 46261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 27, Last known committed idx: 26, Time since last communication: 0.000s
I20240503 23:26:33.467628 3908 raft_consensus.cc:1261] T 98c9500acb43423b86f54ce7a4d98a72 P a3eecbaec43d4954b1096bc3121e824c [term 1 FOLLOWER]: Refusing update from remote peer 83ffd7184ec94302bd1ee74b0fab3e17: Log matching property violated. Preceding OpId in replica: term: 1 index: 26. Preceding OpId from leader: term: 1 index: 27. (index mismatch)
I20240503 23:26:33.467875 4454 consensus_queue.cc:1035] T 98c9500acb43423b86f54ce7a4d98a72 P 83ffd7184ec94302bd1ee74b0fab3e17 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d1d4e261584467fbbce4c72fa18045a" member_type: VOTER last_known_addr { host: "127.0.100.6" port: 35349 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 27, Last known committed idx: 26, Time since last communication: 0.000s
I20240503 23:26:33.468178 4478 consensus_queue.cc:1035] T 98c9500acb43423b86f54ce7a4d98a72 P 83ffd7184ec94302bd1ee74b0fab3e17 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a3eecbaec43d4954b1096bc3121e824c" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 40649 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 27, Last known committed idx: 26, Time since last communication: 0.000s
I20240503 23:26:33.472018 4454 raft_consensus.cc:2932] T 98c9500acb43423b86f54ce7a4d98a72 P 83ffd7184ec94302bd1ee74b0fab3e17 [term 1 LEADER]: Committing config change with OpId 1.27: config changed from index 26 to 27, 4d1d4e261584467fbbce4c72fa18045a (127.0.100.6) changed from NON_VOTER to VOTER. New config: { opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "83ffd7184ec94302bd1ee74b0fab3e17" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 32909 } } peers { permanent_uuid: "a2f9bf2e5d9b4fc19f804797bca4a3ff" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 46261 } } peers { permanent_uuid: "a3eecbaec43d4954b1096bc3121e824c" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 40649 } attrs { replace: true } } peers { permanent_uuid: "4d1d4e261584467fbbce4c72fa18045a" member_type: VOTER last_known_addr { host: "127.0.100.6" port: 35349 } attrs { promote: false } } }
I20240503 23:26:33.472714 4367 raft_consensus.cc:2932] T 98c9500acb43423b86f54ce7a4d98a72 P 4d1d4e261584467fbbce4c72fa18045a [term 1 FOLLOWER]: Committing config change with OpId 1.27: config changed from index 26 to 27, 4d1d4e261584467fbbce4c72fa18045a (127.0.100.6) changed from NON_VOTER to VOTER. New config: { opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "83ffd7184ec94302bd1ee74b0fab3e17" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 32909 } } peers { permanent_uuid: "a2f9bf2e5d9b4fc19f804797bca4a3ff" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 46261 } } peers { permanent_uuid: "a3eecbaec43d4954b1096bc3121e824c" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 40649 } attrs { replace: true } } peers { permanent_uuid: "4d1d4e261584467fbbce4c72fa18045a" member_type: VOTER last_known_addr { host: "127.0.100.6" port: 35349 } attrs { promote: false } } }
I20240503 23:26:33.476791 3838 raft_consensus.cc:2932] T 98c9500acb43423b86f54ce7a4d98a72 P a2f9bf2e5d9b4fc19f804797bca4a3ff [term 1 FOLLOWER]: Committing config change with OpId 1.27: config changed from index 26 to 27, 4d1d4e261584467fbbce4c72fa18045a (127.0.100.6) changed from NON_VOTER to VOTER. New config: { opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "83ffd7184ec94302bd1ee74b0fab3e17" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 32909 } } peers { permanent_uuid: "a2f9bf2e5d9b4fc19f804797bca4a3ff" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 46261 } } peers { permanent_uuid: "a3eecbaec43d4954b1096bc3121e824c" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 40649 } attrs { replace: true } } peers { permanent_uuid: "4d1d4e261584467fbbce4c72fa18045a" member_type: VOTER last_known_addr { host: "127.0.100.6" port: 35349 } attrs { promote: false } } }
I20240503 23:26:33.477677 3655 catalog_manager.cc:5484] T 98c9500acb43423b86f54ce7a4d98a72 P 83ffd7184ec94302bd1ee74b0fab3e17 reported cstate change: config changed from index 26 to 27, 4d1d4e261584467fbbce4c72fa18045a (127.0.100.6) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "83ffd7184ec94302bd1ee74b0fab3e17" committed_config { opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "83ffd7184ec94302bd1ee74b0fab3e17" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 32909 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a2f9bf2e5d9b4fc19f804797bca4a3ff" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 46261 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a3eecbaec43d4954b1096bc3121e824c" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 40649 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4d1d4e261584467fbbce4c72fa18045a" member_type: VOTER last_known_addr { host: "127.0.100.6" port: 35349 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 23:26:33.479738 3908 raft_consensus.cc:2932] T 98c9500acb43423b86f54ce7a4d98a72 P a3eecbaec43d4954b1096bc3121e824c [term 1 FOLLOWER]: Committing config change with OpId 1.27: config changed from index 26 to 27, 4d1d4e261584467fbbce4c72fa18045a (127.0.100.6) changed from NON_VOTER to VOTER. New config: { opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "83ffd7184ec94302bd1ee74b0fab3e17" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 32909 } } peers { permanent_uuid: "a2f9bf2e5d9b4fc19f804797bca4a3ff" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 46261 } } peers { permanent_uuid: "a3eecbaec43d4954b1096bc3121e824c" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 40649 } attrs { replace: true } } peers { permanent_uuid: "4d1d4e261584467fbbce4c72fa18045a" member_type: VOTER last_known_addr { host: "127.0.100.6" port: 35349 } attrs { promote: false } } }
I20240503 23:26:33.486064 3761 consensus_queue.cc:237] T 98c9500acb43423b86f54ce7a4d98a72 P 83ffd7184ec94302bd1ee74b0fab3e17 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 27, Committed index: 27, Last appended: 1.27, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "83ffd7184ec94302bd1ee74b0fab3e17" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 32909 } } peers { permanent_uuid: "a2f9bf2e5d9b4fc19f804797bca4a3ff" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 46261 } } peers { permanent_uuid: "4d1d4e261584467fbbce4c72fa18045a" member_type: VOTER last_known_addr { host: "127.0.100.6" port: 35349 } attrs { promote: false } }
I20240503 23:26:33.488150 4367 raft_consensus.cc:1261] T 98c9500acb43423b86f54ce7a4d98a72 P 4d1d4e261584467fbbce4c72fa18045a [term 1 FOLLOWER]: Refusing update from remote peer 83ffd7184ec94302bd1ee74b0fab3e17: Log matching property violated. Preceding OpId in replica: term: 1 index: 27. Preceding OpId from leader: term: 1 index: 28. (index mismatch)
I20240503 23:26:33.488435 3838 raft_consensus.cc:1261] T 98c9500acb43423b86f54ce7a4d98a72 P a2f9bf2e5d9b4fc19f804797bca4a3ff [term 1 FOLLOWER]: Refusing update from remote peer 83ffd7184ec94302bd1ee74b0fab3e17: Log matching property violated. Preceding OpId in replica: term: 1 index: 27. Preceding OpId from leader: term: 1 index: 28. (index mismatch)
I20240503 23:26:33.488899 4454 consensus_queue.cc:1035] T 98c9500acb43423b86f54ce7a4d98a72 P 83ffd7184ec94302bd1ee74b0fab3e17 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d1d4e261584467fbbce4c72fa18045a" member_type: VOTER last_known_addr { host: "127.0.100.6" port: 35349 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 28, Last known committed idx: 27, Time since last communication: 0.000s
I20240503 23:26:33.489254 4478 consensus_queue.cc:1035] T 98c9500acb43423b86f54ce7a4d98a72 P 83ffd7184ec94302bd1ee74b0fab3e17 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2f9bf2e5d9b4fc19f804797bca4a3ff" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 46261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 28, Last known committed idx: 27, Time since last communication: 0.000s
I20240503 23:26:33.492511 4454 raft_consensus.cc:2932] T 98c9500acb43423b86f54ce7a4d98a72 P 83ffd7184ec94302bd1ee74b0fab3e17 [term 1 LEADER]: Committing config change with OpId 1.28: config changed from index 27 to 28, VOTER a3eecbaec43d4954b1096bc3121e824c (127.0.100.3) evicted. New config: { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "83ffd7184ec94302bd1ee74b0fab3e17" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 32909 } } peers { permanent_uuid: "a2f9bf2e5d9b4fc19f804797bca4a3ff" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 46261 } } peers { permanent_uuid: "4d1d4e261584467fbbce4c72fa18045a" member_type: VOTER last_known_addr { host: "127.0.100.6" port: 35349 } attrs { promote: false } } }
I20240503 23:26:33.494525 3838 raft_consensus.cc:2932] T 98c9500acb43423b86f54ce7a4d98a72 P a2f9bf2e5d9b4fc19f804797bca4a3ff [term 1 FOLLOWER]: Committing config change with OpId 1.28: config changed from index 27 to 28, VOTER a3eecbaec43d4954b1096bc3121e824c (127.0.100.3) evicted. New config: { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "83ffd7184ec94302bd1ee74b0fab3e17" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 32909 } } peers { permanent_uuid: "a2f9bf2e5d9b4fc19f804797bca4a3ff" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 46261 } } peers { permanent_uuid: "4d1d4e261584467fbbce4c72fa18045a" member_type: VOTER last_known_addr { host: "127.0.100.6" port: 35349 } attrs { promote: false } } }
I20240503 23:26:33.497306 3648 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 98c9500acb43423b86f54ce7a4d98a72 with cas_config_opid_index 27: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 23:26:33.498011 4367 raft_consensus.cc:2932] T 98c9500acb43423b86f54ce7a4d98a72 P 4d1d4e261584467fbbce4c72fa18045a [term 1 FOLLOWER]: Committing config change with OpId 1.28: config changed from index 27 to 28, VOTER a3eecbaec43d4954b1096bc3121e824c (127.0.100.3) evicted. New config: { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "83ffd7184ec94302bd1ee74b0fab3e17" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 32909 } } peers { permanent_uuid: "a2f9bf2e5d9b4fc19f804797bca4a3ff" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 46261 } } peers { permanent_uuid: "4d1d4e261584467fbbce4c72fa18045a" member_type: VOTER last_known_addr { host: "127.0.100.6" port: 35349 } attrs { promote: false } } }
I20240503 23:26:33.500247 3659 catalog_manager.cc:5484] T 98c9500acb43423b86f54ce7a4d98a72 P 83ffd7184ec94302bd1ee74b0fab3e17 reported cstate change: config changed from index 27 to 28, VOTER a3eecbaec43d4954b1096bc3121e824c (127.0.100.3) evicted. New cstate: current_term: 1 leader_uuid: "83ffd7184ec94302bd1ee74b0fab3e17" committed_config { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "83ffd7184ec94302bd1ee74b0fab3e17" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 32909 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a2f9bf2e5d9b4fc19f804797bca4a3ff" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 46261 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4d1d4e261584467fbbce4c72fa18045a" member_type: VOTER last_known_addr { host: "127.0.100.6" port: 35349 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 23:26:33.505821 3902 tablet_service.cc:1514] Processing DeleteTablet for tablet 98c9500acb43423b86f54ce7a4d98a72 with delete_type TABLET_DATA_TOMBSTONED (TS a3eecbaec43d4954b1096bc3121e824c not found in new config with opid_index 28) from {username='slave'} at 127.0.0.1:40632
I20240503 23:26:33.506229 4476 tablet_replica.cc:330] T 98c9500acb43423b86f54ce7a4d98a72 P a3eecbaec43d4954b1096bc3121e824c: stopping tablet replica
I20240503 23:26:33.506479 4476 raft_consensus.cc:2229] T 98c9500acb43423b86f54ce7a4d98a72 P a3eecbaec43d4954b1096bc3121e824c [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 23:26:33.506837 4476 raft_consensus.cc:2258] T 98c9500acb43423b86f54ce7a4d98a72 P a3eecbaec43d4954b1096bc3121e824c [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 23:26:33.507897 4476 ts_tablet_manager.cc:1875] T 98c9500acb43423b86f54ce7a4d98a72 P a3eecbaec43d4954b1096bc3121e824c: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 23:26:33.513255 4476 ts_tablet_manager.cc:1888] T 98c9500acb43423b86f54ce7a4d98a72 P a3eecbaec43d4954b1096bc3121e824c: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.27
I20240503 23:26:33.513427 4476 log.cc:1196] T 98c9500acb43423b86f54ce7a4d98a72 P a3eecbaec43d4954b1096bc3121e824c: Deleting WAL directory at /tmp/dist-test-taskZhIiPz/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714778688451250-400-0/minicluster-data/ts-2-root/wals/98c9500acb43423b86f54ce7a4d98a72
I20240503 23:26:33.514088 3646 catalog_manager.cc:4830] TS a3eecbaec43d4954b1096bc3121e824c (127.0.100.3:40649): tablet 98c9500acb43423b86f54ce7a4d98a72 (table test-workload [id=1345b2a336c84ec2b1de62853e7fa3f6]) successfully deleted
I20240503 23:26:33.550345 4483 raft_consensus.cc:1050] T 6039230222ff4e57929bed49cd56caf5 P a3eecbaec43d4954b1096bc3121e824c: attempting to promote NON_VOTER 692da0319a2d4a81ad81f5d8f9f4c86b to VOTER
I20240503 23:26:33.551337 4483 consensus_queue.cc:237] T 6039230222ff4e57929bed49cd56caf5 P a3eecbaec43d4954b1096bc3121e824c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 27, Committed index: 27, Last appended: 2.27, Last appended by leader: 0, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "a3eecbaec43d4954b1096bc3121e824c" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 40649 } } peers { permanent_uuid: "a2f9bf2e5d9b4fc19f804797bca4a3ff" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 46261 } } peers { permanent_uuid: "83ffd7184ec94302bd1ee74b0fab3e17" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 32909 } attrs { replace: true } } peers { permanent_uuid: "692da0319a2d4a81ad81f5d8f9f4c86b" member_type: VOTER last_known_addr { host: "127.0.100.5" port: 38561 } attrs { promote: false } }
I20240503 23:26:33.555331 4293 raft_consensus.cc:1261] T 6039230222ff4e57929bed49cd56caf5 P 692da0319a2d4a81ad81f5d8f9f4c86b [term 2 LEARNER]: Refusing update from remote peer a3eecbaec43d4954b1096bc3121e824c: Log matching property violated. Preceding OpId in replica: term: 2 index: 27. Preceding OpId from leader: term: 2 index: 28. (index mismatch)
I20240503 23:26:33.555891 4426 consensus_queue.cc:1035] T 6039230222ff4e57929bed49cd56caf5 P a3eecbaec43d4954b1096bc3121e824c [LEADER]: Connected to new peer: Peer: permanent_uuid: "692da0319a2d4a81ad81f5d8f9f4c86b" member_type: VOTER last_known_addr { host: "127.0.100.5" port: 38561 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 28, Last known committed idx: 27, Time since last communication: 0.000s
I20240503 23:26:33.556128 3838 raft_consensus.cc:1261] T 6039230222ff4e57929bed49cd56caf5 P a2f9bf2e5d9b4fc19f804797bca4a3ff [term 2 FOLLOWER]: Refusing update from remote peer a3eecbaec43d4954b1096bc3121e824c: Log matching property violated. Preceding OpId in replica: term: 2 index: 27. Preceding OpId from leader: term: 2 index: 28. (index mismatch)
I20240503 23:26:33.556720 4426 consensus_queue.cc:1035] T 6039230222ff4e57929bed49cd56caf5 P a3eecbaec43d4954b1096bc3121e824c [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2f9bf2e5d9b4fc19f804797bca4a3ff" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 46261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 28, Last known committed idx: 27, Time since last communication: 0.000s
I20240503 23:26:33.556746 3761 raft_consensus.cc:1261] T 6039230222ff4e57929bed49cd56caf5 P 83ffd7184ec94302bd1ee74b0fab3e17 [term 2 FOLLOWER]: Refusing update from remote peer a3eecbaec43d4954b1096bc3121e824c: Log matching property violated. Preceding OpId in replica: term: 2 index: 27. Preceding OpId from leader: term: 2 index: 28. (index mismatch)
I20240503 23:26:33.557832 4426 consensus_queue.cc:1035] T 6039230222ff4e57929bed49cd56caf5 P a3eecbaec43d4954b1096bc3121e824c [LEADER]: Connected to new peer: Peer: permanent_uuid: "83ffd7184ec94302bd1ee74b0fab3e17" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 32909 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 28, Last known committed idx: 27, Time since last communication: 0.000s
I20240503 23:26:33.560602 4483 raft_consensus.cc:2932] T 6039230222ff4e57929bed49cd56caf5 P a3eecbaec43d4954b1096bc3121e824c [term 2 LEADER]: Committing config change with OpId 2.28: config changed from index 27 to 28, 692da0319a2d4a81ad81f5d8f9f4c86b (127.0.100.5) changed from NON_VOTER to VOTER. New config: { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "a3eecbaec43d4954b1096bc3121e824c" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 40649 } } peers { permanent_uuid: "a2f9bf2e5d9b4fc19f804797bca4a3ff" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 46261 } } peers { permanent_uuid: "83ffd7184ec94302bd1ee74b0fab3e17" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 32909 } attrs { replace: true } } peers { permanent_uuid: "692da0319a2d4a81ad81f5d8f9f4c86b" member_type: VOTER last_known_addr { host: "127.0.100.5" port: 38561 } attrs { promote: false } } }
I20240503 23:26:33.561416 3838 raft_consensus.cc:2932] T 6039230222ff4e57929bed49cd56caf5 P a2f9bf2e5d9b4fc19f804797bca4a3ff [term 2 FOLLOWER]: Committing config change with OpId 2.28: config changed from index 27 to 28, 692da0319a2d4a81ad81f5d8f9f4c86b (127.0.100.5) changed from NON_VOTER to VOTER. New config: { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "a3eecbaec43d4954b1096bc3121e824c" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 40649 } } peers { permanent_uuid: "a2f9bf2e5d9b4fc19f804797bca4a3ff" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 46261 } } peers { permanent_uuid: "83ffd7184ec94302bd1ee74b0fab3e17" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 32909 } attrs { replace: true } } peers { permanent_uuid: "692da0319a2d4a81ad81f5d8f9f4c86b" member_type: VOTER last_known_addr { host: "127.0.100.5" port: 38561 } attrs { promote: false } } }
I20240503 23:26:33.566049 4293 raft_consensus.cc:2932] T 6039230222ff4e57929bed49cd56caf5 P 692da0319a2d4a81ad81f5d8f9f4c86b [term 2 FOLLOWER]: Committing config change with OpId 2.28: config changed from index 27 to 28, 692da0319a2d4a81ad81f5d8f9f4c86b (127.0.100.5) changed from NON_VOTER to VOTER. New config: { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "a3eecbaec43d4954b1096bc3121e824c" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 40649 } } peers { permanent_uuid: "a2f9bf2e5d9b4fc19f804797bca4a3ff" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 46261 } } peers { permanent_uuid: "83ffd7184ec94302bd1ee74b0fab3e17" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 32909 } attrs { replace: true } } peers { permanent_uuid: "692da0319a2d4a81ad81f5d8f9f4c86b" member_type: VOTER last_known_addr { host: "127.0.100.5" port: 38561 } attrs { promote: false } } }
I20240503 23:26:33.567924 3662 catalog_manager.cc:5484] T 6039230222ff4e57929bed49cd56caf5 P a3eecbaec43d4954b1096bc3121e824c reported cstate change: config changed from index 27 to 28, 692da0319a2d4a81ad81f5d8f9f4c86b (127.0.100.5) changed from NON_VOTER to VOTER. New cstate: current_term: 2 leader_uuid: "a3eecbaec43d4954b1096bc3121e824c" committed_config { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "a3eecbaec43d4954b1096bc3121e824c" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 40649 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a2f9bf2e5d9b4fc19f804797bca4a3ff" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 46261 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "83ffd7184ec94302bd1ee74b0fab3e17" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 32909 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "692da0319a2d4a81ad81f5d8f9f4c86b" member_type: VOTER last_known_addr { host: "127.0.100.5" port: 38561 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 23:26:33.577133 3908 consensus_queue.cc:237] T 6039230222ff4e57929bed49cd56caf5 P a3eecbaec43d4954b1096bc3121e824c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 28, Committed index: 28, Last appended: 2.28, Last appended by leader: 0, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "a3eecbaec43d4954b1096bc3121e824c" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 40649 } } peers { permanent_uuid: "a2f9bf2e5d9b4fc19f804797bca4a3ff" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 46261 } } peers { permanent_uuid: "692da0319a2d4a81ad81f5d8f9f4c86b" member_type: VOTER last_known_addr { host: "127.0.100.5" port: 38561 } attrs { promote: false } }
I20240503 23:26:33.579546 4293 raft_consensus.cc:1261] T 6039230222ff4e57929bed49cd56caf5 P 692da0319a2d4a81ad81f5d8f9f4c86b [term 2 FOLLOWER]: Refusing update from remote peer a3eecbaec43d4954b1096bc3121e824c: Log matching property violated. Preceding OpId in replica: term: 2 index: 28. Preceding OpId from leader: term: 2 index: 29. (index mismatch)
I20240503 23:26:33.580186 4483 consensus_queue.cc:1035] T 6039230222ff4e57929bed49cd56caf5 P a3eecbaec43d4954b1096bc3121e824c [LEADER]: Connected to new peer: Peer: permanent_uuid: "692da0319a2d4a81ad81f5d8f9f4c86b" member_type: VOTER last_known_addr { host: "127.0.100.5" port: 38561 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.000s
I20240503 23:26:33.581640 3833 raft_consensus.cc:1261] T 6039230222ff4e57929bed49cd56caf5 P a2f9bf2e5d9b4fc19f804797bca4a3ff [term 2 FOLLOWER]: Refusing update from remote peer a3eecbaec43d4954b1096bc3121e824c: Log matching property violated. Preceding OpId in replica: term: 2 index: 28. Preceding OpId from leader: term: 2 index: 29. (index mismatch)
I20240503 23:26:33.582485 4483 consensus_queue.cc:1035] T 6039230222ff4e57929bed49cd56caf5 P a3eecbaec43d4954b1096bc3121e824c [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2f9bf2e5d9b4fc19f804797bca4a3ff" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 46261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.000s
I20240503 23:26:33.583215 4426 raft_consensus.cc:2932] T 6039230222ff4e57929bed49cd56caf5 P a3eecbaec43d4954b1096bc3121e824c [term 2 LEADER]: Committing config change with OpId 2.29: config changed from index 28 to 29, VOTER 83ffd7184ec94302bd1ee74b0fab3e17 (127.0.100.1) evicted. New config: { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "a3eecbaec43d4954b1096bc3121e824c" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 40649 } } peers { permanent_uuid: "a2f9bf2e5d9b4fc19f804797bca4a3ff" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 46261 } } peers { permanent_uuid: "692da0319a2d4a81ad81f5d8f9f4c86b" member_type: VOTER last_known_addr { host: "127.0.100.5" port: 38561 } attrs { promote: false } } }
I20240503 23:26:33.586403 3838 raft_consensus.cc:2932] T 6039230222ff4e57929bed49cd56caf5 P a2f9bf2e5d9b4fc19f804797bca4a3ff [term 2 FOLLOWER]: Committing config change with OpId 2.29: config changed from index 28 to 29, VOTER 83ffd7184ec94302bd1ee74b0fab3e17 (127.0.100.1) evicted. New config: { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "a3eecbaec43d4954b1096bc3121e824c" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 40649 } } peers { permanent_uuid: "a2f9bf2e5d9b4fc19f804797bca4a3ff" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 46261 } } peers { permanent_uuid: "692da0319a2d4a81ad81f5d8f9f4c86b" member_type: VOTER last_known_addr { host: "127.0.100.5" port: 38561 } attrs { promote: false } } }
I20240503 23:26:33.586326 4293 raft_consensus.cc:2932] T 6039230222ff4e57929bed49cd56caf5 P 692da0319a2d4a81ad81f5d8f9f4c86b [term 2 FOLLOWER]: Committing config change with OpId 2.29: config changed from index 28 to 29, VOTER 83ffd7184ec94302bd1ee74b0fab3e17 (127.0.100.1) evicted. New config: { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "a3eecbaec43d4954b1096bc3121e824c" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 40649 } } peers { permanent_uuid: "a2f9bf2e5d9b4fc19f804797bca4a3ff" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 46261 } } peers { permanent_uuid: "692da0319a2d4a81ad81f5d8f9f4c86b" member_type: VOTER last_known_addr { host: "127.0.100.5" port: 38561 } attrs { promote: false } } }
I20240503 23:26:33.587714 3646 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 6039230222ff4e57929bed49cd56caf5 with cas_config_opid_index 28: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 23:26:33.590047 3662 catalog_manager.cc:5484] T 6039230222ff4e57929bed49cd56caf5 P a3eecbaec43d4954b1096bc3121e824c reported cstate change: config changed from index 28 to 29, VOTER 83ffd7184ec94302bd1ee74b0fab3e17 (127.0.100.1) evicted. New cstate: current_term: 2 leader_uuid: "a3eecbaec43d4954b1096bc3121e824c" committed_config { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "a3eecbaec43d4954b1096bc3121e824c" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 40649 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a2f9bf2e5d9b4fc19f804797bca4a3ff" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 46261 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "692da0319a2d4a81ad81f5d8f9f4c86b" member_type: VOTER last_known_addr { host: "127.0.100.5" port: 38561 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 23:26:33.594523 3747 tablet_service.cc:1514] Processing DeleteTablet for tablet 6039230222ff4e57929bed49cd56caf5 with delete_type TABLET_DATA_TOMBSTONED (TS 83ffd7184ec94302bd1ee74b0fab3e17 not found in new config with opid_index 29) from {username='slave'} at 127.0.0.1:35576
I20240503 23:26:33.595511 4490 tablet_replica.cc:330] T 6039230222ff4e57929bed49cd56caf5 P 83ffd7184ec94302bd1ee74b0fab3e17: stopping tablet replica
I20240503 23:26:33.595767 4490 raft_consensus.cc:2229] T 6039230222ff4e57929bed49cd56caf5 P 83ffd7184ec94302bd1ee74b0fab3e17 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 23:26:33.596103 4490 pending_rounds.cc:62] T 6039230222ff4e57929bed49cd56caf5 P 83ffd7184ec94302bd1ee74b0fab3e17: Trying to abort 1 pending ops.
I20240503 23:26:33.596278 4490 pending_rounds.cc:68] T 6039230222ff4e57929bed49cd56caf5 P 83ffd7184ec94302bd1ee74b0fab3e17: Aborting op as it isn't in flight: id { term: 2 index: 28 } timestamp: 7023733938383994880 op_type: CHANGE_CONFIG_OP change_config_record { tablet_id: "6039230222ff4e57929bed49cd56caf5" old_config { opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "a3eecbaec43d4954b1096bc3121e824c" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 40649 } } peers { permanent_uuid: "a2f9bf2e5d9b4fc19f804797bca4a3ff" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 46261 } } peers { permanent_uuid: "83ffd7184ec94302bd1ee74b0fab3e17" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 32909 } attrs { replace: true } } peers { permanent_uuid: "692da0319a2d4a81ad81f5d8f9f4c86b" member_type: NON_VOTER last_known_addr { host: "127.0.100.5" port: 38561 } attrs { promote: true } } } new_config { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "a3eecbaec43d4954b1096bc3121e824c" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 40649 } } peers { permanent_uuid: "a2f9bf2e5d9b4fc19f804797bca4a3ff" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 46261 } } peers { permanent_uuid: "83ffd7184ec94302bd1ee74b0fab3e17" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 32909 } attrs { replace: true } } peers { permanent_uuid: "692da0319a2d4a81ad81f5d8f9f4c86b" member_type: VOTER last_known_addr { host: "127.0.100.5" port: 38561 } attrs { promote: false } } } }
I20240503 23:26:33.596776 4490 raft_consensus.cc:2894] T 6039230222ff4e57929bed49cd56caf5 P 83ffd7184ec94302bd1ee74b0fab3e17 [term 2 FOLLOWER]: Aborting config change with OpId 2.28: Aborted: Op aborted
I20240503 23:26:33.596954 4490 raft_consensus.cc:2866] T 6039230222ff4e57929bed49cd56caf5 P 83ffd7184ec94302bd1ee74b0fab3e17 [term 2 FOLLOWER]: CHANGE_CONFIG_OP replication failed: Aborted: Op aborted
I20240503 23:26:33.597059 4490 raft_consensus.cc:2258] T 6039230222ff4e57929bed49cd56caf5 P 83ffd7184ec94302bd1ee74b0fab3e17 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 23:26:33.598193 4490 ts_tablet_manager.cc:1875] T 6039230222ff4e57929bed49cd56caf5 P 83ffd7184ec94302bd1ee74b0fab3e17: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 23:26:33.604817 4490 ts_tablet_manager.cc:1888] T 6039230222ff4e57929bed49cd56caf5 P 83ffd7184ec94302bd1ee74b0fab3e17: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.28
I20240503 23:26:33.604979 4490 log.cc:1196] T 6039230222ff4e57929bed49cd56caf5 P 83ffd7184ec94302bd1ee74b0fab3e17: Deleting WAL directory at /tmp/dist-test-taskZhIiPz/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714778688451250-400-0/minicluster-data/ts-0-root/wals/6039230222ff4e57929bed49cd56caf5
I20240503 23:26:33.605799 3648 catalog_manager.cc:4830] TS 83ffd7184ec94302bd1ee74b0fab3e17 (127.0.100.1:32909): tablet 6039230222ff4e57929bed49cd56caf5 (table test-workload [id=1345b2a336c84ec2b1de62853e7fa3f6]) successfully deleted
I20240503 23:26:33.650331 4467 raft_consensus.cc:1050] T 1b9566be6c644cfca852ec6507fab893 P a2f9bf2e5d9b4fc19f804797bca4a3ff: attempting to promote NON_VOTER 692da0319a2d4a81ad81f5d8f9f4c86b to VOTER
I20240503 23:26:33.652112 4467 consensus_queue.cc:237] T 1b9566be6c644cfca852ec6507fab893 P a2f9bf2e5d9b4fc19f804797bca4a3ff [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 27, Committed index: 27, Last appended: 1.27, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "83ffd7184ec94302bd1ee74b0fab3e17" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 32909 } attrs { replace: true } } peers { permanent_uuid: "a2f9bf2e5d9b4fc19f804797bca4a3ff" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 46261 } } peers { permanent_uuid: "a3eecbaec43d4954b1096bc3121e824c" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 40649 } } peers { permanent_uuid: "692da0319a2d4a81ad81f5d8f9f4c86b" member_type: VOTER last_known_addr { host: "127.0.100.5" port: 38561 } attrs { promote: false } }
I20240503 23:26:33.654925 4293 raft_consensus.cc:1261] T 1b9566be6c644cfca852ec6507fab893 P 692da0319a2d4a81ad81f5d8f9f4c86b [term 1 LEARNER]: Refusing update from remote peer a2f9bf2e5d9b4fc19f804797bca4a3ff: Log matching property violated. Preceding OpId in replica: term: 1 index: 27. Preceding OpId from leader: term: 1 index: 28. (index mismatch)
I20240503 23:26:33.654925 3907 raft_consensus.cc:1261] T 1b9566be6c644cfca852ec6507fab893 P a3eecbaec43d4954b1096bc3121e824c [term 1 FOLLOWER]: Refusing update from remote peer a2f9bf2e5d9b4fc19f804797bca4a3ff: Log matching property violated. Preceding OpId in replica: term: 1 index: 27. Preceding OpId from leader: term: 1 index: 28. (index mismatch)
I20240503 23:26:33.655776 3760 raft_consensus.cc:1261] T 1b9566be6c644cfca852ec6507fab893 P 83ffd7184ec94302bd1ee74b0fab3e17 [term 1 FOLLOWER]: Refusing update from remote peer a2f9bf2e5d9b4fc19f804797bca4a3ff: Log matching property violated. Preceding OpId in replica: term: 1 index: 27. Preceding OpId from leader: term: 1 index: 28. (index mismatch)
I20240503 23:26:33.655896 4492 consensus_queue.cc:1035] T 1b9566be6c644cfca852ec6507fab893 P a2f9bf2e5d9b4fc19f804797bca4a3ff [LEADER]: Connected to new peer: Peer: permanent_uuid: "a3eecbaec43d4954b1096bc3121e824c" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 40649 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 28, Last known committed idx: 27, Time since last communication: 0.000s
I20240503 23:26:33.656270 4467 consensus_queue.cc:1035] T 1b9566be6c644cfca852ec6507fab893 P a2f9bf2e5d9b4fc19f804797bca4a3ff [LEADER]: Connected to new peer: Peer: permanent_uuid: "692da0319a2d4a81ad81f5d8f9f4c86b" member_type: VOTER last_known_addr { host: "127.0.100.5" port: 38561 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 28, Last known committed idx: 27, Time since last communication: 0.000s
I20240503 23:26:33.656823 4466 consensus_queue.cc:1035] T 1b9566be6c644cfca852ec6507fab893 P a2f9bf2e5d9b4fc19f804797bca4a3ff [LEADER]: Connected to new peer: Peer: permanent_uuid: "83ffd7184ec94302bd1ee74b0fab3e17" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 32909 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 28, Last known committed idx: 27, Time since last communication: 0.000s
I20240503 23:26:33.666544 4467 raft_consensus.cc:2932] T 1b9566be6c644cfca852ec6507fab893 P a2f9bf2e5d9b4fc19f804797bca4a3ff [term 1 LEADER]: Committing config change with OpId 1.28: config changed from index 27 to 28, 692da0319a2d4a81ad81f5d8f9f4c86b (127.0.100.5) changed from NON_VOTER to VOTER. New config: { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "83ffd7184ec94302bd1ee74b0fab3e17" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 32909 } attrs { replace: true } } peers { permanent_uuid: "a2f9bf2e5d9b4fc19f804797bca4a3ff" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 46261 } } peers { permanent_uuid: "a3eecbaec43d4954b1096bc3121e824c" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 40649 } } peers { permanent_uuid: "692da0319a2d4a81ad81f5d8f9f4c86b" member_type: VOTER last_known_addr { host: "127.0.100.5" port: 38561 } attrs { promote: false } } }
I20240503 23:26:33.667178 3761 raft_consensus.cc:2932] T 1b9566be6c644cfca852ec6507fab893 P 83ffd7184ec94302bd1ee74b0fab3e17 [term 1 FOLLOWER]: Committing config change with OpId 1.28: config changed from index 27 to 28, 692da0319a2d4a81ad81f5d8f9f4c86b (127.0.100.5) changed from NON_VOTER to VOTER. New config: { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "83ffd7184ec94302bd1ee74b0fab3e17" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 32909 } attrs { replace: true } } peers { permanent_uuid: "a2f9bf2e5d9b4fc19f804797bca4a3ff" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 46261 } } peers { permanent_uuid: "a3eecbaec43d4954b1096bc3121e824c" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 40649 } } peers { permanent_uuid: "692da0319a2d4a81ad81f5d8f9f4c86b" member_type: VOTER last_known_addr { host: "127.0.100.5" port: 38561 } attrs { promote: false } } }
I20240503 23:26:33.669026 4293 raft_consensus.cc:2932] T 1b9566be6c644cfca852ec6507fab893 P 692da0319a2d4a81ad81f5d8f9f4c86b [term 1 FOLLOWER]: Committing config change with OpId 1.28: config changed from index 27 to 28, 692da0319a2d4a81ad81f5d8f9f4c86b (127.0.100.5) changed from NON_VOTER to VOTER. New config: { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "83ffd7184ec94302bd1ee74b0fab3e17" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 32909 } attrs { replace: true } } peers { permanent_uuid: "a2f9bf2e5d9b4fc19f804797bca4a3ff" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 46261 } } peers { permanent_uuid: "a3eecbaec43d4954b1096bc3121e824c" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 40649 } } peers { permanent_uuid: "692da0319a2d4a81ad81f5d8f9f4c86b" member_type: VOTER last_known_addr { host: "127.0.100.5" port: 38561 } attrs { promote: false } } }
I20240503 23:26:33.673629 3907 raft_consensus.cc:2932] T 1b9566be6c644cfca852ec6507fab893 P a3eecbaec43d4954b1096bc3121e824c [term 1 FOLLOWER]: Committing config change with OpId 1.28: config changed from index 27 to 28, 692da0319a2d4a81ad81f5d8f9f4c86b (127.0.100.5) changed from NON_VOTER to VOTER. New config: { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "83ffd7184ec94302bd1ee74b0fab3e17" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 32909 } attrs { replace: true } } peers { permanent_uuid: "a2f9bf2e5d9b4fc19f804797bca4a3ff" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 46261 } } peers { permanent_uuid: "a3eecbaec43d4954b1096bc3121e824c" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 40649 } } peers { permanent_uuid: "692da0319a2d4a81ad81f5d8f9f4c86b" member_type: VOTER last_known_addr { host: "127.0.100.5" port: 38561 } attrs { promote: false } } }
I20240503 23:26:33.673907 3659 catalog_manager.cc:5484] T 1b9566be6c644cfca852ec6507fab893 P 83ffd7184ec94302bd1ee74b0fab3e17 reported cstate change: config changed from index 27 to 28, 692da0319a2d4a81ad81f5d8f9f4c86b (127.0.100.5) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "a2f9bf2e5d9b4fc19f804797bca4a3ff" committed_config { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "83ffd7184ec94302bd1ee74b0fab3e17" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 32909 } attrs { replace: true } } peers { permanent_uuid: "a2f9bf2e5d9b4fc19f804797bca4a3ff" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 46261 } } peers { permanent_uuid: "a3eecbaec43d4954b1096bc3121e824c" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 40649 } } peers { permanent_uuid: "692da0319a2d4a81ad81f5d8f9f4c86b" member_type: VOTER last_known_addr { host: "127.0.100.5" port: 38561 } attrs { promote: false } } }
I20240503 23:26:33.681504 3838 consensus_queue.cc:237] T 1b9566be6c644cfca852ec6507fab893 P a2f9bf2e5d9b4fc19f804797bca4a3ff [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 28, Committed index: 28, Last appended: 1.28, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "a2f9bf2e5d9b4fc19f804797bca4a3ff" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 46261 } } peers { permanent_uuid: "a3eecbaec43d4954b1096bc3121e824c" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 40649 } } peers { permanent_uuid: "692da0319a2d4a81ad81f5d8f9f4c86b" member_type: VOTER last_known_addr { host: "127.0.100.5" port: 38561 } attrs { promote: false } }
I20240503 23:26:33.684193 4293 raft_consensus.cc:1261] T 1b9566be6c644cfca852ec6507fab893 P 692da0319a2d4a81ad81f5d8f9f4c86b [term 1 FOLLOWER]: Refusing update from remote peer a2f9bf2e5d9b4fc19f804797bca4a3ff: Log matching property violated. Preceding OpId in replica: term: 1 index: 28. Preceding OpId from leader: term: 1 index: 29. (index mismatch)
I20240503 23:26:33.684144 3907 raft_consensus.cc:1261] T 1b9566be6c644cfca852ec6507fab893 P a3eecbaec43d4954b1096bc3121e824c [term 1 FOLLOWER]: Refusing update from remote peer a2f9bf2e5d9b4fc19f804797bca4a3ff: Log matching property violated. Preceding OpId in replica: term: 1 index: 28. Preceding OpId from leader: term: 1 index: 29. (index mismatch)
I20240503 23:26:33.685217 4467 consensus_queue.cc:1035] T 1b9566be6c644cfca852ec6507fab893 P a2f9bf2e5d9b4fc19f804797bca4a3ff [LEADER]: Connected to new peer: Peer: permanent_uuid: "a3eecbaec43d4954b1096bc3121e824c" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 40649 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.000s
I20240503 23:26:33.685946 3966 consensus_queue.cc:1035] T 1b9566be6c644cfca852ec6507fab893 P a2f9bf2e5d9b4fc19f804797bca4a3ff [LEADER]: Connected to new peer: Peer: permanent_uuid: "692da0319a2d4a81ad81f5d8f9f4c86b" member_type: VOTER last_known_addr { host: "127.0.100.5" port: 38561 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.000s
I20240503 23:26:33.688421 4492 raft_consensus.cc:2932] T 1b9566be6c644cfca852ec6507fab893 P a2f9bf2e5d9b4fc19f804797bca4a3ff [term 1 LEADER]: Committing config change with OpId 1.29: config changed from index 28 to 29, VOTER 83ffd7184ec94302bd1ee74b0fab3e17 (127.0.100.1) evicted. New config: { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "a2f9bf2e5d9b4fc19f804797bca4a3ff" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 46261 } } peers { permanent_uuid: "a3eecbaec43d4954b1096bc3121e824c" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 40649 } } peers { permanent_uuid: "692da0319a2d4a81ad81f5d8f9f4c86b" member_type: VOTER last_known_addr { host: "127.0.100.5" port: 38561 } attrs { promote: false } } }
I20240503 23:26:33.689066 4293 raft_consensus.cc:2932] T 1b9566be6c644cfca852ec6507fab893 P 692da0319a2d4a81ad81f5d8f9f4c86b [term 1 FOLLOWER]: Committing config change with OpId 1.29: config changed from index 28 to 29, VOTER 83ffd7184ec94302bd1ee74b0fab3e17 (127.0.100.1) evicted. New config: { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "a2f9bf2e5d9b4fc19f804797bca4a3ff" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 46261 } } peers { permanent_uuid: "a3eecbaec43d4954b1096bc3121e824c" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 40649 } } peers { permanent_uuid: "692da0319a2d4a81ad81f5d8f9f4c86b" member_type: VOTER last_known_addr { host: "127.0.100.5" port: 38561 } attrs { promote: false } } }
I20240503 23:26:33.692783 3649 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 1b9566be6c644cfca852ec6507fab893 with cas_config_opid_index 28: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 23:26:33.692873 3907 raft_consensus.cc:2932] T 1b9566be6c644cfca852ec6507fab893 P a3eecbaec43d4954b1096bc3121e824c [term 1 FOLLOWER]: Committing config change with OpId 1.29: config changed from index 28 to 29, VOTER 83ffd7184ec94302bd1ee74b0fab3e17 (127.0.100.1) evicted. New config: { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "a2f9bf2e5d9b4fc19f804797bca4a3ff" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 46261 } } peers { permanent_uuid: "a3eecbaec43d4954b1096bc3121e824c" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 40649 } } peers { permanent_uuid: "692da0319a2d4a81ad81f5d8f9f4c86b" member_type: VOTER last_known_addr { host: "127.0.100.5" port: 38561 } attrs { promote: false } } }
I20240503 23:26:33.694231 3655 catalog_manager.cc:5484] T 1b9566be6c644cfca852ec6507fab893 P a2f9bf2e5d9b4fc19f804797bca4a3ff reported cstate change: config changed from index 28 to 29, VOTER 83ffd7184ec94302bd1ee74b0fab3e17 (127.0.100.1) evicted. New cstate: current_term: 1 leader_uuid: "a2f9bf2e5d9b4fc19f804797bca4a3ff" committed_config { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "a2f9bf2e5d9b4fc19f804797bca4a3ff" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 46261 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a3eecbaec43d4954b1096bc3121e824c" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 40649 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "692da0319a2d4a81ad81f5d8f9f4c86b" member_type: VOTER last_known_addr { host: "127.0.100.5" port: 38561 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 23:26:33.699187 3747 tablet_service.cc:1514] Processing DeleteTablet for tablet 1b9566be6c644cfca852ec6507fab893 with delete_type TABLET_DATA_TOMBSTONED (TS 83ffd7184ec94302bd1ee74b0fab3e17 not found in new config with opid_index 29) from {username='slave'} at 127.0.0.1:35576
I20240503 23:26:33.699577 4490 tablet_replica.cc:330] T 1b9566be6c644cfca852ec6507fab893 P 83ffd7184ec94302bd1ee74b0fab3e17: stopping tablet replica
I20240503 23:26:33.699812 4490 raft_consensus.cc:2229] T 1b9566be6c644cfca852ec6507fab893 P 83ffd7184ec94302bd1ee74b0fab3e17 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 23:26:33.700241 4490 raft_consensus.cc:2258] T 1b9566be6c644cfca852ec6507fab893 P 83ffd7184ec94302bd1ee74b0fab3e17 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 23:26:33.701232 4490 ts_tablet_manager.cc:1875] T 1b9566be6c644cfca852ec6507fab893 P 83ffd7184ec94302bd1ee74b0fab3e17: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 23:26:33.707422 4490 ts_tablet_manager.cc:1888] T 1b9566be6c644cfca852ec6507fab893 P 83ffd7184ec94302bd1ee74b0fab3e17: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.28
I20240503 23:26:33.707621 4490 log.cc:1196] T 1b9566be6c644cfca852ec6507fab893 P 83ffd7184ec94302bd1ee74b0fab3e17: Deleting WAL directory at /tmp/dist-test-taskZhIiPz/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714778688451250-400-0/minicluster-data/ts-0-root/wals/1b9566be6c644cfca852ec6507fab893
I20240503 23:26:33.708310 3648 catalog_manager.cc:4830] TS 83ffd7184ec94302bd1ee74b0fab3e17 (127.0.100.1:32909): tablet 1b9566be6c644cfca852ec6507fab893 (table test-workload [id=1345b2a336c84ec2b1de62853e7fa3f6]) successfully deleted
I20240503 23:26:33.868937 4478 raft_consensus.cc:1050] T a14a12115b8d4fec947c6e223f1ef486 P 83ffd7184ec94302bd1ee74b0fab3e17: attempting to promote NON_VOTER 4d1d4e261584467fbbce4c72fa18045a to VOTER
I20240503 23:26:33.869870 4478 consensus_queue.cc:237] T a14a12115b8d4fec947c6e223f1ef486 P 83ffd7184ec94302bd1ee74b0fab3e17 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 27, Committed index: 27, Last appended: 1.27, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "83ffd7184ec94302bd1ee74b0fab3e17" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 32909 } } peers { permanent_uuid: "a2f9bf2e5d9b4fc19f804797bca4a3ff" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 46261 } attrs { replace: true } } peers { permanent_uuid: "a3eecbaec43d4954b1096bc3121e824c" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 40649 } } peers { permanent_uuid: "4d1d4e261584467fbbce4c72fa18045a" member_type: VOTER last_known_addr { host: "127.0.100.6" port: 35349 } attrs { promote: false } }
I20240503 23:26:33.872567 4367 raft_consensus.cc:1261] T a14a12115b8d4fec947c6e223f1ef486 P 4d1d4e261584467fbbce4c72fa18045a [term 1 LEARNER]: Refusing update from remote peer 83ffd7184ec94302bd1ee74b0fab3e17: Log matching property violated. Preceding OpId in replica: term: 1 index: 27. Preceding OpId from leader: term: 1 index: 28. (index mismatch)
I20240503 23:26:33.872567 3907 raft_consensus.cc:1261] T a14a12115b8d4fec947c6e223f1ef486 P a3eecbaec43d4954b1096bc3121e824c [term 1 FOLLOWER]: Refusing update from remote peer 83ffd7184ec94302bd1ee74b0fab3e17: Log matching property violated. Preceding OpId in replica: term: 1 index: 27. Preceding OpId from leader: term: 1 index: 28. (index mismatch)
I20240503 23:26:33.873106 3837 raft_consensus.cc:1261] T a14a12115b8d4fec947c6e223f1ef486 P a2f9bf2e5d9b4fc19f804797bca4a3ff [term 1 FOLLOWER]: Refusing update from remote peer 83ffd7184ec94302bd1ee74b0fab3e17: Log matching property violated. Preceding OpId in replica: term: 1 index: 27. Preceding OpId from leader: term: 1 index: 28. (index mismatch)
I20240503 23:26:33.873284 4454 consensus_queue.cc:1035] T a14a12115b8d4fec947c6e223f1ef486 P 83ffd7184ec94302bd1ee74b0fab3e17 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a3eecbaec43d4954b1096bc3121e824c" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 40649 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 28, Last known committed idx: 27, Time since last communication: 0.000s
I20240503 23:26:33.873553 4497 consensus_queue.cc:1035] T a14a12115b8d4fec947c6e223f1ef486 P 83ffd7184ec94302bd1ee74b0fab3e17 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d1d4e261584467fbbce4c72fa18045a" member_type: VOTER last_known_addr { host: "127.0.100.6" port: 35349 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 28, Last known committed idx: 27, Time since last communication: 0.000s
I20240503 23:26:33.874868 4497 consensus_queue.cc:1035] T a14a12115b8d4fec947c6e223f1ef486 P 83ffd7184ec94302bd1ee74b0fab3e17 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2f9bf2e5d9b4fc19f804797bca4a3ff" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 46261 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 28, Last known committed idx: 27, Time since last communication: 0.000s
I20240503 23:26:33.887132 4454 raft_consensus.cc:2932] T a14a12115b8d4fec947c6e223f1ef486 P 83ffd7184ec94302bd1ee74b0fab3e17 [term 1 LEADER]: Committing config change with OpId 1.28: config changed from index 27 to 28, 4d1d4e261584467fbbce4c72fa18045a (127.0.100.6) changed from NON_VOTER to VOTER. New config: { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "83ffd7184ec94302bd1ee74b0fab3e17" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 32909 } } peers { permanent_uuid: "a2f9bf2e5d9b4fc19f804797bca4a3ff" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 46261 } attrs { replace: true } } peers { permanent_uuid: "a3eecbaec43d4954b1096bc3121e824c" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 40649 } } peers { permanent_uuid: "4d1d4e261584467fbbce4c72fa18045a" member_type: VOTER last_known_addr { host: "127.0.100.6" port: 35349 } attrs { promote: false } } }
I20240503 23:26:33.887863 3907 raft_consensus.cc:2932] T a14a12115b8d4fec947c6e223f1ef486 P a3eecbaec43d4954b1096bc3121e824c [term 1 FOLLOWER]: Committing config change with OpId 1.28: config changed from index 27 to 28, 4d1d4e261584467fbbce4c72fa18045a (127.0.100.6) changed from NON_VOTER to VOTER. New config: { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "83ffd7184ec94302bd1ee74b0fab3e17" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 32909 } } peers { permanent_uuid: "a2f9bf2e5d9b4fc19f804797bca4a3ff" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 46261 } attrs { replace: true } } peers { permanent_uuid: "a3eecbaec43d4954b1096bc3121e824c" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 40649 } } peers { permanent_uuid: "4d1d4e261584467fbbce4c72fa18045a" member_type: VOTER last_known_addr { host: "127.0.100.6" port: 35349 } attrs { promote: false } } }
I20240503 23:26:33.893389 4367 raft_consensus.cc:2932] T a14a12115b8d4fec947c6e223f1ef486 P 4d1d4e261584467fbbce4c72fa18045a [term 1 FOLLOWER]: Committing config change with OpId 1.28: config changed from index 27 to 28, 4d1d4e261584467fbbce4c72fa18045a (127.0.100.6) changed from NON_VOTER to VOTER. New config: { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "83ffd7184ec94302bd1ee74b0fab3e17" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 32909 } } peers { permanent_uuid: "a2f9bf2e5d9b4fc19f804797bca4a3ff" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 46261 } attrs { replace: true } } peers { permanent_uuid: "a3eecbaec43d4954b1096bc3121e824c" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 40649 } } peers { permanent_uuid: "4d1d4e261584467fbbce4c72fa18045a" member_type: VOTER last_known_addr { host: "127.0.100.6" port: 35349 } attrs { promote: false } } }
I20240503 23:26:33.893386 3837 raft_consensus.cc:2932] T a14a12115b8d4fec947c6e223f1ef486 P a2f9bf2e5d9b4fc19f804797bca4a3ff [term 1 FOLLOWER]: Committing config change with OpId 1.28: config changed from index 27 to 28, 4d1d4e261584467fbbce4c72fa18045a (127.0.100.6) changed from NON_VOTER to VOTER. New config: { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "83ffd7184ec94302bd1ee74b0fab3e17" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 32909 } } peers { permanent_uuid: "a2f9bf2e5d9b4fc19f804797bca4a3ff" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 46261 } attrs { replace: true } } peers { permanent_uuid: "a3eecbaec43d4954b1096bc3121e824c" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 40649 } } peers { permanent_uuid: "4d1d4e261584467fbbce4c72fa18045a" member_type: VOTER last_known_addr { host: "127.0.100.6" port: 35349 } attrs { promote: false } } }
I20240503 23:26:33.893900 3659 catalog_manager.cc:5484] T a14a12115b8d4fec947c6e223f1ef486 P 83ffd7184ec94302bd1ee74b0fab3e17 reported cstate change: config changed from index 27 to 28, 4d1d4e261584467fbbce4c72fa18045a (127.0.100.6) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "83ffd7184ec94302bd1ee74b0fab3e17" committed_config { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "83ffd7184ec94302bd1ee74b0fab3e17" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 32909 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a2f9bf2e5d9b4fc19f804797bca4a3ff" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 46261 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a3eecbaec43d4954b1096bc3121e824c" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 40649 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4d1d4e261584467fbbce4c72fa18045a" member_type: VOTER last_known_addr { host: "127.0.100.6" port: 35349 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240503 23:26:33.903802 3762 consensus_queue.cc:237] T a14a12115b8d4fec947c6e223f1ef486 P 83ffd7184ec94302bd1ee74b0fab3e17 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 28, Committed index: 28, Last appended: 1.28, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "83ffd7184ec94302bd1ee74b0fab3e17" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 32909 } } peers { permanent_uuid: "a3eecbaec43d4954b1096bc3121e824c" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 40649 } } peers { permanent_uuid: "4d1d4e261584467fbbce4c72fa18045a" member_type: VOTER last_known_addr { host: "127.0.100.6" port: 35349 } attrs { promote: false } }
I20240503 23:26:33.906154 4367 raft_consensus.cc:1261] T a14a12115b8d4fec947c6e223f1ef486 P 4d1d4e261584467fbbce4c72fa18045a [term 1 FOLLOWER]: Refusing update from remote peer 83ffd7184ec94302bd1ee74b0fab3e17: Log matching property violated. Preceding OpId in replica: term: 1 index: 28. Preceding OpId from leader: term: 1 index: 29. (index mismatch)
I20240503 23:26:33.906158 3907 raft_consensus.cc:1261] T a14a12115b8d4fec947c6e223f1ef486 P a3eecbaec43d4954b1096bc3121e824c [term 1 FOLLOWER]: Refusing update from remote peer 83ffd7184ec94302bd1ee74b0fab3e17: Log matching property violated. Preceding OpId in replica: term: 1 index: 28. Preceding OpId from leader: term: 1 index: 29. (index mismatch)
I20240503 23:26:33.906737 4497 consensus_queue.cc:1035] T a14a12115b8d4fec947c6e223f1ef486 P 83ffd7184ec94302bd1ee74b0fab3e17 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a3eecbaec43d4954b1096bc3121e824c" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 40649 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.000s
I20240503 23:26:33.907068 4478 consensus_queue.cc:1035] T a14a12115b8d4fec947c6e223f1ef486 P 83ffd7184ec94302bd1ee74b0fab3e17 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d1d4e261584467fbbce4c72fa18045a" member_type: VOTER last_known_addr { host: "127.0.100.6" port: 35349 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.000s
I20240503 23:26:33.909478 4454 raft_consensus.cc:2932] T a14a12115b8d4fec947c6e223f1ef486 P 83ffd7184ec94302bd1ee74b0fab3e17 [term 1 LEADER]: Committing config change with OpId 1.29: config changed from index 28 to 29, VOTER a2f9bf2e5d9b4fc19f804797bca4a3ff (127.0.100.2) evicted. New config: { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "83ffd7184ec94302bd1ee74b0fab3e17" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 32909 } } peers { permanent_uuid: "a3eecbaec43d4954b1096bc3121e824c" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 40649 } } peers { permanent_uuid: "4d1d4e261584467fbbce4c72fa18045a" member_type: VOTER last_known_addr { host: "127.0.100.6" port: 35349 } attrs { promote: false } } }
I20240503 23:26:33.910401 3907 raft_consensus.cc:2932] T a14a12115b8d4fec947c6e223f1ef486 P a3eecbaec43d4954b1096bc3121e824c [term 1 FOLLOWER]: Committing config change with OpId 1.29: config changed from index 28 to 29, VOTER a2f9bf2e5d9b4fc19f804797bca4a3ff (127.0.100.2) evicted. New config: { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "83ffd7184ec94302bd1ee74b0fab3e17" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 32909 } } peers { permanent_uuid: "a3eecbaec43d4954b1096bc3121e824c" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 40649 } } peers { permanent_uuid: "4d1d4e261584467fbbce4c72fa18045a" member_type: VOTER last_known_addr { host: "127.0.100.6" port: 35349 } attrs { promote: false } } }
I20240503 23:26:33.912894 4367 raft_consensus.cc:2932] T a14a12115b8d4fec947c6e223f1ef486 P 4d1d4e261584467fbbce4c72fa18045a [term 1 FOLLOWER]: Committing config change with OpId 1.29: config changed from index 28 to 29, VOTER a2f9bf2e5d9b4fc19f804797bca4a3ff (127.0.100.2) evicted. New config: { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "83ffd7184ec94302bd1ee74b0fab3e17" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 32909 } } peers { permanent_uuid: "a3eecbaec43d4954b1096bc3121e824c" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 40649 } } peers { permanent_uuid: "4d1d4e261584467fbbce4c72fa18045a" member_type: VOTER last_known_addr { host: "127.0.100.6" port: 35349 } attrs { promote: false } } }
I20240503 23:26:33.915339 3659 catalog_manager.cc:5484] T a14a12115b8d4fec947c6e223f1ef486 P a3eecbaec43d4954b1096bc3121e824c reported cstate change: config changed from index 28 to 29, VOTER a2f9bf2e5d9b4fc19f804797bca4a3ff (127.0.100.2) evicted. New cstate: current_term: 1 leader_uuid: "83ffd7184ec94302bd1ee74b0fab3e17" committed_config { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "83ffd7184ec94302bd1ee74b0fab3e17" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 32909 } } peers { permanent_uuid: "a3eecbaec43d4954b1096bc3121e824c" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 40649 } } peers { permanent_uuid: "4d1d4e261584467fbbce4c72fa18045a" member_type: VOTER last_known_addr { host: "127.0.100.6" port: 35349 } attrs { promote: false } } }
I20240503 23:26:33.917217 3648 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet a14a12115b8d4fec947c6e223f1ef486 with cas_config_opid_index 28: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 23:26:33.917994 4454 raft_consensus.cc:1050] T 7181e4e8c0324a8eb095dde23651a1ab P 83ffd7184ec94302bd1ee74b0fab3e17: attempting to promote NON_VOTER e4086c099d7f4447b2b4733b6ca364df to VOTER
I20240503 23:26:33.920013 4454 consensus_queue.cc:237] T 7181e4e8c0324a8eb095dde23651a1ab P 83ffd7184ec94302bd1ee74b0fab3e17 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 26, Committed index: 26, Last appended: 1.26, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "83ffd7184ec94302bd1ee74b0fab3e17" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 32909 } } peers { permanent_uuid: "a2f9bf2e5d9b4fc19f804797bca4a3ff" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 46261 } attrs { replace: true } } peers { permanent_uuid: "a3eecbaec43d4954b1096bc3121e824c" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 40649 } } peers { permanent_uuid: "e4086c099d7f4447b2b4733b6ca364df" member_type: VOTER last_known_addr { host: "127.0.100.4" port: 35011 } attrs { promote: false } }
I20240503 23:26:33.921826 3824 tablet_service.cc:1514] Processing DeleteTablet for tablet a14a12115b8d4fec947c6e223f1ef486 with delete_type TABLET_DATA_TOMBSTONED (TS a2f9bf2e5d9b4fc19f804797bca4a3ff not found in new config with opid_index 29) from {username='slave'} at 127.0.0.1:45354
I20240503 23:26:33.923739 4502 tablet_replica.cc:330] T a14a12115b8d4fec947c6e223f1ef486 P a2f9bf2e5d9b4fc19f804797bca4a3ff: stopping tablet replica
I20240503 23:26:33.924521 4218 raft_consensus.cc:1261] T 7181e4e8c0324a8eb095dde23651a1ab P e4086c099d7f4447b2b4733b6ca364df [term 1 LEARNER]: Refusing update from remote peer 83ffd7184ec94302bd1ee74b0fab3e17: Log matching property violated. Preceding OpId in replica: term: 1 index: 26. Preceding OpId from leader: term: 1 index: 27. (index mismatch)
I20240503 23:26:33.924990 3907 raft_consensus.cc:1261] T 7181e4e8c0324a8eb095dde23651a1ab P a3eecbaec43d4954b1096bc3121e824c [term 1 FOLLOWER]: Refusing update from remote peer 83ffd7184ec94302bd1ee74b0fab3e17: Log matching property violated. Preceding OpId in replica: term: 1 index: 26. Preceding OpId from leader: term: 1 index: 27. (index mismatch)
I20240503 23:26:33.925146 4502 raft_consensus.cc:2229] T a14a12115b8d4fec947c6e223f1ef486 P a2f9bf2e5d9b4fc19f804797bca4a3ff [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 23:26:33.925567 4454 consensus_queue.cc:1035] T 7181e4e8c0324a8eb095dde23651a1ab P 83ffd7184ec94302bd1ee74b0fab3e17 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4086c099d7f4447b2b4733b6ca364df" member_type: VOTER last_known_addr { host: "127.0.100.4" port: 35011 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 27, Last known committed idx: 26, Time since last communication: 0.000s
I20240503 23:26:33.925709 4502 raft_consensus.cc:2258] T a14a12115b8d4fec947c6e223f1ef486 P a2f9bf2e5d9b4fc19f804797bca4a3ff [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 23:26:33.925952 4478 consensus_queue.cc:1035] T 7181e4e8c0324a8eb095dde23651a1ab P 83ffd7184ec94302bd1ee74b0fab3e17 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a3eecbaec43d4954b1096bc3121e824c" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 40649 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 27, Last known committed idx: 26, Time since last communication: 0.000s
I20240503 23:26:33.926873 3837 raft_consensus.cc:1261] T 7181e4e8c0324a8eb095dde23651a1ab P a2f9bf2e5d9b4fc19f804797bca4a3ff [term 1 FOLLOWER]: Refusing update from remote peer 83ffd7184ec94302bd1ee74b0fab3e17: Log matching property violated. Preceding OpId in replica: term: 1 index: 26. Preceding OpId from leader: term: 1 index: 27. (index mismatch)
I20240503 23:26:33.926929 4502 ts_tablet_manager.cc:1875] T a14a12115b8d4fec947c6e223f1ef486 P a2f9bf2e5d9b4fc19f804797bca4a3ff: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 23:26:33.928227 4478 consensus_queue.cc:1035] T 7181e4e8c0324a8eb095dde23651a1ab P 83ffd7184ec94302bd1ee74b0fab3e17 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2f9bf2e5d9b4fc19f804797bca4a3ff" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 46261 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 27, Last known committed idx: 26, Time since last communication: 0.000s
I20240503 23:26:33.937808 4502 ts_tablet_manager.cc:1888] T a14a12115b8d4fec947c6e223f1ef486 P a2f9bf2e5d9b4fc19f804797bca4a3ff: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.28
I20240503 23:26:33.939183 4502 log.cc:1196] T a14a12115b8d4fec947c6e223f1ef486 P a2f9bf2e5d9b4fc19f804797bca4a3ff: Deleting WAL directory at /tmp/dist-test-taskZhIiPz/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714778688451250-400-0/minicluster-data/ts-1-root/wals/a14a12115b8d4fec947c6e223f1ef486
I20240503 23:26:33.940335 3649 catalog_manager.cc:4830] TS a2f9bf2e5d9b4fc19f804797bca4a3ff (127.0.100.2:46261): tablet a14a12115b8d4fec947c6e223f1ef486 (table test-workload [id=1345b2a336c84ec2b1de62853e7fa3f6]) successfully deleted
I20240503 23:26:33.941241 4478 raft_consensus.cc:2932] T 7181e4e8c0324a8eb095dde23651a1ab P 83ffd7184ec94302bd1ee74b0fab3e17 [term 1 LEADER]: Committing config change with OpId 1.27: config changed from index 26 to 27, e4086c099d7f4447b2b4733b6ca364df (127.0.100.4) changed from NON_VOTER to VOTER. New config: { opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "83ffd7184ec94302bd1ee74b0fab3e17" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 32909 } } peers { permanent_uuid: "a2f9bf2e5d9b4fc19f804797bca4a3ff" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 46261 } attrs { replace: true } } peers { permanent_uuid: "a3eecbaec43d4954b1096bc3121e824c" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 40649 } } peers { permanent_uuid: "e4086c099d7f4447b2b4733b6ca364df" member_type: VOTER last_known_addr { host: "127.0.100.4" port: 35011 } attrs { promote: false } } }
I20240503 23:26:33.942302 4218 raft_consensus.cc:2932] T 7181e4e8c0324a8eb095dde23651a1ab P e4086c099d7f4447b2b4733b6ca364df [term 1 FOLLOWER]: Committing config change with OpId 1.27: config changed from index 26 to 27, e4086c099d7f4447b2b4733b6ca364df (127.0.100.4) changed from NON_VOTER to VOTER. New config: { opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "83ffd7184ec94302bd1ee74b0fab3e17" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 32909 } } peers { permanent_uuid: "a2f9bf2e5d9b4fc19f804797bca4a3ff" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 46261 } attrs { replace: true } } peers { permanent_uuid: "a3eecbaec43d4954b1096bc3121e824c" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 40649 } } peers { permanent_uuid: "e4086c099d7f4447b2b4733b6ca364df" member_type: VOTER last_known_addr { host: "127.0.100.4" port: 35011 } attrs { promote: false } } }
I20240503 23:26:33.942302 3907 raft_consensus.cc:2932] T 7181e4e8c0324a8eb095dde23651a1ab P a3eecbaec43d4954b1096bc3121e824c [term 1 FOLLOWER]: Committing config change with OpId 1.27: config changed from index 26 to 27, e4086c099d7f4447b2b4733b6ca364df (127.0.100.4) changed from NON_VOTER to VOTER. New config: { opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "83ffd7184ec94302bd1ee74b0fab3e17" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 32909 } } peers { permanent_uuid: "a2f9bf2e5d9b4fc19f804797bca4a3ff" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 46261 } attrs { replace: true } } peers { permanent_uuid: "a3eecbaec43d4954b1096bc3121e824c" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 40649 } } peers { permanent_uuid: "e4086c099d7f4447b2b4733b6ca364df" member_type: VOTER last_known_addr { host: "127.0.100.4" port: 35011 } attrs { promote: false } } }
I20240503 23:26:33.948807 3659 catalog_manager.cc:5484] T 7181e4e8c0324a8eb095dde23651a1ab P 83ffd7184ec94302bd1ee74b0fab3e17 reported cstate change: config changed from index 26 to 27, e4086c099d7f4447b2b4733b6ca364df (127.0.100.4) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "83ffd7184ec94302bd1ee74b0fab3e17" committed_config { opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "83ffd7184ec94302bd1ee74b0fab3e17" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 32909 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a2f9bf2e5d9b4fc19f804797bca4a3ff" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 46261 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a3eecbaec43d4954b1096bc3121e824c" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 40649 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e4086c099d7f4447b2b4733b6ca364df" member_type: VOTER last_known_addr { host: "127.0.100.4" port: 35011 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 23:26:33.949213 3838 raft_consensus.cc:2932] T 7181e4e8c0324a8eb095dde23651a1ab P a2f9bf2e5d9b4fc19f804797bca4a3ff [term 1 FOLLOWER]: Committing config change with OpId 1.27: config changed from index 26 to 27, e4086c099d7f4447b2b4733b6ca364df (127.0.100.4) changed from NON_VOTER to VOTER. New config: { opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "83ffd7184ec94302bd1ee74b0fab3e17" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 32909 } } peers { permanent_uuid: "a2f9bf2e5d9b4fc19f804797bca4a3ff" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 46261 } attrs { replace: true } } peers { permanent_uuid: "a3eecbaec43d4954b1096bc3121e824c" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 40649 } } peers { permanent_uuid: "e4086c099d7f4447b2b4733b6ca364df" member_type: VOTER last_known_addr { host: "127.0.100.4" port: 35011 } attrs { promote: false } } }
I20240503 23:26:33.959805 3761 consensus_queue.cc:237] T 7181e4e8c0324a8eb095dde23651a1ab P 83ffd7184ec94302bd1ee74b0fab3e17 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 27, Committed index: 27, Last appended: 1.27, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "83ffd7184ec94302bd1ee74b0fab3e17" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 32909 } } peers { permanent_uuid: "a3eecbaec43d4954b1096bc3121e824c" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 40649 } } peers { permanent_uuid: "e4086c099d7f4447b2b4733b6ca364df" member_type: VOTER last_known_addr { host: "127.0.100.4" port: 35011 } attrs { promote: false } }
I20240503 23:26:33.964354 4217 raft_consensus.cc:1261] T 7181e4e8c0324a8eb095dde23651a1ab P e4086c099d7f4447b2b4733b6ca364df [term 1 FOLLOWER]: Refusing update from remote peer 83ffd7184ec94302bd1ee74b0fab3e17: Log matching property violated. Preceding OpId in replica: term: 1 index: 27. Preceding OpId from leader: term: 1 index: 28. (index mismatch)
I20240503 23:26:33.965382 3907 raft_consensus.cc:1261] T 7181e4e8c0324a8eb095dde23651a1ab P a3eecbaec43d4954b1096bc3121e824c [term 1 FOLLOWER]: Refusing update from remote peer 83ffd7184ec94302bd1ee74b0fab3e17: Log matching property violated. Preceding OpId in replica: term: 1 index: 27. Preceding OpId from leader: term: 1 index: 28. (index mismatch)
I20240503 23:26:33.966012 4478 consensus_queue.cc:1035] T 7181e4e8c0324a8eb095dde23651a1ab P 83ffd7184ec94302bd1ee74b0fab3e17 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4086c099d7f4447b2b4733b6ca364df" member_type: VOTER last_known_addr { host: "127.0.100.4" port: 35011 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 28, Last known committed idx: 27, Time since last communication: 0.000s
I20240503 23:26:33.967023 4477 consensus_queue.cc:1035] T 7181e4e8c0324a8eb095dde23651a1ab P 83ffd7184ec94302bd1ee74b0fab3e17 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a3eecbaec43d4954b1096bc3121e824c" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 40649 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 28, Last known committed idx: 27, Time since last communication: 0.000s
I20240503 23:26:33.970103 4497 raft_consensus.cc:2932] T 7181e4e8c0324a8eb095dde23651a1ab P 83ffd7184ec94302bd1ee74b0fab3e17 [term 1 LEADER]: Committing config change with OpId 1.28: config changed from index 27 to 28, VOTER a2f9bf2e5d9b4fc19f804797bca4a3ff (127.0.100.2) evicted. New config: { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "83ffd7184ec94302bd1ee74b0fab3e17" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 32909 } } peers { permanent_uuid: "a3eecbaec43d4954b1096bc3121e824c" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 40649 } } peers { permanent_uuid: "e4086c099d7f4447b2b4733b6ca364df" member_type: VOTER last_known_addr { host: "127.0.100.4" port: 35011 } attrs { promote: false } } }
I20240503 23:26:33.974292 3648 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 7181e4e8c0324a8eb095dde23651a1ab with cas_config_opid_index 27: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 23:26:33.974308 4218 raft_consensus.cc:2932] T 7181e4e8c0324a8eb095dde23651a1ab P e4086c099d7f4447b2b4733b6ca364df [term 1 FOLLOWER]: Committing config change with OpId 1.28: config changed from index 27 to 28, VOTER a2f9bf2e5d9b4fc19f804797bca4a3ff (127.0.100.2) evicted. New config: { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "83ffd7184ec94302bd1ee74b0fab3e17" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 32909 } } peers { permanent_uuid: "a3eecbaec43d4954b1096bc3121e824c" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 40649 } } peers { permanent_uuid: "e4086c099d7f4447b2b4733b6ca364df" member_type: VOTER last_known_addr { host: "127.0.100.4" port: 35011 } attrs { promote: false } } }
I20240503 23:26:33.975769 3655 catalog_manager.cc:5484] T 7181e4e8c0324a8eb095dde23651a1ab P 83ffd7184ec94302bd1ee74b0fab3e17 reported cstate change: config changed from index 27 to 28, VOTER a2f9bf2e5d9b4fc19f804797bca4a3ff (127.0.100.2) evicted. New cstate: current_term: 1 leader_uuid: "83ffd7184ec94302bd1ee74b0fab3e17" committed_config { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "83ffd7184ec94302bd1ee74b0fab3e17" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 32909 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a3eecbaec43d4954b1096bc3121e824c" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 40649 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e4086c099d7f4447b2b4733b6ca364df" member_type: VOTER last_known_addr { host: "127.0.100.4" port: 35011 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 23:26:33.976931 3907 raft_consensus.cc:2932] T 7181e4e8c0324a8eb095dde23651a1ab P a3eecbaec43d4954b1096bc3121e824c [term 1 FOLLOWER]: Committing config change with OpId 1.28: config changed from index 27 to 28, VOTER a2f9bf2e5d9b4fc19f804797bca4a3ff (127.0.100.2) evicted. New config: { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "83ffd7184ec94302bd1ee74b0fab3e17" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 32909 } } peers { permanent_uuid: "a3eecbaec43d4954b1096bc3121e824c" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 40649 } } peers { permanent_uuid: "e4086c099d7f4447b2b4733b6ca364df" member_type: VOTER last_known_addr { host: "127.0.100.4" port: 35011 } attrs { promote: false } } }
I20240503 23:26:33.980687 3824 tablet_service.cc:1514] Processing DeleteTablet for tablet 7181e4e8c0324a8eb095dde23651a1ab with delete_type TABLET_DATA_TOMBSTONED (TS a2f9bf2e5d9b4fc19f804797bca4a3ff not found in new config with opid_index 28) from {username='slave'} at 127.0.0.1:45354
I20240503 23:26:33.981104 4502 tablet_replica.cc:330] T 7181e4e8c0324a8eb095dde23651a1ab P a2f9bf2e5d9b4fc19f804797bca4a3ff: stopping tablet replica
I20240503 23:26:33.981356 4502 raft_consensus.cc:2229] T 7181e4e8c0324a8eb095dde23651a1ab P a2f9bf2e5d9b4fc19f804797bca4a3ff [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 23:26:33.981734 4502 raft_consensus.cc:2258] T 7181e4e8c0324a8eb095dde23651a1ab P a2f9bf2e5d9b4fc19f804797bca4a3ff [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 23:26:33.982733 4502 ts_tablet_manager.cc:1875] T 7181e4e8c0324a8eb095dde23651a1ab P a2f9bf2e5d9b4fc19f804797bca4a3ff: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 23:26:33.989172 4502 ts_tablet_manager.cc:1888] T 7181e4e8c0324a8eb095dde23651a1ab P a2f9bf2e5d9b4fc19f804797bca4a3ff: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.27
I20240503 23:26:33.989351 4502 log.cc:1196] T 7181e4e8c0324a8eb095dde23651a1ab P a2f9bf2e5d9b4fc19f804797bca4a3ff: Deleting WAL directory at /tmp/dist-test-taskZhIiPz/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714778688451250-400-0/minicluster-data/ts-1-root/wals/7181e4e8c0324a8eb095dde23651a1ab
I20240503 23:26:33.990121 3649 catalog_manager.cc:4830] TS a2f9bf2e5d9b4fc19f804797bca4a3ff (127.0.100.2:46261): tablet 7181e4e8c0324a8eb095dde23651a1ab (table test-workload [id=1345b2a336c84ec2b1de62853e7fa3f6]) successfully deleted
I20240503 23:26:34.983045 3907 consensus_queue.cc:237] T 60267cb4849d4823a0afb2380305905f P a3eecbaec43d4954b1096bc3121e824c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 25, Committed index: 25, Last appended: 1.25, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 26 OBSOLETE_local: false peers { permanent_uuid: "83ffd7184ec94302bd1ee74b0fab3e17" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 32909 } } peers { permanent_uuid: "a2f9bf2e5d9b4fc19f804797bca4a3ff" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 46261 } attrs { replace: true } } peers { permanent_uuid: "a3eecbaec43d4954b1096bc3121e824c" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 40649 } } peers { permanent_uuid: "e4086c099d7f4447b2b4733b6ca364df" member_type: NON_VOTER last_known_addr { host: "127.0.100.4" port: 35011 } attrs { promote: true } }
I20240503 23:26:34.987257 3761 raft_consensus.cc:1261] T 60267cb4849d4823a0afb2380305905f P 83ffd7184ec94302bd1ee74b0fab3e17 [term 1 FOLLOWER]: Refusing update from remote peer a3eecbaec43d4954b1096bc3121e824c: Log matching property violated. Preceding OpId in replica: term: 1 index: 25. Preceding OpId from leader: term: 1 index: 26. (index mismatch)
I20240503 23:26:34.987996 3838 raft_consensus.cc:1261] T 60267cb4849d4823a0afb2380305905f P a2f9bf2e5d9b4fc19f804797bca4a3ff [term 1 FOLLOWER]: Refusing update from remote peer a3eecbaec43d4954b1096bc3121e824c: Log matching property violated. Preceding OpId in replica: term: 1 index: 25. Preceding OpId from leader: term: 1 index: 26. (index mismatch)
I20240503 23:26:34.988514 4510 consensus_queue.cc:1035] T 60267cb4849d4823a0afb2380305905f P a3eecbaec43d4954b1096bc3121e824c [LEADER]: Connected to new peer: Peer: permanent_uuid: "83ffd7184ec94302bd1ee74b0fab3e17" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 32909 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 26, Last known committed idx: 25, Time since last communication: 0.000s
I20240503 23:26:34.988915 4426 consensus_queue.cc:1035] T 60267cb4849d4823a0afb2380305905f P a3eecbaec43d4954b1096bc3121e824c [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2f9bf2e5d9b4fc19f804797bca4a3ff" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 46261 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 26, Last known committed idx: 25, Time since last communication: 0.000s
I20240503 23:26:34.991997 4426 raft_consensus.cc:2932] T 60267cb4849d4823a0afb2380305905f P a3eecbaec43d4954b1096bc3121e824c [term 1 LEADER]: Committing config change with OpId 1.26: config changed from index -1 to 26, NON_VOTER e4086c099d7f4447b2b4733b6ca364df (127.0.100.4) added. New config: { opid_index: 26 OBSOLETE_local: false peers { permanent_uuid: "83ffd7184ec94302bd1ee74b0fab3e17" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 32909 } } peers { permanent_uuid: "a2f9bf2e5d9b4fc19f804797bca4a3ff" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 46261 } attrs { replace: true } } peers { permanent_uuid: "a3eecbaec43d4954b1096bc3121e824c" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 40649 } } peers { permanent_uuid: "e4086c099d7f4447b2b4733b6ca364df" member_type: NON_VOTER last_known_addr { host: "127.0.100.4" port: 35011 } attrs { promote: true } } }
I20240503 23:26:34.995083 3838 raft_consensus.cc:2932] T 60267cb4849d4823a0afb2380305905f P a2f9bf2e5d9b4fc19f804797bca4a3ff [term 1 FOLLOWER]: Committing config change with OpId 1.26: config changed from index -1 to 26, NON_VOTER e4086c099d7f4447b2b4733b6ca364df (127.0.100.4) added. New config: { opid_index: 26 OBSOLETE_local: false peers { permanent_uuid: "83ffd7184ec94302bd1ee74b0fab3e17" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 32909 } } peers { permanent_uuid: "a2f9bf2e5d9b4fc19f804797bca4a3ff" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 46261 } attrs { replace: true } } peers { permanent_uuid: "a3eecbaec43d4954b1096bc3121e824c" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 40649 } } peers { permanent_uuid: "e4086c099d7f4447b2b4733b6ca364df" member_type: NON_VOTER last_known_addr { host: "127.0.100.4" port: 35011 } attrs { promote: true } } }
W20240503 23:26:34.998963 3876 consensus_peers.cc:485] T 60267cb4849d4823a0afb2380305905f P a3eecbaec43d4954b1096bc3121e824c -> Peer e4086c099d7f4447b2b4733b6ca364df (127.0.100.4:35011): Couldn't send request to peer e4086c099d7f4447b2b4733b6ca364df. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 60267cb4849d4823a0afb2380305905f. This is attempt 1: this message will repeat every 5th retry.
I20240503 23:26:35.000409 3837 consensus_queue.cc:237] T a0e18d0975a54e3dbeb0a13cb226fefd P a2f9bf2e5d9b4fc19f804797bca4a3ff [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 26, Committed index: 26, Last appended: 1.26, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "83ffd7184ec94302bd1ee74b0fab3e17" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 32909 } } peers { permanent_uuid: "a2f9bf2e5d9b4fc19f804797bca4a3ff" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 46261 } } peers { permanent_uuid: "a3eecbaec43d4954b1096bc3121e824c" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 40649 } attrs { replace: true } } peers { permanent_uuid: "692da0319a2d4a81ad81f5d8f9f4c86b" member_type: NON_VOTER last_known_addr { host: "127.0.100.5" port: 38561 } attrs { promote: true } }
I20240503 23:26:35.000674 3659 catalog_manager.cc:5484] T 60267cb4849d4823a0afb2380305905f P a3eecbaec43d4954b1096bc3121e824c reported cstate change: config changed from index -1 to 26, NON_VOTER e4086c099d7f4447b2b4733b6ca364df (127.0.100.4) added. New cstate: current_term: 1 leader_uuid: "a3eecbaec43d4954b1096bc3121e824c" committed_config { opid_index: 26 OBSOLETE_local: false peers { permanent_uuid: "83ffd7184ec94302bd1ee74b0fab3e17" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 32909 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a2f9bf2e5d9b4fc19f804797bca4a3ff" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 46261 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a3eecbaec43d4954b1096bc3121e824c" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 40649 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e4086c099d7f4447b2b4733b6ca364df" member_type: NON_VOTER last_known_addr { host: "127.0.100.4" port: 35011 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 23:26:35.004434 3761 raft_consensus.cc:1261] T a0e18d0975a54e3dbeb0a13cb226fefd P 83ffd7184ec94302bd1ee74b0fab3e17 [term 1 FOLLOWER]: Refusing update from remote peer a2f9bf2e5d9b4fc19f804797bca4a3ff: Log matching property violated. Preceding OpId in replica: term: 1 index: 26. Preceding OpId from leader: term: 1 index: 27. (index mismatch)
W20240503 23:26:35.004959 3802 consensus_peers.cc:485] T a0e18d0975a54e3dbeb0a13cb226fefd P a2f9bf2e5d9b4fc19f804797bca4a3ff -> Peer 692da0319a2d4a81ad81f5d8f9f4c86b (127.0.100.5:38561): Couldn't send request to peer 692da0319a2d4a81ad81f5d8f9f4c86b. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: a0e18d0975a54e3dbeb0a13cb226fefd. This is attempt 1: this message will repeat every 5th retry.
I20240503 23:26:35.005129 4467 consensus_queue.cc:1035] T a0e18d0975a54e3dbeb0a13cb226fefd P a2f9bf2e5d9b4fc19f804797bca4a3ff [LEADER]: Connected to new peer: Peer: permanent_uuid: "83ffd7184ec94302bd1ee74b0fab3e17" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 32909 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 27, Last known committed idx: 26, Time since last communication: 0.000s
I20240503 23:26:35.012130 3762 raft_consensus.cc:2932] T 60267cb4849d4823a0afb2380305905f P 83ffd7184ec94302bd1ee74b0fab3e17 [term 1 FOLLOWER]: Committing config change with OpId 1.26: config changed from index -1 to 26, NON_VOTER e4086c099d7f4447b2b4733b6ca364df (127.0.100.4) added. New config: { opid_index: 26 OBSOLETE_local: false peers { permanent_uuid: "83ffd7184ec94302bd1ee74b0fab3e17" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 32909 } } peers { permanent_uuid: "a2f9bf2e5d9b4fc19f804797bca4a3ff" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 46261 } attrs { replace: true } } peers { permanent_uuid: "a3eecbaec43d4954b1096bc3121e824c" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 40649 } } peers { permanent_uuid: "e4086c099d7f4447b2b4733b6ca364df" member_type: NON_VOTER last_known_addr { host: "127.0.100.4" port: 35011 } attrs { promote: true } } }
I20240503 23:26:35.013760 3907 raft_consensus.cc:1261] T a0e18d0975a54e3dbeb0a13cb226fefd P a3eecbaec43d4954b1096bc3121e824c [term 1 FOLLOWER]: Refusing update from remote peer a2f9bf2e5d9b4fc19f804797bca4a3ff: Log matching property violated. Preceding OpId in replica: term: 1 index: 26. Preceding OpId from leader: term: 1 index: 27. (index mismatch)
I20240503 23:26:35.014670 4518 consensus_queue.cc:1035] T a0e18d0975a54e3dbeb0a13cb226fefd P a2f9bf2e5d9b4fc19f804797bca4a3ff [LEADER]: Connected to new peer: Peer: permanent_uuid: "a3eecbaec43d4954b1096bc3121e824c" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 40649 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 27, Last known committed idx: 26, Time since last communication: 0.000s
I20240503 23:26:35.017585 4467 raft_consensus.cc:2932] T a0e18d0975a54e3dbeb0a13cb226fefd P a2f9bf2e5d9b4fc19f804797bca4a3ff [term 1 LEADER]: Committing config change with OpId 1.27: config changed from index -1 to 27, NON_VOTER 692da0319a2d4a81ad81f5d8f9f4c86b (127.0.100.5) added. New config: { opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "83ffd7184ec94302bd1ee74b0fab3e17" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 32909 } } peers { permanent_uuid: "a2f9bf2e5d9b4fc19f804797bca4a3ff" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 46261 } } peers { permanent_uuid: "a3eecbaec43d4954b1096bc3121e824c" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 40649 } attrs { replace: true } } peers { permanent_uuid: "692da0319a2d4a81ad81f5d8f9f4c86b" member_type: NON_VOTER last_known_addr { host: "127.0.100.5" port: 38561 } attrs { promote: true } } }
I20240503 23:26:35.021476 3762 raft_consensus.cc:2932] T a0e18d0975a54e3dbeb0a13cb226fefd P 83ffd7184ec94302bd1ee74b0fab3e17 [term 1 FOLLOWER]: Committing config change with OpId 1.27: config changed from index -1 to 27, NON_VOTER 692da0319a2d4a81ad81f5d8f9f4c86b (127.0.100.5) added. New config: { opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "83ffd7184ec94302bd1ee74b0fab3e17" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 32909 } } peers { permanent_uuid: "a2f9bf2e5d9b4fc19f804797bca4a3ff" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 46261 } } peers { permanent_uuid: "a3eecbaec43d4954b1096bc3121e824c" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 40649 } attrs { replace: true } } peers { permanent_uuid: "692da0319a2d4a81ad81f5d8f9f4c86b" member_type: NON_VOTER last_known_addr { host: "127.0.100.5" port: 38561 } attrs { promote: true } } }
I20240503 23:26:35.025344 3837 consensus_queue.cc:237] T 55cf664d5e874c0d9b9adf8bdc9c43ed P a2f9bf2e5d9b4fc19f804797bca4a3ff [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 29, Committed index: 29, Last appended: 1.29, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "83ffd7184ec94302bd1ee74b0fab3e17" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 32909 } attrs { replace: true } } peers { permanent_uuid: "a2f9bf2e5d9b4fc19f804797bca4a3ff" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 46261 } } peers { permanent_uuid: "e4086c099d7f4447b2b4733b6ca364df" member_type: VOTER last_known_addr { host: "127.0.100.4" port: 35011 } attrs { promote: false } } peers { permanent_uuid: "4d1d4e261584467fbbce4c72fa18045a" member_type: NON_VOTER last_known_addr { host: "127.0.100.6" port: 35349 } attrs { promote: true } }
I20240503 23:26:35.027582 3659 catalog_manager.cc:5484] T a0e18d0975a54e3dbeb0a13cb226fefd P a2f9bf2e5d9b4fc19f804797bca4a3ff reported cstate change: config changed from index -1 to 27, NON_VOTER 692da0319a2d4a81ad81f5d8f9f4c86b (127.0.100.5) added. New cstate: current_term: 1 leader_uuid: "a2f9bf2e5d9b4fc19f804797bca4a3ff" committed_config { opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "83ffd7184ec94302bd1ee74b0fab3e17" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 32909 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a2f9bf2e5d9b4fc19f804797bca4a3ff" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 46261 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a3eecbaec43d4954b1096bc3121e824c" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 40649 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "692da0319a2d4a81ad81f5d8f9f4c86b" member_type: NON_VOTER last_known_addr { host: "127.0.100.5" port: 38561 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 23:26:35.036875 4218 raft_consensus.cc:1261] T 55cf664d5e874c0d9b9adf8bdc9c43ed P e4086c099d7f4447b2b4733b6ca364df [term 1 FOLLOWER]: Refusing update from remote peer a2f9bf2e5d9b4fc19f804797bca4a3ff: Log matching property violated. Preceding OpId in replica: term: 1 index: 29. Preceding OpId from leader: term: 1 index: 30. (index mismatch)
I20240503 23:26:35.038152 4467 consensus_queue.cc:1035] T 55cf664d5e874c0d9b9adf8bdc9c43ed P a2f9bf2e5d9b4fc19f804797bca4a3ff [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4086c099d7f4447b2b4733b6ca364df" member_type: VOTER last_known_addr { host: "127.0.100.4" port: 35011 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 30, Last known committed idx: 29, Time since last communication: 0.000s
I20240503 23:26:35.038756 3762 raft_consensus.cc:1261] T 55cf664d5e874c0d9b9adf8bdc9c43ed P 83ffd7184ec94302bd1ee74b0fab3e17 [term 1 FOLLOWER]: Refusing update from remote peer a2f9bf2e5d9b4fc19f804797bca4a3ff: Log matching property violated. Preceding OpId in replica: term: 1 index: 29. Preceding OpId from leader: term: 1 index: 30. (index mismatch)
I20240503 23:26:35.038549 3907 raft_consensus.cc:2932] T a0e18d0975a54e3dbeb0a13cb226fefd P a3eecbaec43d4954b1096bc3121e824c [term 1 FOLLOWER]: Committing config change with OpId 1.27: config changed from index -1 to 27, NON_VOTER 692da0319a2d4a81ad81f5d8f9f4c86b (127.0.100.5) added. New config: { opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "83ffd7184ec94302bd1ee74b0fab3e17" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 32909 } } peers { permanent_uuid: "a2f9bf2e5d9b4fc19f804797bca4a3ff" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 46261 } } peers { permanent_uuid: "a3eecbaec43d4954b1096bc3121e824c" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 40649 } attrs { replace: true } } peers { permanent_uuid: "692da0319a2d4a81ad81f5d8f9f4c86b" member_type: NON_VOTER last_known_addr { host: "127.0.100.5" port: 38561 } attrs { promote: true } } }
I20240503 23:26:35.039600 4467 consensus_queue.cc:1035] T 55cf664d5e874c0d9b9adf8bdc9c43ed P a2f9bf2e5d9b4fc19f804797bca4a3ff [LEADER]: Connected to new peer: Peer: permanent_uuid: "83ffd7184ec94302bd1ee74b0fab3e17" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 32909 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 30, Last known committed idx: 29, Time since last communication: 0.000s
I20240503 23:26:35.050601 4518 raft_consensus.cc:2932] T 55cf664d5e874c0d9b9adf8bdc9c43ed P a2f9bf2e5d9b4fc19f804797bca4a3ff [term 1 LEADER]: Committing config change with OpId 1.30: config changed from index 29 to 30, NON_VOTER 4d1d4e261584467fbbce4c72fa18045a (127.0.100.6) added. New config: { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "83ffd7184ec94302bd1ee74b0fab3e17" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 32909 } attrs { replace: true } } peers { permanent_uuid: "a2f9bf2e5d9b4fc19f804797bca4a3ff" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 46261 } } peers { permanent_uuid: "e4086c099d7f4447b2b4733b6ca364df" member_type: VOTER last_known_addr { host: "127.0.100.4" port: 35011 } attrs { promote: false } } peers { permanent_uuid: "4d1d4e261584467fbbce4c72fa18045a" member_type: NON_VOTER last_known_addr { host: "127.0.100.6" port: 35349 } attrs { promote: true } } }
I20240503 23:26:35.052177 4218 raft_consensus.cc:2932] T 55cf664d5e874c0d9b9adf8bdc9c43ed P e4086c099d7f4447b2b4733b6ca364df [term 1 FOLLOWER]: Committing config change with OpId 1.30: config changed from index 29 to 30, NON_VOTER 4d1d4e261584467fbbce4c72fa18045a (127.0.100.6) added. New config: { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "83ffd7184ec94302bd1ee74b0fab3e17" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 32909 } attrs { replace: true } } peers { permanent_uuid: "a2f9bf2e5d9b4fc19f804797bca4a3ff" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 46261 } } peers { permanent_uuid: "e4086c099d7f4447b2b4733b6ca364df" member_type: VOTER last_known_addr { host: "127.0.100.4" port: 35011 } attrs { promote: false } } peers { permanent_uuid: "4d1d4e261584467fbbce4c72fa18045a" member_type: NON_VOTER last_known_addr { host: "127.0.100.6" port: 35349 } attrs { promote: true } } }
I20240503 23:26:35.052243 3761 raft_consensus.cc:2932] T 55cf664d5e874c0d9b9adf8bdc9c43ed P 83ffd7184ec94302bd1ee74b0fab3e17 [term 1 FOLLOWER]: Committing config change with OpId 1.30: config changed from index 29 to 30, NON_VOTER 4d1d4e261584467fbbce4c72fa18045a (127.0.100.6) added. New config: { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "83ffd7184ec94302bd1ee74b0fab3e17" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 32909 } attrs { replace: true } } peers { permanent_uuid: "a2f9bf2e5d9b4fc19f804797bca4a3ff" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 46261 } } peers { permanent_uuid: "e4086c099d7f4447b2b4733b6ca364df" member_type: VOTER last_known_addr { host: "127.0.100.4" port: 35011 } attrs { promote: false } } peers { permanent_uuid: "4d1d4e261584467fbbce4c72fa18045a" member_type: NON_VOTER last_known_addr { host: "127.0.100.6" port: 35349 } attrs { promote: true } } }
W20240503 23:26:35.057118 3803 consensus_peers.cc:485] T 55cf664d5e874c0d9b9adf8bdc9c43ed P a2f9bf2e5d9b4fc19f804797bca4a3ff -> Peer 4d1d4e261584467fbbce4c72fa18045a (127.0.100.6:35349): Couldn't send request to peer 4d1d4e261584467fbbce4c72fa18045a. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 55cf664d5e874c0d9b9adf8bdc9c43ed. This is attempt 1: this message will repeat every 5th retry.
I20240503 23:26:35.058346 3762 consensus_queue.cc:237] T 98c9500acb43423b86f54ce7a4d98a72 P 83ffd7184ec94302bd1ee74b0fab3e17 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 28, Committed index: 28, Last appended: 1.28, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "83ffd7184ec94302bd1ee74b0fab3e17" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 32909 } } peers { permanent_uuid: "a2f9bf2e5d9b4fc19f804797bca4a3ff" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 46261 } attrs { replace: true } } peers { permanent_uuid: "4d1d4e261584467fbbce4c72fa18045a" member_type: VOTER last_known_addr { host: "127.0.100.6" port: 35349 } attrs { promote: false } } peers { permanent_uuid: "e4086c099d7f4447b2b4733b6ca364df" member_type: NON_VOTER last_known_addr { host: "127.0.100.4" port: 35011 } attrs { promote: true } }
I20240503 23:26:35.059509 3659 catalog_manager.cc:5484] T 55cf664d5e874c0d9b9adf8bdc9c43ed P a2f9bf2e5d9b4fc19f804797bca4a3ff reported cstate change: config changed from index 29 to 30, NON_VOTER 4d1d4e261584467fbbce4c72fa18045a (127.0.100.6) added. New cstate: current_term: 1 leader_uuid: "a2f9bf2e5d9b4fc19f804797bca4a3ff" committed_config { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "83ffd7184ec94302bd1ee74b0fab3e17" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 32909 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a2f9bf2e5d9b4fc19f804797bca4a3ff" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 46261 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e4086c099d7f4447b2b4733b6ca364df" member_type: VOTER last_known_addr { host: "127.0.100.4" port: 35011 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4d1d4e261584467fbbce4c72fa18045a" member_type: NON_VOTER last_known_addr { host: "127.0.100.6" port: 35349 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 23:26:35.063999 4367 raft_consensus.cc:1261] T 98c9500acb43423b86f54ce7a4d98a72 P 4d1d4e261584467fbbce4c72fa18045a [term 1 FOLLOWER]: Refusing update from remote peer 83ffd7184ec94302bd1ee74b0fab3e17: Log matching property violated. Preceding OpId in replica: term: 1 index: 28. Preceding OpId from leader: term: 1 index: 29. (index mismatch)
I20240503 23:26:35.065163 4508 consensus_queue.cc:1035] T 98c9500acb43423b86f54ce7a4d98a72 P 83ffd7184ec94302bd1ee74b0fab3e17 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d1d4e261584467fbbce4c72fa18045a" member_type: VOTER last_known_addr { host: "127.0.100.6" port: 35349 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.000s
W20240503 23:26:35.066218 3727 consensus_peers.cc:485] T 98c9500acb43423b86f54ce7a4d98a72 P 83ffd7184ec94302bd1ee74b0fab3e17 -> Peer e4086c099d7f4447b2b4733b6ca364df (127.0.100.4:35011): Couldn't send request to peer e4086c099d7f4447b2b4733b6ca364df. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 98c9500acb43423b86f54ce7a4d98a72. This is attempt 1: this message will repeat every 5th retry.
I20240503 23:26:35.069443 4508 raft_consensus.cc:2932] T 98c9500acb43423b86f54ce7a4d98a72 P 83ffd7184ec94302bd1ee74b0fab3e17 [term 1 LEADER]: Committing config change with OpId 1.29: config changed from index 28 to 29, NON_VOTER e4086c099d7f4447b2b4733b6ca364df (127.0.100.4) added. New config: { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "83ffd7184ec94302bd1ee74b0fab3e17" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 32909 } } peers { permanent_uuid: "a2f9bf2e5d9b4fc19f804797bca4a3ff" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 46261 } attrs { replace: true } } peers { permanent_uuid: "4d1d4e261584467fbbce4c72fa18045a" member_type: VOTER last_known_addr { host: "127.0.100.6" port: 35349 } attrs { promote: false } } peers { permanent_uuid: "e4086c099d7f4447b2b4733b6ca364df" member_type: NON_VOTER last_known_addr { host: "127.0.100.4" port: 35011 } attrs { promote: true } } }
I20240503 23:26:35.070258 4367 raft_consensus.cc:2932] T 98c9500acb43423b86f54ce7a4d98a72 P 4d1d4e261584467fbbce4c72fa18045a [term 1 FOLLOWER]: Committing config change with OpId 1.29: config changed from index 28 to 29, NON_VOTER e4086c099d7f4447b2b4733b6ca364df (127.0.100.4) added. New config: { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "83ffd7184ec94302bd1ee74b0fab3e17" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 32909 } } peers { permanent_uuid: "a2f9bf2e5d9b4fc19f804797bca4a3ff" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 46261 } attrs { replace: true } } peers { permanent_uuid: "4d1d4e261584467fbbce4c72fa18045a" member_type: VOTER last_known_addr { host: "127.0.100.6" port: 35349 } attrs { promote: false } } peers { permanent_uuid: "e4086c099d7f4447b2b4733b6ca364df" member_type: NON_VOTER last_known_addr { host: "127.0.100.4" port: 35011 } attrs { promote: true } } }
I20240503 23:26:35.073453 3837 raft_consensus.cc:1261] T 98c9500acb43423b86f54ce7a4d98a72 P a2f9bf2e5d9b4fc19f804797bca4a3ff [term 1 FOLLOWER]: Refusing update from remote peer 83ffd7184ec94302bd1ee74b0fab3e17: Log matching property violated. Preceding OpId in replica: term: 1 index: 28. Preceding OpId from leader: term: 1 index: 29. (index mismatch)
I20240503 23:26:35.076651 3907 consensus_queue.cc:237] T d0c4430fc56644cb9f87deffe4fe9407 P a3eecbaec43d4954b1096bc3121e824c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 26, Committed index: 26, Last appended: 1.26, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "83ffd7184ec94302bd1ee74b0fab3e17" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 32909 } attrs { replace: true } } peers { permanent_uuid: "a2f9bf2e5d9b4fc19f804797bca4a3ff" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 46261 } } peers { permanent_uuid: "a3eecbaec43d4954b1096bc3121e824c" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 40649 } } peers { permanent_uuid: "4d1d4e261584467fbbce4c72fa18045a" member_type: NON_VOTER last_known_addr { host: "127.0.100.6" port: 35349 } attrs { promote: true } }
I20240503 23:26:35.077754 3659 catalog_manager.cc:5484] T 98c9500acb43423b86f54ce7a4d98a72 P 4d1d4e261584467fbbce4c72fa18045a reported cstate change: config changed from index 28 to 29, NON_VOTER e4086c099d7f4447b2b4733b6ca364df (127.0.100.4) added. New cstate: current_term: 1 leader_uuid: "83ffd7184ec94302bd1ee74b0fab3e17" committed_config { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "83ffd7184ec94302bd1ee74b0fab3e17" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 32909 } } peers { permanent_uuid: "a2f9bf2e5d9b4fc19f804797bca4a3ff" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 46261 } attrs { replace: true } } peers { permanent_uuid: "4d1d4e261584467fbbce4c72fa18045a" member_type: VOTER last_known_addr { host: "127.0.100.6" port: 35349 } attrs { promote: false } } peers { permanent_uuid: "e4086c099d7f4447b2b4733b6ca364df" member_type: NON_VOTER last_known_addr { host: "127.0.100.4" port: 35011 } attrs { promote: true } } }
I20240503 23:26:35.074142 4532 consensus_queue.cc:1035] T 98c9500acb43423b86f54ce7a4d98a72 P 83ffd7184ec94302bd1ee74b0fab3e17 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2f9bf2e5d9b4fc19f804797bca4a3ff" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 46261 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.007s
I20240503 23:26:35.093041 3837 raft_consensus.cc:2932] T 98c9500acb43423b86f54ce7a4d98a72 P a2f9bf2e5d9b4fc19f804797bca4a3ff [term 1 FOLLOWER]: Committing config change with OpId 1.29: config changed from index 28 to 29, NON_VOTER e4086c099d7f4447b2b4733b6ca364df (127.0.100.4) added. New config: { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "83ffd7184ec94302bd1ee74b0fab3e17" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 32909 } } peers { permanent_uuid: "a2f9bf2e5d9b4fc19f804797bca4a3ff" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 46261 } attrs { replace: true } } peers { permanent_uuid: "4d1d4e261584467fbbce4c72fa18045a" member_type: VOTER last_known_addr { host: "127.0.100.6" port: 35349 } attrs { promote: false } } peers { permanent_uuid: "e4086c099d7f4447b2b4733b6ca364df" member_type: NON_VOTER last_known_addr { host: "127.0.100.4" port: 35011 } attrs { promote: true } } }
I20240503 23:26:35.099133 3838 raft_consensus.cc:1261] T d0c4430fc56644cb9f87deffe4fe9407 P a2f9bf2e5d9b4fc19f804797bca4a3ff [term 1 FOLLOWER]: Refusing update from remote peer a3eecbaec43d4954b1096bc3121e824c: Log matching property violated. Preceding OpId in replica: term: 1 index: 26. Preceding OpId from leader: term: 1 index: 27. (index mismatch)
I20240503 23:26:35.099579 3762 raft_consensus.cc:1261] T d0c4430fc56644cb9f87deffe4fe9407 P 83ffd7184ec94302bd1ee74b0fab3e17 [term 1 FOLLOWER]: Refusing update from remote peer a3eecbaec43d4954b1096bc3121e824c: Log matching property violated. Preceding OpId in replica: term: 1 index: 26. Preceding OpId from leader: term: 1 index: 27. (index mismatch)
I20240503 23:26:35.100116 4426 consensus_queue.cc:1035] T d0c4430fc56644cb9f87deffe4fe9407 P a3eecbaec43d4954b1096bc3121e824c [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2f9bf2e5d9b4fc19f804797bca4a3ff" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 46261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 27, Last known committed idx: 26, Time since last communication: 0.000s
I20240503 23:26:35.100695 4515 consensus_queue.cc:1035] T d0c4430fc56644cb9f87deffe4fe9407 P a3eecbaec43d4954b1096bc3121e824c [LEADER]: Connected to new peer: Peer: permanent_uuid: "83ffd7184ec94302bd1ee74b0fab3e17" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 32909 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 27, Last known committed idx: 26, Time since last communication: 0.000s
I20240503 23:26:35.105391 4426 raft_consensus.cc:2932] T d0c4430fc56644cb9f87deffe4fe9407 P a3eecbaec43d4954b1096bc3121e824c [term 1 LEADER]: Committing config change with OpId 1.27: config changed from index -1 to 27, NON_VOTER 4d1d4e261584467fbbce4c72fa18045a (127.0.100.6) added. New config: { opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "83ffd7184ec94302bd1ee74b0fab3e17" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 32909 } attrs { replace: true } } peers { permanent_uuid: "a2f9bf2e5d9b4fc19f804797bca4a3ff" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 46261 } } peers { permanent_uuid: "a3eecbaec43d4954b1096bc3121e824c" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 40649 } } peers { permanent_uuid: "4d1d4e261584467fbbce4c72fa18045a" member_type: NON_VOTER last_known_addr { host: "127.0.100.6" port: 35349 } attrs { promote: true } } }
I20240503 23:26:35.111922 3837 raft_consensus.cc:2932] T d0c4430fc56644cb9f87deffe4fe9407 P a2f9bf2e5d9b4fc19f804797bca4a3ff [term 1 FOLLOWER]: Committing config change with OpId 1.27: config changed from index -1 to 27, NON_VOTER 4d1d4e261584467fbbce4c72fa18045a (127.0.100.6) added. New config: { opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "83ffd7184ec94302bd1ee74b0fab3e17" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 32909 } attrs { replace: true } } peers { permanent_uuid: "a2f9bf2e5d9b4fc19f804797bca4a3ff" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 46261 } } peers { permanent_uuid: "a3eecbaec43d4954b1096bc3121e824c" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 40649 } } peers { permanent_uuid: "4d1d4e261584467fbbce4c72fa18045a" member_type: NON_VOTER last_known_addr { host: "127.0.100.6" port: 35349 } attrs { promote: true } } }
I20240503 23:26:35.114451 3655 catalog_manager.cc:5484] T d0c4430fc56644cb9f87deffe4fe9407 P a3eecbaec43d4954b1096bc3121e824c reported cstate change: config changed from index -1 to 27, NON_VOTER 4d1d4e261584467fbbce4c72fa18045a (127.0.100.6) added. New cstate: current_term: 1 leader_uuid: "a3eecbaec43d4954b1096bc3121e824c" committed_config { opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "83ffd7184ec94302bd1ee74b0fab3e17" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 32909 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a2f9bf2e5d9b4fc19f804797bca4a3ff" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 46261 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a3eecbaec43d4954b1096bc3121e824c" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 40649 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4d1d4e261584467fbbce4c72fa18045a" member_type: NON_VOTER last_known_addr { host: "127.0.100.6" port: 35349 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20240503 23:26:35.114995 3877 consensus_peers.cc:485] T d0c4430fc56644cb9f87deffe4fe9407 P a3eecbaec43d4954b1096bc3121e824c -> Peer 4d1d4e261584467fbbce4c72fa18045a (127.0.100.6:35349): Couldn't send request to peer 4d1d4e261584467fbbce4c72fa18045a. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d0c4430fc56644cb9f87deffe4fe9407. This is attempt 1: this message will repeat every 5th retry.
I20240503 23:26:35.115847 3761 raft_consensus.cc:2932] T d0c4430fc56644cb9f87deffe4fe9407 P 83ffd7184ec94302bd1ee74b0fab3e17 [term 1 FOLLOWER]: Committing config change with OpId 1.27: config changed from index -1 to 27, NON_VOTER 4d1d4e261584467fbbce4c72fa18045a (127.0.100.6) added. New config: { opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "83ffd7184ec94302bd1ee74b0fab3e17" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 32909 } attrs { replace: true } } peers { permanent_uuid: "a2f9bf2e5d9b4fc19f804797bca4a3ff" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 46261 } } peers { permanent_uuid: "a3eecbaec43d4954b1096bc3121e824c" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 40649 } } peers { permanent_uuid: "4d1d4e261584467fbbce4c72fa18045a" member_type: NON_VOTER last_known_addr { host: "127.0.100.6" port: 35349 } attrs { promote: true } } }
I20240503 23:26:35.116355 3762 consensus_queue.cc:237] T 487ac5b9bf2d4de2b80a1b13387a2a77 P 83ffd7184ec94302bd1ee74b0fab3e17 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 26, Committed index: 26, Last appended: 1.26, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "83ffd7184ec94302bd1ee74b0fab3e17" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 32909 } } peers { permanent_uuid: "a2f9bf2e5d9b4fc19f804797bca4a3ff" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 46261 } } peers { permanent_uuid: "a3eecbaec43d4954b1096bc3121e824c" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 40649 } attrs { replace: true } } peers { permanent_uuid: "692da0319a2d4a81ad81f5d8f9f4c86b" member_type: NON_VOTER last_known_addr { host: "127.0.100.5" port: 38561 } attrs { promote: true } }
I20240503 23:26:35.119997 3838 raft_consensus.cc:1261] T 487ac5b9bf2d4de2b80a1b13387a2a77 P a2f9bf2e5d9b4fc19f804797bca4a3ff [term 1 FOLLOWER]: Refusing update from remote peer 83ffd7184ec94302bd1ee74b0fab3e17: Log matching property violated. Preceding OpId in replica: term: 1 index: 26. Preceding OpId from leader: term: 1 index: 27. (index mismatch)
I20240503 23:26:35.120606 3907 raft_consensus.cc:1261] T 487ac5b9bf2d4de2b80a1b13387a2a77 P a3eecbaec43d4954b1096bc3121e824c [term 1 FOLLOWER]: Refusing update from remote peer 83ffd7184ec94302bd1ee74b0fab3e17: Log matching property violated. Preceding OpId in replica: term: 1 index: 26. Preceding OpId from leader: term: 1 index: 27. (index mismatch)
I20240503 23:26:35.121860 4537 consensus_queue.cc:1035] T 487ac5b9bf2d4de2b80a1b13387a2a77 P 83ffd7184ec94302bd1ee74b0fab3e17 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2f9bf2e5d9b4fc19f804797bca4a3ff" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 46261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 27, Last known committed idx: 26, Time since last communication: 0.000s
I20240503 23:26:35.122249 4532 consensus_queue.cc:1035] T 487ac5b9bf2d4de2b80a1b13387a2a77 P 83ffd7184ec94302bd1ee74b0fab3e17 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a3eecbaec43d4954b1096bc3121e824c" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 40649 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 27, Last known committed idx: 26, Time since last communication: 0.000s
I20240503 23:26:35.128175 4532 raft_consensus.cc:2932] T 487ac5b9bf2d4de2b80a1b13387a2a77 P 83ffd7184ec94302bd1ee74b0fab3e17 [term 1 LEADER]: Committing config change with OpId 1.27: config changed from index -1 to 27, NON_VOTER 692da0319a2d4a81ad81f5d8f9f4c86b (127.0.100.5) added. New config: { opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "83ffd7184ec94302bd1ee74b0fab3e17" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 32909 } } peers { permanent_uuid: "a2f9bf2e5d9b4fc19f804797bca4a3ff" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 46261 } } peers { permanent_uuid: "a3eecbaec43d4954b1096bc3121e824c" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 40649 } attrs { replace: true } } peers { permanent_uuid: "692da0319a2d4a81ad81f5d8f9f4c86b" member_type: NON_VOTER last_known_addr { host: "127.0.100.5" port: 38561 } attrs { promote: true } } }
I20240503 23:26:35.133198 3838 raft_consensus.cc:2932] T 487ac5b9bf2d4de2b80a1b13387a2a77 P a2f9bf2e5d9b4fc19f804797bca4a3ff [term 1 FOLLOWER]: Committing config change with OpId 1.27: config changed from index -1 to 27, NON_VOTER 692da0319a2d4a81ad81f5d8f9f4c86b (127.0.100.5) added. New config: { opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "83ffd7184ec94302bd1ee74b0fab3e17" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 32909 } } peers { permanent_uuid: "a2f9bf2e5d9b4fc19f804797bca4a3ff" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 46261 } } peers { permanent_uuid: "a3eecbaec43d4954b1096bc3121e824c" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 40649 } attrs { replace: true } } peers { permanent_uuid: "692da0319a2d4a81ad81f5d8f9f4c86b" member_type: NON_VOTER last_known_addr { host: "127.0.100.5" port: 38561 } attrs { promote: true } } }
I20240503 23:26:35.141542 3659 catalog_manager.cc:5484] T 487ac5b9bf2d4de2b80a1b13387a2a77 P 83ffd7184ec94302bd1ee74b0fab3e17 reported cstate change: config changed from index -1 to 27, NON_VOTER 692da0319a2d4a81ad81f5d8f9f4c86b (127.0.100.5) added. New cstate: current_term: 1 leader_uuid: "83ffd7184ec94302bd1ee74b0fab3e17" committed_config { opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "83ffd7184ec94302bd1ee74b0fab3e17" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 32909 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a2f9bf2e5d9b4fc19f804797bca4a3ff" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 46261 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a3eecbaec43d4954b1096bc3121e824c" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 40649 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "692da0319a2d4a81ad81f5d8f9f4c86b" member_type: NON_VOTER last_known_addr { host: "127.0.100.5" port: 38561 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 23:26:35.151069 3907 raft_consensus.cc:2932] T 487ac5b9bf2d4de2b80a1b13387a2a77 P a3eecbaec43d4954b1096bc3121e824c [term 1 FOLLOWER]: Committing config change with OpId 1.27: config changed from index -1 to 27, NON_VOTER 692da0319a2d4a81ad81f5d8f9f4c86b (127.0.100.5) added. New config: { opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "83ffd7184ec94302bd1ee74b0fab3e17" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 32909 } } peers { permanent_uuid: "a2f9bf2e5d9b4fc19f804797bca4a3ff" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 46261 } } peers { permanent_uuid: "a3eecbaec43d4954b1096bc3121e824c" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 40649 } attrs { replace: true } } peers { permanent_uuid: "692da0319a2d4a81ad81f5d8f9f4c86b" member_type: NON_VOTER last_known_addr { host: "127.0.100.5" port: 38561 } attrs { promote: true } } }
W20240503 23:26:35.154071 3727 consensus_peers.cc:485] T 487ac5b9bf2d4de2b80a1b13387a2a77 P 83ffd7184ec94302bd1ee74b0fab3e17 -> Peer 692da0319a2d4a81ad81f5d8f9f4c86b (127.0.100.5:38561): Couldn't send request to peer 692da0319a2d4a81ad81f5d8f9f4c86b. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 487ac5b9bf2d4de2b80a1b13387a2a77. This is attempt 1: this message will repeat every 5th retry.
I20240503 23:26:35.199573 3715 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240503 23:26:35.224309 3838 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "1b9566be6c644cfca852ec6507fab893"
dest_uuid: "a2f9bf2e5d9b4fc19f804797bca4a3ff"
mode: GRACEFUL
new_leader_uuid: "692da0319a2d4a81ad81f5d8f9f4c86b"
from {username='slave'} at 127.0.0.1:55472
I20240503 23:26:35.224632 3838 raft_consensus.cc:595] T 1b9566be6c644cfca852ec6507fab893 P a2f9bf2e5d9b4fc19f804797bca4a3ff [term 1 LEADER]: Received request to transfer leadership to TS 692da0319a2d4a81ad81f5d8f9f4c86b
I20240503 23:26:35.225831 3907 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "481cd0f8c8d147adac217915a53ff6d9"
dest_uuid: "a3eecbaec43d4954b1096bc3121e824c"
mode: GRACEFUL
new_leader_uuid: "83ffd7184ec94302bd1ee74b0fab3e17"
from {username='slave'} at 127.0.0.1:40650
I20240503 23:26:35.226058 3907 raft_consensus.cc:595] T 481cd0f8c8d147adac217915a53ff6d9 P a3eecbaec43d4954b1096bc3121e824c [term 1 LEADER]: Received request to transfer leadership to TS 83ffd7184ec94302bd1ee74b0fab3e17
I20240503 23:26:35.234498 3761 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "487ac5b9bf2d4de2b80a1b13387a2a77"
dest_uuid: "83ffd7184ec94302bd1ee74b0fab3e17"
mode: GRACEFUL
new_leader_uuid: "a2f9bf2e5d9b4fc19f804797bca4a3ff"
from {username='slave'} at 127.0.0.1:57936
I20240503 23:26:35.234776 3761 raft_consensus.cc:595] T 487ac5b9bf2d4de2b80a1b13387a2a77 P 83ffd7184ec94302bd1ee74b0fab3e17 [term 1 LEADER]: Received request to transfer leadership to TS a2f9bf2e5d9b4fc19f804797bca4a3ff
I20240503 23:26:35.235939 3838 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "55cf664d5e874c0d9b9adf8bdc9c43ed"
dest_uuid: "a2f9bf2e5d9b4fc19f804797bca4a3ff"
mode: GRACEFUL
new_leader_uuid: "e4086c099d7f4447b2b4733b6ca364df"
from {username='slave'} at 127.0.0.1:55472
I20240503 23:26:35.236177 3838 raft_consensus.cc:595] T 55cf664d5e874c0d9b9adf8bdc9c43ed P a2f9bf2e5d9b4fc19f804797bca4a3ff [term 1 LEADER]: Received request to transfer leadership to TS e4086c099d7f4447b2b4733b6ca364df
I20240503 23:26:35.237243 3762 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "a14a12115b8d4fec947c6e223f1ef486"
dest_uuid: "83ffd7184ec94302bd1ee74b0fab3e17"
mode: GRACEFUL
new_leader_uuid: "4d1d4e261584467fbbce4c72fa18045a"
from {username='slave'} at 127.0.0.1:57936
I20240503 23:26:35.237464 3762 raft_consensus.cc:595] T a14a12115b8d4fec947c6e223f1ef486 P 83ffd7184ec94302bd1ee74b0fab3e17 [term 1 LEADER]: Received request to transfer leadership to TS 4d1d4e261584467fbbce4c72fa18045a
I20240503 23:26:35.238533 3907 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "d0c4430fc56644cb9f87deffe4fe9407"
dest_uuid: "a3eecbaec43d4954b1096bc3121e824c"
mode: GRACEFUL
new_leader_uuid: "83ffd7184ec94302bd1ee74b0fab3e17"
from {username='slave'} at 127.0.0.1:40650
I20240503 23:26:35.238709 3907 raft_consensus.cc:595] T d0c4430fc56644cb9f87deffe4fe9407 P a3eecbaec43d4954b1096bc3121e824c [term 1 LEADER]: Received request to transfer leadership to TS 83ffd7184ec94302bd1ee74b0fab3e17
I20240503 23:26:35.239142 3715 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 6
I20240503 23:26:35.239411 3715 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240503 23:26:35.362452 4550 ts_tablet_manager.cc:902] T 60267cb4849d4823a0afb2380305905f P e4086c099d7f4447b2b4733b6ca364df: Initiating tablet copy from peer a3eecbaec43d4954b1096bc3121e824c (127.0.100.3:40649)
I20240503 23:26:35.363391 4550 tablet_copy_client.cc:316] T 60267cb4849d4823a0afb2380305905f P e4086c099d7f4447b2b4733b6ca364df: tablet copy: Beginning tablet copy session from remote peer at address 127.0.100.3:40649
I20240503 23:26:35.372126 3923 tablet_copy_service.cc:140] P a3eecbaec43d4954b1096bc3121e824c: Received BeginTabletCopySession request for tablet 60267cb4849d4823a0afb2380305905f from peer e4086c099d7f4447b2b4733b6ca364df ({username='slave'} at 127.0.0.1:39458)
I20240503 23:26:35.372378 3923 tablet_copy_service.cc:161] P a3eecbaec43d4954b1096bc3121e824c: Beginning new tablet copy session on tablet 60267cb4849d4823a0afb2380305905f from peer e4086c099d7f4447b2b4733b6ca364df at {username='slave'} at 127.0.0.1:39458: session id = e4086c099d7f4447b2b4733b6ca364df-60267cb4849d4823a0afb2380305905f
I20240503 23:26:35.375159 3923 tablet_copy_source_session.cc:205] T 60267cb4849d4823a0afb2380305905f P a3eecbaec43d4954b1096bc3121e824c: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 23:26:35.376717 4550 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 60267cb4849d4823a0afb2380305905f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 23:26:35.382494 4550 tablet_copy_client.cc:791] T 60267cb4849d4823a0afb2380305905f P e4086c099d7f4447b2b4733b6ca364df: tablet copy: Starting download of 0 data blocks...
I20240503 23:26:35.382810 4550 tablet_copy_client.cc:655] T 60267cb4849d4823a0afb2380305905f P e4086c099d7f4447b2b4733b6ca364df: tablet copy: Starting download of 1 WAL segments...
I20240503 23:26:35.384601 4551 tablet_copy_client.cc:1010] T 60267cb4849d4823a0afb2380305905f P e4086c099d7f4447b2b4733b6ca364df: tablet copy: Injecting latency into file download: 2000
I20240503 23:26:35.510252 4510 raft_consensus.cc:979] T d0c4430fc56644cb9f87deffe4fe9407 P a3eecbaec43d4954b1096bc3121e824c: : Instructing follower 83ffd7184ec94302bd1ee74b0fab3e17 to start an election
I20240503 23:26:35.510509 4426 raft_consensus.cc:1067] T d0c4430fc56644cb9f87deffe4fe9407 P a3eecbaec43d4954b1096bc3121e824c [term 1 LEADER]: Signalling peer 83ffd7184ec94302bd1ee74b0fab3e17 to start an election
I20240503 23:26:35.511113 3762 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "d0c4430fc56644cb9f87deffe4fe9407"
dest_uuid: "83ffd7184ec94302bd1ee74b0fab3e17"
from {username='slave'} at 127.0.0.1:35592
I20240503 23:26:35.511400 3762 raft_consensus.cc:482] T d0c4430fc56644cb9f87deffe4fe9407 P 83ffd7184ec94302bd1ee74b0fab3e17 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 23:26:35.511543 3762 raft_consensus.cc:3037] T d0c4430fc56644cb9f87deffe4fe9407 P 83ffd7184ec94302bd1ee74b0fab3e17 [term 1 FOLLOWER]: Advancing to term 2
I20240503 23:26:35.514632 3762 raft_consensus.cc:504] T d0c4430fc56644cb9f87deffe4fe9407 P 83ffd7184ec94302bd1ee74b0fab3e17 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "83ffd7184ec94302bd1ee74b0fab3e17" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 32909 } attrs { replace: true } } peers { permanent_uuid: "a2f9bf2e5d9b4fc19f804797bca4a3ff" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 46261 } } peers { permanent_uuid: "a3eecbaec43d4954b1096bc3121e824c" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 40649 } } peers { permanent_uuid: "4d1d4e261584467fbbce4c72fa18045a" member_type: NON_VOTER last_known_addr { host: "127.0.100.6" port: 35349 } attrs { promote: true } }
I20240503 23:26:35.515542 3762 leader_election.cc:288] T d0c4430fc56644cb9f87deffe4fe9407 P 83ffd7184ec94302bd1ee74b0fab3e17 [CANDIDATE]: Term 2 election: Requested vote from peers a2f9bf2e5d9b4fc19f804797bca4a3ff (127.0.100.2:46261), a3eecbaec43d4954b1096bc3121e824c (127.0.100.3:40649)
I20240503 23:26:35.515933 3837 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0c4430fc56644cb9f87deffe4fe9407" candidate_uuid: "83ffd7184ec94302bd1ee74b0fab3e17" candidate_term: 2 candidate_status { last_received { term: 1 index: 27 } } ignore_live_leader: true dest_uuid: "a2f9bf2e5d9b4fc19f804797bca4a3ff"
I20240503 23:26:35.516173 3837 raft_consensus.cc:3037] T d0c4430fc56644cb9f87deffe4fe9407 P a2f9bf2e5d9b4fc19f804797bca4a3ff [term 1 FOLLOWER]: Advancing to term 2
I20240503 23:26:35.516170 3907 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0c4430fc56644cb9f87deffe4fe9407" candidate_uuid: "83ffd7184ec94302bd1ee74b0fab3e17" candidate_term: 2 candidate_status { last_received { term: 1 index: 27 } } ignore_live_leader: true dest_uuid: "a3eecbaec43d4954b1096bc3121e824c"
I20240503 23:26:35.516490 3907 raft_consensus.cc:3032] T d0c4430fc56644cb9f87deffe4fe9407 P a3eecbaec43d4954b1096bc3121e824c [term 1 LEADER]: Stepping down as leader of term 1
I20240503 23:26:35.516626 3907 raft_consensus.cc:729] T d0c4430fc56644cb9f87deffe4fe9407 P a3eecbaec43d4954b1096bc3121e824c [term 1 LEADER]: Becoming Follower/Learner. State: Replica: a3eecbaec43d4954b1096bc3121e824c, State: Running, Role: LEADER
I20240503 23:26:35.516988 3907 consensus_queue.cc:260] T d0c4430fc56644cb9f87deffe4fe9407 P a3eecbaec43d4954b1096bc3121e824c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 27, Committed index: 27, Last appended: 1.27, Last appended by leader: 27, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "83ffd7184ec94302bd1ee74b0fab3e17" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 32909 } attrs { replace: true } } peers { permanent_uuid: "a2f9bf2e5d9b4fc19f804797bca4a3ff" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 46261 } } peers { permanent_uuid: "a3eecbaec43d4954b1096bc3121e824c" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 40649 } } peers { permanent_uuid: "4d1d4e261584467fbbce4c72fa18045a" member_type: NON_VOTER last_known_addr { host: "127.0.100.6" port: 35349 } attrs { promote: true } }
I20240503 23:26:35.517503 3907 raft_consensus.cc:3037] T d0c4430fc56644cb9f87deffe4fe9407 P a3eecbaec43d4954b1096bc3121e824c [term 1 FOLLOWER]: Advancing to term 2
I20240503 23:26:35.519387 3837 raft_consensus.cc:2454] T d0c4430fc56644cb9f87deffe4fe9407 P a2f9bf2e5d9b4fc19f804797bca4a3ff [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83ffd7184ec94302bd1ee74b0fab3e17 in term 2.
I20240503 23:26:35.519963 3730 leader_election.cc:302] T d0c4430fc56644cb9f87deffe4fe9407 P 83ffd7184ec94302bd1ee74b0fab3e17 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 83ffd7184ec94302bd1ee74b0fab3e17, a2f9bf2e5d9b4fc19f804797bca4a3ff; no voters:
I20240503 23:26:35.520295 4532 raft_consensus.cc:2781] T d0c4430fc56644cb9f87deffe4fe9407 P 83ffd7184ec94302bd1ee74b0fab3e17 [term 2 FOLLOWER]: Leader election won for term 2
I20240503 23:26:35.520488 4532 raft_consensus.cc:686] T d0c4430fc56644cb9f87deffe4fe9407 P 83ffd7184ec94302bd1ee74b0fab3e17 [term 2 LEADER]: Becoming Leader. State: Replica: 83ffd7184ec94302bd1ee74b0fab3e17, State: Running, Role: LEADER
I20240503 23:26:35.520663 3907 raft_consensus.cc:2454] T d0c4430fc56644cb9f87deffe4fe9407 P a3eecbaec43d4954b1096bc3121e824c [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83ffd7184ec94302bd1ee74b0fab3e17 in term 2.
I20240503 23:26:35.520902 4554 ts_tablet_manager.cc:902] T a0e18d0975a54e3dbeb0a13cb226fefd P 692da0319a2d4a81ad81f5d8f9f4c86b: Initiating tablet copy from peer a2f9bf2e5d9b4fc19f804797bca4a3ff (127.0.100.2:46261)
I20240503 23:26:35.520895 4532 consensus_queue.cc:237] T d0c4430fc56644cb9f87deffe4fe9407 P 83ffd7184ec94302bd1ee74b0fab3e17 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 27, Committed index: 27, Last appended: 1.27, Last appended by leader: 27, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "83ffd7184ec94302bd1ee74b0fab3e17" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 32909 } attrs { replace: true } } peers { permanent_uuid: "a2f9bf2e5d9b4fc19f804797bca4a3ff" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 46261 } } peers { permanent_uuid: "a3eecbaec43d4954b1096bc3121e824c" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 40649 } } peers { permanent_uuid: "4d1d4e261584467fbbce4c72fa18045a" member_type: NON_VOTER last_known_addr { host: "127.0.100.6" port: 35349 } attrs { promote: true } }
I20240503 23:26:35.521833 4554 tablet_copy_client.cc:316] T a0e18d0975a54e3dbeb0a13cb226fefd P 692da0319a2d4a81ad81f5d8f9f4c86b: tablet copy: Beginning tablet copy session from remote peer at address 127.0.100.2:46261
I20240503 23:26:35.522490 3848 tablet_copy_service.cc:140] P a2f9bf2e5d9b4fc19f804797bca4a3ff: Received BeginTabletCopySession request for tablet a0e18d0975a54e3dbeb0a13cb226fefd from peer 692da0319a2d4a81ad81f5d8f9f4c86b ({username='slave'} at 127.0.0.1:55456)
I20240503 23:26:35.522693 3848 tablet_copy_service.cc:161] P a2f9bf2e5d9b4fc19f804797bca4a3ff: Beginning new tablet copy session on tablet a0e18d0975a54e3dbeb0a13cb226fefd from peer 692da0319a2d4a81ad81f5d8f9f4c86b at {username='slave'} at 127.0.0.1:55456: session id = 692da0319a2d4a81ad81f5d8f9f4c86b-a0e18d0975a54e3dbeb0a13cb226fefd
I20240503 23:26:35.525532 3848 tablet_copy_source_session.cc:205] T a0e18d0975a54e3dbeb0a13cb226fefd P a2f9bf2e5d9b4fc19f804797bca4a3ff: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 23:26:35.527011 4554 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a0e18d0975a54e3dbeb0a13cb226fefd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 23:26:35.527061 3655 catalog_manager.cc:5484] T d0c4430fc56644cb9f87deffe4fe9407 P 83ffd7184ec94302bd1ee74b0fab3e17 reported cstate change: term changed from 1 to 2, leader changed from a3eecbaec43d4954b1096bc3121e824c (127.0.100.3) to 83ffd7184ec94302bd1ee74b0fab3e17 (127.0.100.1). New cstate: current_term: 2 leader_uuid: "83ffd7184ec94302bd1ee74b0fab3e17" committed_config { opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "83ffd7184ec94302bd1ee74b0fab3e17" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 32909 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a2f9bf2e5d9b4fc19f804797bca4a3ff" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 46261 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a3eecbaec43d4954b1096bc3121e824c" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 40649 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4d1d4e261584467fbbce4c72fa18045a" member_type: NON_VOTER last_known_addr { host: "127.0.100.6" port: 35349 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 23:26:35.533520 4554 tablet_copy_client.cc:791] T a0e18d0975a54e3dbeb0a13cb226fefd P 692da0319a2d4a81ad81f5d8f9f4c86b: tablet copy: Starting download of 0 data blocks...
I20240503 23:26:35.533834 4554 tablet_copy_client.cc:655] T a0e18d0975a54e3dbeb0a13cb226fefd P 692da0319a2d4a81ad81f5d8f9f4c86b: tablet copy: Starting download of 1 WAL segments...
I20240503 23:26:35.535548 4555 tablet_copy_client.cc:1010] T a0e18d0975a54e3dbeb0a13cb226fefd P 692da0319a2d4a81ad81f5d8f9f4c86b: tablet copy: Injecting latency into file download: 2000
I20240503 23:26:35.539443 3762 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "d0c4430fc56644cb9f87deffe4fe9407"
dest_uuid: "83ffd7184ec94302bd1ee74b0fab3e17"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:35576
I20240503 23:26:35.539860 3762 raft_consensus.cc:595] T d0c4430fc56644cb9f87deffe4fe9407 P 83ffd7184ec94302bd1ee74b0fab3e17 [term 2 LEADER]: Received request to transfer leadership
I20240503 23:26:35.540009 4467 raft_consensus.cc:979] T 55cf664d5e874c0d9b9adf8bdc9c43ed P a2f9bf2e5d9b4fc19f804797bca4a3ff: : Instructing follower e4086c099d7f4447b2b4733b6ca364df to start an election
I20240503 23:26:35.540158 4467 raft_consensus.cc:1067] T 55cf664d5e874c0d9b9adf8bdc9c43ed P a2f9bf2e5d9b4fc19f804797bca4a3ff [term 1 LEADER]: Signalling peer e4086c099d7f4447b2b4733b6ca364df to start an election
I20240503 23:26:35.540813 4218 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "55cf664d5e874c0d9b9adf8bdc9c43ed"
dest_uuid: "e4086c099d7f4447b2b4733b6ca364df"
from {username='slave'} at 127.0.0.1:50808
I20240503 23:26:35.541071 4218 raft_consensus.cc:482] T 55cf664d5e874c0d9b9adf8bdc9c43ed P e4086c099d7f4447b2b4733b6ca364df [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 23:26:35.541196 4218 raft_consensus.cc:3037] T 55cf664d5e874c0d9b9adf8bdc9c43ed P e4086c099d7f4447b2b4733b6ca364df [term 1 FOLLOWER]: Advancing to term 2
I20240503 23:26:35.544579 4556 ts_tablet_manager.cc:902] T 98c9500acb43423b86f54ce7a4d98a72 P e4086c099d7f4447b2b4733b6ca364df: Initiating tablet copy from peer 83ffd7184ec94302bd1ee74b0fab3e17 (127.0.100.1:32909)
I20240503 23:26:35.544502 4218 raft_consensus.cc:504] T 55cf664d5e874c0d9b9adf8bdc9c43ed P e4086c099d7f4447b2b4733b6ca364df [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "83ffd7184ec94302bd1ee74b0fab3e17" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 32909 } attrs { replace: true } } peers { permanent_uuid: "a2f9bf2e5d9b4fc19f804797bca4a3ff" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 46261 } } peers { permanent_uuid: "e4086c099d7f4447b2b4733b6ca364df" member_type: VOTER last_known_addr { host: "127.0.100.4" port: 35011 } attrs { promote: false } } peers { permanent_uuid: "4d1d4e261584467fbbce4c72fa18045a" member_type: NON_VOTER last_known_addr { host: "127.0.100.6" port: 35349 } attrs { promote: true } }
I20240503 23:26:35.545847 4218 leader_election.cc:288] T 55cf664d5e874c0d9b9adf8bdc9c43ed P e4086c099d7f4447b2b4733b6ca364df [CANDIDATE]: Term 2 election: Requested vote from peers 83ffd7184ec94302bd1ee74b0fab3e17 (127.0.100.1:32909), a2f9bf2e5d9b4fc19f804797bca4a3ff (127.0.100.2:46261)
I20240503 23:26:35.546168 3837 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "55cf664d5e874c0d9b9adf8bdc9c43ed" candidate_uuid: "e4086c099d7f4447b2b4733b6ca364df" candidate_term: 2 candidate_status { last_received { term: 1 index: 30 } } ignore_live_leader: true dest_uuid: "a2f9bf2e5d9b4fc19f804797bca4a3ff"
I20240503 23:26:35.546612 3762 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "55cf664d5e874c0d9b9adf8bdc9c43ed" candidate_uuid: "e4086c099d7f4447b2b4733b6ca364df" candidate_term: 2 candidate_status { last_received { term: 1 index: 30 } } ignore_live_leader: true dest_uuid: "83ffd7184ec94302bd1ee74b0fab3e17"
I20240503 23:26:35.546608 3837 raft_consensus.cc:3032] T 55cf664d5e874c0d9b9adf8bdc9c43ed P a2f9bf2e5d9b4fc19f804797bca4a3ff [term 1 LEADER]: Stepping down as leader of term 1
I20240503 23:26:35.546929 3837 raft_consensus.cc:729] T 55cf664d5e874c0d9b9adf8bdc9c43ed P a2f9bf2e5d9b4fc19f804797bca4a3ff [term 1 LEADER]: Becoming Follower/Learner. State: Replica: a2f9bf2e5d9b4fc19f804797bca4a3ff, State: Running, Role: LEADER
I20240503 23:26:35.546975 3762 raft_consensus.cc:3037] T 55cf664d5e874c0d9b9adf8bdc9c43ed P 83ffd7184ec94302bd1ee74b0fab3e17 [term 1 FOLLOWER]: Advancing to term 2
I20240503 23:26:35.547267 3837 consensus_queue.cc:260] T 55cf664d5e874c0d9b9adf8bdc9c43ed P a2f9bf2e5d9b4fc19f804797bca4a3ff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 30, Committed index: 30, Last appended: 1.30, Last appended by leader: 30, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "83ffd7184ec94302bd1ee74b0fab3e17" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 32909 } attrs { replace: true } } peers { permanent_uuid: "a2f9bf2e5d9b4fc19f804797bca4a3ff" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 46261 } } peers { permanent_uuid: "e4086c099d7f4447b2b4733b6ca364df" member_type: VOTER last_known_addr { host: "127.0.100.4" port: 35011 } attrs { promote: false } } peers { permanent_uuid: "4d1d4e261584467fbbce4c72fa18045a" member_type: NON_VOTER last_known_addr { host: "127.0.100.6" port: 35349 } attrs { promote: true } }
I20240503 23:26:35.547884 3837 raft_consensus.cc:3037] T 55cf664d5e874c0d9b9adf8bdc9c43ed P a2f9bf2e5d9b4fc19f804797bca4a3ff [term 1 FOLLOWER]: Advancing to term 2
I20240503 23:26:35.549054 4556 tablet_copy_client.cc:316] T 98c9500acb43423b86f54ce7a4d98a72 P e4086c099d7f4447b2b4733b6ca364df: tablet copy: Beginning tablet copy session from remote peer at address 127.0.100.1:32909
I20240503 23:26:35.551324 3761 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "d0c4430fc56644cb9f87deffe4fe9407"
dest_uuid: "83ffd7184ec94302bd1ee74b0fab3e17"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:35576
I20240503 23:26:35.551549 3837 raft_consensus.cc:2454] T 55cf664d5e874c0d9b9adf8bdc9c43ed P a2f9bf2e5d9b4fc19f804797bca4a3ff [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e4086c099d7f4447b2b4733b6ca364df in term 2.
I20240503 23:26:35.551587 3761 raft_consensus.cc:595] T d0c4430fc56644cb9f87deffe4fe9407 P 83ffd7184ec94302bd1ee74b0fab3e17 [term 2 LEADER]: Received request to transfer leadership
I20240503 23:26:35.552037 4184 leader_election.cc:302] T 55cf664d5e874c0d9b9adf8bdc9c43ed P e4086c099d7f4447b2b4733b6ca364df [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a2f9bf2e5d9b4fc19f804797bca4a3ff, e4086c099d7f4447b2b4733b6ca364df; no voters:
I20240503 23:26:35.552340 4530 raft_consensus.cc:2781] T 55cf664d5e874c0d9b9adf8bdc9c43ed P e4086c099d7f4447b2b4733b6ca364df [term 2 FOLLOWER]: Leader election won for term 2
I20240503 23:26:35.552810 3774 tablet_copy_service.cc:140] P 83ffd7184ec94302bd1ee74b0fab3e17: Received BeginTabletCopySession request for tablet 98c9500acb43423b86f54ce7a4d98a72 from peer e4086c099d7f4447b2b4733b6ca364df ({username='slave'} at 127.0.0.1:57918)
I20240503 23:26:35.552994 3774 tablet_copy_service.cc:161] P 83ffd7184ec94302bd1ee74b0fab3e17: Beginning new tablet copy session on tablet 98c9500acb43423b86f54ce7a4d98a72 from peer e4086c099d7f4447b2b4733b6ca364df at {username='slave'} at 127.0.0.1:57918: session id = e4086c099d7f4447b2b4733b6ca364df-98c9500acb43423b86f54ce7a4d98a72
I20240503 23:26:35.553422 4530 raft_consensus.cc:686] T 55cf664d5e874c0d9b9adf8bdc9c43ed P e4086c099d7f4447b2b4733b6ca364df [term 2 LEADER]: Becoming Leader. State: Replica: e4086c099d7f4447b2b4733b6ca364df, State: Running, Role: LEADER
I20240503 23:26:35.553838 4530 consensus_queue.cc:237] T 55cf664d5e874c0d9b9adf8bdc9c43ed P e4086c099d7f4447b2b4733b6ca364df [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 30, Committed index: 30, Last appended: 1.30, Last appended by leader: 30, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "83ffd7184ec94302bd1ee74b0fab3e17" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 32909 } attrs { replace: true } } peers { permanent_uuid: "a2f9bf2e5d9b4fc19f804797bca4a3ff" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 46261 } } peers { permanent_uuid: "e4086c099d7f4447b2b4733b6ca364df" member_type: VOTER last_known_addr { host: "127.0.100.4" port: 35011 } attrs { promote: false } } peers { permanent_uuid: "4d1d4e261584467fbbce4c72fa18045a" member_type: NON_VOTER last_known_addr { host: "127.0.100.6" port: 35349 } attrs { promote: true } }
I20240503 23:26:35.556520 3774 tablet_copy_source_session.cc:205] T 98c9500acb43423b86f54ce7a4d98a72 P 83ffd7184ec94302bd1ee74b0fab3e17: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 23:26:35.556470 3762 raft_consensus.cc:2454] T 55cf664d5e874c0d9b9adf8bdc9c43ed P 83ffd7184ec94302bd1ee74b0fab3e17 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e4086c099d7f4447b2b4733b6ca364df in term 2.
W20240503 23:26:35.556907 3648 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: leadership transfer for d0c4430fc56644cb9f87deffe4fe9407 already in progress [suppressed 3 similar messages]
I20240503 23:26:35.557533 3655 catalog_manager.cc:5484] T 55cf664d5e874c0d9b9adf8bdc9c43ed P e4086c099d7f4447b2b4733b6ca364df reported cstate change: term changed from 1 to 2, leader changed from a2f9bf2e5d9b4fc19f804797bca4a3ff (127.0.100.2) to e4086c099d7f4447b2b4733b6ca364df (127.0.100.4). New cstate: current_term: 2 leader_uuid: "e4086c099d7f4447b2b4733b6ca364df" committed_config { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "83ffd7184ec94302bd1ee74b0fab3e17" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 32909 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a2f9bf2e5d9b4fc19f804797bca4a3ff" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 46261 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e4086c099d7f4447b2b4733b6ca364df" member_type: VOTER last_known_addr { host: "127.0.100.4" port: 35011 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4d1d4e261584467fbbce4c72fa18045a" member_type: NON_VOTER last_known_addr { host: "127.0.100.6" port: 35349 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 23:26:35.559712 4560 ts_tablet_manager.cc:902] T 487ac5b9bf2d4de2b80a1b13387a2a77 P 692da0319a2d4a81ad81f5d8f9f4c86b: Initiating tablet copy from peer 83ffd7184ec94302bd1ee74b0fab3e17 (127.0.100.1:32909)
I20240503 23:26:35.562922 4560 tablet_copy_client.cc:316] T 487ac5b9bf2d4de2b80a1b13387a2a77 P 692da0319a2d4a81ad81f5d8f9f4c86b: tablet copy: Beginning tablet copy session from remote peer at address 127.0.100.1:32909
I20240503 23:26:35.566287 3762 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "d0c4430fc56644cb9f87deffe4fe9407"
dest_uuid: "83ffd7184ec94302bd1ee74b0fab3e17"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:35576
I20240503 23:26:35.566612 3762 raft_consensus.cc:595] T d0c4430fc56644cb9f87deffe4fe9407 P 83ffd7184ec94302bd1ee74b0fab3e17 [term 2 LEADER]: Received request to transfer leadership
W20240503 23:26:35.567273 3710 auto_rebalancer.cc:663] Could not move replica: Remote error: Service unavailable: leadership transfer for d0c4430fc56644cb9f87deffe4fe9407 already in progress
W20240503 23:26:35.567360 3710 auto_rebalancer.cc:264] scheduled replica move failed to complete: Remote error: Service unavailable: leadership transfer for d0c4430fc56644cb9f87deffe4fe9407 already in progress
I20240503 23:26:35.572243 3773 tablet_copy_service.cc:140] P 83ffd7184ec94302bd1ee74b0fab3e17: Received BeginTabletCopySession request for tablet 487ac5b9bf2d4de2b80a1b13387a2a77 from peer 692da0319a2d4a81ad81f5d8f9f4c86b ({username='slave'} at 127.0.0.1:57952)
I20240503 23:26:35.572424 3773 tablet_copy_service.cc:161] P 83ffd7184ec94302bd1ee74b0fab3e17: Beginning new tablet copy session on tablet 487ac5b9bf2d4de2b80a1b13387a2a77 from peer 692da0319a2d4a81ad81f5d8f9f4c86b at {username='slave'} at 127.0.0.1:57952: session id = 692da0319a2d4a81ad81f5d8f9f4c86b-487ac5b9bf2d4de2b80a1b13387a2a77
I20240503 23:26:35.574486 3773 tablet_copy_source_session.cc:205] T 487ac5b9bf2d4de2b80a1b13387a2a77 P 83ffd7184ec94302bd1ee74b0fab3e17: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 23:26:35.575773 4560 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 487ac5b9bf2d4de2b80a1b13387a2a77. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 23:26:35.579362 4556 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 98c9500acb43423b86f54ce7a4d98a72. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 23:26:35.581287 4560 tablet_copy_client.cc:791] T 487ac5b9bf2d4de2b80a1b13387a2a77 P 692da0319a2d4a81ad81f5d8f9f4c86b: tablet copy: Starting download of 0 data blocks...
I20240503 23:26:35.581534 4560 tablet_copy_client.cc:655] T 487ac5b9bf2d4de2b80a1b13387a2a77 P 692da0319a2d4a81ad81f5d8f9f4c86b: tablet copy: Starting download of 1 WAL segments...
I20240503 23:26:35.585430 4561 tablet_copy_client.cc:1010] T 487ac5b9bf2d4de2b80a1b13387a2a77 P 692da0319a2d4a81ad81f5d8f9f4c86b: tablet copy: Injecting latency into file download: 2000
I20240503 23:26:35.586161 4556 tablet_copy_client.cc:791] T 98c9500acb43423b86f54ce7a4d98a72 P e4086c099d7f4447b2b4733b6ca364df: tablet copy: Starting download of 0 data blocks...
I20240503 23:26:35.586504 4556 tablet_copy_client.cc:655] T 98c9500acb43423b86f54ce7a4d98a72 P e4086c099d7f4447b2b4733b6ca364df: tablet copy: Starting download of 1 WAL segments...
I20240503 23:26:35.588328 4557 tablet_copy_client.cc:1010] T 98c9500acb43423b86f54ce7a4d98a72 P e4086c099d7f4447b2b4733b6ca364df: tablet copy: Injecting latency into file download: 2000
I20240503 23:26:35.637424 4534 raft_consensus.cc:979] T 487ac5b9bf2d4de2b80a1b13387a2a77 P 83ffd7184ec94302bd1ee74b0fab3e17: : Instructing follower a2f9bf2e5d9b4fc19f804797bca4a3ff to start an election
I20240503 23:26:35.637607 4534 raft_consensus.cc:1067] T 487ac5b9bf2d4de2b80a1b13387a2a77 P 83ffd7184ec94302bd1ee74b0fab3e17 [term 1 LEADER]: Signalling peer a2f9bf2e5d9b4fc19f804797bca4a3ff to start an election
I20240503 23:26:35.638186 3837 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "487ac5b9bf2d4de2b80a1b13387a2a77"
dest_uuid: "a2f9bf2e5d9b4fc19f804797bca4a3ff"
from {username='slave'} at 127.0.0.1:45370
I20240503 23:26:35.638401 3837 raft_consensus.cc:482] T 487ac5b9bf2d4de2b80a1b13387a2a77 P a2f9bf2e5d9b4fc19f804797bca4a3ff [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 23:26:35.638527 3837 raft_consensus.cc:3037] T 487ac5b9bf2d4de2b80a1b13387a2a77 P a2f9bf2e5d9b4fc19f804797bca4a3ff [term 1 FOLLOWER]: Advancing to term 2
I20240503 23:26:35.641086 3837 raft_consensus.cc:504] T 487ac5b9bf2d4de2b80a1b13387a2a77 P a2f9bf2e5d9b4fc19f804797bca4a3ff [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "83ffd7184ec94302bd1ee74b0fab3e17" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 32909 } } peers { permanent_uuid: "a2f9bf2e5d9b4fc19f804797bca4a3ff" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 46261 } } peers { permanent_uuid: "a3eecbaec43d4954b1096bc3121e824c" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 40649 } attrs { replace: true } } peers { permanent_uuid: "692da0319a2d4a81ad81f5d8f9f4c86b" member_type: NON_VOTER last_known_addr { host: "127.0.100.5" port: 38561 } attrs { promote: true } }
I20240503 23:26:35.641839 3837 leader_election.cc:288] T 487ac5b9bf2d4de2b80a1b13387a2a77 P a2f9bf2e5d9b4fc19f804797bca4a3ff [CANDIDATE]: Term 2 election: Requested vote from peers 83ffd7184ec94302bd1ee74b0fab3e17 (127.0.100.1:32909), a3eecbaec43d4954b1096bc3121e824c (127.0.100.3:40649)
I20240503 23:26:35.642382 3762 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "487ac5b9bf2d4de2b80a1b13387a2a77" candidate_uuid: "a2f9bf2e5d9b4fc19f804797bca4a3ff" candidate_term: 2 candidate_status { last_received { term: 1 index: 27 } } ignore_live_leader: true dest_uuid: "83ffd7184ec94302bd1ee74b0fab3e17"
I20240503 23:26:35.642565 3762 raft_consensus.cc:3032] T 487ac5b9bf2d4de2b80a1b13387a2a77 P 83ffd7184ec94302bd1ee74b0fab3e17 [term 1 LEADER]: Stepping down as leader of term 1
I20240503 23:26:35.642505 3907 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "487ac5b9bf2d4de2b80a1b13387a2a77" candidate_uuid: "a2f9bf2e5d9b4fc19f804797bca4a3ff" candidate_term: 2 candidate_status { last_received { term: 1 index: 27 } } ignore_live_leader: true dest_uuid: "a3eecbaec43d4954b1096bc3121e824c"
I20240503 23:26:35.642642 3762 raft_consensus.cc:729] T 487ac5b9bf2d4de2b80a1b13387a2a77 P 83ffd7184ec94302bd1ee74b0fab3e17 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 83ffd7184ec94302bd1ee74b0fab3e17, State: Running, Role: LEADER
I20240503 23:26:35.642704 3907 raft_consensus.cc:3037] T 487ac5b9bf2d4de2b80a1b13387a2a77 P a3eecbaec43d4954b1096bc3121e824c [term 1 FOLLOWER]: Advancing to term 2
I20240503 23:26:35.642966 3762 consensus_queue.cc:260] T 487ac5b9bf2d4de2b80a1b13387a2a77 P 83ffd7184ec94302bd1ee74b0fab3e17 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 27, Committed index: 27, Last appended: 1.27, Last appended by leader: 27, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "83ffd7184ec94302bd1ee74b0fab3e17" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 32909 } } peers { permanent_uuid: "a2f9bf2e5d9b4fc19f804797bca4a3ff" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 46261 } } peers { permanent_uuid: "a3eecbaec43d4954b1096bc3121e824c" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 40649 } attrs { replace: true } } peers { permanent_uuid: "692da0319a2d4a81ad81f5d8f9f4c86b" member_type: NON_VOTER last_known_addr { host: "127.0.100.5" port: 38561 } attrs { promote: true } }
I20240503 23:26:35.643368 3762 raft_consensus.cc:3037] T 487ac5b9bf2d4de2b80a1b13387a2a77 P 83ffd7184ec94302bd1ee74b0fab3e17 [term 1 FOLLOWER]: Advancing to term 2
I20240503 23:26:35.645098 3907 raft_consensus.cc:2454] T 487ac5b9bf2d4de2b80a1b13387a2a77 P a3eecbaec43d4954b1096bc3121e824c [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2f9bf2e5d9b4fc19f804797bca4a3ff in term 2.
I20240503 23:26:35.645349 3762 raft_consensus.cc:2454] T 487ac5b9bf2d4de2b80a1b13387a2a77 P 83ffd7184ec94302bd1ee74b0fab3e17 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2f9bf2e5d9b4fc19f804797bca4a3ff in term 2.
I20240503 23:26:35.645638 3802 leader_election.cc:302] T 487ac5b9bf2d4de2b80a1b13387a2a77 P a2f9bf2e5d9b4fc19f804797bca4a3ff [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a2f9bf2e5d9b4fc19f804797bca4a3ff, a3eecbaec43d4954b1096bc3121e824c; no voters:
I20240503 23:26:35.645985 4518 raft_consensus.cc:2781] T 487ac5b9bf2d4de2b80a1b13387a2a77 P a2f9bf2e5d9b4fc19f804797bca4a3ff [term 2 FOLLOWER]: Leader election won for term 2
I20240503 23:26:35.646124 4518 raft_consensus.cc:686] T 487ac5b9bf2d4de2b80a1b13387a2a77 P a2f9bf2e5d9b4fc19f804797bca4a3ff [term 2 LEADER]: Becoming Leader. State: Replica: a2f9bf2e5d9b4fc19f804797bca4a3ff, State: Running, Role: LEADER
I20240503 23:26:35.646369 4518 consensus_queue.cc:237] T 487ac5b9bf2d4de2b80a1b13387a2a77 P a2f9bf2e5d9b4fc19f804797bca4a3ff [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 27, Committed index: 27, Last appended: 1.27, Last appended by leader: 27, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "83ffd7184ec94302bd1ee74b0fab3e17" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 32909 } } peers { permanent_uuid: "a2f9bf2e5d9b4fc19f804797bca4a3ff" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 46261 } } peers { permanent_uuid: "a3eecbaec43d4954b1096bc3121e824c" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 40649 } attrs { replace: true } } peers { permanent_uuid: "692da0319a2d4a81ad81f5d8f9f4c86b" member_type: NON_VOTER last_known_addr { host: "127.0.100.5" port: 38561 } attrs { promote: true } }
I20240503 23:26:35.649395 3655 catalog_manager.cc:5484] T 487ac5b9bf2d4de2b80a1b13387a2a77 P a2f9bf2e5d9b4fc19f804797bca4a3ff reported cstate change: term changed from 1 to 2, leader changed from 83ffd7184ec94302bd1ee74b0fab3e17 (127.0.100.1) to a2f9bf2e5d9b4fc19f804797bca4a3ff (127.0.100.2). New cstate: current_term: 2 leader_uuid: "a2f9bf2e5d9b4fc19f804797bca4a3ff" committed_config { opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "83ffd7184ec94302bd1ee74b0fab3e17" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 32909 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a2f9bf2e5d9b4fc19f804797bca4a3ff" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 46261 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a3eecbaec43d4954b1096bc3121e824c" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 40649 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "692da0319a2d4a81ad81f5d8f9f4c86b" member_type: NON_VOTER last_known_addr { host: "127.0.100.5" port: 38561 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 23:26:35.661823 4532 raft_consensus.cc:979] T a14a12115b8d4fec947c6e223f1ef486 P 83ffd7184ec94302bd1ee74b0fab3e17: : Instructing follower 4d1d4e261584467fbbce4c72fa18045a to start an election
I20240503 23:26:35.662055 4534 raft_consensus.cc:1067] T a14a12115b8d4fec947c6e223f1ef486 P 83ffd7184ec94302bd1ee74b0fab3e17 [term 1 LEADER]: Signalling peer 4d1d4e261584467fbbce4c72fa18045a to start an election
I20240503 23:26:35.662731 4367 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "a14a12115b8d4fec947c6e223f1ef486"
dest_uuid: "4d1d4e261584467fbbce4c72fa18045a"
from {username='slave'} at 127.0.0.1:47554
I20240503 23:26:35.662952 4367 raft_consensus.cc:482] T a14a12115b8d4fec947c6e223f1ef486 P 4d1d4e261584467fbbce4c72fa18045a [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 23:26:35.663085 4367 raft_consensus.cc:3037] T a14a12115b8d4fec947c6e223f1ef486 P 4d1d4e261584467fbbce4c72fa18045a [term 1 FOLLOWER]: Advancing to term 2
I20240503 23:26:35.665949 4367 raft_consensus.cc:504] T a14a12115b8d4fec947c6e223f1ef486 P 4d1d4e261584467fbbce4c72fa18045a [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "83ffd7184ec94302bd1ee74b0fab3e17" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 32909 } } peers { permanent_uuid: "a3eecbaec43d4954b1096bc3121e824c" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 40649 } } peers { permanent_uuid: "4d1d4e261584467fbbce4c72fa18045a" member_type: VOTER last_known_addr { host: "127.0.100.6" port: 35349 } attrs { promote: false } }
I20240503 23:26:35.666967 4367 leader_election.cc:288] T a14a12115b8d4fec947c6e223f1ef486 P 4d1d4e261584467fbbce4c72fa18045a [CANDIDATE]: Term 2 election: Requested vote from peers 83ffd7184ec94302bd1ee74b0fab3e17 (127.0.100.1:32909), a3eecbaec43d4954b1096bc3121e824c (127.0.100.3:40649)
I20240503 23:26:35.667965 3762 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a14a12115b8d4fec947c6e223f1ef486" candidate_uuid: "4d1d4e261584467fbbce4c72fa18045a" candidate_term: 2 candidate_status { last_received { term: 1 index: 29 } } ignore_live_leader: true dest_uuid: "83ffd7184ec94302bd1ee74b0fab3e17"
I20240503 23:26:35.668175 3762 raft_consensus.cc:3032] T a14a12115b8d4fec947c6e223f1ef486 P 83ffd7184ec94302bd1ee74b0fab3e17 [term 1 LEADER]: Stepping down as leader of term 1
I20240503 23:26:35.668247 3762 raft_consensus.cc:729] T a14a12115b8d4fec947c6e223f1ef486 P 83ffd7184ec94302bd1ee74b0fab3e17 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 83ffd7184ec94302bd1ee74b0fab3e17, State: Running, Role: LEADER
I20240503 23:26:35.668578 3762 consensus_queue.cc:260] T a14a12115b8d4fec947c6e223f1ef486 P 83ffd7184ec94302bd1ee74b0fab3e17 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 29, Committed index: 29, Last appended: 1.29, Last appended by leader: 29, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "83ffd7184ec94302bd1ee74b0fab3e17" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 32909 } } peers { permanent_uuid: "a3eecbaec43d4954b1096bc3121e824c" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 40649 } } peers { permanent_uuid: "4d1d4e261584467fbbce4c72fa18045a" member_type: VOTER last_known_addr { host: "127.0.100.6" port: 35349 } attrs { promote: false } }
I20240503 23:26:35.669024 3762 raft_consensus.cc:3037] T a14a12115b8d4fec947c6e223f1ef486 P 83ffd7184ec94302bd1ee74b0fab3e17 [term 1 FOLLOWER]: Advancing to term 2
I20240503 23:26:35.671384 3762 raft_consensus.cc:2454] T a14a12115b8d4fec947c6e223f1ef486 P 83ffd7184ec94302bd1ee74b0fab3e17 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d1d4e261584467fbbce4c72fa18045a in term 2.
I20240503 23:26:35.671748 4332 leader_election.cc:302] T a14a12115b8d4fec947c6e223f1ef486 P 4d1d4e261584467fbbce4c72fa18045a [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d1d4e261584467fbbce4c72fa18045a, 83ffd7184ec94302bd1ee74b0fab3e17; no voters:
I20240503 23:26:35.672425 4566 raft_consensus.cc:2781] T a14a12115b8d4fec947c6e223f1ef486 P 4d1d4e261584467fbbce4c72fa18045a [term 2 FOLLOWER]: Leader election won for term 2
I20240503 23:26:35.672899 4566 raft_consensus.cc:686] T a14a12115b8d4fec947c6e223f1ef486 P 4d1d4e261584467fbbce4c72fa18045a [term 2 LEADER]: Becoming Leader. State: Replica: 4d1d4e261584467fbbce4c72fa18045a, State: Running, Role: LEADER
I20240503 23:26:35.673260 4566 consensus_queue.cc:237] T a14a12115b8d4fec947c6e223f1ef486 P 4d1d4e261584467fbbce4c72fa18045a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 29, Committed index: 29, Last appended: 1.29, Last appended by leader: 29, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "83ffd7184ec94302bd1ee74b0fab3e17" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 32909 } } peers { permanent_uuid: "a3eecbaec43d4954b1096bc3121e824c" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 40649 } } peers { permanent_uuid: "4d1d4e261584467fbbce4c72fa18045a" member_type: VOTER last_known_addr { host: "127.0.100.6" port: 35349 } attrs { promote: false } }
I20240503 23:26:35.673799 3908 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a14a12115b8d4fec947c6e223f1ef486" candidate_uuid: "4d1d4e261584467fbbce4c72fa18045a" candidate_term: 2 candidate_status { last_received { term: 1 index: 29 } } ignore_live_leader: true dest_uuid: "a3eecbaec43d4954b1096bc3121e824c"
I20240503 23:26:35.673990 3908 raft_consensus.cc:3037] T a14a12115b8d4fec947c6e223f1ef486 P a3eecbaec43d4954b1096bc3121e824c [term 1 FOLLOWER]: Advancing to term 2
I20240503 23:26:35.675855 3655 catalog_manager.cc:5484] T a14a12115b8d4fec947c6e223f1ef486 P 4d1d4e261584467fbbce4c72fa18045a reported cstate change: term changed from 1 to 2, leader changed from 83ffd7184ec94302bd1ee74b0fab3e17 (127.0.100.1) to 4d1d4e261584467fbbce4c72fa18045a (127.0.100.6). New cstate: current_term: 2 leader_uuid: "4d1d4e261584467fbbce4c72fa18045a" committed_config { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "83ffd7184ec94302bd1ee74b0fab3e17" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 32909 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a3eecbaec43d4954b1096bc3121e824c" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 40649 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4d1d4e261584467fbbce4c72fa18045a" member_type: VOTER last_known_addr { host: "127.0.100.6" port: 35349 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 23:26:35.676479 3908 raft_consensus.cc:2454] T a14a12115b8d4fec947c6e223f1ef486 P a3eecbaec43d4954b1096bc3121e824c [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d1d4e261584467fbbce4c72fa18045a in term 2.
I20240503 23:26:35.716408 4510 raft_consensus.cc:979] T 481cd0f8c8d147adac217915a53ff6d9 P a3eecbaec43d4954b1096bc3121e824c: : Instructing follower 83ffd7184ec94302bd1ee74b0fab3e17 to start an election
I20240503 23:26:35.716638 4426 raft_consensus.cc:1067] T 481cd0f8c8d147adac217915a53ff6d9 P a3eecbaec43d4954b1096bc3121e824c [term 1 LEADER]: Signalling peer 83ffd7184ec94302bd1ee74b0fab3e17 to start an election
I20240503 23:26:35.717278 3762 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "481cd0f8c8d147adac217915a53ff6d9"
dest_uuid: "83ffd7184ec94302bd1ee74b0fab3e17"
from {username='slave'} at 127.0.0.1:35592
I20240503 23:26:35.717500 3762 raft_consensus.cc:482] T 481cd0f8c8d147adac217915a53ff6d9 P 83ffd7184ec94302bd1ee74b0fab3e17 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 23:26:35.717618 3762 raft_consensus.cc:3037] T 481cd0f8c8d147adac217915a53ff6d9 P 83ffd7184ec94302bd1ee74b0fab3e17 [term 1 FOLLOWER]: Advancing to term 2
I20240503 23:26:35.720582 3762 raft_consensus.cc:504] T 481cd0f8c8d147adac217915a53ff6d9 P 83ffd7184ec94302bd1ee74b0fab3e17 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83ffd7184ec94302bd1ee74b0fab3e17" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 32909 } } peers { permanent_uuid: "a2f9bf2e5d9b4fc19f804797bca4a3ff" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 46261 } } peers { permanent_uuid: "a3eecbaec43d4954b1096bc3121e824c" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 40649 } }
I20240503 23:26:35.721357 3762 leader_election.cc:288] T 481cd0f8c8d147adac217915a53ff6d9 P 83ffd7184ec94302bd1ee74b0fab3e17 [CANDIDATE]: Term 2 election: Requested vote from peers a2f9bf2e5d9b4fc19f804797bca4a3ff (127.0.100.2:46261), a3eecbaec43d4954b1096bc3121e824c (127.0.100.3:40649)
I20240503 23:26:35.721683 3908 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "481cd0f8c8d147adac217915a53ff6d9" candidate_uuid: "83ffd7184ec94302bd1ee74b0fab3e17" candidate_term: 2 candidate_status { last_received { term: 1 index: 26 } } ignore_live_leader: true dest_uuid: "a3eecbaec43d4954b1096bc3121e824c"
I20240503 23:26:35.721712 3838 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "481cd0f8c8d147adac217915a53ff6d9" candidate_uuid: "83ffd7184ec94302bd1ee74b0fab3e17" candidate_term: 2 candidate_status { last_received { term: 1 index: 26 } } ignore_live_leader: true dest_uuid: "a2f9bf2e5d9b4fc19f804797bca4a3ff"
I20240503 23:26:35.721906 3908 raft_consensus.cc:3032] T 481cd0f8c8d147adac217915a53ff6d9 P a3eecbaec43d4954b1096bc3121e824c [term 1 LEADER]: Stepping down as leader of term 1
I20240503 23:26:35.721906 3838 raft_consensus.cc:3037] T 481cd0f8c8d147adac217915a53ff6d9 P a2f9bf2e5d9b4fc19f804797bca4a3ff [term 1 FOLLOWER]: Advancing to term 2
I20240503 23:26:35.722000 3908 raft_consensus.cc:729] T 481cd0f8c8d147adac217915a53ff6d9 P a3eecbaec43d4954b1096bc3121e824c [term 1 LEADER]: Becoming Follower/Learner. State: Replica: a3eecbaec43d4954b1096bc3121e824c, State: Running, Role: LEADER
I20240503 23:26:35.722272 3908 consensus_queue.cc:260] T 481cd0f8c8d147adac217915a53ff6d9 P a3eecbaec43d4954b1096bc3121e824c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 26, Committed index: 26, Last appended: 1.26, Last appended by leader: 26, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83ffd7184ec94302bd1ee74b0fab3e17" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 32909 } } peers { permanent_uuid: "a2f9bf2e5d9b4fc19f804797bca4a3ff" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 46261 } } peers { permanent_uuid: "a3eecbaec43d4954b1096bc3121e824c" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 40649 } }
I20240503 23:26:35.722599 3908 raft_consensus.cc:3037] T 481cd0f8c8d147adac217915a53ff6d9 P a3eecbaec43d4954b1096bc3121e824c [term 1 FOLLOWER]: Advancing to term 2
I20240503 23:26:35.725142 3838 raft_consensus.cc:2454] T 481cd0f8c8d147adac217915a53ff6d9 P a2f9bf2e5d9b4fc19f804797bca4a3ff [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83ffd7184ec94302bd1ee74b0fab3e17 in term 2.
I20240503 23:26:35.725142 3908 raft_consensus.cc:2454] T 481cd0f8c8d147adac217915a53ff6d9 P a3eecbaec43d4954b1096bc3121e824c [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83ffd7184ec94302bd1ee74b0fab3e17 in term 2.
I20240503 23:26:35.725663 3730 leader_election.cc:302] T 481cd0f8c8d147adac217915a53ff6d9 P 83ffd7184ec94302bd1ee74b0fab3e17 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 83ffd7184ec94302bd1ee74b0fab3e17, a2f9bf2e5d9b4fc19f804797bca4a3ff; no voters:
I20240503 23:26:35.725972 4569 raft_consensus.cc:2781] T 481cd0f8c8d147adac217915a53ff6d9 P 83ffd7184ec94302bd1ee74b0fab3e17 [term 2 FOLLOWER]: Leader election won for term 2
I20240503 23:26:35.726131 4569 raft_consensus.cc:686] T 481cd0f8c8d147adac217915a53ff6d9 P 83ffd7184ec94302bd1ee74b0fab3e17 [term 2 LEADER]: Becoming Leader. State: Replica: 83ffd7184ec94302bd1ee74b0fab3e17, State: Running, Role: LEADER
I20240503 23:26:35.726377 4569 consensus_queue.cc:237] T 481cd0f8c8d147adac217915a53ff6d9 P 83ffd7184ec94302bd1ee74b0fab3e17 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 26, Committed index: 26, Last appended: 1.26, Last appended by leader: 26, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83ffd7184ec94302bd1ee74b0fab3e17" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 32909 } } peers { permanent_uuid: "a2f9bf2e5d9b4fc19f804797bca4a3ff" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 46261 } } peers { permanent_uuid: "a3eecbaec43d4954b1096bc3121e824c" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 40649 } }
I20240503 23:26:35.729897 3655 catalog_manager.cc:5484] T 481cd0f8c8d147adac217915a53ff6d9 P 83ffd7184ec94302bd1ee74b0fab3e17 reported cstate change: term changed from 1 to 2, leader changed from a3eecbaec43d4954b1096bc3121e824c (127.0.100.3) to 83ffd7184ec94302bd1ee74b0fab3e17 (127.0.100.1). New cstate: current_term: 2 leader_uuid: "83ffd7184ec94302bd1ee74b0fab3e17" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83ffd7184ec94302bd1ee74b0fab3e17" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 32909 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a2f9bf2e5d9b4fc19f804797bca4a3ff" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 46261 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a3eecbaec43d4954b1096bc3121e824c" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 40649 } health_report { overall_health: UNKNOWN } } }
I20240503 23:26:35.789254 4467 raft_consensus.cc:979] T 1b9566be6c644cfca852ec6507fab893 P a2f9bf2e5d9b4fc19f804797bca4a3ff: : Instructing follower 692da0319a2d4a81ad81f5d8f9f4c86b to start an election
I20240503 23:26:35.789469 4467 raft_consensus.cc:1067] T 1b9566be6c644cfca852ec6507fab893 P a2f9bf2e5d9b4fc19f804797bca4a3ff [term 1 LEADER]: Signalling peer 692da0319a2d4a81ad81f5d8f9f4c86b to start an election
I20240503 23:26:35.790004 4293 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "1b9566be6c644cfca852ec6507fab893"
dest_uuid: "692da0319a2d4a81ad81f5d8f9f4c86b"
from {username='slave'} at 127.0.0.1:56164
I20240503 23:26:35.790206 4293 raft_consensus.cc:482] T 1b9566be6c644cfca852ec6507fab893 P 692da0319a2d4a81ad81f5d8f9f4c86b [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 23:26:35.790311 4293 raft_consensus.cc:3037] T 1b9566be6c644cfca852ec6507fab893 P 692da0319a2d4a81ad81f5d8f9f4c86b [term 1 FOLLOWER]: Advancing to term 2
I20240503 23:26:35.793212 4293 raft_consensus.cc:504] T 1b9566be6c644cfca852ec6507fab893 P 692da0319a2d4a81ad81f5d8f9f4c86b [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "a2f9bf2e5d9b4fc19f804797bca4a3ff" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 46261 } } peers { permanent_uuid: "a3eecbaec43d4954b1096bc3121e824c" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 40649 } } peers { permanent_uuid: "692da0319a2d4a81ad81f5d8f9f4c86b" member_type: VOTER last_known_addr { host: "127.0.100.5" port: 38561 } attrs { promote: false } }
I20240503 23:26:35.793983 4293 leader_election.cc:288] T 1b9566be6c644cfca852ec6507fab893 P 692da0319a2d4a81ad81f5d8f9f4c86b [CANDIDATE]: Term 2 election: Requested vote from peers a2f9bf2e5d9b4fc19f804797bca4a3ff (127.0.100.2:46261), a3eecbaec43d4954b1096bc3121e824c (127.0.100.3:40649)
I20240503 23:26:35.794317 3837 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b9566be6c644cfca852ec6507fab893" candidate_uuid: "692da0319a2d4a81ad81f5d8f9f4c86b" candidate_term: 2 candidate_status { last_received { term: 1 index: 29 } } ignore_live_leader: true dest_uuid: "a2f9bf2e5d9b4fc19f804797bca4a3ff"
I20240503 23:26:35.794402 3907 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b9566be6c644cfca852ec6507fab893" candidate_uuid: "692da0319a2d4a81ad81f5d8f9f4c86b" candidate_term: 2 candidate_status { last_received { term: 1 index: 29 } } ignore_live_leader: true dest_uuid: "a3eecbaec43d4954b1096bc3121e824c"
I20240503 23:26:35.794535 3907 raft_consensus.cc:3037] T 1b9566be6c644cfca852ec6507fab893 P a3eecbaec43d4954b1096bc3121e824c [term 1 FOLLOWER]: Advancing to term 2
I20240503 23:26:35.794557 3837 raft_consensus.cc:3032] T 1b9566be6c644cfca852ec6507fab893 P a2f9bf2e5d9b4fc19f804797bca4a3ff [term 1 LEADER]: Stepping down as leader of term 1
I20240503 23:26:35.794617 3837 raft_consensus.cc:729] T 1b9566be6c644cfca852ec6507fab893 P a2f9bf2e5d9b4fc19f804797bca4a3ff [term 1 LEADER]: Becoming Follower/Learner. State: Replica: a2f9bf2e5d9b4fc19f804797bca4a3ff, State: Running, Role: LEADER
I20240503 23:26:35.794894 3837 consensus_queue.cc:260] T 1b9566be6c644cfca852ec6507fab893 P a2f9bf2e5d9b4fc19f804797bca4a3ff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 29, Committed index: 29, Last appended: 1.29, Last appended by leader: 29, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "a2f9bf2e5d9b4fc19f804797bca4a3ff" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 46261 } } peers { permanent_uuid: "a3eecbaec43d4954b1096bc3121e824c" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 40649 } } peers { permanent_uuid: "692da0319a2d4a81ad81f5d8f9f4c86b" member_type: VOTER last_known_addr { host: "127.0.100.5" port: 38561 } attrs { promote: false } }
I20240503 23:26:35.795248 3837 raft_consensus.cc:3037] T 1b9566be6c644cfca852ec6507fab893 P a2f9bf2e5d9b4fc19f804797bca4a3ff [term 1 FOLLOWER]: Advancing to term 2
I20240503 23:26:35.797849 3907 raft_consensus.cc:2454] T 1b9566be6c644cfca852ec6507fab893 P a3eecbaec43d4954b1096bc3121e824c [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 692da0319a2d4a81ad81f5d8f9f4c86b in term 2.
I20240503 23:26:35.797981 3837 raft_consensus.cc:2454] T 1b9566be6c644cfca852ec6507fab893 P a2f9bf2e5d9b4fc19f804797bca4a3ff [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 692da0319a2d4a81ad81f5d8f9f4c86b in term 2.
I20240503 23:26:35.798337 4256 leader_election.cc:302] T 1b9566be6c644cfca852ec6507fab893 P 692da0319a2d4a81ad81f5d8f9f4c86b [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 692da0319a2d4a81ad81f5d8f9f4c86b, a3eecbaec43d4954b1096bc3121e824c; no voters:
I20240503 23:26:35.798995 4571 raft_consensus.cc:2781] T 1b9566be6c644cfca852ec6507fab893 P 692da0319a2d4a81ad81f5d8f9f4c86b [term 2 FOLLOWER]: Leader election won for term 2
I20240503 23:26:35.799466 4571 raft_consensus.cc:686] T 1b9566be6c644cfca852ec6507fab893 P 692da0319a2d4a81ad81f5d8f9f4c86b [term 2 LEADER]: Becoming Leader. State: Replica: 692da0319a2d4a81ad81f5d8f9f4c86b, State: Running, Role: LEADER
I20240503 23:26:35.799713 4571 consensus_queue.cc:237] T 1b9566be6c644cfca852ec6507fab893 P 692da0319a2d4a81ad81f5d8f9f4c86b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 29, Committed index: 29, Last appended: 1.29, Last appended by leader: 29, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "a2f9bf2e5d9b4fc19f804797bca4a3ff" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 46261 } } peers { permanent_uuid: "a3eecbaec43d4954b1096bc3121e824c" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 40649 } } peers { permanent_uuid: "692da0319a2d4a81ad81f5d8f9f4c86b" member_type: VOTER last_known_addr { host: "127.0.100.5" port: 38561 } attrs { promote: false } }
I20240503 23:26:35.802388 3655 catalog_manager.cc:5484] T 1b9566be6c644cfca852ec6507fab893 P 692da0319a2d4a81ad81f5d8f9f4c86b reported cstate change: term changed from 1 to 2, leader changed from a2f9bf2e5d9b4fc19f804797bca4a3ff (127.0.100.2) to 692da0319a2d4a81ad81f5d8f9f4c86b (127.0.100.5). New cstate: current_term: 2 leader_uuid: "692da0319a2d4a81ad81f5d8f9f4c86b" committed_config { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "a2f9bf2e5d9b4fc19f804797bca4a3ff" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 46261 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a3eecbaec43d4954b1096bc3121e824c" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 40649 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "692da0319a2d4a81ad81f5d8f9f4c86b" member_type: VOTER last_known_addr { host: "127.0.100.5" port: 38561 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 23:26:35.921095 3837 raft_consensus.cc:1261] T d0c4430fc56644cb9f87deffe4fe9407 P a2f9bf2e5d9b4fc19f804797bca4a3ff [term 2 FOLLOWER]: Refusing update from remote peer 83ffd7184ec94302bd1ee74b0fab3e17: Log matching property violated. Preceding OpId in replica: term: 1 index: 27. Preceding OpId from leader: term: 2 index: 28. (index mismatch)
I20240503 23:26:35.921769 4569 consensus_queue.cc:1035] T d0c4430fc56644cb9f87deffe4fe9407 P 83ffd7184ec94302bd1ee74b0fab3e17 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2f9bf2e5d9b4fc19f804797bca4a3ff" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 46261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 28, Last known committed idx: 27, Time since last communication: 0.000s
I20240503 23:26:35.924041 4532 raft_consensus.cc:979] T d0c4430fc56644cb9f87deffe4fe9407 P 83ffd7184ec94302bd1ee74b0fab3e17: : Instructing follower a2f9bf2e5d9b4fc19f804797bca4a3ff to start an election
I20240503 23:26:35.924214 4534 raft_consensus.cc:1067] T d0c4430fc56644cb9f87deffe4fe9407 P 83ffd7184ec94302bd1ee74b0fab3e17 [term 2 LEADER]: Signalling peer a2f9bf2e5d9b4fc19f804797bca4a3ff to start an election
W20240503 23:26:35.927562 3728 consensus_peers.cc:485] T d0c4430fc56644cb9f87deffe4fe9407 P 83ffd7184ec94302bd1ee74b0fab3e17 -> Peer 4d1d4e261584467fbbce4c72fa18045a (127.0.100.6:35349): Couldn't send request to peer 4d1d4e261584467fbbce4c72fa18045a. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d0c4430fc56644cb9f87deffe4fe9407. This is attempt 1: this message will repeat every 5th retry.
I20240503 23:26:35.928596 3837 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "d0c4430fc56644cb9f87deffe4fe9407"
dest_uuid: "a2f9bf2e5d9b4fc19f804797bca4a3ff"
from {username='slave'} at 127.0.0.1:45370
I20240503 23:26:35.928840 3837 raft_consensus.cc:482] T d0c4430fc56644cb9f87deffe4fe9407 P a2f9bf2e5d9b4fc19f804797bca4a3ff [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 23:26:35.928941 3837 raft_consensus.cc:3037] T d0c4430fc56644cb9f87deffe4fe9407 P a2f9bf2e5d9b4fc19f804797bca4a3ff [term 2 FOLLOWER]: Advancing to term 3
I20240503 23:26:35.929003 3907 raft_consensus.cc:1261] T d0c4430fc56644cb9f87deffe4fe9407 P a3eecbaec43d4954b1096bc3121e824c [term 2 FOLLOWER]: Refusing update from remote peer 83ffd7184ec94302bd1ee74b0fab3e17: Log matching property violated. Preceding OpId in replica: term: 1 index: 27. Preceding OpId from leader: term: 2 index: 28. (index mismatch)
I20240503 23:26:35.931984 3837 raft_consensus.cc:504] T d0c4430fc56644cb9f87deffe4fe9407 P a2f9bf2e5d9b4fc19f804797bca4a3ff [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "83ffd7184ec94302bd1ee74b0fab3e17" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 32909 } attrs { replace: true } } peers { permanent_uuid: "a2f9bf2e5d9b4fc19f804797bca4a3ff" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 46261 } } peers { permanent_uuid: "a3eecbaec43d4954b1096bc3121e824c" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 40649 } } peers { permanent_uuid: "4d1d4e261584467fbbce4c72fa18045a" member_type: NON_VOTER last_known_addr { host: "127.0.100.6" port: 35349 } attrs { promote: true } }
I20240503 23:26:35.932897 4569 consensus_queue.cc:1035] T d0c4430fc56644cb9f87deffe4fe9407 P 83ffd7184ec94302bd1ee74b0fab3e17 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a3eecbaec43d4954b1096bc3121e824c" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 40649 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 28, Last known committed idx: 27, Time since last communication: 0.000s
I20240503 23:26:35.934100 3837 leader_election.cc:288] T d0c4430fc56644cb9f87deffe4fe9407 P a2f9bf2e5d9b4fc19f804797bca4a3ff [CANDIDATE]: Term 3 election: Requested vote from peers 83ffd7184ec94302bd1ee74b0fab3e17 (127.0.100.1:32909), a3eecbaec43d4954b1096bc3121e824c (127.0.100.3:40649)
I20240503 23:26:35.934778 3908 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0c4430fc56644cb9f87deffe4fe9407" candidate_uuid: "a2f9bf2e5d9b4fc19f804797bca4a3ff" candidate_term: 3 candidate_status { last_received { term: 2 index: 28 } } ignore_live_leader: true dest_uuid: "a3eecbaec43d4954b1096bc3121e824c"
I20240503 23:26:35.934609 3762 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0c4430fc56644cb9f87deffe4fe9407" candidate_uuid: "a2f9bf2e5d9b4fc19f804797bca4a3ff" candidate_term: 3 candidate_status { last_received { term: 2 index: 28 } } ignore_live_leader: true dest_uuid: "83ffd7184ec94302bd1ee74b0fab3e17"
I20240503 23:26:35.935317 3762 raft_consensus.cc:3032] T d0c4430fc56644cb9f87deffe4fe9407 P 83ffd7184ec94302bd1ee74b0fab3e17 [term 2 LEADER]: Stepping down as leader of term 2
I20240503 23:26:35.935416 3762 raft_consensus.cc:729] T d0c4430fc56644cb9f87deffe4fe9407 P 83ffd7184ec94302bd1ee74b0fab3e17 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 83ffd7184ec94302bd1ee74b0fab3e17, State: Running, Role: LEADER
I20240503 23:26:35.935930 3762 consensus_queue.cc:260] T d0c4430fc56644cb9f87deffe4fe9407 P 83ffd7184ec94302bd1ee74b0fab3e17 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 28, Committed index: 28, Last appended: 2.28, Last appended by leader: 28, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "83ffd7184ec94302bd1ee74b0fab3e17" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 32909 } attrs { replace: true } } peers { permanent_uuid: "a2f9bf2e5d9b4fc19f804797bca4a3ff" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 46261 } } peers { permanent_uuid: "a3eecbaec43d4954b1096bc3121e824c" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 40649 } } peers { permanent_uuid: "4d1d4e261584467fbbce4c72fa18045a" member_type: NON_VOTER last_known_addr { host: "127.0.100.6" port: 35349 } attrs { promote: true } }
I20240503 23:26:35.936482 3762 raft_consensus.cc:3037] T d0c4430fc56644cb9f87deffe4fe9407 P 83ffd7184ec94302bd1ee74b0fab3e17 [term 2 FOLLOWER]: Advancing to term 3
I20240503 23:26:35.939833 3762 raft_consensus.cc:2454] T d0c4430fc56644cb9f87deffe4fe9407 P 83ffd7184ec94302bd1ee74b0fab3e17 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2f9bf2e5d9b4fc19f804797bca4a3ff in term 3.
I20240503 23:26:35.940332 3804 leader_election.cc:302] T d0c4430fc56644cb9f87deffe4fe9407 P a2f9bf2e5d9b4fc19f804797bca4a3ff [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 83ffd7184ec94302bd1ee74b0fab3e17, a2f9bf2e5d9b4fc19f804797bca4a3ff; no voters: a3eecbaec43d4954b1096bc3121e824c
I20240503 23:26:35.940707 4574 raft_consensus.cc:2781] T d0c4430fc56644cb9f87deffe4fe9407 P a2f9bf2e5d9b4fc19f804797bca4a3ff [term 3 FOLLOWER]: Leader election won for term 3
I20240503 23:26:35.941288 4574 raft_consensus.cc:686] T d0c4430fc56644cb9f87deffe4fe9407 P a2f9bf2e5d9b4fc19f804797bca4a3ff [term 3 LEADER]: Becoming Leader. State: Replica: a2f9bf2e5d9b4fc19f804797bca4a3ff, State: Running, Role: LEADER
I20240503 23:26:35.941691 4574 consensus_queue.cc:237] T d0c4430fc56644cb9f87deffe4fe9407 P a2f9bf2e5d9b4fc19f804797bca4a3ff [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 28, Committed index: 28, Last appended: 2.28, Last appended by leader: 28, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "83ffd7184ec94302bd1ee74b0fab3e17" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 32909 } attrs { replace: true } } peers { permanent_uuid: "a2f9bf2e5d9b4fc19f804797bca4a3ff" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 46261 } } peers { permanent_uuid: "a3eecbaec43d4954b1096bc3121e824c" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 40649 } } peers { permanent_uuid: "4d1d4e261584467fbbce4c72fa18045a" member_type: NON_VOTER last_known_addr { host: "127.0.100.6" port: 35349 } attrs { promote: true } }
I20240503 23:26:35.945004 3655 catalog_manager.cc:5484] T d0c4430fc56644cb9f87deffe4fe9407 P a2f9bf2e5d9b4fc19f804797bca4a3ff reported cstate change: term changed from 2 to 3, leader changed from 83ffd7184ec94302bd1ee74b0fab3e17 (127.0.100.1) to a2f9bf2e5d9b4fc19f804797bca4a3ff (127.0.100.2). New cstate: current_term: 3 leader_uuid: "a2f9bf2e5d9b4fc19f804797bca4a3ff" committed_config { opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "83ffd7184ec94302bd1ee74b0fab3e17" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 32909 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a2f9bf2e5d9b4fc19f804797bca4a3ff" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 46261 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a3eecbaec43d4954b1096bc3121e824c" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 40649 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4d1d4e261584467fbbce4c72fa18045a" member_type: NON_VOTER last_known_addr { host: "127.0.100.6" port: 35349 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 23:26:35.994065 3761 raft_consensus.cc:1261] T 55cf664d5e874c0d9b9adf8bdc9c43ed P 83ffd7184ec94302bd1ee74b0fab3e17 [term 2 FOLLOWER]: Refusing update from remote peer e4086c099d7f4447b2b4733b6ca364df: Log matching property violated. Preceding OpId in replica: term: 1 index: 30. Preceding OpId from leader: term: 2 index: 31. (index mismatch)
I20240503 23:26:35.994714 4530 consensus_queue.cc:1035] T 55cf664d5e874c0d9b9adf8bdc9c43ed P e4086c099d7f4447b2b4733b6ca364df [LEADER]: Connected to new peer: Peer: permanent_uuid: "83ffd7184ec94302bd1ee74b0fab3e17" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 32909 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 31, Last known committed idx: 30, Time since last communication: 0.000s
I20240503 23:26:36.010553 3837 raft_consensus.cc:1261] T 55cf664d5e874c0d9b9adf8bdc9c43ed P a2f9bf2e5d9b4fc19f804797bca4a3ff [term 2 FOLLOWER]: Refusing update from remote peer e4086c099d7f4447b2b4733b6ca364df: Log matching property violated. Preceding OpId in replica: term: 1 index: 30. Preceding OpId from leader: term: 2 index: 31. (index mismatch)
I20240503 23:26:36.011969 4580 consensus_queue.cc:1035] T 55cf664d5e874c0d9b9adf8bdc9c43ed P e4086c099d7f4447b2b4733b6ca364df [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2f9bf2e5d9b4fc19f804797bca4a3ff" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 46261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 31, Last known committed idx: 30, Time since last communication: 0.000s
W20240503 23:26:36.020579 4182 consensus_peers.cc:485] T 55cf664d5e874c0d9b9adf8bdc9c43ed P e4086c099d7f4447b2b4733b6ca364df -> Peer 4d1d4e261584467fbbce4c72fa18045a (127.0.100.6:35349): Couldn't send request to peer 4d1d4e261584467fbbce4c72fa18045a. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 55cf664d5e874c0d9b9adf8bdc9c43ed. This is attempt 1: this message will repeat every 5th retry.
W20240503 23:26:36.100314 3802 consensus_peers.cc:485] T 487ac5b9bf2d4de2b80a1b13387a2a77 P a2f9bf2e5d9b4fc19f804797bca4a3ff -> Peer 692da0319a2d4a81ad81f5d8f9f4c86b (127.0.100.5:38561): Couldn't send request to peer 692da0319a2d4a81ad81f5d8f9f4c86b. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20240503 23:26:36.104130 3837 raft_consensus.cc:1261] T 481cd0f8c8d147adac217915a53ff6d9 P a2f9bf2e5d9b4fc19f804797bca4a3ff [term 2 FOLLOWER]: Refusing update from remote peer 83ffd7184ec94302bd1ee74b0fab3e17: Log matching property violated. Preceding OpId in replica: term: 1 index: 26. Preceding OpId from leader: term: 2 index: 27. (index mismatch)
I20240503 23:26:36.104842 4569 consensus_queue.cc:1035] T 481cd0f8c8d147adac217915a53ff6d9 P 83ffd7184ec94302bd1ee74b0fab3e17 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2f9bf2e5d9b4fc19f804797bca4a3ff" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 46261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 27, Last known committed idx: 26, Time since last communication: 0.000s
I20240503 23:26:36.112579 3907 raft_consensus.cc:1261] T 481cd0f8c8d147adac217915a53ff6d9 P a3eecbaec43d4954b1096bc3121e824c [term 2 FOLLOWER]: Refusing update from remote peer 83ffd7184ec94302bd1ee74b0fab3e17: Log matching property violated. Preceding OpId in replica: term: 1 index: 26. Preceding OpId from leader: term: 2 index: 27. (index mismatch)
I20240503 23:26:36.113162 4569 consensus_queue.cc:1035] T 481cd0f8c8d147adac217915a53ff6d9 P 83ffd7184ec94302bd1ee74b0fab3e17 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a3eecbaec43d4954b1096bc3121e824c" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 40649 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 27, Last known committed idx: 26, Time since last communication: 0.000s
I20240503 23:26:36.124370 3907 raft_consensus.cc:1261] T a14a12115b8d4fec947c6e223f1ef486 P a3eecbaec43d4954b1096bc3121e824c [term 2 FOLLOWER]: Refusing update from remote peer 4d1d4e261584467fbbce4c72fa18045a: Log matching property violated. Preceding OpId in replica: term: 1 index: 29. Preceding OpId from leader: term: 2 index: 30. (index mismatch)
I20240503 23:26:36.125198 4566 consensus_queue.cc:1035] T a14a12115b8d4fec947c6e223f1ef486 P 4d1d4e261584467fbbce4c72fa18045a [LEADER]: Connected to new peer: Peer: permanent_uuid: "a3eecbaec43d4954b1096bc3121e824c" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 40649 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 30, Last known committed idx: 29, Time since last communication: 0.000s
I20240503 23:26:36.130560 3908 raft_consensus.cc:1261] T 487ac5b9bf2d4de2b80a1b13387a2a77 P a3eecbaec43d4954b1096bc3121e824c [term 2 FOLLOWER]: Refusing update from remote peer a2f9bf2e5d9b4fc19f804797bca4a3ff: Log matching property violated. Preceding OpId in replica: term: 1 index: 27. Preceding OpId from leader: term: 2 index: 28. (index mismatch)
I20240503 23:26:36.131112 4574 consensus_queue.cc:1035] T 487ac5b9bf2d4de2b80a1b13387a2a77 P a2f9bf2e5d9b4fc19f804797bca4a3ff [LEADER]: Connected to new peer: Peer: permanent_uuid: "a3eecbaec43d4954b1096bc3121e824c" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 40649 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 28, Last known committed idx: 27, Time since last communication: 0.000s
I20240503 23:26:36.135468 3761 raft_consensus.cc:1261] T 487ac5b9bf2d4de2b80a1b13387a2a77 P 83ffd7184ec94302bd1ee74b0fab3e17 [term 2 FOLLOWER]: Refusing update from remote peer a2f9bf2e5d9b4fc19f804797bca4a3ff: Log matching property violated. Preceding OpId in replica: term: 1 index: 27. Preceding OpId from leader: term: 2 index: 28. (index mismatch)
I20240503 23:26:36.136270 4574 consensus_queue.cc:1035] T 487ac5b9bf2d4de2b80a1b13387a2a77 P a2f9bf2e5d9b4fc19f804797bca4a3ff [LEADER]: Connected to new peer: Peer: permanent_uuid: "83ffd7184ec94302bd1ee74b0fab3e17" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 32909 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 28, Last known committed idx: 27, Time since last communication: 0.000s
I20240503 23:26:36.149169 3761 raft_consensus.cc:1261] T a14a12115b8d4fec947c6e223f1ef486 P 83ffd7184ec94302bd1ee74b0fab3e17 [term 2 FOLLOWER]: Refusing update from remote peer 4d1d4e261584467fbbce4c72fa18045a: Log matching property violated. Preceding OpId in replica: term: 1 index: 29. Preceding OpId from leader: term: 2 index: 30. (index mismatch)
I20240503 23:26:36.149931 4567 consensus_queue.cc:1035] T a14a12115b8d4fec947c6e223f1ef486 P 4d1d4e261584467fbbce4c72fa18045a [LEADER]: Connected to new peer: Peer: permanent_uuid: "83ffd7184ec94302bd1ee74b0fab3e17" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 32909 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 30, Last known committed idx: 29, Time since last communication: 0.000s
I20240503 23:26:36.326073 3908 raft_consensus.cc:3037] T d0c4430fc56644cb9f87deffe4fe9407 P a3eecbaec43d4954b1096bc3121e824c [term 2 FOLLOWER]: Advancing to term 3
I20240503 23:26:36.329381 3908 raft_consensus.cc:1261] T d0c4430fc56644cb9f87deffe4fe9407 P a3eecbaec43d4954b1096bc3121e824c [term 3 FOLLOWER]: Refusing update from remote peer a2f9bf2e5d9b4fc19f804797bca4a3ff: Log matching property violated. Preceding OpId in replica: term: 2 index: 28. Preceding OpId from leader: term: 3 index: 29. (index mismatch)
I20240503 23:26:36.330013 4588 consensus_queue.cc:1035] T d0c4430fc56644cb9f87deffe4fe9407 P a2f9bf2e5d9b4fc19f804797bca4a3ff [LEADER]: Connected to new peer: Peer: permanent_uuid: "a3eecbaec43d4954b1096bc3121e824c" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 40649 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.000s
I20240503 23:26:36.333623 3761 raft_consensus.cc:1261] T d0c4430fc56644cb9f87deffe4fe9407 P 83ffd7184ec94302bd1ee74b0fab3e17 [term 3 FOLLOWER]: Refusing update from remote peer a2f9bf2e5d9b4fc19f804797bca4a3ff: Log matching property violated. Preceding OpId in replica: term: 2 index: 28. Preceding OpId from leader: term: 3 index: 29. (index mismatch)
W20240503 23:26:36.333945 3803 consensus_peers.cc:485] T d0c4430fc56644cb9f87deffe4fe9407 P a2f9bf2e5d9b4fc19f804797bca4a3ff -> Peer 4d1d4e261584467fbbce4c72fa18045a (127.0.100.6:35349): Couldn't send request to peer 4d1d4e261584467fbbce4c72fa18045a. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d0c4430fc56644cb9f87deffe4fe9407. This is attempt 1: this message will repeat every 5th retry.
I20240503 23:26:36.334426 4467 consensus_queue.cc:1035] T d0c4430fc56644cb9f87deffe4fe9407 P a2f9bf2e5d9b4fc19f804797bca4a3ff [LEADER]: Connected to new peer: Peer: permanent_uuid: "83ffd7184ec94302bd1ee74b0fab3e17" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 32909 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.000s
I20240503 23:26:36.340741 3908 raft_consensus.cc:1261] T 1b9566be6c644cfca852ec6507fab893 P a3eecbaec43d4954b1096bc3121e824c [term 2 FOLLOWER]: Refusing update from remote peer 692da0319a2d4a81ad81f5d8f9f4c86b: Log matching property violated. Preceding OpId in replica: term: 1 index: 29. Preceding OpId from leader: term: 2 index: 30. (index mismatch)
I20240503 23:26:36.341295 4592 consensus_queue.cc:1035] T 1b9566be6c644cfca852ec6507fab893 P 692da0319a2d4a81ad81f5d8f9f4c86b [LEADER]: Connected to new peer: Peer: permanent_uuid: "a3eecbaec43d4954b1096bc3121e824c" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 40649 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 30, Last known committed idx: 29, Time since last communication: 0.000s
I20240503 23:26:36.351105 3837 raft_consensus.cc:1261] T 1b9566be6c644cfca852ec6507fab893 P a2f9bf2e5d9b4fc19f804797bca4a3ff [term 2 FOLLOWER]: Refusing update from remote peer 692da0319a2d4a81ad81f5d8f9f4c86b: Log matching property violated. Preceding OpId in replica: term: 1 index: 29. Preceding OpId from leader: term: 2 index: 30. (index mismatch)
I20240503 23:26:36.352800 4599 consensus_queue.cc:1035] T 1b9566be6c644cfca852ec6507fab893 P 692da0319a2d4a81ad81f5d8f9f4c86b [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2f9bf2e5d9b4fc19f804797bca4a3ff" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 46261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 30, Last known committed idx: 29, Time since last communication: 0.001s
I20240503 23:26:36.496219 4601 ts_tablet_manager.cc:902] T 55cf664d5e874c0d9b9adf8bdc9c43ed P 4d1d4e261584467fbbce4c72fa18045a: Initiating tablet copy from peer e4086c099d7f4447b2b4733b6ca364df (127.0.100.4:35011)
I20240503 23:26:36.496982 4601 tablet_copy_client.cc:316] T 55cf664d5e874c0d9b9adf8bdc9c43ed P 4d1d4e261584467fbbce4c72fa18045a: tablet copy: Beginning tablet copy session from remote peer at address 127.0.100.4:35011
I20240503 23:26:36.515664 4228 tablet_copy_service.cc:140] P e4086c099d7f4447b2b4733b6ca364df: Received BeginTabletCopySession request for tablet 55cf664d5e874c0d9b9adf8bdc9c43ed from peer 4d1d4e261584467fbbce4c72fa18045a ({username='slave'} at 127.0.0.1:50852)
I20240503 23:26:36.515900 4228 tablet_copy_service.cc:161] P e4086c099d7f4447b2b4733b6ca364df: Beginning new tablet copy session on tablet 55cf664d5e874c0d9b9adf8bdc9c43ed from peer 4d1d4e261584467fbbce4c72fa18045a at {username='slave'} at 127.0.0.1:50852: session id = 4d1d4e261584467fbbce4c72fa18045a-55cf664d5e874c0d9b9adf8bdc9c43ed
I20240503 23:26:36.518589 4228 tablet_copy_source_session.cc:205] T 55cf664d5e874c0d9b9adf8bdc9c43ed P e4086c099d7f4447b2b4733b6ca364df: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 23:26:36.519807 4601 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 55cf664d5e874c0d9b9adf8bdc9c43ed. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 23:26:36.525229 4601 tablet_copy_client.cc:791] T 55cf664d5e874c0d9b9adf8bdc9c43ed P 4d1d4e261584467fbbce4c72fa18045a: tablet copy: Starting download of 0 data blocks...
I20240503 23:26:36.525517 4601 tablet_copy_client.cc:655] T 55cf664d5e874c0d9b9adf8bdc9c43ed P 4d1d4e261584467fbbce4c72fa18045a: tablet copy: Starting download of 1 WAL segments...
I20240503 23:26:36.527060 4602 tablet_copy_client.cc:1010] T 55cf664d5e874c0d9b9adf8bdc9c43ed P 4d1d4e261584467fbbce4c72fa18045a: tablet copy: Injecting latency into file download: 2000
I20240503 23:26:36.817606 4607 ts_tablet_manager.cc:902] T d0c4430fc56644cb9f87deffe4fe9407 P 4d1d4e261584467fbbce4c72fa18045a: Initiating tablet copy from peer a2f9bf2e5d9b4fc19f804797bca4a3ff (127.0.100.2:46261)
I20240503 23:26:36.818775 4607 tablet_copy_client.cc:316] T d0c4430fc56644cb9f87deffe4fe9407 P 4d1d4e261584467fbbce4c72fa18045a: tablet copy: Beginning tablet copy session from remote peer at address 127.0.100.2:46261
I20240503 23:26:36.826177 3848 tablet_copy_service.cc:140] P a2f9bf2e5d9b4fc19f804797bca4a3ff: Received BeginTabletCopySession request for tablet d0c4430fc56644cb9f87deffe4fe9407 from peer 4d1d4e261584467fbbce4c72fa18045a ({username='slave'} at 127.0.0.1:55484)
I20240503 23:26:36.826356 3848 tablet_copy_service.cc:161] P a2f9bf2e5d9b4fc19f804797bca4a3ff: Beginning new tablet copy session on tablet d0c4430fc56644cb9f87deffe4fe9407 from peer 4d1d4e261584467fbbce4c72fa18045a at {username='slave'} at 127.0.0.1:55484: session id = 4d1d4e261584467fbbce4c72fa18045a-d0c4430fc56644cb9f87deffe4fe9407
I20240503 23:26:36.829280 3848 tablet_copy_source_session.cc:205] T d0c4430fc56644cb9f87deffe4fe9407 P a2f9bf2e5d9b4fc19f804797bca4a3ff: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 23:26:36.830528 4607 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d0c4430fc56644cb9f87deffe4fe9407. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 23:26:36.835829 4607 tablet_copy_client.cc:791] T d0c4430fc56644cb9f87deffe4fe9407 P 4d1d4e261584467fbbce4c72fa18045a: tablet copy: Starting download of 0 data blocks...
I20240503 23:26:36.836215 4607 tablet_copy_client.cc:655] T d0c4430fc56644cb9f87deffe4fe9407 P 4d1d4e261584467fbbce4c72fa18045a: tablet copy: Starting download of 1 WAL segments...
I20240503 23:26:36.837776 4608 tablet_copy_client.cc:1010] T d0c4430fc56644cb9f87deffe4fe9407 P 4d1d4e261584467fbbce4c72fa18045a: tablet copy: Injecting latency into file download: 2000
I20240503 23:26:37.239820 3715 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240503 23:26:37.243474 3761 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "7181e4e8c0324a8eb095dde23651a1ab"
dest_uuid: "83ffd7184ec94302bd1ee74b0fab3e17"
mode: GRACEFUL
new_leader_uuid: "a3eecbaec43d4954b1096bc3121e824c"
from {username='slave'} at 127.0.0.1:57936
I20240503 23:26:37.243803 3761 raft_consensus.cc:595] T 7181e4e8c0324a8eb095dde23651a1ab P 83ffd7184ec94302bd1ee74b0fab3e17 [term 1 LEADER]: Received request to transfer leadership to TS a3eecbaec43d4954b1096bc3121e824c
I20240503 23:26:37.245446 3838 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "a0e18d0975a54e3dbeb0a13cb226fefd"
dest_uuid: "a2f9bf2e5d9b4fc19f804797bca4a3ff"
mode: GRACEFUL
new_leader_uuid: "83ffd7184ec94302bd1ee74b0fab3e17"
from {username='slave'} at 127.0.0.1:55472
I20240503 23:26:37.245744 3838 raft_consensus.cc:595] T a0e18d0975a54e3dbeb0a13cb226fefd P a2f9bf2e5d9b4fc19f804797bca4a3ff [term 1 LEADER]: Received request to transfer leadership to TS 83ffd7184ec94302bd1ee74b0fab3e17
I20240503 23:26:37.246737 3838 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "d0c4430fc56644cb9f87deffe4fe9407"
dest_uuid: "a2f9bf2e5d9b4fc19f804797bca4a3ff"
mode: GRACEFUL
new_leader_uuid: "83ffd7184ec94302bd1ee74b0fab3e17"
from {username='slave'} at 127.0.0.1:55472
I20240503 23:26:37.246954 3838 raft_consensus.cc:595] T d0c4430fc56644cb9f87deffe4fe9407 P a2f9bf2e5d9b4fc19f804797bca4a3ff [term 3 LEADER]: Received request to transfer leadership to TS 83ffd7184ec94302bd1ee74b0fab3e17
I20240503 23:26:37.247408 3715 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 3
I20240503 23:26:37.247669 3715 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240503 23:26:37.385041 4550 tablet_copy_client.cc:531] T 60267cb4849d4823a0afb2380305905f P e4086c099d7f4447b2b4733b6ca364df: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 23:26:37.388496 4550 tablet_bootstrap.cc:492] T 60267cb4849d4823a0afb2380305905f P e4086c099d7f4447b2b4733b6ca364df: Bootstrap starting.
I20240503 23:26:37.423866 4550 tablet_bootstrap.cc:492] T 60267cb4849d4823a0afb2380305905f P e4086c099d7f4447b2b4733b6ca364df: Bootstrap replayed 1/1 log segments. Stats: ops{read=26 overwritten=0 applied=26 ignored=0} inserts{seen=94 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 23:26:37.424350 4550 tablet_bootstrap.cc:492] T 60267cb4849d4823a0afb2380305905f P e4086c099d7f4447b2b4733b6ca364df: Bootstrap complete.
I20240503 23:26:37.424656 4550 ts_tablet_manager.cc:1367] T 60267cb4849d4823a0afb2380305905f P e4086c099d7f4447b2b4733b6ca364df: Time spent bootstrapping tablet: real 0.036s user 0.028s sys 0.012s
I20240503 23:26:37.425412 4550 raft_consensus.cc:348] T 60267cb4849d4823a0afb2380305905f P e4086c099d7f4447b2b4733b6ca364df [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 26 OBSOLETE_local: false peers { permanent_uuid: "83ffd7184ec94302bd1ee74b0fab3e17" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 32909 } } peers { permanent_uuid: "a2f9bf2e5d9b4fc19f804797bca4a3ff" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 46261 } attrs { replace: true } } peers { permanent_uuid: "a3eecbaec43d4954b1096bc3121e824c" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 40649 } } peers { permanent_uuid: "e4086c099d7f4447b2b4733b6ca364df" member_type: NON_VOTER last_known_addr { host: "127.0.100.4" port: 35011 } attrs { promote: true } }
I20240503 23:26:37.425639 4550 raft_consensus.cc:729] T 60267cb4849d4823a0afb2380305905f P e4086c099d7f4447b2b4733b6ca364df [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: e4086c099d7f4447b2b4733b6ca364df, State: Initialized, Role: LEARNER
I20240503 23:26:37.425848 4550 consensus_queue.cc:260] T 60267cb4849d4823a0afb2380305905f P e4086c099d7f4447b2b4733b6ca364df [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 26, Last appended: 1.26, Last appended by leader: 26, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 26 OBSOLETE_local: false peers { permanent_uuid: "83ffd7184ec94302bd1ee74b0fab3e17" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 32909 } } peers { permanent_uuid: "a2f9bf2e5d9b4fc19f804797bca4a3ff" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 46261 } attrs { replace: true } } peers { permanent_uuid: "a3eecbaec43d4954b1096bc3121e824c" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 40649 } } peers { permanent_uuid: "e4086c099d7f4447b2b4733b6ca364df" member_type: NON_VOTER last_known_addr { host: "127.0.100.4" port: 35011 } attrs { promote: true } }
I20240503 23:26:37.426514 4550 ts_tablet_manager.cc:1398] T 60267cb4849d4823a0afb2380305905f P e4086c099d7f4447b2b4733b6ca364df: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240503 23:26:37.427244 3923 tablet_copy_service.cc:342] P a3eecbaec43d4954b1096bc3121e824c: Request end of tablet copy session e4086c099d7f4447b2b4733b6ca364df-60267cb4849d4823a0afb2380305905f received from {username='slave'} at 127.0.0.1:39458
I20240503 23:26:37.427346 3923 tablet_copy_service.cc:433] P a3eecbaec43d4954b1096bc3121e824c: ending tablet copy session e4086c099d7f4447b2b4733b6ca364df-60267cb4849d4823a0afb2380305905f on tablet 60267cb4849d4823a0afb2380305905f with peer e4086c099d7f4447b2b4733b6ca364df
I20240503 23:26:37.438004 4613 raft_consensus.cc:979] T 7181e4e8c0324a8eb095dde23651a1ab P 83ffd7184ec94302bd1ee74b0fab3e17: : Instructing follower a3eecbaec43d4954b1096bc3121e824c to start an election
I20240503 23:26:37.438197 4613 raft_consensus.cc:1067] T 7181e4e8c0324a8eb095dde23651a1ab P 83ffd7184ec94302bd1ee74b0fab3e17 [term 1 LEADER]: Signalling peer a3eecbaec43d4954b1096bc3121e824c to start an election
I20240503 23:26:37.438795 3907 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "7181e4e8c0324a8eb095dde23651a1ab"
dest_uuid: "a3eecbaec43d4954b1096bc3121e824c"
from {username='slave'} at 127.0.0.1:40648
I20240503 23:26:37.438977 3907 raft_consensus.cc:482] T 7181e4e8c0324a8eb095dde23651a1ab P a3eecbaec43d4954b1096bc3121e824c [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 23:26:37.439077 3907 raft_consensus.cc:3037] T 7181e4e8c0324a8eb095dde23651a1ab P a3eecbaec43d4954b1096bc3121e824c [term 1 FOLLOWER]: Advancing to term 2
I20240503 23:26:37.441757 3907 raft_consensus.cc:504] T 7181e4e8c0324a8eb095dde23651a1ab P a3eecbaec43d4954b1096bc3121e824c [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "83ffd7184ec94302bd1ee74b0fab3e17" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 32909 } } peers { permanent_uuid: "a3eecbaec43d4954b1096bc3121e824c" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 40649 } } peers { permanent_uuid: "e4086c099d7f4447b2b4733b6ca364df" member_type: VOTER last_known_addr { host: "127.0.100.4" port: 35011 } attrs { promote: false } }
I20240503 23:26:37.442509 3907 leader_election.cc:288] T 7181e4e8c0324a8eb095dde23651a1ab P a3eecbaec43d4954b1096bc3121e824c [CANDIDATE]: Term 2 election: Requested vote from peers 83ffd7184ec94302bd1ee74b0fab3e17 (127.0.100.1:32909), e4086c099d7f4447b2b4733b6ca364df (127.0.100.4:35011)
I20240503 23:26:37.442929 3762 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7181e4e8c0324a8eb095dde23651a1ab" candidate_uuid: "a3eecbaec43d4954b1096bc3121e824c" candidate_term: 2 candidate_status { last_received { term: 1 index: 28 } } ignore_live_leader: true dest_uuid: "83ffd7184ec94302bd1ee74b0fab3e17"
I20240503 23:26:37.443122 3762 raft_consensus.cc:3032] T 7181e4e8c0324a8eb095dde23651a1ab P 83ffd7184ec94302bd1ee74b0fab3e17 [term 1 LEADER]: Stepping down as leader of term 1
I20240503 23:26:37.443171 3762 raft_consensus.cc:729] T 7181e4e8c0324a8eb095dde23651a1ab P 83ffd7184ec94302bd1ee74b0fab3e17 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 83ffd7184ec94302bd1ee74b0fab3e17, State: Running, Role: LEADER
I20240503 23:26:37.443296 4218 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7181e4e8c0324a8eb095dde23651a1ab" candidate_uuid: "a3eecbaec43d4954b1096bc3121e824c" candidate_term: 2 candidate_status { last_received { term: 1 index: 28 } } ignore_live_leader: true dest_uuid: "e4086c099d7f4447b2b4733b6ca364df"
I20240503 23:26:37.443481 4218 raft_consensus.cc:3037] T 7181e4e8c0324a8eb095dde23651a1ab P e4086c099d7f4447b2b4733b6ca364df [term 1 FOLLOWER]: Advancing to term 2
I20240503 23:26:37.443396 3762 consensus_queue.cc:260] T 7181e4e8c0324a8eb095dde23651a1ab P 83ffd7184ec94302bd1ee74b0fab3e17 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 28, Committed index: 28, Last appended: 1.28, Last appended by leader: 28, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "83ffd7184ec94302bd1ee74b0fab3e17" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 32909 } } peers { permanent_uuid: "a3eecbaec43d4954b1096bc3121e824c" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 40649 } } peers { permanent_uuid: "e4086c099d7f4447b2b4733b6ca364df" member_type: VOTER last_known_addr { host: "127.0.100.4" port: 35011 } attrs { promote: false } }
I20240503 23:26:37.443691 3762 raft_consensus.cc:3037] T 7181e4e8c0324a8eb095dde23651a1ab P 83ffd7184ec94302bd1ee74b0fab3e17 [term 1 FOLLOWER]: Advancing to term 2
I20240503 23:26:37.445813 3762 raft_consensus.cc:2454] T 7181e4e8c0324a8eb095dde23651a1ab P 83ffd7184ec94302bd1ee74b0fab3e17 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a3eecbaec43d4954b1096bc3121e824c in term 2.
I20240503 23:26:37.445813 4218 raft_consensus.cc:2454] T 7181e4e8c0324a8eb095dde23651a1ab P e4086c099d7f4447b2b4733b6ca364df [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a3eecbaec43d4954b1096bc3121e824c in term 2.
I20240503 23:26:37.446580 3878 leader_election.cc:302] T 7181e4e8c0324a8eb095dde23651a1ab P a3eecbaec43d4954b1096bc3121e824c [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 83ffd7184ec94302bd1ee74b0fab3e17, a3eecbaec43d4954b1096bc3121e824c; no voters:
I20240503 23:26:37.446947 4595 raft_consensus.cc:2781] T 7181e4e8c0324a8eb095dde23651a1ab P a3eecbaec43d4954b1096bc3121e824c [term 2 FOLLOWER]: Leader election won for term 2
I20240503 23:26:37.447582 4595 raft_consensus.cc:686] T 7181e4e8c0324a8eb095dde23651a1ab P a3eecbaec43d4954b1096bc3121e824c [term 2 LEADER]: Becoming Leader. State: Replica: a3eecbaec43d4954b1096bc3121e824c, State: Running, Role: LEADER
I20240503 23:26:37.447901 4595 consensus_queue.cc:237] T 7181e4e8c0324a8eb095dde23651a1ab P a3eecbaec43d4954b1096bc3121e824c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 28, Committed index: 28, Last appended: 1.28, Last appended by leader: 28, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "83ffd7184ec94302bd1ee74b0fab3e17" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 32909 } } peers { permanent_uuid: "a3eecbaec43d4954b1096bc3121e824c" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 40649 } } peers { permanent_uuid: "e4086c099d7f4447b2b4733b6ca364df" member_type: VOTER last_known_addr { host: "127.0.100.4" port: 35011 } attrs { promote: false } }
I20240503 23:26:37.451462 3655 catalog_manager.cc:5484] T 7181e4e8c0324a8eb095dde23651a1ab P a3eecbaec43d4954b1096bc3121e824c reported cstate change: term changed from 1 to 2, leader changed from 83ffd7184ec94302bd1ee74b0fab3e17 (127.0.100.1) to a3eecbaec43d4954b1096bc3121e824c (127.0.100.3). New cstate: current_term: 2 leader_uuid: "a3eecbaec43d4954b1096bc3121e824c" committed_config { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "83ffd7184ec94302bd1ee74b0fab3e17" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 32909 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a3eecbaec43d4954b1096bc3121e824c" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 40649 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e4086c099d7f4447b2b4733b6ca364df" member_type: VOTER last_known_addr { host: "127.0.100.4" port: 35011 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240503 23:26:37.536141 4554 tablet_copy_client.cc:531] T a0e18d0975a54e3dbeb0a13cb226fefd P 692da0319a2d4a81ad81f5d8f9f4c86b: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 23:26:37.539516 4554 tablet_bootstrap.cc:492] T a0e18d0975a54e3dbeb0a13cb226fefd P 692da0319a2d4a81ad81f5d8f9f4c86b: Bootstrap starting.
I20240503 23:26:37.570979 4610 raft_consensus.cc:979] T a0e18d0975a54e3dbeb0a13cb226fefd P a2f9bf2e5d9b4fc19f804797bca4a3ff: : Instructing follower 83ffd7184ec94302bd1ee74b0fab3e17 to start an election
I20240503 23:26:37.571179 4611 raft_consensus.cc:1067] T a0e18d0975a54e3dbeb0a13cb226fefd P a2f9bf2e5d9b4fc19f804797bca4a3ff [term 1 LEADER]: Signalling peer 83ffd7184ec94302bd1ee74b0fab3e17 to start an election
I20240503 23:26:37.571872 3762 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "a0e18d0975a54e3dbeb0a13cb226fefd"
dest_uuid: "83ffd7184ec94302bd1ee74b0fab3e17"
from {username='slave'} at 127.0.0.1:35586
I20240503 23:26:37.572100 3762 raft_consensus.cc:482] T a0e18d0975a54e3dbeb0a13cb226fefd P 83ffd7184ec94302bd1ee74b0fab3e17 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 23:26:37.572216 3762 raft_consensus.cc:3037] T a0e18d0975a54e3dbeb0a13cb226fefd P 83ffd7184ec94302bd1ee74b0fab3e17 [term 1 FOLLOWER]: Advancing to term 2
I20240503 23:26:37.575629 3762 raft_consensus.cc:504] T a0e18d0975a54e3dbeb0a13cb226fefd P 83ffd7184ec94302bd1ee74b0fab3e17 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "83ffd7184ec94302bd1ee74b0fab3e17" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 32909 } } peers { permanent_uuid: "a2f9bf2e5d9b4fc19f804797bca4a3ff" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 46261 } } peers { permanent_uuid: "a3eecbaec43d4954b1096bc3121e824c" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 40649 } attrs { replace: true } } peers { permanent_uuid: "692da0319a2d4a81ad81f5d8f9f4c86b" member_type: NON_VOTER last_known_addr { host: "127.0.100.5" port: 38561 } attrs { promote: true } }
I20240503 23:26:37.576877 3837 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a0e18d0975a54e3dbeb0a13cb226fefd" candidate_uuid: "83ffd7184ec94302bd1ee74b0fab3e17" candidate_term: 2 candidate_status { last_received { term: 1 index: 27 } } ignore_live_leader: true dest_uuid: "a2f9bf2e5d9b4fc19f804797bca4a3ff"
I20240503 23:26:37.577128 3837 raft_consensus.cc:3032] T a0e18d0975a54e3dbeb0a13cb226fefd P a2f9bf2e5d9b4fc19f804797bca4a3ff [term 1 LEADER]: Stepping down as leader of term 1
I20240503 23:26:37.577199 3837 raft_consensus.cc:729] T a0e18d0975a54e3dbeb0a13cb226fefd P a2f9bf2e5d9b4fc19f804797bca4a3ff [term 1 LEADER]: Becoming Follower/Learner. State: Replica: a2f9bf2e5d9b4fc19f804797bca4a3ff, State: Running, Role: LEADER
I20240503 23:26:37.577600 3907 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a0e18d0975a54e3dbeb0a13cb226fefd" candidate_uuid: "83ffd7184ec94302bd1ee74b0fab3e17" candidate_term: 2 candidate_status { last_received { term: 1 index: 27 } } ignore_live_leader: true dest_uuid: "a3eecbaec43d4954b1096bc3121e824c"
I20240503 23:26:37.577710 3762 leader_election.cc:288] T a0e18d0975a54e3dbeb0a13cb226fefd P 83ffd7184ec94302bd1ee74b0fab3e17 [CANDIDATE]: Term 2 election: Requested vote from peers a2f9bf2e5d9b4fc19f804797bca4a3ff (127.0.100.2:46261), a3eecbaec43d4954b1096bc3121e824c (127.0.100.3:40649)
I20240503 23:26:37.577754 3907 raft_consensus.cc:3037] T a0e18d0975a54e3dbeb0a13cb226fefd P a3eecbaec43d4954b1096bc3121e824c [term 1 FOLLOWER]: Advancing to term 2
I20240503 23:26:37.577935 4554 tablet_bootstrap.cc:492] T a0e18d0975a54e3dbeb0a13cb226fefd P 692da0319a2d4a81ad81f5d8f9f4c86b: Bootstrap replayed 1/1 log segments. Stats: ops{read=27 overwritten=0 applied=27 ignored=0} inserts{seen=94 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 23:26:37.578382 4554 tablet_bootstrap.cc:492] T a0e18d0975a54e3dbeb0a13cb226fefd P 692da0319a2d4a81ad81f5d8f9f4c86b: Bootstrap complete.
I20240503 23:26:37.578684 4554 ts_tablet_manager.cc:1367] T a0e18d0975a54e3dbeb0a13cb226fefd P 692da0319a2d4a81ad81f5d8f9f4c86b: Time spent bootstrapping tablet: real 0.039s user 0.039s sys 0.001s
I20240503 23:26:37.579341 4554 raft_consensus.cc:348] T a0e18d0975a54e3dbeb0a13cb226fefd P 692da0319a2d4a81ad81f5d8f9f4c86b [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "83ffd7184ec94302bd1ee74b0fab3e17" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 32909 } } peers { permanent_uuid: "a2f9bf2e5d9b4fc19f804797bca4a3ff" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 46261 } } peers { permanent_uuid: "a3eecbaec43d4954b1096bc3121e824c" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 40649 } attrs { replace: true } } peers { permanent_uuid: "692da0319a2d4a81ad81f5d8f9f4c86b" member_type: NON_VOTER last_known_addr { host: "127.0.100.5" port: 38561 } attrs { promote: true } }
I20240503 23:26:37.579501 4554 raft_consensus.cc:729] T a0e18d0975a54e3dbeb0a13cb226fefd P 692da0319a2d4a81ad81f5d8f9f4c86b [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 692da0319a2d4a81ad81f5d8f9f4c86b, State: Initialized, Role: LEARNER
I20240503 23:26:37.579643 4554 consensus_queue.cc:260] T a0e18d0975a54e3dbeb0a13cb226fefd P 692da0319a2d4a81ad81f5d8f9f4c86b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 27, Last appended: 1.27, Last appended by leader: 27, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "83ffd7184ec94302bd1ee74b0fab3e17" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 32909 } } peers { permanent_uuid: "a2f9bf2e5d9b4fc19f804797bca4a3ff" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 46261 } } peers { permanent_uuid: "a3eecbaec43d4954b1096bc3121e824c" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 40649 } attrs { replace: true } } peers { permanent_uuid: "692da0319a2d4a81ad81f5d8f9f4c86b" member_type: NON_VOTER last_known_addr { host: "127.0.100.5" port: 38561 } attrs { promote: true } }
I20240503 23:26:37.580121 4554 ts_tablet_manager.cc:1398] T a0e18d0975a54e3dbeb0a13cb226fefd P 692da0319a2d4a81ad81f5d8f9f4c86b: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 23:26:37.580662 3907 raft_consensus.cc:2454] T a0e18d0975a54e3dbeb0a13cb226fefd P a3eecbaec43d4954b1096bc3121e824c [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83ffd7184ec94302bd1ee74b0fab3e17 in term 2.
I20240503 23:26:37.580966 3848 tablet_copy_service.cc:342] P a2f9bf2e5d9b4fc19f804797bca4a3ff: Request end of tablet copy session 692da0319a2d4a81ad81f5d8f9f4c86b-a0e18d0975a54e3dbeb0a13cb226fefd received from {username='slave'} at 127.0.0.1:55456
I20240503 23:26:37.581068 3848 tablet_copy_service.cc:433] P a2f9bf2e5d9b4fc19f804797bca4a3ff: ending tablet copy session 692da0319a2d4a81ad81f5d8f9f4c86b-a0e18d0975a54e3dbeb0a13cb226fefd on tablet a0e18d0975a54e3dbeb0a13cb226fefd with peer 692da0319a2d4a81ad81f5d8f9f4c86b
I20240503 23:26:37.581493 3727 leader_election.cc:302] T a0e18d0975a54e3dbeb0a13cb226fefd P 83ffd7184ec94302bd1ee74b0fab3e17 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 83ffd7184ec94302bd1ee74b0fab3e17, a3eecbaec43d4954b1096bc3121e824c; no voters:
I20240503 23:26:37.581794 4613 raft_consensus.cc:2781] T a0e18d0975a54e3dbeb0a13cb226fefd P 83ffd7184ec94302bd1ee74b0fab3e17 [term 2 FOLLOWER]: Leader election won for term 2
I20240503 23:26:37.581923 4613 raft_consensus.cc:686] T a0e18d0975a54e3dbeb0a13cb226fefd P 83ffd7184ec94302bd1ee74b0fab3e17 [term 2 LEADER]: Becoming Leader. State: Replica: 83ffd7184ec94302bd1ee74b0fab3e17, State: Running, Role: LEADER
I20240503 23:26:37.582247 4613 consensus_queue.cc:237] T a0e18d0975a54e3dbeb0a13cb226fefd P 83ffd7184ec94302bd1ee74b0fab3e17 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 27, Committed index: 27, Last appended: 1.27, Last appended by leader: 27, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "83ffd7184ec94302bd1ee74b0fab3e17" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 32909 } } peers { permanent_uuid: "a2f9bf2e5d9b4fc19f804797bca4a3ff" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 46261 } } peers { permanent_uuid: "a3eecbaec43d4954b1096bc3121e824c" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 40649 } attrs { replace: true } } peers { permanent_uuid: "692da0319a2d4a81ad81f5d8f9f4c86b" member_type: NON_VOTER last_known_addr { host: "127.0.100.5" port: 38561 } attrs { promote: true } }
I20240503 23:26:37.585842 4560 tablet_copy_client.cc:531] T 487ac5b9bf2d4de2b80a1b13387a2a77 P 692da0319a2d4a81ad81f5d8f9f4c86b: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 23:26:37.585582 3655 catalog_manager.cc:5484] T a0e18d0975a54e3dbeb0a13cb226fefd P 83ffd7184ec94302bd1ee74b0fab3e17 reported cstate change: term changed from 1 to 2, leader changed from a2f9bf2e5d9b4fc19f804797bca4a3ff (127.0.100.2) to 83ffd7184ec94302bd1ee74b0fab3e17 (127.0.100.1). New cstate: current_term: 2 leader_uuid: "83ffd7184ec94302bd1ee74b0fab3e17" committed_config { opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "83ffd7184ec94302bd1ee74b0fab3e17" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 32909 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a2f9bf2e5d9b4fc19f804797bca4a3ff" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 46261 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a3eecbaec43d4954b1096bc3121e824c" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 40649 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "692da0319a2d4a81ad81f5d8f9f4c86b" member_type: NON_VOTER last_known_addr { host: "127.0.100.5" port: 38561 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 23:26:37.588328 3837 consensus_queue.cc:260] T a0e18d0975a54e3dbeb0a13cb226fefd P a2f9bf2e5d9b4fc19f804797bca4a3ff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 27, Committed index: 27, Last appended: 1.27, Last appended by leader: 27, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "83ffd7184ec94302bd1ee74b0fab3e17" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 32909 } } peers { permanent_uuid: "a2f9bf2e5d9b4fc19f804797bca4a3ff" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 46261 } } peers { permanent_uuid: "a3eecbaec43d4954b1096bc3121e824c" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 40649 } attrs { replace: true } } peers { permanent_uuid: "692da0319a2d4a81ad81f5d8f9f4c86b" member_type: NON_VOTER last_known_addr { host: "127.0.100.5" port: 38561 } attrs { promote: true } }
I20240503 23:26:37.588721 4556 tablet_copy_client.cc:531] T 98c9500acb43423b86f54ce7a4d98a72 P e4086c099d7f4447b2b4733b6ca364df: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 23:26:37.588871 3837 raft_consensus.cc:3037] T a0e18d0975a54e3dbeb0a13cb226fefd P a2f9bf2e5d9b4fc19f804797bca4a3ff [term 1 FOLLOWER]: Advancing to term 2
I20240503 23:26:37.590771 4560 tablet_bootstrap.cc:492] T 487ac5b9bf2d4de2b80a1b13387a2a77 P 692da0319a2d4a81ad81f5d8f9f4c86b: Bootstrap starting.
I20240503 23:26:37.591511 4556 tablet_bootstrap.cc:492] T 98c9500acb43423b86f54ce7a4d98a72 P e4086c099d7f4447b2b4733b6ca364df: Bootstrap starting.
I20240503 23:26:37.591614 3837 raft_consensus.cc:2454] T a0e18d0975a54e3dbeb0a13cb226fefd P a2f9bf2e5d9b4fc19f804797bca4a3ff [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83ffd7184ec94302bd1ee74b0fab3e17 in term 2.
I20240503 23:26:37.629338 4556 tablet_bootstrap.cc:492] T 98c9500acb43423b86f54ce7a4d98a72 P e4086c099d7f4447b2b4733b6ca364df: Bootstrap replayed 1/1 log segments. Stats: ops{read=29 overwritten=0 applied=29 ignored=0} inserts{seen=102 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 23:26:37.629853 4556 tablet_bootstrap.cc:492] T 98c9500acb43423b86f54ce7a4d98a72 P e4086c099d7f4447b2b4733b6ca364df: Bootstrap complete.
I20240503 23:26:37.630129 4556 ts_tablet_manager.cc:1367] T 98c9500acb43423b86f54ce7a4d98a72 P e4086c099d7f4447b2b4733b6ca364df: Time spent bootstrapping tablet: real 0.039s user 0.032s sys 0.004s
I20240503 23:26:37.630919 4556 raft_consensus.cc:348] T 98c9500acb43423b86f54ce7a4d98a72 P e4086c099d7f4447b2b4733b6ca364df [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "83ffd7184ec94302bd1ee74b0fab3e17" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 32909 } } peers { permanent_uuid: "a2f9bf2e5d9b4fc19f804797bca4a3ff" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 46261 } attrs { replace: true } } peers { permanent_uuid: "4d1d4e261584467fbbce4c72fa18045a" member_type: VOTER last_known_addr { host: "127.0.100.6" port: 35349 } attrs { promote: false } } peers { permanent_uuid: "e4086c099d7f4447b2b4733b6ca364df" member_type: NON_VOTER last_known_addr { host: "127.0.100.4" port: 35011 } attrs { promote: true } }
I20240503 23:26:37.631161 4556 raft_consensus.cc:729] T 98c9500acb43423b86f54ce7a4d98a72 P e4086c099d7f4447b2b4733b6ca364df [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: e4086c099d7f4447b2b4733b6ca364df, State: Initialized, Role: LEARNER
I20240503 23:26:37.631387 4556 consensus_queue.cc:260] T 98c9500acb43423b86f54ce7a4d98a72 P e4086c099d7f4447b2b4733b6ca364df [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 29, Last appended: 1.29, Last appended by leader: 29, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "83ffd7184ec94302bd1ee74b0fab3e17" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 32909 } } peers { permanent_uuid: "a2f9bf2e5d9b4fc19f804797bca4a3ff" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 46261 } attrs { replace: true } } peers { permanent_uuid: "4d1d4e261584467fbbce4c72fa18045a" member_type: VOTER last_known_addr { host: "127.0.100.6" port: 35349 } attrs { promote: false } } peers { permanent_uuid: "e4086c099d7f4447b2b4733b6ca364df" member_type: NON_VOTER last_known_addr { host: "127.0.100.4" port: 35011 } attrs { promote: true } }
I20240503 23:26:37.632098 4556 ts_tablet_manager.cc:1398] T 98c9500acb43423b86f54ce7a4d98a72 P e4086c099d7f4447b2b4733b6ca364df: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20240503 23:26:37.632758 3774 tablet_copy_service.cc:342] P 83ffd7184ec94302bd1ee74b0fab3e17: Request end of tablet copy session e4086c099d7f4447b2b4733b6ca364df-98c9500acb43423b86f54ce7a4d98a72 received from {username='slave'} at 127.0.0.1:57918
I20240503 23:26:37.632861 3774 tablet_copy_service.cc:433] P 83ffd7184ec94302bd1ee74b0fab3e17: ending tablet copy session e4086c099d7f4447b2b4733b6ca364df-98c9500acb43423b86f54ce7a4d98a72 on tablet 98c9500acb43423b86f54ce7a4d98a72 with peer e4086c099d7f4447b2b4733b6ca364df
I20240503 23:26:37.633095 4560 tablet_bootstrap.cc:492] T 487ac5b9bf2d4de2b80a1b13387a2a77 P 692da0319a2d4a81ad81f5d8f9f4c86b: Bootstrap replayed 1/1 log segments. Stats: ops{read=27 overwritten=0 applied=27 ignored=0} inserts{seen=113 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 23:26:37.633467 4560 tablet_bootstrap.cc:492] T 487ac5b9bf2d4de2b80a1b13387a2a77 P 692da0319a2d4a81ad81f5d8f9f4c86b: Bootstrap complete.
I20240503 23:26:37.633790 4560 ts_tablet_manager.cc:1367] T 487ac5b9bf2d4de2b80a1b13387a2a77 P 692da0319a2d4a81ad81f5d8f9f4c86b: Time spent bootstrapping tablet: real 0.043s user 0.040s sys 0.001s
I20240503 23:26:37.634595 4560 raft_consensus.cc:348] T 487ac5b9bf2d4de2b80a1b13387a2a77 P 692da0319a2d4a81ad81f5d8f9f4c86b [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "83ffd7184ec94302bd1ee74b0fab3e17" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 32909 } } peers { permanent_uuid: "a2f9bf2e5d9b4fc19f804797bca4a3ff" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 46261 } } peers { permanent_uuid: "a3eecbaec43d4954b1096bc3121e824c" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 40649 } attrs { replace: true } } peers { permanent_uuid: "692da0319a2d4a81ad81f5d8f9f4c86b" member_type: NON_VOTER last_known_addr { host: "127.0.100.5" port: 38561 } attrs { promote: true } }
I20240503 23:26:37.634874 4560 raft_consensus.cc:729] T 487ac5b9bf2d4de2b80a1b13387a2a77 P 692da0319a2d4a81ad81f5d8f9f4c86b [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 692da0319a2d4a81ad81f5d8f9f4c86b, State: Initialized, Role: LEARNER
I20240503 23:26:37.635094 4560 consensus_queue.cc:260] T 487ac5b9bf2d4de2b80a1b13387a2a77 P 692da0319a2d4a81ad81f5d8f9f4c86b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 27, Last appended: 1.27, Last appended by leader: 27, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "83ffd7184ec94302bd1ee74b0fab3e17" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 32909 } } peers { permanent_uuid: "a2f9bf2e5d9b4fc19f804797bca4a3ff" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 46261 } } peers { permanent_uuid: "a3eecbaec43d4954b1096bc3121e824c" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 40649 } attrs { replace: true } } peers { permanent_uuid: "692da0319a2d4a81ad81f5d8f9f4c86b" member_type: NON_VOTER last_known_addr { host: "127.0.100.5" port: 38561 } attrs { promote: true } }
I20240503 23:26:37.635807 4560 ts_tablet_manager.cc:1398] T 487ac5b9bf2d4de2b80a1b13387a2a77 P 692da0319a2d4a81ad81f5d8f9f4c86b: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20240503 23:26:37.636411 3774 tablet_copy_service.cc:342] P 83ffd7184ec94302bd1ee74b0fab3e17: Request end of tablet copy session 692da0319a2d4a81ad81f5d8f9f4c86b-487ac5b9bf2d4de2b80a1b13387a2a77 received from {username='slave'} at 127.0.0.1:57952
I20240503 23:26:37.636528 3774 tablet_copy_service.cc:433] P 83ffd7184ec94302bd1ee74b0fab3e17: ending tablet copy session 692da0319a2d4a81ad81f5d8f9f4c86b-487ac5b9bf2d4de2b80a1b13387a2a77 on tablet 487ac5b9bf2d4de2b80a1b13387a2a77 with peer 692da0319a2d4a81ad81f5d8f9f4c86b
I20240503 23:26:37.784600 4610 raft_consensus.cc:979] T d0c4430fc56644cb9f87deffe4fe9407 P a2f9bf2e5d9b4fc19f804797bca4a3ff: : Instructing follower 83ffd7184ec94302bd1ee74b0fab3e17 to start an election
I20240503 23:26:37.784785 4611 raft_consensus.cc:1067] T d0c4430fc56644cb9f87deffe4fe9407 P a2f9bf2e5d9b4fc19f804797bca4a3ff [term 3 LEADER]: Signalling peer 83ffd7184ec94302bd1ee74b0fab3e17 to start an election
I20240503 23:26:37.785327 3762 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "d0c4430fc56644cb9f87deffe4fe9407"
dest_uuid: "83ffd7184ec94302bd1ee74b0fab3e17"
from {username='slave'} at 127.0.0.1:35586
I20240503 23:26:37.785511 3762 raft_consensus.cc:482] T d0c4430fc56644cb9f87deffe4fe9407 P 83ffd7184ec94302bd1ee74b0fab3e17 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 23:26:37.785612 3762 raft_consensus.cc:3037] T d0c4430fc56644cb9f87deffe4fe9407 P 83ffd7184ec94302bd1ee74b0fab3e17 [term 3 FOLLOWER]: Advancing to term 4
I20240503 23:26:37.788390 3762 raft_consensus.cc:504] T d0c4430fc56644cb9f87deffe4fe9407 P 83ffd7184ec94302bd1ee74b0fab3e17 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "83ffd7184ec94302bd1ee74b0fab3e17" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 32909 } attrs { replace: true } } peers { permanent_uuid: "a2f9bf2e5d9b4fc19f804797bca4a3ff" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 46261 } } peers { permanent_uuid: "a3eecbaec43d4954b1096bc3121e824c" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 40649 } } peers { permanent_uuid: "4d1d4e261584467fbbce4c72fa18045a" member_type: NON_VOTER last_known_addr { host: "127.0.100.6" port: 35349 } attrs { promote: true } }
I20240503 23:26:37.789604 3837 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0c4430fc56644cb9f87deffe4fe9407" candidate_uuid: "83ffd7184ec94302bd1ee74b0fab3e17" candidate_term: 4 candidate_status { last_received { term: 3 index: 29 } } ignore_live_leader: true dest_uuid: "a2f9bf2e5d9b4fc19f804797bca4a3ff"
I20240503 23:26:37.789758 3762 leader_election.cc:288] T d0c4430fc56644cb9f87deffe4fe9407 P 83ffd7184ec94302bd1ee74b0fab3e17 [CANDIDATE]: Term 4 election: Requested vote from peers a2f9bf2e5d9b4fc19f804797bca4a3ff (127.0.100.2:46261), a3eecbaec43d4954b1096bc3121e824c (127.0.100.3:40649)
I20240503 23:26:37.789863 3837 raft_consensus.cc:3032] T d0c4430fc56644cb9f87deffe4fe9407 P a2f9bf2e5d9b4fc19f804797bca4a3ff [term 3 LEADER]: Stepping down as leader of term 3
I20240503 23:26:37.789940 3837 raft_consensus.cc:729] T d0c4430fc56644cb9f87deffe4fe9407 P a2f9bf2e5d9b4fc19f804797bca4a3ff [term 3 LEADER]: Becoming Follower/Learner. State: Replica: a2f9bf2e5d9b4fc19f804797bca4a3ff, State: Running, Role: LEADER
I20240503 23:26:37.790282 3837 consensus_queue.cc:260] T d0c4430fc56644cb9f87deffe4fe9407 P a2f9bf2e5d9b4fc19f804797bca4a3ff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 29, Committed index: 29, Last appended: 3.29, Last appended by leader: 29, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "83ffd7184ec94302bd1ee74b0fab3e17" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 32909 } attrs { replace: true } } peers { permanent_uuid: "a2f9bf2e5d9b4fc19f804797bca4a3ff" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 46261 } } peers { permanent_uuid: "a3eecbaec43d4954b1096bc3121e824c" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 40649 } } peers { permanent_uuid: "4d1d4e261584467fbbce4c72fa18045a" member_type: NON_VOTER last_known_addr { host: "127.0.100.6" port: 35349 } attrs { promote: true } }
I20240503 23:26:37.790848 3837 raft_consensus.cc:3037] T d0c4430fc56644cb9f87deffe4fe9407 P a2f9bf2e5d9b4fc19f804797bca4a3ff [term 3 FOLLOWER]: Advancing to term 4
I20240503 23:26:37.793417 3907 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0c4430fc56644cb9f87deffe4fe9407" candidate_uuid: "83ffd7184ec94302bd1ee74b0fab3e17" candidate_term: 4 candidate_status { last_received { term: 3 index: 29 } } ignore_live_leader: true dest_uuid: "a3eecbaec43d4954b1096bc3121e824c"
I20240503 23:26:37.793722 3907 raft_consensus.cc:3037] T d0c4430fc56644cb9f87deffe4fe9407 P a3eecbaec43d4954b1096bc3121e824c [term 3 FOLLOWER]: Advancing to term 4
I20240503 23:26:37.794647 3837 raft_consensus.cc:2454] T d0c4430fc56644cb9f87deffe4fe9407 P a2f9bf2e5d9b4fc19f804797bca4a3ff [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83ffd7184ec94302bd1ee74b0fab3e17 in term 4.
I20240503 23:26:37.795212 3730 leader_election.cc:302] T d0c4430fc56644cb9f87deffe4fe9407 P 83ffd7184ec94302bd1ee74b0fab3e17 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 83ffd7184ec94302bd1ee74b0fab3e17, a2f9bf2e5d9b4fc19f804797bca4a3ff; no voters:
I20240503 23:26:37.795590 4613 raft_consensus.cc:2781] T d0c4430fc56644cb9f87deffe4fe9407 P 83ffd7184ec94302bd1ee74b0fab3e17 [term 4 FOLLOWER]: Leader election won for term 4
I20240503 23:26:37.795727 4613 raft_consensus.cc:686] T d0c4430fc56644cb9f87deffe4fe9407 P 83ffd7184ec94302bd1ee74b0fab3e17 [term 4 LEADER]: Becoming Leader. State: Replica: 83ffd7184ec94302bd1ee74b0fab3e17, State: Running, Role: LEADER
I20240503 23:26:37.796083 4613 consensus_queue.cc:237] T d0c4430fc56644cb9f87deffe4fe9407 P 83ffd7184ec94302bd1ee74b0fab3e17 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 29, Committed index: 29, Last appended: 3.29, Last appended by leader: 29, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "83ffd7184ec94302bd1ee74b0fab3e17" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 32909 } attrs { replace: true } } peers { permanent_uuid: "a2f9bf2e5d9b4fc19f804797bca4a3ff" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 46261 } } peers { permanent_uuid: "a3eecbaec43d4954b1096bc3121e824c" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 40649 } } peers { permanent_uuid: "4d1d4e261584467fbbce4c72fa18045a" member_type: NON_VOTER last_known_addr { host: "127.0.100.6" port: 35349 } attrs { promote: true } }
I20240503 23:26:37.796830 3907 raft_consensus.cc:2454] T d0c4430fc56644cb9f87deffe4fe9407 P a3eecbaec43d4954b1096bc3121e824c [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83ffd7184ec94302bd1ee74b0fab3e17 in term 4.
I20240503 23:26:37.800048 3655 catalog_manager.cc:5484] T d0c4430fc56644cb9f87deffe4fe9407 P 83ffd7184ec94302bd1ee74b0fab3e17 reported cstate change: term changed from 3 to 4, leader changed from a2f9bf2e5d9b4fc19f804797bca4a3ff (127.0.100.2) to 83ffd7184ec94302bd1ee74b0fab3e17 (127.0.100.1). New cstate: current_term: 4 leader_uuid: "83ffd7184ec94302bd1ee74b0fab3e17" committed_config { opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "83ffd7184ec94302bd1ee74b0fab3e17" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 32909 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a2f9bf2e5d9b4fc19f804797bca4a3ff" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 46261 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a3eecbaec43d4954b1096bc3121e824c" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 40649 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4d1d4e261584467fbbce4c72fa18045a" member_type: NON_VOTER last_known_addr { host: "127.0.100.6" port: 35349 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 23:26:37.924825 4217 raft_consensus.cc:1261] T 7181e4e8c0324a8eb095dde23651a1ab P e4086c099d7f4447b2b4733b6ca364df [term 2 FOLLOWER]: Refusing update from remote peer a3eecbaec43d4954b1096bc3121e824c: Log matching property violated. Preceding OpId in replica: term: 1 index: 28. Preceding OpId from leader: term: 2 index: 29. (index mismatch)
I20240503 23:26:37.925432 4595 consensus_queue.cc:1035] T 7181e4e8c0324a8eb095dde23651a1ab P a3eecbaec43d4954b1096bc3121e824c [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4086c099d7f4447b2b4733b6ca364df" member_type: VOTER last_known_addr { host: "127.0.100.4" port: 35011 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.000s
I20240503 23:26:37.932142 4218 raft_consensus.cc:1203] T 60267cb4849d4823a0afb2380305905f P e4086c099d7f4447b2b4733b6ca364df [term 1 LEARNER]: Deduplicated request from leader. Original: 1.25->[1.26-1.26] Dedup: 1.26->[]
I20240503 23:26:37.946784 3762 raft_consensus.cc:1261] T 7181e4e8c0324a8eb095dde23651a1ab P 83ffd7184ec94302bd1ee74b0fab3e17 [term 2 FOLLOWER]: Refusing update from remote peer a3eecbaec43d4954b1096bc3121e824c: Log matching property violated. Preceding OpId in replica: term: 1 index: 28. Preceding OpId from leader: term: 2 index: 29. (index mismatch)
I20240503 23:26:37.947803 4595 consensus_queue.cc:1035] T 7181e4e8c0324a8eb095dde23651a1ab P a3eecbaec43d4954b1096bc3121e824c [LEADER]: Connected to new peer: Peer: permanent_uuid: "83ffd7184ec94302bd1ee74b0fab3e17" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 32909 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.000s
I20240503 23:26:37.977572 4293 raft_consensus.cc:3037] T a0e18d0975a54e3dbeb0a13cb226fefd P 692da0319a2d4a81ad81f5d8f9f4c86b [term 1 LEARNER]: Advancing to term 2
I20240503 23:26:37.980876 4293 raft_consensus.cc:1261] T a0e18d0975a54e3dbeb0a13cb226fefd P 692da0319a2d4a81ad81f5d8f9f4c86b [term 2 LEARNER]: Refusing update from remote peer 83ffd7184ec94302bd1ee74b0fab3e17: Log matching property violated. Preceding OpId in replica: term: 1 index: 27. Preceding OpId from leader: term: 2 index: 28. (index mismatch)
I20240503 23:26:37.981599 4613 consensus_queue.cc:1035] T a0e18d0975a54e3dbeb0a13cb226fefd P 83ffd7184ec94302bd1ee74b0fab3e17 [LEADER]: Connected to new peer: Peer: permanent_uuid: "692da0319a2d4a81ad81f5d8f9f4c86b" member_type: NON_VOTER last_known_addr { host: "127.0.100.5" port: 38561 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 28, Last known committed idx: 27, Time since last communication: 0.000s
I20240503 23:26:37.986968 4613 raft_consensus.cc:1050] T a0e18d0975a54e3dbeb0a13cb226fefd P 83ffd7184ec94302bd1ee74b0fab3e17: attempting to promote NON_VOTER 692da0319a2d4a81ad81f5d8f9f4c86b to VOTER
W20240503 23:26:37.987181 4613 raft_consensus.cc:1054] T a0e18d0975a54e3dbeb0a13cb226fefd P 83ffd7184ec94302bd1ee74b0fab3e17: Unable to promote non-voter 692da0319a2d4a81ad81f5d8f9f4c86b: Illegal state: Leader has not yet committed an operation in its own term
I20240503 23:26:37.990854 3907 raft_consensus.cc:1261] T a0e18d0975a54e3dbeb0a13cb226fefd P a3eecbaec43d4954b1096bc3121e824c [term 2 FOLLOWER]: Refusing update from remote peer 83ffd7184ec94302bd1ee74b0fab3e17: Log matching property violated. Preceding OpId in replica: term: 1 index: 27. Preceding OpId from leader: term: 2 index: 28. (index mismatch)
I20240503 23:26:37.991470 4613 consensus_queue.cc:1035] T a0e18d0975a54e3dbeb0a13cb226fefd P 83ffd7184ec94302bd1ee74b0fab3e17 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a3eecbaec43d4954b1096bc3121e824c" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 40649 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 28, Last known committed idx: 27, Time since last communication: 0.000s
I20240503 23:26:37.997310 3837 raft_consensus.cc:1261] T a0e18d0975a54e3dbeb0a13cb226fefd P a2f9bf2e5d9b4fc19f804797bca4a3ff [term 2 FOLLOWER]: Refusing update from remote peer 83ffd7184ec94302bd1ee74b0fab3e17: Log matching property violated. Preceding OpId in replica: term: 1 index: 27. Preceding OpId from leader: term: 2 index: 28. (index mismatch)
I20240503 23:26:37.998195 4532 consensus_queue.cc:1035] T a0e18d0975a54e3dbeb0a13cb226fefd P 83ffd7184ec94302bd1ee74b0fab3e17 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2f9bf2e5d9b4fc19f804797bca4a3ff" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 46261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 28, Last known committed idx: 27, Time since last communication: 0.000s
I20240503 23:26:38.007395 4532 raft_consensus.cc:1050] T a0e18d0975a54e3dbeb0a13cb226fefd P 83ffd7184ec94302bd1ee74b0fab3e17: attempting to promote NON_VOTER 692da0319a2d4a81ad81f5d8f9f4c86b to VOTER
I20240503 23:26:38.009125 4532 consensus_queue.cc:237] T a0e18d0975a54e3dbeb0a13cb226fefd P 83ffd7184ec94302bd1ee74b0fab3e17 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 28, Committed index: 28, Last appended: 2.28, Last appended by leader: 27, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "83ffd7184ec94302bd1ee74b0fab3e17" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 32909 } } peers { permanent_uuid: "a2f9bf2e5d9b4fc19f804797bca4a3ff" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 46261 } } peers { permanent_uuid: "a3eecbaec43d4954b1096bc3121e824c" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 40649 } attrs { replace: true } } peers { permanent_uuid: "692da0319a2d4a81ad81f5d8f9f4c86b" member_type: VOTER last_known_addr { host: "127.0.100.5" port: 38561 } attrs { promote: false } }
I20240503 23:26:38.014917 3907 raft_consensus.cc:1261] T a0e18d0975a54e3dbeb0a13cb226fefd P a3eecbaec43d4954b1096bc3121e824c [term 2 FOLLOWER]: Refusing update from remote peer 83ffd7184ec94302bd1ee74b0fab3e17: Log matching property violated. Preceding OpId in replica: term: 2 index: 28. Preceding OpId from leader: term: 2 index: 29. (index mismatch)
I20240503 23:26:38.016405 4293 raft_consensus.cc:1261] T a0e18d0975a54e3dbeb0a13cb226fefd P 692da0319a2d4a81ad81f5d8f9f4c86b [term 2 LEARNER]: Refusing update from remote peer 83ffd7184ec94302bd1ee74b0fab3e17: Log matching property violated. Preceding OpId in replica: term: 2 index: 28. Preceding OpId from leader: term: 2 index: 29. (index mismatch)
I20240503 23:26:38.018461 4635 consensus_queue.cc:1035] T a0e18d0975a54e3dbeb0a13cb226fefd P 83ffd7184ec94302bd1ee74b0fab3e17 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a3eecbaec43d4954b1096bc3121e824c" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 40649 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.000s
I20240503 23:26:38.019168 4613 consensus_queue.cc:1035] T a0e18d0975a54e3dbeb0a13cb226fefd P 83ffd7184ec94302bd1ee74b0fab3e17 [LEADER]: Connected to new peer: Peer: permanent_uuid: "692da0319a2d4a81ad81f5d8f9f4c86b" member_type: VOTER last_known_addr { host: "127.0.100.5" port: 38561 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.000s
I20240503 23:26:38.023551 4635 raft_consensus.cc:2932] T a0e18d0975a54e3dbeb0a13cb226fefd P 83ffd7184ec94302bd1ee74b0fab3e17 [term 2 LEADER]: Committing config change with OpId 2.29: config changed from index 27 to 29, 692da0319a2d4a81ad81f5d8f9f4c86b (127.0.100.5) changed from NON_VOTER to VOTER. New config: { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "83ffd7184ec94302bd1ee74b0fab3e17" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 32909 } } peers { permanent_uuid: "a2f9bf2e5d9b4fc19f804797bca4a3ff" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 46261 } } peers { permanent_uuid: "a3eecbaec43d4954b1096bc3121e824c" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 40649 } attrs { replace: true } } peers { permanent_uuid: "692da0319a2d4a81ad81f5d8f9f4c86b" member_type: VOTER last_known_addr { host: "127.0.100.5" port: 38561 } attrs { promote: false } } }
I20240503 23:26:38.024819 3838 raft_consensus.cc:1261] T a0e18d0975a54e3dbeb0a13cb226fefd P a2f9bf2e5d9b4fc19f804797bca4a3ff [term 2 FOLLOWER]: Refusing update from remote peer 83ffd7184ec94302bd1ee74b0fab3e17: Log matching property violated. Preceding OpId in replica: term: 2 index: 28. Preceding OpId from leader: term: 2 index: 29. (index mismatch)
I20240503 23:26:38.024757 3907 raft_consensus.cc:2932] T a0e18d0975a54e3dbeb0a13cb226fefd P a3eecbaec43d4954b1096bc3121e824c [term 2 FOLLOWER]: Committing config change with OpId 2.29: config changed from index 27 to 29, 692da0319a2d4a81ad81f5d8f9f4c86b (127.0.100.5) changed from NON_VOTER to VOTER. New config: { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "83ffd7184ec94302bd1ee74b0fab3e17" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 32909 } } peers { permanent_uuid: "a2f9bf2e5d9b4fc19f804797bca4a3ff" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 46261 } } peers { permanent_uuid: "a3eecbaec43d4954b1096bc3121e824c" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 40649 } attrs { replace: true } } peers { permanent_uuid: "692da0319a2d4a81ad81f5d8f9f4c86b" member_type: VOTER last_known_addr { host: "127.0.100.5" port: 38561 } attrs { promote: false } } }
I20240503 23:26:38.025804 4613 consensus_queue.cc:1035] T a0e18d0975a54e3dbeb0a13cb226fefd P 83ffd7184ec94302bd1ee74b0fab3e17 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2f9bf2e5d9b4fc19f804797bca4a3ff" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 46261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.000s
I20240503 23:26:38.027483 3837 raft_consensus.cc:2932] T a0e18d0975a54e3dbeb0a13cb226fefd P a2f9bf2e5d9b4fc19f804797bca4a3ff [term 2 FOLLOWER]: Committing config change with OpId 2.29: config changed from index 27 to 29, 692da0319a2d4a81ad81f5d8f9f4c86b (127.0.100.5) changed from NON_VOTER to VOTER. New config: { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "83ffd7184ec94302bd1ee74b0fab3e17" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 32909 } } peers { permanent_uuid: "a2f9bf2e5d9b4fc19f804797bca4a3ff" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 46261 } } peers { permanent_uuid: "a3eecbaec43d4954b1096bc3121e824c" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 40649 } attrs { replace: true } } peers { permanent_uuid: "692da0319a2d4a81ad81f5d8f9f4c86b" member_type: VOTER last_known_addr { host: "127.0.100.5" port: 38561 } attrs { promote: false } } }
I20240503 23:26:38.033272 3659 catalog_manager.cc:5484] T a0e18d0975a54e3dbeb0a13cb226fefd P a3eecbaec43d4954b1096bc3121e824c reported cstate change: config changed from index 27 to 29, 692da0319a2d4a81ad81f5d8f9f4c86b (127.0.100.5) changed from NON_VOTER to VOTER. New cstate: current_term: 2 leader_uuid: "83ffd7184ec94302bd1ee74b0fab3e17" committed_config { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "83ffd7184ec94302bd1ee74b0fab3e17" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 32909 } } peers { permanent_uuid: "a2f9bf2e5d9b4fc19f804797bca4a3ff" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 46261 } } peers { permanent_uuid: "a3eecbaec43d4954b1096bc3121e824c" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 40649 } attrs { replace: true } } peers { permanent_uuid: "692da0319a2d4a81ad81f5d8f9f4c86b" member_type: VOTER last_known_addr { host: "127.0.100.5" port: 38561 } attrs { promote: false } } }
I20240503 23:26:38.038671 4293 raft_consensus.cc:2932] T a0e18d0975a54e3dbeb0a13cb226fefd P 692da0319a2d4a81ad81f5d8f9f4c86b [term 2 FOLLOWER]: Committing config change with OpId 2.29: config changed from index 27 to 29, 692da0319a2d4a81ad81f5d8f9f4c86b (127.0.100.5) changed from NON_VOTER to VOTER. New config: { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "83ffd7184ec94302bd1ee74b0fab3e17" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 32909 } } peers { permanent_uuid: "a2f9bf2e5d9b4fc19f804797bca4a3ff" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 46261 } } peers { permanent_uuid: "a3eecbaec43d4954b1096bc3121e824c" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 40649 } attrs { replace: true } } peers { permanent_uuid: "692da0319a2d4a81ad81f5d8f9f4c86b" member_type: VOTER last_known_addr { host: "127.0.100.5" port: 38561 } attrs { promote: false } } }
I20240503 23:26:38.050693 3762 consensus_queue.cc:237] T a0e18d0975a54e3dbeb0a13cb226fefd P 83ffd7184ec94302bd1ee74b0fab3e17 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 29, Committed index: 29, Last appended: 2.29, Last appended by leader: 27, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "83ffd7184ec94302bd1ee74b0fab3e17" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 32909 } } peers { permanent_uuid: "a2f9bf2e5d9b4fc19f804797bca4a3ff" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 46261 } } peers { permanent_uuid: "692da0319a2d4a81ad81f5d8f9f4c86b" member_type: VOTER last_known_addr { host: "127.0.100.5" port: 38561 } attrs { promote: false } }
I20240503 23:26:38.055070 4293 raft_consensus.cc:1261] T a0e18d0975a54e3dbeb0a13cb226fefd P 692da0319a2d4a81ad81f5d8f9f4c86b [term 2 FOLLOWER]: Refusing update from remote peer 83ffd7184ec94302bd1ee74b0fab3e17: Log matching property violated. Preceding OpId in replica: term: 2 index: 29. Preceding OpId from leader: term: 2 index: 30. (index mismatch)
I20240503 23:26:38.056216 3837 raft_consensus.cc:1261] T a0e18d0975a54e3dbeb0a13cb226fefd P a2f9bf2e5d9b4fc19f804797bca4a3ff [term 2 FOLLOWER]: Refusing update from remote peer 83ffd7184ec94302bd1ee74b0fab3e17: Log matching property violated. Preceding OpId in replica: term: 2 index: 29. Preceding OpId from leader: term: 2 index: 30. (index mismatch)
I20240503 23:26:38.056721 4613 consensus_queue.cc:1035] T a0e18d0975a54e3dbeb0a13cb226fefd P 83ffd7184ec94302bd1ee74b0fab3e17 [LEADER]: Connected to new peer: Peer: permanent_uuid: "692da0319a2d4a81ad81f5d8f9f4c86b" member_type: VOTER last_known_addr { host: "127.0.100.5" port: 38561 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 30, Last known committed idx: 29, Time since last communication: 0.000s
I20240503 23:26:38.057384 4613 consensus_queue.cc:1035] T a0e18d0975a54e3dbeb0a13cb226fefd P 83ffd7184ec94302bd1ee74b0fab3e17 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2f9bf2e5d9b4fc19f804797bca4a3ff" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 46261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 30, Last known committed idx: 29, Time since last communication: 0.000s
I20240503 23:26:38.060895 4640 raft_consensus.cc:2932] T a0e18d0975a54e3dbeb0a13cb226fefd P 83ffd7184ec94302bd1ee74b0fab3e17 [term 2 LEADER]: Committing config change with OpId 2.30: config changed from index 29 to 30, VOTER a3eecbaec43d4954b1096bc3121e824c (127.0.100.3) evicted. New config: { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "83ffd7184ec94302bd1ee74b0fab3e17" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 32909 } } peers { permanent_uuid: "a2f9bf2e5d9b4fc19f804797bca4a3ff" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 46261 } } peers { permanent_uuid: "692da0319a2d4a81ad81f5d8f9f4c86b" member_type: VOTER last_known_addr { host: "127.0.100.5" port: 38561 } attrs { promote: false } } }
I20240503 23:26:38.062297 4293 raft_consensus.cc:2932] T a0e18d0975a54e3dbeb0a13cb226fefd P 692da0319a2d4a81ad81f5d8f9f4c86b [term 2 FOLLOWER]: Committing config change with OpId 2.30: config changed from index 29 to 30, VOTER a3eecbaec43d4954b1096bc3121e824c (127.0.100.3) evicted. New config: { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "83ffd7184ec94302bd1ee74b0fab3e17" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 32909 } } peers { permanent_uuid: "a2f9bf2e5d9b4fc19f804797bca4a3ff" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 46261 } } peers { permanent_uuid: "692da0319a2d4a81ad81f5d8f9f4c86b" member_type: VOTER last_known_addr { host: "127.0.100.5" port: 38561 } attrs { promote: false } } }
I20240503 23:26:38.063623 3837 raft_consensus.cc:2932] T a0e18d0975a54e3dbeb0a13cb226fefd P a2f9bf2e5d9b4fc19f804797bca4a3ff [term 2 FOLLOWER]: Committing config change with OpId 2.30: config changed from index 29 to 30, VOTER a3eecbaec43d4954b1096bc3121e824c (127.0.100.3) evicted. New config: { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "83ffd7184ec94302bd1ee74b0fab3e17" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 32909 } } peers { permanent_uuid: "a2f9bf2e5d9b4fc19f804797bca4a3ff" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 46261 } } peers { permanent_uuid: "692da0319a2d4a81ad81f5d8f9f4c86b" member_type: VOTER last_known_addr { host: "127.0.100.5" port: 38561 } attrs { promote: false } } }
I20240503 23:26:38.068783 3659 catalog_manager.cc:5484] T a0e18d0975a54e3dbeb0a13cb226fefd P 692da0319a2d4a81ad81f5d8f9f4c86b reported cstate change: config changed from index 29 to 30, VOTER a3eecbaec43d4954b1096bc3121e824c (127.0.100.3) evicted. New cstate: current_term: 2 leader_uuid: "83ffd7184ec94302bd1ee74b0fab3e17" committed_config { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "83ffd7184ec94302bd1ee74b0fab3e17" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 32909 } } peers { permanent_uuid: "a2f9bf2e5d9b4fc19f804797bca4a3ff" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 46261 } } peers { permanent_uuid: "692da0319a2d4a81ad81f5d8f9f4c86b" member_type: VOTER last_known_addr { host: "127.0.100.5" port: 38561 } attrs { promote: false } } }
I20240503 23:26:38.071960 3648 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet a0e18d0975a54e3dbeb0a13cb226fefd with cas_config_opid_index 29: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 23:26:38.076488 3902 tablet_service.cc:1514] Processing DeleteTablet for tablet a0e18d0975a54e3dbeb0a13cb226fefd with delete_type TABLET_DATA_TOMBSTONED (TS a3eecbaec43d4954b1096bc3121e824c not found in new config with opid_index 30) from {username='slave'} at 127.0.0.1:40632
I20240503 23:26:38.081286 4641 tablet_replica.cc:330] T a0e18d0975a54e3dbeb0a13cb226fefd P a3eecbaec43d4954b1096bc3121e824c: stopping tablet replica
I20240503 23:26:38.081648 4641 raft_consensus.cc:2229] T a0e18d0975a54e3dbeb0a13cb226fefd P a3eecbaec43d4954b1096bc3121e824c [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 23:26:38.081960 4641 raft_consensus.cc:2258] T a0e18d0975a54e3dbeb0a13cb226fefd P a3eecbaec43d4954b1096bc3121e824c [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 23:26:38.083747 4641 ts_tablet_manager.cc:1875] T a0e18d0975a54e3dbeb0a13cb226fefd P a3eecbaec43d4954b1096bc3121e824c: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 23:26:38.091017 4641 ts_tablet_manager.cc:1888] T a0e18d0975a54e3dbeb0a13cb226fefd P a3eecbaec43d4954b1096bc3121e824c: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.29
I20240503 23:26:38.091358 4641 log.cc:1196] T a0e18d0975a54e3dbeb0a13cb226fefd P a3eecbaec43d4954b1096bc3121e824c: Deleting WAL directory at /tmp/dist-test-taskZhIiPz/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714778688451250-400-0/minicluster-data/ts-2-root/wals/a0e18d0975a54e3dbeb0a13cb226fefd
I20240503 23:26:38.092970 3646 catalog_manager.cc:4830] TS a3eecbaec43d4954b1096bc3121e824c (127.0.100.3:40649): tablet a0e18d0975a54e3dbeb0a13cb226fefd (table test-workload [id=1345b2a336c84ec2b1de62853e7fa3f6]) successfully deleted
I20240503 23:26:38.103809 4293 raft_consensus.cc:3037] T 487ac5b9bf2d4de2b80a1b13387a2a77 P 692da0319a2d4a81ad81f5d8f9f4c86b [term 1 LEARNER]: Advancing to term 2
I20240503 23:26:38.150122 4217 raft_consensus.cc:1203] T 98c9500acb43423b86f54ce7a4d98a72 P e4086c099d7f4447b2b4733b6ca364df [term 1 LEARNER]: Deduplicated request from leader. Original: 1.28->[1.29-1.29] Dedup: 1.29->[]
W20240503 23:26:38.367206 3728 consensus_peers.cc:485] T d0c4430fc56644cb9f87deffe4fe9407 P 83ffd7184ec94302bd1ee74b0fab3e17 -> Peer 4d1d4e261584467fbbce4c72fa18045a (127.0.100.6:35349): Couldn't send request to peer 4d1d4e261584467fbbce4c72fa18045a. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20240503 23:26:38.384128 3907 raft_consensus.cc:1261] T d0c4430fc56644cb9f87deffe4fe9407 P a3eecbaec43d4954b1096bc3121e824c [term 4 FOLLOWER]: Refusing update from remote peer 83ffd7184ec94302bd1ee74b0fab3e17: Log matching property violated. Preceding OpId in replica: term: 3 index: 29. Preceding OpId from leader: term: 4 index: 30. (index mismatch)
I20240503 23:26:38.384727 4640 consensus_queue.cc:1035] T d0c4430fc56644cb9f87deffe4fe9407 P 83ffd7184ec94302bd1ee74b0fab3e17 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a3eecbaec43d4954b1096bc3121e824c" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 40649 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 30, Last known committed idx: 29, Time since last communication: 0.000s
I20240503 23:26:38.394047 3837 raft_consensus.cc:1261] T d0c4430fc56644cb9f87deffe4fe9407 P a2f9bf2e5d9b4fc19f804797bca4a3ff [term 4 FOLLOWER]: Refusing update from remote peer 83ffd7184ec94302bd1ee74b0fab3e17: Log matching property violated. Preceding OpId in replica: term: 3 index: 29. Preceding OpId from leader: term: 4 index: 30. (index mismatch)
I20240503 23:26:38.396714 4640 consensus_queue.cc:1035] T d0c4430fc56644cb9f87deffe4fe9407 P 83ffd7184ec94302bd1ee74b0fab3e17 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2f9bf2e5d9b4fc19f804797bca4a3ff" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 46261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 30, Last known committed idx: 29, Time since last communication: 0.000s
I20240503 23:26:38.460603 4614 raft_consensus.cc:1050] T 60267cb4849d4823a0afb2380305905f P a3eecbaec43d4954b1096bc3121e824c: attempting to promote NON_VOTER e4086c099d7f4447b2b4733b6ca364df to VOTER
I20240503 23:26:38.461486 4614 consensus_queue.cc:237] T 60267cb4849d4823a0afb2380305905f P a3eecbaec43d4954b1096bc3121e824c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 26, Committed index: 26, Last appended: 1.26, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "83ffd7184ec94302bd1ee74b0fab3e17" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 32909 } } peers { permanent_uuid: "a2f9bf2e5d9b4fc19f804797bca4a3ff" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 46261 } attrs { replace: true } } peers { permanent_uuid: "a3eecbaec43d4954b1096bc3121e824c" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 40649 } } peers { permanent_uuid: "e4086c099d7f4447b2b4733b6ca364df" member_type: VOTER last_known_addr { host: "127.0.100.4" port: 35011 } attrs { promote: false } }
I20240503 23:26:38.464623 3760 raft_consensus.cc:1261] T 60267cb4849d4823a0afb2380305905f P 83ffd7184ec94302bd1ee74b0fab3e17 [term 1 FOLLOWER]: Refusing update from remote peer a3eecbaec43d4954b1096bc3121e824c: Log matching property violated. Preceding OpId in replica: term: 1 index: 26. Preceding OpId from leader: term: 1 index: 27. (index mismatch)
I20240503 23:26:38.464623 4217 raft_consensus.cc:1261] T 60267cb4849d4823a0afb2380305905f P e4086c099d7f4447b2b4733b6ca364df [term 1 LEARNER]: Refusing update from remote peer a3eecbaec43d4954b1096bc3121e824c: Log matching property violated. Preceding OpId in replica: term: 1 index: 26. Preceding OpId from leader: term: 1 index: 27. (index mismatch)
I20240503 23:26:38.464823 3837 raft_consensus.cc:1261] T 60267cb4849d4823a0afb2380305905f P a2f9bf2e5d9b4fc19f804797bca4a3ff [term 1 FOLLOWER]: Refusing update from remote peer a3eecbaec43d4954b1096bc3121e824c: Log matching property violated. Preceding OpId in replica: term: 1 index: 26. Preceding OpId from leader: term: 1 index: 27. (index mismatch)
I20240503 23:26:38.465179 4614 consensus_queue.cc:1035] T 60267cb4849d4823a0afb2380305905f P a3eecbaec43d4954b1096bc3121e824c [LEADER]: Connected to new peer: Peer: permanent_uuid: "83ffd7184ec94302bd1ee74b0fab3e17" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 32909 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 27, Last known committed idx: 26, Time since last communication: 0.000s
I20240503 23:26:38.465487 4595 consensus_queue.cc:1035] T 60267cb4849d4823a0afb2380305905f P a3eecbaec43d4954b1096bc3121e824c [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2f9bf2e5d9b4fc19f804797bca4a3ff" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 46261 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 27, Last known committed idx: 26, Time since last communication: 0.000s
I20240503 23:26:38.466100 4630 consensus_queue.cc:1035] T 60267cb4849d4823a0afb2380305905f P a3eecbaec43d4954b1096bc3121e824c [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4086c099d7f4447b2b4733b6ca364df" member_type: VOTER last_known_addr { host: "127.0.100.4" port: 35011 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 27, Last known committed idx: 26, Time since last communication: 0.000s
I20240503 23:26:38.477530 4614 raft_consensus.cc:2932] T 60267cb4849d4823a0afb2380305905f P a3eecbaec43d4954b1096bc3121e824c [term 1 LEADER]: Committing config change with OpId 1.27: config changed from index 26 to 27, e4086c099d7f4447b2b4733b6ca364df (127.0.100.4) changed from NON_VOTER to VOTER. New config: { opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "83ffd7184ec94302bd1ee74b0fab3e17" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 32909 } } peers { permanent_uuid: "a2f9bf2e5d9b4fc19f804797bca4a3ff" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 46261 } attrs { replace: true } } peers { permanent_uuid: "a3eecbaec43d4954b1096bc3121e824c" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 40649 } } peers { permanent_uuid: "e4086c099d7f4447b2b4733b6ca364df" member_type: VOTER last_known_addr { host: "127.0.100.4" port: 35011 } attrs { promote: false } } }
I20240503 23:26:38.478281 3837 raft_consensus.cc:2932] T 60267cb4849d4823a0afb2380305905f P a2f9bf2e5d9b4fc19f804797bca4a3ff [term 1 FOLLOWER]: Committing config change with OpId 1.27: config changed from index 26 to 27, e4086c099d7f4447b2b4733b6ca364df (127.0.100.4) changed from NON_VOTER to VOTER. New config: { opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "83ffd7184ec94302bd1ee74b0fab3e17" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 32909 } } peers { permanent_uuid: "a2f9bf2e5d9b4fc19f804797bca4a3ff" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 46261 } attrs { replace: true } } peers { permanent_uuid: "a3eecbaec43d4954b1096bc3121e824c" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 40649 } } peers { permanent_uuid: "e4086c099d7f4447b2b4733b6ca364df" member_type: VOTER last_known_addr { host: "127.0.100.4" port: 35011 } attrs { promote: false } } }
I20240503 23:26:38.482683 4218 raft_consensus.cc:2932] T 60267cb4849d4823a0afb2380305905f P e4086c099d7f4447b2b4733b6ca364df [term 1 FOLLOWER]: Committing config change with OpId 1.27: config changed from index 26 to 27, e4086c099d7f4447b2b4733b6ca364df (127.0.100.4) changed from NON_VOTER to VOTER. New config: { opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "83ffd7184ec94302bd1ee74b0fab3e17" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 32909 } } peers { permanent_uuid: "a2f9bf2e5d9b4fc19f804797bca4a3ff" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 46261 } attrs { replace: true } } peers { permanent_uuid: "a3eecbaec43d4954b1096bc3121e824c" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 40649 } } peers { permanent_uuid: "e4086c099d7f4447b2b4733b6ca364df" member_type: VOTER last_known_addr { host: "127.0.100.4" port: 35011 } attrs { promote: false } } }
I20240503 23:26:38.484194 3662 catalog_manager.cc:5484] T 60267cb4849d4823a0afb2380305905f P a3eecbaec43d4954b1096bc3121e824c reported cstate change: config changed from index 26 to 27, e4086c099d7f4447b2b4733b6ca364df (127.0.100.4) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "a3eecbaec43d4954b1096bc3121e824c" committed_config { opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "83ffd7184ec94302bd1ee74b0fab3e17" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 32909 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a2f9bf2e5d9b4fc19f804797bca4a3ff" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 46261 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a3eecbaec43d4954b1096bc3121e824c" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 40649 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e4086c099d7f4447b2b4733b6ca364df" member_type: VOTER last_known_addr { host: "127.0.100.4" port: 35011 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 23:26:38.487968 3760 raft_consensus.cc:2932] T 60267cb4849d4823a0afb2380305905f P 83ffd7184ec94302bd1ee74b0fab3e17 [term 1 FOLLOWER]: Committing config change with OpId 1.27: config changed from index 26 to 27, e4086c099d7f4447b2b4733b6ca364df (127.0.100.4) changed from NON_VOTER to VOTER. New config: { opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "83ffd7184ec94302bd1ee74b0fab3e17" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 32909 } } peers { permanent_uuid: "a2f9bf2e5d9b4fc19f804797bca4a3ff" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 46261 } attrs { replace: true } } peers { permanent_uuid: "a3eecbaec43d4954b1096bc3121e824c" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 40649 } } peers { permanent_uuid: "e4086c099d7f4447b2b4733b6ca364df" member_type: VOTER last_known_addr { host: "127.0.100.4" port: 35011 } attrs { promote: false } } }
I20240503 23:26:38.496354 3908 consensus_queue.cc:237] T 60267cb4849d4823a0afb2380305905f P a3eecbaec43d4954b1096bc3121e824c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 27, Committed index: 27, Last appended: 1.27, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "83ffd7184ec94302bd1ee74b0fab3e17" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 32909 } } peers { permanent_uuid: "a3eecbaec43d4954b1096bc3121e824c" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 40649 } } peers { permanent_uuid: "e4086c099d7f4447b2b4733b6ca364df" member_type: VOTER last_known_addr { host: "127.0.100.4" port: 35011 } attrs { promote: false } }
I20240503 23:26:38.498744 3760 raft_consensus.cc:1261] T 60267cb4849d4823a0afb2380305905f P 83ffd7184ec94302bd1ee74b0fab3e17 [term 1 FOLLOWER]: Refusing update from remote peer a3eecbaec43d4954b1096bc3121e824c: Log matching property violated. Preceding OpId in replica: term: 1 index: 27. Preceding OpId from leader: term: 1 index: 28. (index mismatch)
I20240503 23:26:38.498744 4218 raft_consensus.cc:1261] T 60267cb4849d4823a0afb2380305905f P e4086c099d7f4447b2b4733b6ca364df [term 1 FOLLOWER]: Refusing update from remote peer a3eecbaec43d4954b1096bc3121e824c: Log matching property violated. Preceding OpId in replica: term: 1 index: 27. Preceding OpId from leader: term: 1 index: 28. (index mismatch)
I20240503 23:26:38.499490 4646 consensus_queue.cc:1035] T 60267cb4849d4823a0afb2380305905f P a3eecbaec43d4954b1096bc3121e824c [LEADER]: Connected to new peer: Peer: permanent_uuid: "83ffd7184ec94302bd1ee74b0fab3e17" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 32909 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 28, Last known committed idx: 27, Time since last communication: 0.000s
I20240503 23:26:38.500057 4595 consensus_queue.cc:1035] T 60267cb4849d4823a0afb2380305905f P a3eecbaec43d4954b1096bc3121e824c [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4086c099d7f4447b2b4733b6ca364df" member_type: VOTER last_known_addr { host: "127.0.100.4" port: 35011 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 28, Last known committed idx: 27, Time since last communication: 0.000s
I20240503 23:26:38.502945 4595 raft_consensus.cc:2932] T 60267cb4849d4823a0afb2380305905f P a3eecbaec43d4954b1096bc3121e824c [term 1 LEADER]: Committing config change with OpId 1.28: config changed from index 27 to 28, VOTER a2f9bf2e5d9b4fc19f804797bca4a3ff (127.0.100.2) evicted. New config: { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "83ffd7184ec94302bd1ee74b0fab3e17" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 32909 } } peers { permanent_uuid: "a3eecbaec43d4954b1096bc3121e824c" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 40649 } } peers { permanent_uuid: "e4086c099d7f4447b2b4733b6ca364df" member_type: VOTER last_known_addr { host: "127.0.100.4" port: 35011 } attrs { promote: false } } }
I20240503 23:26:38.503506 4217 raft_consensus.cc:2932] T 60267cb4849d4823a0afb2380305905f P e4086c099d7f4447b2b4733b6ca364df [term 1 FOLLOWER]: Committing config change with OpId 1.28: config changed from index 27 to 28, VOTER a2f9bf2e5d9b4fc19f804797bca4a3ff (127.0.100.2) evicted. New config: { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "83ffd7184ec94302bd1ee74b0fab3e17" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 32909 } } peers { permanent_uuid: "a3eecbaec43d4954b1096bc3121e824c" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 40649 } } peers { permanent_uuid: "e4086c099d7f4447b2b4733b6ca364df" member_type: VOTER last_known_addr { host: "127.0.100.4" port: 35011 } attrs { promote: false } } }
I20240503 23:26:38.503628 3762 raft_consensus.cc:2932] T 60267cb4849d4823a0afb2380305905f P 83ffd7184ec94302bd1ee74b0fab3e17 [term 1 FOLLOWER]: Committing config change with OpId 1.28: config changed from index 27 to 28, VOTER a2f9bf2e5d9b4fc19f804797bca4a3ff (127.0.100.2) evicted. New config: { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "83ffd7184ec94302bd1ee74b0fab3e17" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 32909 } } peers { permanent_uuid: "a3eecbaec43d4954b1096bc3121e824c" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 40649 } } peers { permanent_uuid: "e4086c099d7f4447b2b4733b6ca364df" member_type: VOTER last_known_addr { host: "127.0.100.4" port: 35011 } attrs { promote: false } } }
I20240503 23:26:38.510008 3646 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 60267cb4849d4823a0afb2380305905f with cas_config_opid_index 27: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 23:26:38.510597 3662 catalog_manager.cc:5484] T 60267cb4849d4823a0afb2380305905f P 83ffd7184ec94302bd1ee74b0fab3e17 reported cstate change: config changed from index 27 to 28, VOTER a2f9bf2e5d9b4fc19f804797bca4a3ff (127.0.100.2) evicted. New cstate: current_term: 1 leader_uuid: "a3eecbaec43d4954b1096bc3121e824c" committed_config { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "83ffd7184ec94302bd1ee74b0fab3e17" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 32909 } } peers { permanent_uuid: "a3eecbaec43d4954b1096bc3121e824c" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 40649 } } peers { permanent_uuid: "e4086c099d7f4447b2b4733b6ca364df" member_type: VOTER last_known_addr { host: "127.0.100.4" port: 35011 } attrs { promote: false } } }
I20240503 23:26:38.515108 3824 tablet_service.cc:1514] Processing DeleteTablet for tablet 60267cb4849d4823a0afb2380305905f with delete_type TABLET_DATA_TOMBSTONED (TS a2f9bf2e5d9b4fc19f804797bca4a3ff not found in new config with opid_index 28) from {username='slave'} at 127.0.0.1:45354
I20240503 23:26:38.516283 4650 tablet_replica.cc:330] T 60267cb4849d4823a0afb2380305905f P a2f9bf2e5d9b4fc19f804797bca4a3ff: stopping tablet replica
I20240503 23:26:38.516534 4650 raft_consensus.cc:2229] T 60267cb4849d4823a0afb2380305905f P a2f9bf2e5d9b4fc19f804797bca4a3ff [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 23:26:38.516899 4650 raft_consensus.cc:2258] T 60267cb4849d4823a0afb2380305905f P a2f9bf2e5d9b4fc19f804797bca4a3ff [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 23:26:38.518077 4650 ts_tablet_manager.cc:1875] T 60267cb4849d4823a0afb2380305905f P a2f9bf2e5d9b4fc19f804797bca4a3ff: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 23:26:38.524564 4650 ts_tablet_manager.cc:1888] T 60267cb4849d4823a0afb2380305905f P a2f9bf2e5d9b4fc19f804797bca4a3ff: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.27
I20240503 23:26:38.524739 4650 log.cc:1196] T 60267cb4849d4823a0afb2380305905f P a2f9bf2e5d9b4fc19f804797bca4a3ff: Deleting WAL directory at /tmp/dist-test-taskZhIiPz/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714778688451250-400-0/minicluster-data/ts-1-root/wals/60267cb4849d4823a0afb2380305905f
I20240503 23:26:38.525527 3649 catalog_manager.cc:4830] TS a2f9bf2e5d9b4fc19f804797bca4a3ff (127.0.100.2:46261): tablet 60267cb4849d4823a0afb2380305905f (table test-workload [id=1345b2a336c84ec2b1de62853e7fa3f6]) successfully deleted
I20240503 23:26:38.529181 4601 tablet_copy_client.cc:531] T 55cf664d5e874c0d9b9adf8bdc9c43ed P 4d1d4e261584467fbbce4c72fa18045a: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 23:26:38.533599 4601 tablet_bootstrap.cc:492] T 55cf664d5e874c0d9b9adf8bdc9c43ed P 4d1d4e261584467fbbce4c72fa18045a: Bootstrap starting.
I20240503 23:26:38.537097 4640 raft_consensus.cc:1050] T 98c9500acb43423b86f54ce7a4d98a72 P 83ffd7184ec94302bd1ee74b0fab3e17: attempting to promote NON_VOTER e4086c099d7f4447b2b4733b6ca364df to VOTER
I20240503 23:26:38.538206 4640 consensus_queue.cc:237] T 98c9500acb43423b86f54ce7a4d98a72 P 83ffd7184ec94302bd1ee74b0fab3e17 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 29, Committed index: 29, Last appended: 1.29, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "83ffd7184ec94302bd1ee74b0fab3e17" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 32909 } } peers { permanent_uuid: "a2f9bf2e5d9b4fc19f804797bca4a3ff" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 46261 } attrs { replace: true } } peers { permanent_uuid: "4d1d4e261584467fbbce4c72fa18045a" member_type: VOTER last_known_addr { host: "127.0.100.6" port: 35349 } attrs { promote: false } } peers { permanent_uuid: "e4086c099d7f4447b2b4733b6ca364df" member_type: VOTER last_known_addr { host: "127.0.100.4" port: 35011 } attrs { promote: false } }
I20240503 23:26:38.541546 4218 raft_consensus.cc:1261] T 98c9500acb43423b86f54ce7a4d98a72 P e4086c099d7f4447b2b4733b6ca364df [term 1 LEARNER]: Refusing update from remote peer 83ffd7184ec94302bd1ee74b0fab3e17: Log matching property violated. Preceding OpId in replica: term: 1 index: 29. Preceding OpId from leader: term: 1 index: 30. (index mismatch)
I20240503 23:26:38.542235 4635 consensus_queue.cc:1035] T 98c9500acb43423b86f54ce7a4d98a72 P 83ffd7184ec94302bd1ee74b0fab3e17 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4086c099d7f4447b2b4733b6ca364df" member_type: VOTER last_known_addr { host: "127.0.100.4" port: 35011 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 30, Last known committed idx: 29, Time since last communication: 0.000s
I20240503 23:26:38.543596 3837 raft_consensus.cc:1261] T 98c9500acb43423b86f54ce7a4d98a72 P a2f9bf2e5d9b4fc19f804797bca4a3ff [term 1 FOLLOWER]: Refusing update from remote peer 83ffd7184ec94302bd1ee74b0fab3e17: Log matching property violated. Preceding OpId in replica: term: 1 index: 29. Preceding OpId from leader: term: 1 index: 30. (index mismatch)
I20240503 23:26:38.543769 4367 raft_consensus.cc:1261] T 98c9500acb43423b86f54ce7a4d98a72 P 4d1d4e261584467fbbce4c72fa18045a [term 1 FOLLOWER]: Refusing update from remote peer 83ffd7184ec94302bd1ee74b0fab3e17: Log matching property violated. Preceding OpId in replica: term: 1 index: 29. Preceding OpId from leader: term: 1 index: 30. (index mismatch)
I20240503 23:26:38.544143 4635 consensus_queue.cc:1035] T 98c9500acb43423b86f54ce7a4d98a72 P 83ffd7184ec94302bd1ee74b0fab3e17 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2f9bf2e5d9b4fc19f804797bca4a3ff" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 46261 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 30, Last known committed idx: 29, Time since last communication: 0.000s
I20240503 23:26:38.545485 4635 consensus_queue.cc:1035] T 98c9500acb43423b86f54ce7a4d98a72 P 83ffd7184ec94302bd1ee74b0fab3e17 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d1d4e261584467fbbce4c72fa18045a" member_type: VOTER last_known_addr { host: "127.0.100.6" port: 35349 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 30, Last known committed idx: 29, Time since last communication: 0.000s
I20240503 23:26:38.564582 4635 raft_consensus.cc:2932] T 98c9500acb43423b86f54ce7a4d98a72 P 83ffd7184ec94302bd1ee74b0fab3e17 [term 1 LEADER]: Committing config change with OpId 1.30: config changed from index 29 to 30, e4086c099d7f4447b2b4733b6ca364df (127.0.100.4) changed from NON_VOTER to VOTER. New config: { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "83ffd7184ec94302bd1ee74b0fab3e17" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 32909 } } peers { permanent_uuid: "a2f9bf2e5d9b4fc19f804797bca4a3ff" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 46261 } attrs { replace: true } } peers { permanent_uuid: "4d1d4e261584467fbbce4c72fa18045a" member_type: VOTER last_known_addr { host: "127.0.100.6" port: 35349 } attrs { promote: false } } peers { permanent_uuid: "e4086c099d7f4447b2b4733b6ca364df" member_type: VOTER last_known_addr { host: "127.0.100.4" port: 35011 } attrs { promote: false } } }
I20240503 23:26:38.569713 4367 raft_consensus.cc:2932] T 98c9500acb43423b86f54ce7a4d98a72 P 4d1d4e261584467fbbce4c72fa18045a [term 1 FOLLOWER]: Committing config change with OpId 1.30: config changed from index 29 to 30, e4086c099d7f4447b2b4733b6ca364df (127.0.100.4) changed from NON_VOTER to VOTER. New config: { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "83ffd7184ec94302bd1ee74b0fab3e17" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 32909 } } peers { permanent_uuid: "a2f9bf2e5d9b4fc19f804797bca4a3ff" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 46261 } attrs { replace: true } } peers { permanent_uuid: "4d1d4e261584467fbbce4c72fa18045a" member_type: VOTER last_known_addr { host: "127.0.100.6" port: 35349 } attrs { promote: false } } peers { permanent_uuid: "e4086c099d7f4447b2b4733b6ca364df" member_type: VOTER last_known_addr { host: "127.0.100.4" port: 35011 } attrs { promote: false } } }
I20240503 23:26:38.572422 3838 raft_consensus.cc:2932] T 98c9500acb43423b86f54ce7a4d98a72 P a2f9bf2e5d9b4fc19f804797bca4a3ff [term 1 FOLLOWER]: Committing config change with OpId 1.30: config changed from index 29 to 30, e4086c099d7f4447b2b4733b6ca364df (127.0.100.4) changed from NON_VOTER to VOTER. New config: { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "83ffd7184ec94302bd1ee74b0fab3e17" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 32909 } } peers { permanent_uuid: "a2f9bf2e5d9b4fc19f804797bca4a3ff" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 46261 } attrs { replace: true } } peers { permanent_uuid: "4d1d4e261584467fbbce4c72fa18045a" member_type: VOTER last_known_addr { host: "127.0.100.6" port: 35349 } attrs { promote: false } } peers { permanent_uuid: "e4086c099d7f4447b2b4733b6ca364df" member_type: VOTER last_known_addr { host: "127.0.100.4" port: 35011 } attrs { promote: false } } }
I20240503 23:26:38.572563 4218 raft_consensus.cc:2932] T 98c9500acb43423b86f54ce7a4d98a72 P e4086c099d7f4447b2b4733b6ca364df [term 1 FOLLOWER]: Committing config change with OpId 1.30: config changed from index 29 to 30, e4086c099d7f4447b2b4733b6ca364df (127.0.100.4) changed from NON_VOTER to VOTER. New config: { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "83ffd7184ec94302bd1ee74b0fab3e17" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 32909 } } peers { permanent_uuid: "a2f9bf2e5d9b4fc19f804797bca4a3ff" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 46261 } attrs { replace: true } } peers { permanent_uuid: "4d1d4e261584467fbbce4c72fa18045a" member_type: VOTER last_known_addr { host: "127.0.100.6" port: 35349 } attrs { promote: false } } peers { permanent_uuid: "e4086c099d7f4447b2b4733b6ca364df" member_type: VOTER last_known_addr { host: "127.0.100.4" port: 35011 } attrs { promote: false } } }
I20240503 23:26:38.578753 3659 catalog_manager.cc:5484] T 98c9500acb43423b86f54ce7a4d98a72 P 4d1d4e261584467fbbce4c72fa18045a reported cstate change: config changed from index 29 to 30, e4086c099d7f4447b2b4733b6ca364df (127.0.100.4) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "83ffd7184ec94302bd1ee74b0fab3e17" committed_config { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "83ffd7184ec94302bd1ee74b0fab3e17" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 32909 } } peers { permanent_uuid: "a2f9bf2e5d9b4fc19f804797bca4a3ff" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 46261 } attrs { replace: true } } peers { permanent_uuid: "4d1d4e261584467fbbce4c72fa18045a" member_type: VOTER last_known_addr { host: "127.0.100.6" port: 35349 } attrs { promote: false } } peers { permanent_uuid: "e4086c099d7f4447b2b4733b6ca364df" member_type: VOTER last_known_addr { host: "127.0.100.4" port: 35011 } attrs { promote: false } } }
I20240503 23:26:38.585896 3762 consensus_queue.cc:237] T 98c9500acb43423b86f54ce7a4d98a72 P 83ffd7184ec94302bd1ee74b0fab3e17 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 30, Committed index: 30, Last appended: 1.30, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "83ffd7184ec94302bd1ee74b0fab3e17" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 32909 } } peers { permanent_uuid: "4d1d4e261584467fbbce4c72fa18045a" member_type: VOTER last_known_addr { host: "127.0.100.6" port: 35349 } attrs { promote: false } } peers { permanent_uuid: "e4086c099d7f4447b2b4733b6ca364df" member_type: VOTER last_known_addr { host: "127.0.100.4" port: 35011 } attrs { promote: false } }
I20240503 23:26:38.588716 4367 raft_consensus.cc:1261] T 98c9500acb43423b86f54ce7a4d98a72 P 4d1d4e261584467fbbce4c72fa18045a [term 1 FOLLOWER]: Refusing update from remote peer 83ffd7184ec94302bd1ee74b0fab3e17: Log matching property violated. Preceding OpId in replica: term: 1 index: 30. Preceding OpId from leader: term: 1 index: 31. (index mismatch)
I20240503 23:26:38.588744 4218 raft_consensus.cc:1261] T 98c9500acb43423b86f54ce7a4d98a72 P e4086c099d7f4447b2b4733b6ca364df [term 1 FOLLOWER]: Refusing update from remote peer 83ffd7184ec94302bd1ee74b0fab3e17: Log matching property violated. Preceding OpId in replica: term: 1 index: 30. Preceding OpId from leader: term: 1 index: 31. (index mismatch)
I20240503 23:26:38.590253 4640 consensus_queue.cc:1035] T 98c9500acb43423b86f54ce7a4d98a72 P 83ffd7184ec94302bd1ee74b0fab3e17 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d1d4e261584467fbbce4c72fa18045a" member_type: VOTER last_known_addr { host: "127.0.100.6" port: 35349 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 31, Last known committed idx: 30, Time since last communication: 0.000s
I20240503 23:26:38.590580 4635 consensus_queue.cc:1035] T 98c9500acb43423b86f54ce7a4d98a72 P 83ffd7184ec94302bd1ee74b0fab3e17 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4086c099d7f4447b2b4733b6ca364df" member_type: VOTER last_known_addr { host: "127.0.100.4" port: 35011 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 31, Last known committed idx: 30, Time since last communication: 0.000s
I20240503 23:26:38.595177 4532 raft_consensus.cc:2932] T 98c9500acb43423b86f54ce7a4d98a72 P 83ffd7184ec94302bd1ee74b0fab3e17 [term 1 LEADER]: Committing config change with OpId 1.31: config changed from index 30 to 31, VOTER a2f9bf2e5d9b4fc19f804797bca4a3ff (127.0.100.2) evicted. New config: { opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "83ffd7184ec94302bd1ee74b0fab3e17" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 32909 } } peers { permanent_uuid: "4d1d4e261584467fbbce4c72fa18045a" member_type: VOTER last_known_addr { host: "127.0.100.6" port: 35349 } attrs { promote: false } } peers { permanent_uuid: "e4086c099d7f4447b2b4733b6ca364df" member_type: VOTER last_known_addr { host: "127.0.100.4" port: 35011 } attrs { promote: false } } }
I20240503 23:26:38.595631 4367 raft_consensus.cc:2932] T 98c9500acb43423b86f54ce7a4d98a72 P 4d1d4e261584467fbbce4c72fa18045a [term 1 FOLLOWER]: Committing config change with OpId 1.31: config changed from index 30 to 31, VOTER a2f9bf2e5d9b4fc19f804797bca4a3ff (127.0.100.2) evicted. New config: { opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "83ffd7184ec94302bd1ee74b0fab3e17" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 32909 } } peers { permanent_uuid: "4d1d4e261584467fbbce4c72fa18045a" member_type: VOTER last_known_addr { host: "127.0.100.6" port: 35349 } attrs { promote: false } } peers { permanent_uuid: "e4086c099d7f4447b2b4733b6ca364df" member_type: VOTER last_known_addr { host: "127.0.100.4" port: 35011 } attrs { promote: false } } }
I20240503 23:26:38.600643 3648 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 98c9500acb43423b86f54ce7a4d98a72 with cas_config_opid_index 30: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 23:26:38.601059 4218 raft_consensus.cc:2932] T 98c9500acb43423b86f54ce7a4d98a72 P e4086c099d7f4447b2b4733b6ca364df [term 1 FOLLOWER]: Committing config change with OpId 1.31: config changed from index 30 to 31, VOTER a2f9bf2e5d9b4fc19f804797bca4a3ff (127.0.100.2) evicted. New config: { opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "83ffd7184ec94302bd1ee74b0fab3e17" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 32909 } } peers { permanent_uuid: "4d1d4e261584467fbbce4c72fa18045a" member_type: VOTER last_known_addr { host: "127.0.100.6" port: 35349 } attrs { promote: false } } peers { permanent_uuid: "e4086c099d7f4447b2b4733b6ca364df" member_type: VOTER last_known_addr { host: "127.0.100.4" port: 35011 } attrs { promote: false } } }
I20240503 23:26:38.602877 4601 tablet_bootstrap.cc:492] T 55cf664d5e874c0d9b9adf8bdc9c43ed P 4d1d4e261584467fbbce4c72fa18045a: Bootstrap replayed 1/1 log segments. Stats: ops{read=31 overwritten=0 applied=31 ignored=0} inserts{seen=116 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 23:26:38.602947 3659 catalog_manager.cc:5484] T 98c9500acb43423b86f54ce7a4d98a72 P 4d1d4e261584467fbbce4c72fa18045a reported cstate change: config changed from index 30 to 31, VOTER a2f9bf2e5d9b4fc19f804797bca4a3ff (127.0.100.2) evicted. New cstate: current_term: 1 leader_uuid: "83ffd7184ec94302bd1ee74b0fab3e17" committed_config { opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "83ffd7184ec94302bd1ee74b0fab3e17" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 32909 } } peers { permanent_uuid: "4d1d4e261584467fbbce4c72fa18045a" member_type: VOTER last_known_addr { host: "127.0.100.6" port: 35349 } attrs { promote: false } } peers { permanent_uuid: "e4086c099d7f4447b2b4733b6ca364df" member_type: VOTER last_known_addr { host: "127.0.100.4" port: 35011 } attrs { promote: false } } }
I20240503 23:26:38.603430 4601 tablet_bootstrap.cc:492] T 55cf664d5e874c0d9b9adf8bdc9c43ed P 4d1d4e261584467fbbce4c72fa18045a: Bootstrap complete.
I20240503 23:26:38.603951 4601 ts_tablet_manager.cc:1367] T 55cf664d5e874c0d9b9adf8bdc9c43ed P 4d1d4e261584467fbbce4c72fa18045a: Time spent bootstrapping tablet: real 0.070s user 0.055s sys 0.009s
I20240503 23:26:38.604907 4601 raft_consensus.cc:348] T 55cf664d5e874c0d9b9adf8bdc9c43ed P 4d1d4e261584467fbbce4c72fa18045a [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "83ffd7184ec94302bd1ee74b0fab3e17" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 32909 } attrs { replace: true } } peers { permanent_uuid: "a2f9bf2e5d9b4fc19f804797bca4a3ff" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 46261 } } peers { permanent_uuid: "e4086c099d7f4447b2b4733b6ca364df" member_type: VOTER last_known_addr { host: "127.0.100.4" port: 35011 } attrs { promote: false } } peers { permanent_uuid: "4d1d4e261584467fbbce4c72fa18045a" member_type: NON_VOTER last_known_addr { host: "127.0.100.6" port: 35349 } attrs { promote: true } }
I20240503 23:26:38.605226 4601 raft_consensus.cc:729] T 55cf664d5e874c0d9b9adf8bdc9c43ed P 4d1d4e261584467fbbce4c72fa18045a [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: 4d1d4e261584467fbbce4c72fa18045a, State: Initialized, Role: LEARNER
I20240503 23:26:38.605463 4601 consensus_queue.cc:260] T 55cf664d5e874c0d9b9adf8bdc9c43ed P 4d1d4e261584467fbbce4c72fa18045a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 31, Last appended: 2.31, Last appended by leader: 31, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "83ffd7184ec94302bd1ee74b0fab3e17" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 32909 } attrs { replace: true } } peers { permanent_uuid: "a2f9bf2e5d9b4fc19f804797bca4a3ff" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 46261 } } peers { permanent_uuid: "e4086c099d7f4447b2b4733b6ca364df" member_type: VOTER last_known_addr { host: "127.0.100.4" port: 35011 } attrs { promote: false } } peers { permanent_uuid: "4d1d4e261584467fbbce4c72fa18045a" member_type: NON_VOTER last_known_addr { host: "127.0.100.6" port: 35349 } attrs { promote: true } }
I20240503 23:26:38.606170 4601 ts_tablet_manager.cc:1398] T 55cf664d5e874c0d9b9adf8bdc9c43ed P 4d1d4e261584467fbbce4c72fa18045a: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240503 23:26:38.607033 4228 tablet_copy_service.cc:342] P e4086c099d7f4447b2b4733b6ca364df: Request end of tablet copy session 4d1d4e261584467fbbce4c72fa18045a-55cf664d5e874c0d9b9adf8bdc9c43ed received from {username='slave'} at 127.0.0.1:50852
I20240503 23:26:38.607156 4228 tablet_copy_service.cc:433] P e4086c099d7f4447b2b4733b6ca364df: ending tablet copy session 4d1d4e261584467fbbce4c72fa18045a-55cf664d5e874c0d9b9adf8bdc9c43ed on tablet 55cf664d5e874c0d9b9adf8bdc9c43ed with peer 4d1d4e261584467fbbce4c72fa18045a
I20240503 23:26:38.609653 3824 tablet_service.cc:1514] Processing DeleteTablet for tablet 98c9500acb43423b86f54ce7a4d98a72 with delete_type TABLET_DATA_TOMBSTONED (TS a2f9bf2e5d9b4fc19f804797bca4a3ff not found in new config with opid_index 31) from {username='slave'} at 127.0.0.1:45354
I20240503 23:26:38.610014 4650 tablet_replica.cc:330] T 98c9500acb43423b86f54ce7a4d98a72 P a2f9bf2e5d9b4fc19f804797bca4a3ff: stopping tablet replica
I20240503 23:26:38.610255 4650 raft_consensus.cc:2229] T 98c9500acb43423b86f54ce7a4d98a72 P a2f9bf2e5d9b4fc19f804797bca4a3ff [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 23:26:38.610582 4650 raft_consensus.cc:2258] T 98c9500acb43423b86f54ce7a4d98a72 P a2f9bf2e5d9b4fc19f804797bca4a3ff [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 23:26:38.612123 4650 ts_tablet_manager.cc:1875] T 98c9500acb43423b86f54ce7a4d98a72 P a2f9bf2e5d9b4fc19f804797bca4a3ff: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 23:26:38.618836 4650 ts_tablet_manager.cc:1888] T 98c9500acb43423b86f54ce7a4d98a72 P a2f9bf2e5d9b4fc19f804797bca4a3ff: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.30
I20240503 23:26:38.618993 4650 log.cc:1196] T 98c9500acb43423b86f54ce7a4d98a72 P a2f9bf2e5d9b4fc19f804797bca4a3ff: Deleting WAL directory at /tmp/dist-test-taskZhIiPz/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714778688451250-400-0/minicluster-data/ts-1-root/wals/98c9500acb43423b86f54ce7a4d98a72
I20240503 23:26:38.628587 3649 catalog_manager.cc:4830] TS a2f9bf2e5d9b4fc19f804797bca4a3ff (127.0.100.2:46261): tablet 98c9500acb43423b86f54ce7a4d98a72 (table test-workload [id=1345b2a336c84ec2b1de62853e7fa3f6]) successfully deleted
I20240503 23:26:38.714933 4656 raft_consensus.cc:1050] T 487ac5b9bf2d4de2b80a1b13387a2a77 P a2f9bf2e5d9b4fc19f804797bca4a3ff: attempting to promote NON_VOTER 692da0319a2d4a81ad81f5d8f9f4c86b to VOTER
I20240503 23:26:38.715862 4656 consensus_queue.cc:237] T 487ac5b9bf2d4de2b80a1b13387a2a77 P a2f9bf2e5d9b4fc19f804797bca4a3ff [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 28, Committed index: 28, Last appended: 2.28, Last appended by leader: 27, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "83ffd7184ec94302bd1ee74b0fab3e17" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 32909 } } peers { permanent_uuid: "a2f9bf2e5d9b4fc19f804797bca4a3ff" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 46261 } } peers { permanent_uuid: "a3eecbaec43d4954b1096bc3121e824c" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 40649 } attrs { replace: true } } peers { permanent_uuid: "692da0319a2d4a81ad81f5d8f9f4c86b" member_type: VOTER last_known_addr { host: "127.0.100.5" port: 38561 } attrs { promote: false } }
I20240503 23:26:38.718429 3908 raft_consensus.cc:1261] T 487ac5b9bf2d4de2b80a1b13387a2a77 P a3eecbaec43d4954b1096bc3121e824c [term 2 FOLLOWER]: Refusing update from remote peer a2f9bf2e5d9b4fc19f804797bca4a3ff: Log matching property violated. Preceding OpId in replica: term: 2 index: 28. Preceding OpId from leader: term: 2 index: 29. (index mismatch)
I20240503 23:26:38.718432 4293 raft_consensus.cc:1261] T 487ac5b9bf2d4de2b80a1b13387a2a77 P 692da0319a2d4a81ad81f5d8f9f4c86b [term 2 LEARNER]: Refusing update from remote peer a2f9bf2e5d9b4fc19f804797bca4a3ff: Log matching property violated. Preceding OpId in replica: term: 2 index: 28. Preceding OpId from leader: term: 2 index: 29. (index mismatch)
I20240503 23:26:38.718955 3760 raft_consensus.cc:1261] T 487ac5b9bf2d4de2b80a1b13387a2a77 P 83ffd7184ec94302bd1ee74b0fab3e17 [term 2 FOLLOWER]: Refusing update from remote peer a2f9bf2e5d9b4fc19f804797bca4a3ff: Log matching property violated. Preceding OpId in replica: term: 2 index: 28. Preceding OpId from leader: term: 2 index: 29. (index mismatch)
I20240503 23:26:38.719074 4657 consensus_queue.cc:1035] T 487ac5b9bf2d4de2b80a1b13387a2a77 P a2f9bf2e5d9b4fc19f804797bca4a3ff [LEADER]: Connected to new peer: Peer: permanent_uuid: "a3eecbaec43d4954b1096bc3121e824c" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 40649 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.000s
I20240503 23:26:38.719484 4611 consensus_queue.cc:1035] T 487ac5b9bf2d4de2b80a1b13387a2a77 P a2f9bf2e5d9b4fc19f804797bca4a3ff [LEADER]: Connected to new peer: Peer: permanent_uuid: "83ffd7184ec94302bd1ee74b0fab3e17" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 32909 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.000s
I20240503 23:26:38.719703 4656 consensus_queue.cc:1035] T 487ac5b9bf2d4de2b80a1b13387a2a77 P a2f9bf2e5d9b4fc19f804797bca4a3ff [LEADER]: Connected to new peer: Peer: permanent_uuid: "692da0319a2d4a81ad81f5d8f9f4c86b" member_type: VOTER last_known_addr { host: "127.0.100.5" port: 38561 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.000s
I20240503 23:26:38.722734 4656 raft_consensus.cc:2932] T 487ac5b9bf2d4de2b80a1b13387a2a77 P a2f9bf2e5d9b4fc19f804797bca4a3ff [term 2 LEADER]: Committing config change with OpId 2.29: config changed from index 27 to 29, 692da0319a2d4a81ad81f5d8f9f4c86b (127.0.100.5) changed from NON_VOTER to VOTER. New config: { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "83ffd7184ec94302bd1ee74b0fab3e17" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 32909 } } peers { permanent_uuid: "a2f9bf2e5d9b4fc19f804797bca4a3ff" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 46261 } } peers { permanent_uuid: "a3eecbaec43d4954b1096bc3121e824c" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 40649 } attrs { replace: true } } peers { permanent_uuid: "692da0319a2d4a81ad81f5d8f9f4c86b" member_type: VOTER last_known_addr { host: "127.0.100.5" port: 38561 } attrs { promote: false } } }
I20240503 23:26:38.723845 3760 raft_consensus.cc:2932] T 487ac5b9bf2d4de2b80a1b13387a2a77 P 83ffd7184ec94302bd1ee74b0fab3e17 [term 2 FOLLOWER]: Committing config change with OpId 2.29: config changed from index 27 to 29, 692da0319a2d4a81ad81f5d8f9f4c86b (127.0.100.5) changed from NON_VOTER to VOTER. New config: { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "83ffd7184ec94302bd1ee74b0fab3e17" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 32909 } } peers { permanent_uuid: "a2f9bf2e5d9b4fc19f804797bca4a3ff" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 46261 } } peers { permanent_uuid: "a3eecbaec43d4954b1096bc3121e824c" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 40649 } attrs { replace: true } } peers { permanent_uuid: "692da0319a2d4a81ad81f5d8f9f4c86b" member_type: VOTER last_known_addr { host: "127.0.100.5" port: 38561 } attrs { promote: false } } }
I20240503 23:26:38.728042 3659 catalog_manager.cc:5484] T 487ac5b9bf2d4de2b80a1b13387a2a77 P a2f9bf2e5d9b4fc19f804797bca4a3ff reported cstate change: config changed from index 27 to 29, 692da0319a2d4a81ad81f5d8f9f4c86b (127.0.100.5) changed from NON_VOTER to VOTER. New cstate: current_term: 2 leader_uuid: "a2f9bf2e5d9b4fc19f804797bca4a3ff" committed_config { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "83ffd7184ec94302bd1ee74b0fab3e17" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 32909 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a2f9bf2e5d9b4fc19f804797bca4a3ff" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 46261 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a3eecbaec43d4954b1096bc3121e824c" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 40649 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "692da0319a2d4a81ad81f5d8f9f4c86b" member_type: VOTER last_known_addr { host: "127.0.100.5" port: 38561 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240503 23:26:38.729476 4293 raft_consensus.cc:2932] T 487ac5b9bf2d4de2b80a1b13387a2a77 P 692da0319a2d4a81ad81f5d8f9f4c86b [term 2 FOLLOWER]: Committing config change with OpId 2.29: config changed from index 27 to 29, 692da0319a2d4a81ad81f5d8f9f4c86b (127.0.100.5) changed from NON_VOTER to VOTER. New config: { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "83ffd7184ec94302bd1ee74b0fab3e17" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 32909 } } peers { permanent_uuid: "a2f9bf2e5d9b4fc19f804797bca4a3ff" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 46261 } } peers { permanent_uuid: "a3eecbaec43d4954b1096bc3121e824c" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 40649 } attrs { replace: true } } peers { permanent_uuid: "692da0319a2d4a81ad81f5d8f9f4c86b" member_type: VOTER last_known_addr { host: "127.0.100.5" port: 38561 } attrs { promote: false } } }
I20240503 23:26:38.735235 3837 consensus_queue.cc:237] T 487ac5b9bf2d4de2b80a1b13387a2a77 P a2f9bf2e5d9b4fc19f804797bca4a3ff [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 29, Committed index: 29, Last appended: 2.29, Last appended by leader: 27, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "83ffd7184ec94302bd1ee74b0fab3e17" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 32909 } } peers { permanent_uuid: "a2f9bf2e5d9b4fc19f804797bca4a3ff" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 46261 } } peers { permanent_uuid: "692da0319a2d4a81ad81f5d8f9f4c86b" member_type: VOTER last_known_addr { host: "127.0.100.5" port: 38561 } attrs { promote: false } }
I20240503 23:26:38.737938 3762 raft_consensus.cc:1261] T 487ac5b9bf2d4de2b80a1b13387a2a77 P 83ffd7184ec94302bd1ee74b0fab3e17 [term 2 FOLLOWER]: Refusing update from remote peer a2f9bf2e5d9b4fc19f804797bca4a3ff: Log matching property violated. Preceding OpId in replica: term: 2 index: 29. Preceding OpId from leader: term: 2 index: 30. (index mismatch)
I20240503 23:26:38.738484 4611 consensus_queue.cc:1035] T 487ac5b9bf2d4de2b80a1b13387a2a77 P a2f9bf2e5d9b4fc19f804797bca4a3ff [LEADER]: Connected to new peer: Peer: permanent_uuid: "83ffd7184ec94302bd1ee74b0fab3e17" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 32909 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 30, Last known committed idx: 29, Time since last communication: 0.000s
I20240503 23:26:38.741062 4292 raft_consensus.cc:1261] T 487ac5b9bf2d4de2b80a1b13387a2a77 P 692da0319a2d4a81ad81f5d8f9f4c86b [term 2 FOLLOWER]: Refusing update from remote peer a2f9bf2e5d9b4fc19f804797bca4a3ff: Log matching property violated. Preceding OpId in replica: term: 2 index: 29. Preceding OpId from leader: term: 2 index: 30. (index mismatch)
I20240503 23:26:38.742213 4657 consensus_queue.cc:1035] T 487ac5b9bf2d4de2b80a1b13387a2a77 P a2f9bf2e5d9b4fc19f804797bca4a3ff [LEADER]: Connected to new peer: Peer: permanent_uuid: "692da0319a2d4a81ad81f5d8f9f4c86b" member_type: VOTER last_known_addr { host: "127.0.100.5" port: 38561 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 30, Last known committed idx: 29, Time since last communication: 0.000s
I20240503 23:26:38.742214 3760 raft_consensus.cc:2932] T 487ac5b9bf2d4de2b80a1b13387a2a77 P 83ffd7184ec94302bd1ee74b0fab3e17 [term 2 FOLLOWER]: Committing config change with OpId 2.30: config changed from index 29 to 30, VOTER a3eecbaec43d4954b1096bc3121e824c (127.0.100.3) evicted. New config: { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "83ffd7184ec94302bd1ee74b0fab3e17" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 32909 } } peers { permanent_uuid: "a2f9bf2e5d9b4fc19f804797bca4a3ff" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 46261 } } peers { permanent_uuid: "692da0319a2d4a81ad81f5d8f9f4c86b" member_type: VOTER last_known_addr { host: "127.0.100.5" port: 38561 } attrs { promote: false } } }
I20240503 23:26:38.742046 4656 raft_consensus.cc:2932] T 487ac5b9bf2d4de2b80a1b13387a2a77 P a2f9bf2e5d9b4fc19f804797bca4a3ff [term 2 LEADER]: Committing config change with OpId 2.30: config changed from index 29 to 30, VOTER a3eecbaec43d4954b1096bc3121e824c (127.0.100.3) evicted. New config: { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "83ffd7184ec94302bd1ee74b0fab3e17" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 32909 } } peers { permanent_uuid: "a2f9bf2e5d9b4fc19f804797bca4a3ff" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 46261 } } peers { permanent_uuid: "692da0319a2d4a81ad81f5d8f9f4c86b" member_type: VOTER last_known_addr { host: "127.0.100.5" port: 38561 } attrs { promote: false } } }
I20240503 23:26:38.743568 4293 raft_consensus.cc:2932] T 487ac5b9bf2d4de2b80a1b13387a2a77 P 692da0319a2d4a81ad81f5d8f9f4c86b [term 2 FOLLOWER]: Committing config change with OpId 2.30: config changed from index 29 to 30, VOTER a3eecbaec43d4954b1096bc3121e824c (127.0.100.3) evicted. New config: { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "83ffd7184ec94302bd1ee74b0fab3e17" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 32909 } } peers { permanent_uuid: "a2f9bf2e5d9b4fc19f804797bca4a3ff" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 46261 } } peers { permanent_uuid: "692da0319a2d4a81ad81f5d8f9f4c86b" member_type: VOTER last_known_addr { host: "127.0.100.5" port: 38561 } attrs { promote: false } } }
I20240503 23:26:38.746052 3649 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 487ac5b9bf2d4de2b80a1b13387a2a77 with cas_config_opid_index 29: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 23:26:38.747012 3655 catalog_manager.cc:5484] T 487ac5b9bf2d4de2b80a1b13387a2a77 P a2f9bf2e5d9b4fc19f804797bca4a3ff reported cstate change: config changed from index 29 to 30, VOTER a3eecbaec43d4954b1096bc3121e824c (127.0.100.3) evicted. New cstate: current_term: 2 leader_uuid: "a2f9bf2e5d9b4fc19f804797bca4a3ff" committed_config { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "83ffd7184ec94302bd1ee74b0fab3e17" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 32909 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a2f9bf2e5d9b4fc19f804797bca4a3ff" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 46261 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "692da0319a2d4a81ad81f5d8f9f4c86b" member_type: VOTER last_known_addr { host: "127.0.100.5" port: 38561 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240503 23:26:38.752633 3902 tablet_service.cc:1514] Processing DeleteTablet for tablet 487ac5b9bf2d4de2b80a1b13387a2a77 with delete_type TABLET_DATA_TOMBSTONED (TS a3eecbaec43d4954b1096bc3121e824c not found in new config with opid_index 30) from {username='slave'} at 127.0.0.1:40632
I20240503 23:26:38.753427 4661 tablet_replica.cc:330] T 487ac5b9bf2d4de2b80a1b13387a2a77 P a3eecbaec43d4954b1096bc3121e824c: stopping tablet replica
I20240503 23:26:38.753701 4661 raft_consensus.cc:2229] T 487ac5b9bf2d4de2b80a1b13387a2a77 P a3eecbaec43d4954b1096bc3121e824c [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 23:26:38.754020 4661 pending_rounds.cc:62] T 487ac5b9bf2d4de2b80a1b13387a2a77 P a3eecbaec43d4954b1096bc3121e824c: Trying to abort 1 pending ops.
I20240503 23:26:38.754089 4661 pending_rounds.cc:68] T 487ac5b9bf2d4de2b80a1b13387a2a77 P a3eecbaec43d4954b1096bc3121e824c: Aborting op as it isn't in flight: id { term: 2 index: 29 } timestamp: 7023733959538159616 op_type: CHANGE_CONFIG_OP change_config_record { tablet_id: "487ac5b9bf2d4de2b80a1b13387a2a77" old_config { opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "83ffd7184ec94302bd1ee74b0fab3e17" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 32909 } } peers { permanent_uuid: "a2f9bf2e5d9b4fc19f804797bca4a3ff" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 46261 } } peers { permanent_uuid: "a3eecbaec43d4954b1096bc3121e824c" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 40649 } attrs { replace: true } } peers { permanent_uuid: "692da0319a2d4a81ad81f5d8f9f4c86b" member_type: NON_VOTER last_known_addr { host: "127.0.100.5" port: 38561 } attrs { promote: true } } } new_config { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "83ffd7184ec94302bd1ee74b0fab3e17" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 32909 } } peers { permanent_uuid: "a2f9bf2e5d9b4fc19f804797bca4a3ff" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 46261 } } peers { permanent_uuid: "a3eecbaec43d4954b1096bc3121e824c" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 40649 } attrs { replace: true } } peers { permanent_uuid: "692da0319a2d4a81ad81f5d8f9f4c86b" member_type: VOTER last_known_addr { host: "127.0.100.5" port: 38561 } attrs { promote: false } } } }
I20240503 23:26:38.754469 4661 raft_consensus.cc:2894] T 487ac5b9bf2d4de2b80a1b13387a2a77 P a3eecbaec43d4954b1096bc3121e824c [term 2 FOLLOWER]: Aborting config change with OpId 2.29: Aborted: Op aborted
I20240503 23:26:38.754601 4661 raft_consensus.cc:2866] T 487ac5b9bf2d4de2b80a1b13387a2a77 P a3eecbaec43d4954b1096bc3121e824c [term 2 FOLLOWER]: CHANGE_CONFIG_OP replication failed: Aborted: Op aborted
I20240503 23:26:38.754688 4661 raft_consensus.cc:2258] T 487ac5b9bf2d4de2b80a1b13387a2a77 P a3eecbaec43d4954b1096bc3121e824c [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 23:26:38.755959 4661 ts_tablet_manager.cc:1875] T 487ac5b9bf2d4de2b80a1b13387a2a77 P a3eecbaec43d4954b1096bc3121e824c: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 23:26:38.762238 4661 ts_tablet_manager.cc:1888] T 487ac5b9bf2d4de2b80a1b13387a2a77 P a3eecbaec43d4954b1096bc3121e824c: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.29
I20240503 23:26:38.762403 4661 log.cc:1196] T 487ac5b9bf2d4de2b80a1b13387a2a77 P a3eecbaec43d4954b1096bc3121e824c: Deleting WAL directory at /tmp/dist-test-taskZhIiPz/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714778688451250-400-0/minicluster-data/ts-2-root/wals/487ac5b9bf2d4de2b80a1b13387a2a77
I20240503 23:26:38.763139 3646 catalog_manager.cc:4830] TS a3eecbaec43d4954b1096bc3121e824c (127.0.100.3:40649): tablet 487ac5b9bf2d4de2b80a1b13387a2a77 (table test-workload [id=1345b2a336c84ec2b1de62853e7fa3f6]) successfully deleted
I20240503 23:26:38.839740 4607 tablet_copy_client.cc:531] T d0c4430fc56644cb9f87deffe4fe9407 P 4d1d4e261584467fbbce4c72fa18045a: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 23:26:38.842809 4607 tablet_bootstrap.cc:492] T d0c4430fc56644cb9f87deffe4fe9407 P 4d1d4e261584467fbbce4c72fa18045a: Bootstrap starting.
I20240503 23:26:38.882301 4607 tablet_bootstrap.cc:492] T d0c4430fc56644cb9f87deffe4fe9407 P 4d1d4e261584467fbbce4c72fa18045a: Bootstrap replayed 1/1 log segments. Stats: ops{read=29 overwritten=0 applied=29 ignored=0} inserts{seen=104 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 23:26:38.882843 4607 tablet_bootstrap.cc:492] T d0c4430fc56644cb9f87deffe4fe9407 P 4d1d4e261584467fbbce4c72fa18045a: Bootstrap complete.
I20240503 23:26:38.883160 4607 ts_tablet_manager.cc:1367] T d0c4430fc56644cb9f87deffe4fe9407 P 4d1d4e261584467fbbce4c72fa18045a: Time spent bootstrapping tablet: real 0.040s user 0.029s sys 0.009s
I20240503 23:26:38.884141 4607 raft_consensus.cc:348] T d0c4430fc56644cb9f87deffe4fe9407 P 4d1d4e261584467fbbce4c72fa18045a [term 3 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "83ffd7184ec94302bd1ee74b0fab3e17" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 32909 } attrs { replace: true } } peers { permanent_uuid: "a2f9bf2e5d9b4fc19f804797bca4a3ff" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 46261 } } peers { permanent_uuid: "a3eecbaec43d4954b1096bc3121e824c" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 40649 } } peers { permanent_uuid: "4d1d4e261584467fbbce4c72fa18045a" member_type: NON_VOTER last_known_addr { host: "127.0.100.6" port: 35349 } attrs { promote: true } }
I20240503 23:26:38.884415 4607 raft_consensus.cc:729] T d0c4430fc56644cb9f87deffe4fe9407 P 4d1d4e261584467fbbce4c72fa18045a [term 3 LEARNER]: Becoming Follower/Learner. State: Replica: 4d1d4e261584467fbbce4c72fa18045a, State: Initialized, Role: LEARNER
I20240503 23:26:38.884670 4607 consensus_queue.cc:260] T d0c4430fc56644cb9f87deffe4fe9407 P 4d1d4e261584467fbbce4c72fa18045a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 29, Last appended: 3.29, Last appended by leader: 29, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "83ffd7184ec94302bd1ee74b0fab3e17" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 32909 } attrs { replace: true } } peers { permanent_uuid: "a2f9bf2e5d9b4fc19f804797bca4a3ff" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 46261 } } peers { permanent_uuid: "a3eecbaec43d4954b1096bc3121e824c" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 40649 } } peers { permanent_uuid: "4d1d4e261584467fbbce4c72fa18045a" member_type: NON_VOTER last_known_addr { host: "127.0.100.6" port: 35349 } attrs { promote: true } }
I20240503 23:26:38.885214 4607 ts_tablet_manager.cc:1398] T d0c4430fc56644cb9f87deffe4fe9407 P 4d1d4e261584467fbbce4c72fa18045a: Time spent starting tablet: real 0.002s user 0.000s sys 0.004s
I20240503 23:26:38.886027 3848 tablet_copy_service.cc:342] P a2f9bf2e5d9b4fc19f804797bca4a3ff: Request end of tablet copy session 4d1d4e261584467fbbce4c72fa18045a-d0c4430fc56644cb9f87deffe4fe9407 received from {username='slave'} at 127.0.0.1:55484
I20240503 23:26:38.886159 3848 tablet_copy_service.cc:433] P a2f9bf2e5d9b4fc19f804797bca4a3ff: ending tablet copy session 4d1d4e261584467fbbce4c72fa18045a-d0c4430fc56644cb9f87deffe4fe9407 on tablet d0c4430fc56644cb9f87deffe4fe9407 with peer 4d1d4e261584467fbbce4c72fa18045a
I20240503 23:26:39.090655 4367 raft_consensus.cc:1203] T 55cf664d5e874c0d9b9adf8bdc9c43ed P 4d1d4e261584467fbbce4c72fa18045a [term 2 LEARNER]: Deduplicated request from leader. Original: 1.30->[2.31-2.31] Dedup: 2.31->[]
I20240503 23:26:39.248076 3715 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240503 23:26:39.252115 3908 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "6039230222ff4e57929bed49cd56caf5"
dest_uuid: "a3eecbaec43d4954b1096bc3121e824c"
mode: GRACEFUL
new_leader_uuid: "a2f9bf2e5d9b4fc19f804797bca4a3ff"
from {username='slave'} at 127.0.0.1:40650
I20240503 23:26:39.252467 3908 raft_consensus.cc:595] T 6039230222ff4e57929bed49cd56caf5 P a3eecbaec43d4954b1096bc3121e824c [term 2 LEADER]: Received request to transfer leadership to TS a2f9bf2e5d9b4fc19f804797bca4a3ff
I20240503 23:26:39.253775 3760 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "a0e18d0975a54e3dbeb0a13cb226fefd"
dest_uuid: "83ffd7184ec94302bd1ee74b0fab3e17"
mode: GRACEFUL
new_leader_uuid: "692da0319a2d4a81ad81f5d8f9f4c86b"
from {username='slave'} at 127.0.0.1:57936
I20240503 23:26:39.254036 3760 raft_consensus.cc:595] T a0e18d0975a54e3dbeb0a13cb226fefd P 83ffd7184ec94302bd1ee74b0fab3e17 [term 2 LEADER]: Received request to transfer leadership to TS 692da0319a2d4a81ad81f5d8f9f4c86b
I20240503 23:26:39.255124 3760 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "d0c4430fc56644cb9f87deffe4fe9407"
dest_uuid: "83ffd7184ec94302bd1ee74b0fab3e17"
mode: GRACEFUL
new_leader_uuid: "a2f9bf2e5d9b4fc19f804797bca4a3ff"
from {username='slave'} at 127.0.0.1:57936
I20240503 23:26:39.255384 3760 raft_consensus.cc:595] T d0c4430fc56644cb9f87deffe4fe9407 P 83ffd7184ec94302bd1ee74b0fab3e17 [term 4 LEADER]: Received request to transfer leadership to TS a2f9bf2e5d9b4fc19f804797bca4a3ff
I20240503 23:26:39.255790 3715 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 3
I20240503 23:26:39.256035 3715 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240503 23:26:39.315605 4663 raft_consensus.cc:979] T 6039230222ff4e57929bed49cd56caf5 P a3eecbaec43d4954b1096bc3121e824c: : Instructing follower a2f9bf2e5d9b4fc19f804797bca4a3ff to start an election
I20240503 23:26:39.315883 4663 raft_consensus.cc:1067] T 6039230222ff4e57929bed49cd56caf5 P a3eecbaec43d4954b1096bc3121e824c [term 2 LEADER]: Signalling peer a2f9bf2e5d9b4fc19f804797bca4a3ff to start an election
I20240503 23:26:39.316618 3837 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "6039230222ff4e57929bed49cd56caf5"
dest_uuid: "a2f9bf2e5d9b4fc19f804797bca4a3ff"
from {username='slave'} at 127.0.0.1:45358
I20240503 23:26:39.316845 3837 raft_consensus.cc:482] T 6039230222ff4e57929bed49cd56caf5 P a2f9bf2e5d9b4fc19f804797bca4a3ff [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 23:26:39.316954 3837 raft_consensus.cc:3037] T 6039230222ff4e57929bed49cd56caf5 P a2f9bf2e5d9b4fc19f804797bca4a3ff [term 2 FOLLOWER]: Advancing to term 3
I20240503 23:26:39.319764 3837 raft_consensus.cc:504] T 6039230222ff4e57929bed49cd56caf5 P a2f9bf2e5d9b4fc19f804797bca4a3ff [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "a3eecbaec43d4954b1096bc3121e824c" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 40649 } } peers { permanent_uuid: "a2f9bf2e5d9b4fc19f804797bca4a3ff" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 46261 } } peers { permanent_uuid: "692da0319a2d4a81ad81f5d8f9f4c86b" member_type: VOTER last_known_addr { host: "127.0.100.5" port: 38561 } attrs { promote: false } }
I20240503 23:26:39.320617 3837 leader_election.cc:288] T 6039230222ff4e57929bed49cd56caf5 P a2f9bf2e5d9b4fc19f804797bca4a3ff [CANDIDATE]: Term 3 election: Requested vote from peers a3eecbaec43d4954b1096bc3121e824c (127.0.100.3:40649), 692da0319a2d4a81ad81f5d8f9f4c86b (127.0.100.5:38561)
I20240503 23:26:39.320937 3908 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6039230222ff4e57929bed49cd56caf5" candidate_uuid: "a2f9bf2e5d9b4fc19f804797bca4a3ff" candidate_term: 3 candidate_status { last_received { term: 2 index: 29 } } ignore_live_leader: true dest_uuid: "a3eecbaec43d4954b1096bc3121e824c"
I20240503 23:26:39.321161 3908 raft_consensus.cc:3032] T 6039230222ff4e57929bed49cd56caf5 P a3eecbaec43d4954b1096bc3121e824c [term 2 LEADER]: Stepping down as leader of term 2
I20240503 23:26:39.321240 3908 raft_consensus.cc:729] T 6039230222ff4e57929bed49cd56caf5 P a3eecbaec43d4954b1096bc3121e824c [term 2 LEADER]: Becoming Follower/Learner. State: Replica: a3eecbaec43d4954b1096bc3121e824c, State: Running, Role: LEADER
I20240503 23:26:39.321576 3908 consensus_queue.cc:260] T 6039230222ff4e57929bed49cd56caf5 P a3eecbaec43d4954b1096bc3121e824c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 29, Committed index: 29, Last appended: 2.29, Last appended by leader: 29, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "a3eecbaec43d4954b1096bc3121e824c" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 40649 } } peers { permanent_uuid: "a2f9bf2e5d9b4fc19f804797bca4a3ff" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 46261 } } peers { permanent_uuid: "692da0319a2d4a81ad81f5d8f9f4c86b" member_type: VOTER last_known_addr { host: "127.0.100.5" port: 38561 } attrs { promote: false } }
I20240503 23:26:39.322002 3908 raft_consensus.cc:3037] T 6039230222ff4e57929bed49cd56caf5 P a3eecbaec43d4954b1096bc3121e824c [term 2 FOLLOWER]: Advancing to term 3
I20240503 23:26:39.325699 3908 raft_consensus.cc:2454] T 6039230222ff4e57929bed49cd56caf5 P a3eecbaec43d4954b1096bc3121e824c [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2f9bf2e5d9b4fc19f804797bca4a3ff in term 3.
I20240503 23:26:39.326231 3802 leader_election.cc:302] T 6039230222ff4e57929bed49cd56caf5 P a2f9bf2e5d9b4fc19f804797bca4a3ff [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a2f9bf2e5d9b4fc19f804797bca4a3ff, a3eecbaec43d4954b1096bc3121e824c; no voters:
I20240503 23:26:39.326176 4293 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6039230222ff4e57929bed49cd56caf5" candidate_uuid: "a2f9bf2e5d9b4fc19f804797bca4a3ff" candidate_term: 3 candidate_status { last_received { term: 2 index: 29 } } ignore_live_leader: true dest_uuid: "692da0319a2d4a81ad81f5d8f9f4c86b"
I20240503 23:26:39.326413 4293 raft_consensus.cc:3037] T 6039230222ff4e57929bed49cd56caf5 P 692da0319a2d4a81ad81f5d8f9f4c86b [term 2 FOLLOWER]: Advancing to term 3
I20240503 23:26:39.329115 4293 raft_consensus.cc:2454] T 6039230222ff4e57929bed49cd56caf5 P 692da0319a2d4a81ad81f5d8f9f4c86b [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2f9bf2e5d9b4fc19f804797bca4a3ff in term 3.
I20240503 23:26:39.329933 4656 raft_consensus.cc:2781] T 6039230222ff4e57929bed49cd56caf5 P a2f9bf2e5d9b4fc19f804797bca4a3ff [term 3 FOLLOWER]: Leader election won for term 3
I20240503 23:26:39.330097 4656 raft_consensus.cc:686] T 6039230222ff4e57929bed49cd56caf5 P a2f9bf2e5d9b4fc19f804797bca4a3ff [term 3 LEADER]: Becoming Leader. State: Replica: a2f9bf2e5d9b4fc19f804797bca4a3ff, State: Running, Role: LEADER
I20240503 23:26:39.330353 4656 consensus_queue.cc:237] T 6039230222ff4e57929bed49cd56caf5 P a2f9bf2e5d9b4fc19f804797bca4a3ff [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 29, Committed index: 29, Last appended: 2.29, Last appended by leader: 29, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "a3eecbaec43d4954b1096bc3121e824c" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 40649 } } peers { permanent_uuid: "a2f9bf2e5d9b4fc19f804797bca4a3ff" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 46261 } } peers { permanent_uuid: "692da0319a2d4a81ad81f5d8f9f4c86b" member_type: VOTER last_known_addr { host: "127.0.100.5" port: 38561 } attrs { promote: false } }
I20240503 23:26:39.333382 3655 catalog_manager.cc:5484] T 6039230222ff4e57929bed49cd56caf5 P a2f9bf2e5d9b4fc19f804797bca4a3ff reported cstate change: term changed from 2 to 3, leader changed from a3eecbaec43d4954b1096bc3121e824c (127.0.100.3) to a2f9bf2e5d9b4fc19f804797bca4a3ff (127.0.100.2). New cstate: current_term: 3 leader_uuid: "a2f9bf2e5d9b4fc19f804797bca4a3ff" committed_config { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "a3eecbaec43d4954b1096bc3121e824c" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 40649 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a2f9bf2e5d9b4fc19f804797bca4a3ff" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 46261 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "692da0319a2d4a81ad81f5d8f9f4c86b" member_type: VOTER last_known_addr { host: "127.0.100.5" port: 38561 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240503 23:26:39.381865 4668 raft_consensus.cc:979] T d0c4430fc56644cb9f87deffe4fe9407 P 83ffd7184ec94302bd1ee74b0fab3e17: : Instructing follower a2f9bf2e5d9b4fc19f804797bca4a3ff to start an election
I20240503 23:26:39.382088 4668 raft_consensus.cc:1067] T d0c4430fc56644cb9f87deffe4fe9407 P 83ffd7184ec94302bd1ee74b0fab3e17 [term 4 LEADER]: Signalling peer a2f9bf2e5d9b4fc19f804797bca4a3ff to start an election
I20240503 23:26:39.382828 3837 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "d0c4430fc56644cb9f87deffe4fe9407"
dest_uuid: "a2f9bf2e5d9b4fc19f804797bca4a3ff"
from {username='slave'} at 127.0.0.1:45370
I20240503 23:26:39.383064 3837 raft_consensus.cc:482] T d0c4430fc56644cb9f87deffe4fe9407 P a2f9bf2e5d9b4fc19f804797bca4a3ff [term 4 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 23:26:39.383174 3837 raft_consensus.cc:3037] T d0c4430fc56644cb9f87deffe4fe9407 P a2f9bf2e5d9b4fc19f804797bca4a3ff [term 4 FOLLOWER]: Advancing to term 5
I20240503 23:26:39.386226 3837 raft_consensus.cc:504] T d0c4430fc56644cb9f87deffe4fe9407 P a2f9bf2e5d9b4fc19f804797bca4a3ff [term 5 FOLLOWER]: Starting forced leader election with config: opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "83ffd7184ec94302bd1ee74b0fab3e17" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 32909 } attrs { replace: true } } peers { permanent_uuid: "a2f9bf2e5d9b4fc19f804797bca4a3ff" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 46261 } } peers { permanent_uuid: "a3eecbaec43d4954b1096bc3121e824c" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 40649 } } peers { permanent_uuid: "4d1d4e261584467fbbce4c72fa18045a" member_type: NON_VOTER last_known_addr { host: "127.0.100.6" port: 35349 } attrs { promote: true } }
I20240503 23:26:39.387027 3837 leader_election.cc:288] T d0c4430fc56644cb9f87deffe4fe9407 P a2f9bf2e5d9b4fc19f804797bca4a3ff [CANDIDATE]: Term 5 election: Requested vote from peers 83ffd7184ec94302bd1ee74b0fab3e17 (127.0.100.1:32909), a3eecbaec43d4954b1096bc3121e824c (127.0.100.3:40649)
I20240503 23:26:39.387454 3760 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0c4430fc56644cb9f87deffe4fe9407" candidate_uuid: "a2f9bf2e5d9b4fc19f804797bca4a3ff" candidate_term: 5 candidate_status { last_received { term: 4 index: 30 } } ignore_live_leader: true dest_uuid: "83ffd7184ec94302bd1ee74b0fab3e17"
I20240503 23:26:39.387768 3760 raft_consensus.cc:3032] T d0c4430fc56644cb9f87deffe4fe9407 P 83ffd7184ec94302bd1ee74b0fab3e17 [term 4 LEADER]: Stepping down as leader of term 4
I20240503 23:26:39.387888 3760 raft_consensus.cc:729] T d0c4430fc56644cb9f87deffe4fe9407 P 83ffd7184ec94302bd1ee74b0fab3e17 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 83ffd7184ec94302bd1ee74b0fab3e17, State: Running, Role: LEADER
I20240503 23:26:39.388274 3907 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0c4430fc56644cb9f87deffe4fe9407" candidate_uuid: "a2f9bf2e5d9b4fc19f804797bca4a3ff" candidate_term: 5 candidate_status { last_received { term: 4 index: 30 } } ignore_live_leader: true dest_uuid: "a3eecbaec43d4954b1096bc3121e824c"
I20240503 23:26:39.388442 3907 raft_consensus.cc:3037] T d0c4430fc56644cb9f87deffe4fe9407 P a3eecbaec43d4954b1096bc3121e824c [term 4 FOLLOWER]: Advancing to term 5
I20240503 23:26:39.388237 3760 consensus_queue.cc:260] T d0c4430fc56644cb9f87deffe4fe9407 P 83ffd7184ec94302bd1ee74b0fab3e17 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 30, Committed index: 30, Last appended: 4.30, Last appended by leader: 30, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "83ffd7184ec94302bd1ee74b0fab3e17" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 32909 } attrs { replace: true } } peers { permanent_uuid: "a2f9bf2e5d9b4fc19f804797bca4a3ff" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 46261 } } peers { permanent_uuid: "a3eecbaec43d4954b1096bc3121e824c" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 40649 } } peers { permanent_uuid: "4d1d4e261584467fbbce4c72fa18045a" member_type: NON_VOTER last_known_addr { host: "127.0.100.6" port: 35349 } attrs { promote: true } }
I20240503 23:26:39.388813 3760 raft_consensus.cc:3037] T d0c4430fc56644cb9f87deffe4fe9407 P 83ffd7184ec94302bd1ee74b0fab3e17 [term 4 FOLLOWER]: Advancing to term 5
I20240503 23:26:39.391753 3907 raft_consensus.cc:2454] T d0c4430fc56644cb9f87deffe4fe9407 P a3eecbaec43d4954b1096bc3121e824c [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2f9bf2e5d9b4fc19f804797bca4a3ff in term 5.
I20240503 23:26:39.391753 3760 raft_consensus.cc:2454] T d0c4430fc56644cb9f87deffe4fe9407 P 83ffd7184ec94302bd1ee74b0fab3e17 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2f9bf2e5d9b4fc19f804797bca4a3ff in term 5.
I20240503 23:26:39.392419 3802 leader_election.cc:302] T d0c4430fc56644cb9f87deffe4fe9407 P a2f9bf2e5d9b4fc19f804797bca4a3ff [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a2f9bf2e5d9b4fc19f804797bca4a3ff, a3eecbaec43d4954b1096bc3121e824c; no voters:
I20240503 23:26:39.392786 4656 raft_consensus.cc:2781] T d0c4430fc56644cb9f87deffe4fe9407 P a2f9bf2e5d9b4fc19f804797bca4a3ff [term 5 FOLLOWER]: Leader election won for term 5
I20240503 23:26:39.392935 4656 raft_consensus.cc:686] T d0c4430fc56644cb9f87deffe4fe9407 P a2f9bf2e5d9b4fc19f804797bca4a3ff [term 5 LEADER]: Becoming Leader. State: Replica: a2f9bf2e5d9b4fc19f804797bca4a3ff, State: Running, Role: LEADER
I20240503 23:26:39.393204 4656 consensus_queue.cc:237] T d0c4430fc56644cb9f87deffe4fe9407 P a2f9bf2e5d9b4fc19f804797bca4a3ff [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 30, Committed index: 30, Last appended: 4.30, Last appended by leader: 30, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "83ffd7184ec94302bd1ee74b0fab3e17" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 32909 } attrs { replace: true } } peers { permanent_uuid: "a2f9bf2e5d9b4fc19f804797bca4a3ff" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 46261 } } peers { permanent_uuid: "a3eecbaec43d4954b1096bc3121e824c" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 40649 } } peers { permanent_uuid: "4d1d4e261584467fbbce4c72fa18045a" member_type: NON_VOTER last_known_addr { host: "127.0.100.6" port: 35349 } attrs { promote: true } }
I20240503 23:26:39.395948 3655 catalog_manager.cc:5484] T d0c4430fc56644cb9f87deffe4fe9407 P a2f9bf2e5d9b4fc19f804797bca4a3ff reported cstate change: term changed from 4 to 5, leader changed from 83ffd7184ec94302bd1ee74b0fab3e17 (127.0.100.1) to a2f9bf2e5d9b4fc19f804797bca4a3ff (127.0.100.2). New cstate: current_term: 5 leader_uuid: "a2f9bf2e5d9b4fc19f804797bca4a3ff" committed_config { opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "83ffd7184ec94302bd1ee74b0fab3e17" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 32909 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a2f9bf2e5d9b4fc19f804797bca4a3ff" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 46261 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a3eecbaec43d4954b1096bc3121e824c" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 40649 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4d1d4e261584467fbbce4c72fa18045a" member_type: NON_VOTER last_known_addr { host: "127.0.100.6" port: 35349 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 23:26:39.565176 4613 raft_consensus.cc:979] T a0e18d0975a54e3dbeb0a13cb226fefd P 83ffd7184ec94302bd1ee74b0fab3e17: : Instructing follower 692da0319a2d4a81ad81f5d8f9f4c86b to start an election
I20240503 23:26:39.565372 4613 raft_consensus.cc:1067] T a0e18d0975a54e3dbeb0a13cb226fefd P 83ffd7184ec94302bd1ee74b0fab3e17 [term 2 LEADER]: Signalling peer 692da0319a2d4a81ad81f5d8f9f4c86b to start an election
I20240503 23:26:39.565925 4293 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "a0e18d0975a54e3dbeb0a13cb226fefd"
dest_uuid: "692da0319a2d4a81ad81f5d8f9f4c86b"
from {username='slave'} at 127.0.0.1:56186
I20240503 23:26:39.566129 4293 raft_consensus.cc:482] T a0e18d0975a54e3dbeb0a13cb226fefd P 692da0319a2d4a81ad81f5d8f9f4c86b [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 23:26:39.566231 4293 raft_consensus.cc:3037] T a0e18d0975a54e3dbeb0a13cb226fefd P 692da0319a2d4a81ad81f5d8f9f4c86b [term 2 FOLLOWER]: Advancing to term 3
I20240503 23:26:39.569156 4293 raft_consensus.cc:504] T a0e18d0975a54e3dbeb0a13cb226fefd P 692da0319a2d4a81ad81f5d8f9f4c86b [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "83ffd7184ec94302bd1ee74b0fab3e17" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 32909 } } peers { permanent_uuid: "a2f9bf2e5d9b4fc19f804797bca4a3ff" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 46261 } } peers { permanent_uuid: "692da0319a2d4a81ad81f5d8f9f4c86b" member_type: VOTER last_known_addr { host: "127.0.100.5" port: 38561 } attrs { promote: false } }
I20240503 23:26:39.570151 4293 leader_election.cc:288] T a0e18d0975a54e3dbeb0a13cb226fefd P 692da0319a2d4a81ad81f5d8f9f4c86b [CANDIDATE]: Term 3 election: Requested vote from peers 83ffd7184ec94302bd1ee74b0fab3e17 (127.0.100.1:32909), a2f9bf2e5d9b4fc19f804797bca4a3ff (127.0.100.2:46261)
I20240503 23:26:39.570691 3837 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a0e18d0975a54e3dbeb0a13cb226fefd" candidate_uuid: "692da0319a2d4a81ad81f5d8f9f4c86b" candidate_term: 3 candidate_status { last_received { term: 2 index: 30 } } ignore_live_leader: true dest_uuid: "a2f9bf2e5d9b4fc19f804797bca4a3ff"
I20240503 23:26:39.570688 3760 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a0e18d0975a54e3dbeb0a13cb226fefd" candidate_uuid: "692da0319a2d4a81ad81f5d8f9f4c86b" candidate_term: 3 candidate_status { last_received { term: 2 index: 30 } } ignore_live_leader: true dest_uuid: "83ffd7184ec94302bd1ee74b0fab3e17"
I20240503 23:26:39.570942 3837 raft_consensus.cc:3037] T a0e18d0975a54e3dbeb0a13cb226fefd P a2f9bf2e5d9b4fc19f804797bca4a3ff [term 2 FOLLOWER]: Advancing to term 3
I20240503 23:26:39.570950 3760 raft_consensus.cc:3032] T a0e18d0975a54e3dbeb0a13cb226fefd P 83ffd7184ec94302bd1ee74b0fab3e17 [term 2 LEADER]: Stepping down as leader of term 2
I20240503 23:26:39.571036 3760 raft_consensus.cc:729] T a0e18d0975a54e3dbeb0a13cb226fefd P 83ffd7184ec94302bd1ee74b0fab3e17 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 83ffd7184ec94302bd1ee74b0fab3e17, State: Running, Role: LEADER
I20240503 23:26:39.571285 3760 consensus_queue.cc:260] T a0e18d0975a54e3dbeb0a13cb226fefd P 83ffd7184ec94302bd1ee74b0fab3e17 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 30, Committed index: 30, Last appended: 2.30, Last appended by leader: 30, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "83ffd7184ec94302bd1ee74b0fab3e17" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 32909 } } peers { permanent_uuid: "a2f9bf2e5d9b4fc19f804797bca4a3ff" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 46261 } } peers { permanent_uuid: "692da0319a2d4a81ad81f5d8f9f4c86b" member_type: VOTER last_known_addr { host: "127.0.100.5" port: 38561 } attrs { promote: false } }
I20240503 23:26:39.571667 3760 raft_consensus.cc:3037] T a0e18d0975a54e3dbeb0a13cb226fefd P 83ffd7184ec94302bd1ee74b0fab3e17 [term 2 FOLLOWER]: Advancing to term 3
I20240503 23:26:39.573346 3837 raft_consensus.cc:2454] T a0e18d0975a54e3dbeb0a13cb226fefd P a2f9bf2e5d9b4fc19f804797bca4a3ff [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 692da0319a2d4a81ad81f5d8f9f4c86b in term 3.
I20240503 23:26:39.573901 4259 leader_election.cc:302] T a0e18d0975a54e3dbeb0a13cb226fefd P 692da0319a2d4a81ad81f5d8f9f4c86b [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 692da0319a2d4a81ad81f5d8f9f4c86b, a2f9bf2e5d9b4fc19f804797bca4a3ff; no voters:
I20240503 23:26:39.574262 4592 raft_consensus.cc:2781] T a0e18d0975a54e3dbeb0a13cb226fefd P 692da0319a2d4a81ad81f5d8f9f4c86b [term 3 FOLLOWER]: Leader election won for term 3
I20240503 23:26:39.574501 3760 raft_consensus.cc:2454] T a0e18d0975a54e3dbeb0a13cb226fefd P 83ffd7184ec94302bd1ee74b0fab3e17 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 692da0319a2d4a81ad81f5d8f9f4c86b in term 3.
I20240503 23:26:39.574913 4592 raft_consensus.cc:686] T a0e18d0975a54e3dbeb0a13cb226fefd P 692da0319a2d4a81ad81f5d8f9f4c86b [term 3 LEADER]: Becoming Leader. State: Replica: 692da0319a2d4a81ad81f5d8f9f4c86b, State: Running, Role: LEADER
I20240503 23:26:39.575222 4592 consensus_queue.cc:237] T a0e18d0975a54e3dbeb0a13cb226fefd P 692da0319a2d4a81ad81f5d8f9f4c86b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 30, Committed index: 30, Last appended: 2.30, Last appended by leader: 30, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "83ffd7184ec94302bd1ee74b0fab3e17" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 32909 } } peers { permanent_uuid: "a2f9bf2e5d9b4fc19f804797bca4a3ff" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 46261 } } peers { permanent_uuid: "692da0319a2d4a81ad81f5d8f9f4c86b" member_type: VOTER last_known_addr { host: "127.0.100.5" port: 38561 } attrs { promote: false } }
I20240503 23:26:39.578295 3655 catalog_manager.cc:5484] T a0e18d0975a54e3dbeb0a13cb226fefd P 692da0319a2d4a81ad81f5d8f9f4c86b reported cstate change: term changed from 2 to 3, leader changed from 83ffd7184ec94302bd1ee74b0fab3e17 (127.0.100.1) to 692da0319a2d4a81ad81f5d8f9f4c86b (127.0.100.5). New cstate: current_term: 3 leader_uuid: "692da0319a2d4a81ad81f5d8f9f4c86b" committed_config { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "83ffd7184ec94302bd1ee74b0fab3e17" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 32909 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a2f9bf2e5d9b4fc19f804797bca4a3ff" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 46261 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "692da0319a2d4a81ad81f5d8f9f4c86b" member_type: VOTER last_known_addr { host: "127.0.100.5" port: 38561 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 23:26:39.582330 4671 raft_consensus.cc:1050] T 55cf664d5e874c0d9b9adf8bdc9c43ed P e4086c099d7f4447b2b4733b6ca364df: attempting to promote NON_VOTER 4d1d4e261584467fbbce4c72fa18045a to VOTER
I20240503 23:26:39.586259 4671 consensus_queue.cc:237] T 55cf664d5e874c0d9b9adf8bdc9c43ed P e4086c099d7f4447b2b4733b6ca364df [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 31, Committed index: 31, Last appended: 2.31, Last appended by leader: 30, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "83ffd7184ec94302bd1ee74b0fab3e17" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 32909 } attrs { replace: true } } peers { permanent_uuid: "a2f9bf2e5d9b4fc19f804797bca4a3ff" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 46261 } } peers { permanent_uuid: "e4086c099d7f4447b2b4733b6ca364df" member_type: VOTER last_known_addr { host: "127.0.100.4" port: 35011 } attrs { promote: false } } peers { permanent_uuid: "4d1d4e261584467fbbce4c72fa18045a" member_type: VOTER last_known_addr { host: "127.0.100.6" port: 35349 } attrs { promote: false } }
I20240503 23:26:39.589036 4367 raft_consensus.cc:1261] T 55cf664d5e874c0d9b9adf8bdc9c43ed P 4d1d4e261584467fbbce4c72fa18045a [term 2 LEARNER]: Refusing update from remote peer e4086c099d7f4447b2b4733b6ca364df: Log matching property violated. Preceding OpId in replica: term: 2 index: 31. Preceding OpId from leader: term: 2 index: 32. (index mismatch)
I20240503 23:26:39.589020 3837 raft_consensus.cc:1261] T 55cf664d5e874c0d9b9adf8bdc9c43ed P a2f9bf2e5d9b4fc19f804797bca4a3ff [term 2 FOLLOWER]: Refusing update from remote peer e4086c099d7f4447b2b4733b6ca364df: Log matching property violated. Preceding OpId in replica: term: 2 index: 31. Preceding OpId from leader: term: 2 index: 32. (index mismatch)
I20240503 23:26:39.589020 3760 raft_consensus.cc:1261] T 55cf664d5e874c0d9b9adf8bdc9c43ed P 83ffd7184ec94302bd1ee74b0fab3e17 [term 2 FOLLOWER]: Refusing update from remote peer e4086c099d7f4447b2b4733b6ca364df: Log matching property violated. Preceding OpId in replica: term: 2 index: 31. Preceding OpId from leader: term: 2 index: 32. (index mismatch)
I20240503 23:26:39.589643 4672 consensus_queue.cc:1035] T 55cf664d5e874c0d9b9adf8bdc9c43ed P e4086c099d7f4447b2b4733b6ca364df [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d1d4e261584467fbbce4c72fa18045a" member_type: VOTER last_known_addr { host: "127.0.100.6" port: 35349 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 32, Last known committed idx: 31, Time since last communication: 0.000s
I20240503 23:26:39.590513 4673 consensus_queue.cc:1035] T 55cf664d5e874c0d9b9adf8bdc9c43ed P e4086c099d7f4447b2b4733b6ca364df [LEADER]: Connected to new peer: Peer: permanent_uuid: "83ffd7184ec94302bd1ee74b0fab3e17" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 32909 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 32, Last known committed idx: 31, Time since last communication: 0.000s
I20240503 23:26:39.591547 4621 consensus_queue.cc:1035] T 55cf664d5e874c0d9b9adf8bdc9c43ed P e4086c099d7f4447b2b4733b6ca364df [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2f9bf2e5d9b4fc19f804797bca4a3ff" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 46261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 32, Last known committed idx: 31, Time since last communication: 0.000s
I20240503 23:26:39.595058 4621 raft_consensus.cc:2932] T 55cf664d5e874c0d9b9adf8bdc9c43ed P e4086c099d7f4447b2b4733b6ca364df [term 2 LEADER]: Committing config change with OpId 2.32: config changed from index 30 to 32, 4d1d4e261584467fbbce4c72fa18045a (127.0.100.6) changed from NON_VOTER to VOTER. New config: { opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "83ffd7184ec94302bd1ee74b0fab3e17" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 32909 } attrs { replace: true } } peers { permanent_uuid: "a2f9bf2e5d9b4fc19f804797bca4a3ff" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 46261 } } peers { permanent_uuid: "e4086c099d7f4447b2b4733b6ca364df" member_type: VOTER last_known_addr { host: "127.0.100.4" port: 35011 } attrs { promote: false } } peers { permanent_uuid: "4d1d4e261584467fbbce4c72fa18045a" member_type: VOTER last_known_addr { host: "127.0.100.6" port: 35349 } attrs { promote: false } } }
I20240503 23:26:39.595657 3837 raft_consensus.cc:2932] T 55cf664d5e874c0d9b9adf8bdc9c43ed P a2f9bf2e5d9b4fc19f804797bca4a3ff [term 2 FOLLOWER]: Committing config change with OpId 2.32: config changed from index 30 to 32, 4d1d4e261584467fbbce4c72fa18045a (127.0.100.6) changed from NON_VOTER to VOTER. New config: { opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "83ffd7184ec94302bd1ee74b0fab3e17" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 32909 } attrs { replace: true } } peers { permanent_uuid: "a2f9bf2e5d9b4fc19f804797bca4a3ff" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 46261 } } peers { permanent_uuid: "e4086c099d7f4447b2b4733b6ca364df" member_type: VOTER last_known_addr { host: "127.0.100.4" port: 35011 } attrs { promote: false } } peers { permanent_uuid: "4d1d4e261584467fbbce4c72fa18045a" member_type: VOTER last_known_addr { host: "127.0.100.6" port: 35349 } attrs { promote: false } } }
I20240503 23:26:39.596215 4367 raft_consensus.cc:2932] T 55cf664d5e874c0d9b9adf8bdc9c43ed P 4d1d4e261584467fbbce4c72fa18045a [term 2 FOLLOWER]: Committing config change with OpId 2.32: config changed from index 30 to 32, 4d1d4e261584467fbbce4c72fa18045a (127.0.100.6) changed from NON_VOTER to VOTER. New config: { opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "83ffd7184ec94302bd1ee74b0fab3e17" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 32909 } attrs { replace: true } } peers { permanent_uuid: "a2f9bf2e5d9b4fc19f804797bca4a3ff" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 46261 } } peers { permanent_uuid: "e4086c099d7f4447b2b4733b6ca364df" member_type: VOTER last_known_addr { host: "127.0.100.4" port: 35011 } attrs { promote: false } } peers { permanent_uuid: "4d1d4e261584467fbbce4c72fa18045a" member_type: VOTER last_known_addr { host: "127.0.100.6" port: 35349 } attrs { promote: false } } }
I20240503 23:26:39.601323 3760 raft_consensus.cc:2932] T 55cf664d5e874c0d9b9adf8bdc9c43ed P 83ffd7184ec94302bd1ee74b0fab3e17 [term 2 FOLLOWER]: Committing config change with OpId 2.32: config changed from index 30 to 32, 4d1d4e261584467fbbce4c72fa18045a (127.0.100.6) changed from NON_VOTER to VOTER. New config: { opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "83ffd7184ec94302bd1ee74b0fab3e17" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 32909 } attrs { replace: true } } peers { permanent_uuid: "a2f9bf2e5d9b4fc19f804797bca4a3ff" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 46261 } } peers { permanent_uuid: "e4086c099d7f4447b2b4733b6ca364df" member_type: VOTER last_known_addr { host: "127.0.100.4" port: 35011 } attrs { promote: false } } peers { permanent_uuid: "4d1d4e261584467fbbce4c72fa18045a" member_type: VOTER last_known_addr { host: "127.0.100.6" port: 35349 } attrs { promote: false } } }
I20240503 23:26:39.601482 3655 catalog_manager.cc:5484] T 55cf664d5e874c0d9b9adf8bdc9c43ed P e4086c099d7f4447b2b4733b6ca364df reported cstate change: config changed from index 30 to 32, 4d1d4e261584467fbbce4c72fa18045a (127.0.100.6) changed from NON_VOTER to VOTER. New cstate: current_term: 2 leader_uuid: "e4086c099d7f4447b2b4733b6ca364df" committed_config { opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "83ffd7184ec94302bd1ee74b0fab3e17" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 32909 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a2f9bf2e5d9b4fc19f804797bca4a3ff" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 46261 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e4086c099d7f4447b2b4733b6ca364df" member_type: VOTER last_known_addr { host: "127.0.100.4" port: 35011 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4d1d4e261584467fbbce4c72fa18045a" member_type: VOTER last_known_addr { host: "127.0.100.6" port: 35349 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 23:26:39.607030 4218 consensus_queue.cc:237] T 55cf664d5e874c0d9b9adf8bdc9c43ed P e4086c099d7f4447b2b4733b6ca364df [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 32, Committed index: 32, Last appended: 2.32, Last appended by leader: 30, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 33 OBSOLETE_local: false peers { permanent_uuid: "a2f9bf2e5d9b4fc19f804797bca4a3ff" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 46261 } } peers { permanent_uuid: "e4086c099d7f4447b2b4733b6ca364df" member_type: VOTER last_known_addr { host: "127.0.100.4" port: 35011 } attrs { promote: false } } peers { permanent_uuid: "4d1d4e261584467fbbce4c72fa18045a" member_type: VOTER last_known_addr { host: "127.0.100.6" port: 35349 } attrs { promote: false } }
I20240503 23:26:39.609967 3837 raft_consensus.cc:1261] T 55cf664d5e874c0d9b9adf8bdc9c43ed P a2f9bf2e5d9b4fc19f804797bca4a3ff [term 2 FOLLOWER]: Refusing update from remote peer e4086c099d7f4447b2b4733b6ca364df: Log matching property violated. Preceding OpId in replica: term: 2 index: 32. Preceding OpId from leader: term: 2 index: 33. (index mismatch)
I20240503 23:26:39.610599 4673 consensus_queue.cc:1035] T 55cf664d5e874c0d9b9adf8bdc9c43ed P e4086c099d7f4447b2b4733b6ca364df [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2f9bf2e5d9b4fc19f804797bca4a3ff" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 46261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 33, Last known committed idx: 32, Time since last communication: 0.000s
I20240503 23:26:39.610966 4367 raft_consensus.cc:1261] T 55cf664d5e874c0d9b9adf8bdc9c43ed P 4d1d4e261584467fbbce4c72fa18045a [term 2 FOLLOWER]: Refusing update from remote peer e4086c099d7f4447b2b4733b6ca364df: Log matching property violated. Preceding OpId in replica: term: 2 index: 32. Preceding OpId from leader: term: 2 index: 33. (index mismatch)
I20240503 23:26:39.611438 4673 consensus_queue.cc:1035] T 55cf664d5e874c0d9b9adf8bdc9c43ed P e4086c099d7f4447b2b4733b6ca364df [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d1d4e261584467fbbce4c72fa18045a" member_type: VOTER last_known_addr { host: "127.0.100.6" port: 35349 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 33, Last known committed idx: 32, Time since last communication: 0.000s
I20240503 23:26:39.613843 4672 raft_consensus.cc:2932] T 55cf664d5e874c0d9b9adf8bdc9c43ed P e4086c099d7f4447b2b4733b6ca364df [term 2 LEADER]: Committing config change with OpId 2.33: config changed from index 32 to 33, VOTER 83ffd7184ec94302bd1ee74b0fab3e17 (127.0.100.1) evicted. New config: { opid_index: 33 OBSOLETE_local: false peers { permanent_uuid: "a2f9bf2e5d9b4fc19f804797bca4a3ff" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 46261 } } peers { permanent_uuid: "e4086c099d7f4447b2b4733b6ca364df" member_type: VOTER last_known_addr { host: "127.0.100.4" port: 35011 } attrs { promote: false } } peers { permanent_uuid: "4d1d4e261584467fbbce4c72fa18045a" member_type: VOTER last_known_addr { host: "127.0.100.6" port: 35349 } attrs { promote: false } } }
I20240503 23:26:39.614372 3837 raft_consensus.cc:2932] T 55cf664d5e874c0d9b9adf8bdc9c43ed P a2f9bf2e5d9b4fc19f804797bca4a3ff [term 2 FOLLOWER]: Committing config change with OpId 2.33: config changed from index 32 to 33, VOTER 83ffd7184ec94302bd1ee74b0fab3e17 (127.0.100.1) evicted. New config: { opid_index: 33 OBSOLETE_local: false peers { permanent_uuid: "a2f9bf2e5d9b4fc19f804797bca4a3ff" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 46261 } } peers { permanent_uuid: "e4086c099d7f4447b2b4733b6ca364df" member_type: VOTER last_known_addr { host: "127.0.100.4" port: 35011 } attrs { promote: false } } peers { permanent_uuid: "4d1d4e261584467fbbce4c72fa18045a" member_type: VOTER last_known_addr { host: "127.0.100.6" port: 35349 } attrs { promote: false } } }
I20240503 23:26:39.614782 4367 raft_consensus.cc:2932] T 55cf664d5e874c0d9b9adf8bdc9c43ed P 4d1d4e261584467fbbce4c72fa18045a [term 2 FOLLOWER]: Committing config change with OpId 2.33: config changed from index 32 to 33, VOTER 83ffd7184ec94302bd1ee74b0fab3e17 (127.0.100.1) evicted. New config: { opid_index: 33 OBSOLETE_local: false peers { permanent_uuid: "a2f9bf2e5d9b4fc19f804797bca4a3ff" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 46261 } } peers { permanent_uuid: "e4086c099d7f4447b2b4733b6ca364df" member_type: VOTER last_known_addr { host: "127.0.100.4" port: 35011 } attrs { promote: false } } peers { permanent_uuid: "4d1d4e261584467fbbce4c72fa18045a" member_type: VOTER last_known_addr { host: "127.0.100.6" port: 35349 } attrs { promote: false } } }
I20240503 23:26:39.618162 3704 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 55cf664d5e874c0d9b9adf8bdc9c43ed with cas_config_opid_index 32: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 23:26:39.619933 3662 catalog_manager.cc:5484] T 55cf664d5e874c0d9b9adf8bdc9c43ed P 4d1d4e261584467fbbce4c72fa18045a reported cstate change: config changed from index 32 to 33, VOTER 83ffd7184ec94302bd1ee74b0fab3e17 (127.0.100.1) evicted. New cstate: current_term: 2 leader_uuid: "e4086c099d7f4447b2b4733b6ca364df" committed_config { opid_index: 33 OBSOLETE_local: false peers { permanent_uuid: "a2f9bf2e5d9b4fc19f804797bca4a3ff" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 46261 } } peers { permanent_uuid: "e4086c099d7f4447b2b4733b6ca364df" member_type: VOTER last_known_addr { host: "127.0.100.4" port: 35011 } attrs { promote: false } } peers { permanent_uuid: "4d1d4e261584467fbbce4c72fa18045a" member_type: VOTER last_known_addr { host: "127.0.100.6" port: 35349 } attrs { promote: false } } }
I20240503 23:26:39.624053 3747 tablet_service.cc:1514] Processing DeleteTablet for tablet 55cf664d5e874c0d9b9adf8bdc9c43ed with delete_type TABLET_DATA_TOMBSTONED (TS 83ffd7184ec94302bd1ee74b0fab3e17 not found in new config with opid_index 33) from {username='slave'} at 127.0.0.1:35576
I20240503 23:26:39.624809 4678 tablet_replica.cc:330] T 55cf664d5e874c0d9b9adf8bdc9c43ed P 83ffd7184ec94302bd1ee74b0fab3e17: stopping tablet replica
I20240503 23:26:39.625819 4678 raft_consensus.cc:2229] T 55cf664d5e874c0d9b9adf8bdc9c43ed P 83ffd7184ec94302bd1ee74b0fab3e17 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 23:26:39.626178 4678 raft_consensus.cc:2258] T 55cf664d5e874c0d9b9adf8bdc9c43ed P 83ffd7184ec94302bd1ee74b0fab3e17 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 23:26:39.627161 4678 ts_tablet_manager.cc:1875] T 55cf664d5e874c0d9b9adf8bdc9c43ed P 83ffd7184ec94302bd1ee74b0fab3e17: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 23:26:39.632537 4678 ts_tablet_manager.cc:1888] T 55cf664d5e874c0d9b9adf8bdc9c43ed P 83ffd7184ec94302bd1ee74b0fab3e17: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.32
I20240503 23:26:39.632745 4678 log.cc:1196] T 55cf664d5e874c0d9b9adf8bdc9c43ed P 83ffd7184ec94302bd1ee74b0fab3e17: Deleting WAL directory at /tmp/dist-test-taskZhIiPz/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714778688451250-400-0/minicluster-data/ts-0-root/wals/55cf664d5e874c0d9b9adf8bdc9c43ed
I20240503 23:26:39.633656 3648 catalog_manager.cc:4830] TS 83ffd7184ec94302bd1ee74b0fab3e17 (127.0.100.1:32909): tablet 55cf664d5e874c0d9b9adf8bdc9c43ed (table test-workload [id=1345b2a336c84ec2b1de62853e7fa3f6]) successfully deleted
I20240503 23:26:39.844802 3907 raft_consensus.cc:1261] T 6039230222ff4e57929bed49cd56caf5 P a3eecbaec43d4954b1096bc3121e824c [term 3 FOLLOWER]: Refusing update from remote peer a2f9bf2e5d9b4fc19f804797bca4a3ff: Log matching property violated. Preceding OpId in replica: term: 2 index: 29. Preceding OpId from leader: term: 3 index: 30. (index mismatch)
I20240503 23:26:39.845515 4656 consensus_queue.cc:1035] T 6039230222ff4e57929bed49cd56caf5 P a2f9bf2e5d9b4fc19f804797bca4a3ff [LEADER]: Connected to new peer: Peer: permanent_uuid: "a3eecbaec43d4954b1096bc3121e824c" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 40649 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 30, Last known committed idx: 29, Time since last communication: 0.000s
I20240503 23:26:39.865115 4293 raft_consensus.cc:1261] T 6039230222ff4e57929bed49cd56caf5 P 692da0319a2d4a81ad81f5d8f9f4c86b [term 3 FOLLOWER]: Refusing update from remote peer a2f9bf2e5d9b4fc19f804797bca4a3ff: Log matching property violated. Preceding OpId in replica: term: 2 index: 29. Preceding OpId from leader: term: 3 index: 30. (index mismatch)
I20240503 23:26:39.865923 4683 consensus_queue.cc:1035] T 6039230222ff4e57929bed49cd56caf5 P a2f9bf2e5d9b4fc19f804797bca4a3ff [LEADER]: Connected to new peer: Peer: permanent_uuid: "692da0319a2d4a81ad81f5d8f9f4c86b" member_type: VOTER last_known_addr { host: "127.0.100.5" port: 38561 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 30, Last known committed idx: 29, Time since last communication: 0.000s
I20240503 23:26:39.872197 3907 raft_consensus.cc:1261] T d0c4430fc56644cb9f87deffe4fe9407 P a3eecbaec43d4954b1096bc3121e824c [term 5 FOLLOWER]: Refusing update from remote peer a2f9bf2e5d9b4fc19f804797bca4a3ff: Log matching property violated. Preceding OpId in replica: term: 4 index: 30. Preceding OpId from leader: term: 5 index: 31. (index mismatch)
I20240503 23:26:39.872895 4683 consensus_queue.cc:1035] T d0c4430fc56644cb9f87deffe4fe9407 P a2f9bf2e5d9b4fc19f804797bca4a3ff [LEADER]: Connected to new peer: Peer: permanent_uuid: "a3eecbaec43d4954b1096bc3121e824c" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 40649 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 31, Last known committed idx: 30, Time since last communication: 0.000s
I20240503 23:26:39.879341 3760 raft_consensus.cc:1261] T d0c4430fc56644cb9f87deffe4fe9407 P 83ffd7184ec94302bd1ee74b0fab3e17 [term 5 FOLLOWER]: Refusing update from remote peer a2f9bf2e5d9b4fc19f804797bca4a3ff: Log matching property violated. Preceding OpId in replica: term: 4 index: 30. Preceding OpId from leader: term: 5 index: 31. (index mismatch)
I20240503 23:26:39.880151 4683 consensus_queue.cc:1035] T d0c4430fc56644cb9f87deffe4fe9407 P a2f9bf2e5d9b4fc19f804797bca4a3ff [LEADER]: Connected to new peer: Peer: permanent_uuid: "83ffd7184ec94302bd1ee74b0fab3e17" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 32909 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 31, Last known committed idx: 30, Time since last communication: 0.000s
I20240503 23:26:39.882205 4367 raft_consensus.cc:3037] T d0c4430fc56644cb9f87deffe4fe9407 P 4d1d4e261584467fbbce4c72fa18045a [term 3 LEARNER]: Advancing to term 5
I20240503 23:26:39.887846 4367 raft_consensus.cc:1261] T d0c4430fc56644cb9f87deffe4fe9407 P 4d1d4e261584467fbbce4c72fa18045a [term 5 LEARNER]: Refusing update from remote peer a2f9bf2e5d9b4fc19f804797bca4a3ff: Log matching property violated. Preceding OpId in replica: term: 3 index: 29. Preceding OpId from leader: term: 5 index: 31. (index mismatch)
I20240503 23:26:39.892099 4683 consensus_queue.cc:1035] T d0c4430fc56644cb9f87deffe4fe9407 P a2f9bf2e5d9b4fc19f804797bca4a3ff [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d1d4e261584467fbbce4c72fa18045a" member_type: NON_VOTER last_known_addr { host: "127.0.100.6" port: 35349 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 31, Last known committed idx: 29, Time since last communication: 0.000s
I20240503 23:26:39.897548 4656 raft_consensus.cc:1050] T d0c4430fc56644cb9f87deffe4fe9407 P a2f9bf2e5d9b4fc19f804797bca4a3ff: attempting to promote NON_VOTER 4d1d4e261584467fbbce4c72fa18045a to VOTER
I20240503 23:26:39.898483 4656 consensus_queue.cc:237] T d0c4430fc56644cb9f87deffe4fe9407 P a2f9bf2e5d9b4fc19f804797bca4a3ff [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 31, Committed index: 31, Last appended: 5.31, Last appended by leader: 30, Current term: 5, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "83ffd7184ec94302bd1ee74b0fab3e17" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 32909 } attrs { replace: true } } peers { permanent_uuid: "a2f9bf2e5d9b4fc19f804797bca4a3ff" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 46261 } } peers { permanent_uuid: "a3eecbaec43d4954b1096bc3121e824c" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 40649 } } peers { permanent_uuid: "4d1d4e261584467fbbce4c72fa18045a" member_type: VOTER last_known_addr { host: "127.0.100.6" port: 35349 } attrs { promote: false } }
I20240503 23:26:39.901690 3760 raft_consensus.cc:1261] T d0c4430fc56644cb9f87deffe4fe9407 P 83ffd7184ec94302bd1ee74b0fab3e17 [term 5 FOLLOWER]: Refusing update from remote peer a2f9bf2e5d9b4fc19f804797bca4a3ff: Log matching property violated. Preceding OpId in replica: term: 5 index: 31. Preceding OpId from leader: term: 5 index: 32. (index mismatch)
I20240503 23:26:39.901690 3907 raft_consensus.cc:1261] T d0c4430fc56644cb9f87deffe4fe9407 P a3eecbaec43d4954b1096bc3121e824c [term 5 FOLLOWER]: Refusing update from remote peer a2f9bf2e5d9b4fc19f804797bca4a3ff: Log matching property violated. Preceding OpId in replica: term: 5 index: 31. Preceding OpId from leader: term: 5 index: 32. (index mismatch)
I20240503 23:26:39.902084 4367 raft_consensus.cc:1261] T d0c4430fc56644cb9f87deffe4fe9407 P 4d1d4e261584467fbbce4c72fa18045a [term 5 LEARNER]: Refusing update from remote peer a2f9bf2e5d9b4fc19f804797bca4a3ff: Log matching property violated. Preceding OpId in replica: term: 5 index: 31. Preceding OpId from leader: term: 5 index: 32. (index mismatch)
I20240503 23:26:39.902293 4683 consensus_queue.cc:1035] T d0c4430fc56644cb9f87deffe4fe9407 P a2f9bf2e5d9b4fc19f804797bca4a3ff [LEADER]: Connected to new peer: Peer: permanent_uuid: "83ffd7184ec94302bd1ee74b0fab3e17" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 32909 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 32, Last known committed idx: 31, Time since last communication: 0.000s
I20240503 23:26:39.902894 4682 consensus_queue.cc:1035] T d0c4430fc56644cb9f87deffe4fe9407 P a2f9bf2e5d9b4fc19f804797bca4a3ff [LEADER]: Connected to new peer: Peer: permanent_uuid: "a3eecbaec43d4954b1096bc3121e824c" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 40649 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 32, Last known committed idx: 31, Time since last communication: 0.000s
I20240503 23:26:39.903163 4656 consensus_queue.cc:1035] T d0c4430fc56644cb9f87deffe4fe9407 P a2f9bf2e5d9b4fc19f804797bca4a3ff [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d1d4e261584467fbbce4c72fa18045a" member_type: VOTER last_known_addr { host: "127.0.100.6" port: 35349 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 32, Last known committed idx: 31, Time since last communication: 0.000s
I20240503 23:26:39.907128 4367 raft_consensus.cc:2932] T d0c4430fc56644cb9f87deffe4fe9407 P 4d1d4e261584467fbbce4c72fa18045a [term 5 FOLLOWER]: Committing config change with OpId 5.32: config changed from index 27 to 32, 4d1d4e261584467fbbce4c72fa18045a (127.0.100.6) changed from NON_VOTER to VOTER. New config: { opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "83ffd7184ec94302bd1ee74b0fab3e17" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 32909 } attrs { replace: true } } peers { permanent_uuid: "a2f9bf2e5d9b4fc19f804797bca4a3ff" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 46261 } } peers { permanent_uuid: "a3eecbaec43d4954b1096bc3121e824c" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 40649 } } peers { permanent_uuid: "4d1d4e261584467fbbce4c72fa18045a" member_type: VOTER last_known_addr { host: "127.0.100.6" port: 35349 } attrs { promote: false } } }
I20240503 23:26:39.907155 3907 raft_consensus.cc:2932] T d0c4430fc56644cb9f87deffe4fe9407 P a3eecbaec43d4954b1096bc3121e824c [term 5 FOLLOWER]: Committing config change with OpId 5.32: config changed from index 27 to 32, 4d1d4e261584467fbbce4c72fa18045a (127.0.100.6) changed from NON_VOTER to VOTER. New config: { opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "83ffd7184ec94302bd1ee74b0fab3e17" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 32909 } attrs { replace: true } } peers { permanent_uuid: "a2f9bf2e5d9b4fc19f804797bca4a3ff" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 46261 } } peers { permanent_uuid: "a3eecbaec43d4954b1096bc3121e824c" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 40649 } } peers { permanent_uuid: "4d1d4e261584467fbbce4c72fa18045a" member_type: VOTER last_known_addr { host: "127.0.100.6" port: 35349 } attrs { promote: false } } }
I20240503 23:26:39.906442 4683 raft_consensus.cc:2932] T d0c4430fc56644cb9f87deffe4fe9407 P a2f9bf2e5d9b4fc19f804797bca4a3ff [term 5 LEADER]: Committing config change with OpId 5.32: config changed from index 27 to 32, 4d1d4e261584467fbbce4c72fa18045a (127.0.100.6) changed from NON_VOTER to VOTER. New config: { opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "83ffd7184ec94302bd1ee74b0fab3e17" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 32909 } attrs { replace: true } } peers { permanent_uuid: "a2f9bf2e5d9b4fc19f804797bca4a3ff" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 46261 } } peers { permanent_uuid: "a3eecbaec43d4954b1096bc3121e824c" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 40649 } } peers { permanent_uuid: "4d1d4e261584467fbbce4c72fa18045a" member_type: VOTER last_known_addr { host: "127.0.100.6" port: 35349 } attrs { promote: false } } }
I20240503 23:26:39.913559 3655 catalog_manager.cc:5484] T d0c4430fc56644cb9f87deffe4fe9407 P a3eecbaec43d4954b1096bc3121e824c reported cstate change: config changed from index 27 to 32, 4d1d4e261584467fbbce4c72fa18045a (127.0.100.6) changed from NON_VOTER to VOTER. New cstate: current_term: 5 leader_uuid: "a2f9bf2e5d9b4fc19f804797bca4a3ff" committed_config { opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "83ffd7184ec94302bd1ee74b0fab3e17" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 32909 } attrs { replace: true } } peers { permanent_uuid: "a2f9bf2e5d9b4fc19f804797bca4a3ff" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 46261 } } peers { permanent_uuid: "a3eecbaec43d4954b1096bc3121e824c" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 40649 } } peers { permanent_uuid: "4d1d4e261584467fbbce4c72fa18045a" member_type: VOTER last_known_addr { host: "127.0.100.6" port: 35349 } attrs { promote: false } } }
I20240503 23:26:39.915275 3760 raft_consensus.cc:2932] T d0c4430fc56644cb9f87deffe4fe9407 P 83ffd7184ec94302bd1ee74b0fab3e17 [term 5 FOLLOWER]: Committing config change with OpId 5.32: config changed from index 27 to 32, 4d1d4e261584467fbbce4c72fa18045a (127.0.100.6) changed from NON_VOTER to VOTER. New config: { opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "83ffd7184ec94302bd1ee74b0fab3e17" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 32909 } attrs { replace: true } } peers { permanent_uuid: "a2f9bf2e5d9b4fc19f804797bca4a3ff" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 46261 } } peers { permanent_uuid: "a3eecbaec43d4954b1096bc3121e824c" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 40649 } } peers { permanent_uuid: "4d1d4e261584467fbbce4c72fa18045a" member_type: VOTER last_known_addr { host: "127.0.100.6" port: 35349 } attrs { promote: false } } }
I20240503 23:26:39.921782 3837 consensus_queue.cc:237] T d0c4430fc56644cb9f87deffe4fe9407 P a2f9bf2e5d9b4fc19f804797bca4a3ff [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 32, Committed index: 32, Last appended: 5.32, Last appended by leader: 30, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 33 OBSOLETE_local: false peers { permanent_uuid: "a2f9bf2e5d9b4fc19f804797bca4a3ff" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 46261 } } peers { permanent_uuid: "a3eecbaec43d4954b1096bc3121e824c" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 40649 } } peers { permanent_uuid: "4d1d4e261584467fbbce4c72fa18045a" member_type: VOTER last_known_addr { host: "127.0.100.6" port: 35349 } attrs { promote: false } }
I20240503 23:26:39.924170 4367 raft_consensus.cc:1261] T d0c4430fc56644cb9f87deffe4fe9407 P 4d1d4e261584467fbbce4c72fa18045a [term 5 FOLLOWER]: Refusing update from remote peer a2f9bf2e5d9b4fc19f804797bca4a3ff: Log matching property violated. Preceding OpId in replica: term: 5 index: 32. Preceding OpId from leader: term: 5 index: 33. (index mismatch)
I20240503 23:26:39.924319 3907 raft_consensus.cc:1261] T d0c4430fc56644cb9f87deffe4fe9407 P a3eecbaec43d4954b1096bc3121e824c [term 5 FOLLOWER]: Refusing update from remote peer a2f9bf2e5d9b4fc19f804797bca4a3ff: Log matching property violated. Preceding OpId in replica: term: 5 index: 32. Preceding OpId from leader: term: 5 index: 33. (index mismatch)
I20240503 23:26:39.925033 4683 consensus_queue.cc:1035] T d0c4430fc56644cb9f87deffe4fe9407 P a2f9bf2e5d9b4fc19f804797bca4a3ff [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d1d4e261584467fbbce4c72fa18045a" member_type: VOTER last_known_addr { host: "127.0.100.6" port: 35349 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 33, Last known committed idx: 32, Time since last communication: 0.000s
I20240503 23:26:39.925439 4657 consensus_queue.cc:1035] T d0c4430fc56644cb9f87deffe4fe9407 P a2f9bf2e5d9b4fc19f804797bca4a3ff [LEADER]: Connected to new peer: Peer: permanent_uuid: "a3eecbaec43d4954b1096bc3121e824c" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 40649 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 33, Last known committed idx: 32, Time since last communication: 0.000s
I20240503 23:26:39.928469 4682 raft_consensus.cc:2932] T d0c4430fc56644cb9f87deffe4fe9407 P a2f9bf2e5d9b4fc19f804797bca4a3ff [term 5 LEADER]: Committing config change with OpId 5.33: config changed from index 32 to 33, VOTER 83ffd7184ec94302bd1ee74b0fab3e17 (127.0.100.1) evicted. New config: { opid_index: 33 OBSOLETE_local: false peers { permanent_uuid: "a2f9bf2e5d9b4fc19f804797bca4a3ff" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 46261 } } peers { permanent_uuid: "a3eecbaec43d4954b1096bc3121e824c" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 40649 } } peers { permanent_uuid: "4d1d4e261584467fbbce4c72fa18045a" member_type: VOTER last_known_addr { host: "127.0.100.6" port: 35349 } attrs { promote: false } } }
I20240503 23:26:39.929244 4367 raft_consensus.cc:2932] T d0c4430fc56644cb9f87deffe4fe9407 P 4d1d4e261584467fbbce4c72fa18045a [term 5 FOLLOWER]: Committing config change with OpId 5.33: config changed from index 32 to 33, VOTER 83ffd7184ec94302bd1ee74b0fab3e17 (127.0.100.1) evicted. New config: { opid_index: 33 OBSOLETE_local: false peers { permanent_uuid: "a2f9bf2e5d9b4fc19f804797bca4a3ff" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 46261 } } peers { permanent_uuid: "a3eecbaec43d4954b1096bc3121e824c" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 40649 } } peers { permanent_uuid: "4d1d4e261584467fbbce4c72fa18045a" member_type: VOTER last_known_addr { host: "127.0.100.6" port: 35349 } attrs { promote: false } } }
I20240503 23:26:39.929893 3907 raft_consensus.cc:2932] T d0c4430fc56644cb9f87deffe4fe9407 P a3eecbaec43d4954b1096bc3121e824c [term 5 FOLLOWER]: Committing config change with OpId 5.33: config changed from index 32 to 33, VOTER 83ffd7184ec94302bd1ee74b0fab3e17 (127.0.100.1) evicted. New config: { opid_index: 33 OBSOLETE_local: false peers { permanent_uuid: "a2f9bf2e5d9b4fc19f804797bca4a3ff" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 46261 } } peers { permanent_uuid: "a3eecbaec43d4954b1096bc3121e824c" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 40649 } } peers { permanent_uuid: "4d1d4e261584467fbbce4c72fa18045a" member_type: VOTER last_known_addr { host: "127.0.100.6" port: 35349 } attrs { promote: false } } }
I20240503 23:26:39.934741 3649 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet d0c4430fc56644cb9f87deffe4fe9407 with cas_config_opid_index 32: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 23:26:39.935293 3655 catalog_manager.cc:5484] T d0c4430fc56644cb9f87deffe4fe9407 P 4d1d4e261584467fbbce4c72fa18045a reported cstate change: config changed from index 32 to 33, VOTER 83ffd7184ec94302bd1ee74b0fab3e17 (127.0.100.1) evicted. New cstate: current_term: 5 leader_uuid: "a2f9bf2e5d9b4fc19f804797bca4a3ff" committed_config { opid_index: 33 OBSOLETE_local: false peers { permanent_uuid: "a2f9bf2e5d9b4fc19f804797bca4a3ff" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 46261 } } peers { permanent_uuid: "a3eecbaec43d4954b1096bc3121e824c" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 40649 } } peers { permanent_uuid: "4d1d4e261584467fbbce4c72fa18045a" member_type: VOTER last_known_addr { host: "127.0.100.6" port: 35349 } attrs { promote: false } } }
I20240503 23:26:39.941685 3747 tablet_service.cc:1514] Processing DeleteTablet for tablet d0c4430fc56644cb9f87deffe4fe9407 with delete_type TABLET_DATA_TOMBSTONED (TS 83ffd7184ec94302bd1ee74b0fab3e17 not found in new config with opid_index 33) from {username='slave'} at 127.0.0.1:35576
I20240503 23:26:39.942116 4678 tablet_replica.cc:330] T d0c4430fc56644cb9f87deffe4fe9407 P 83ffd7184ec94302bd1ee74b0fab3e17: stopping tablet replica
I20240503 23:26:39.942353 4678 raft_consensus.cc:2229] T d0c4430fc56644cb9f87deffe4fe9407 P 83ffd7184ec94302bd1ee74b0fab3e17 [term 5 FOLLOWER]: Raft consensus shutting down.
I20240503 23:26:39.942852 4678 raft_consensus.cc:2258] T d0c4430fc56644cb9f87deffe4fe9407 P 83ffd7184ec94302bd1ee74b0fab3e17 [term 5 FOLLOWER]: Raft consensus is shut down!
I20240503 23:26:39.943877 4678 ts_tablet_manager.cc:1875] T d0c4430fc56644cb9f87deffe4fe9407 P 83ffd7184ec94302bd1ee74b0fab3e17: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 23:26:39.951603 4678 ts_tablet_manager.cc:1888] T d0c4430fc56644cb9f87deffe4fe9407 P 83ffd7184ec94302bd1ee74b0fab3e17: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 5.32
I20240503 23:26:39.951766 4678 log.cc:1196] T d0c4430fc56644cb9f87deffe4fe9407 P 83ffd7184ec94302bd1ee74b0fab3e17: Deleting WAL directory at /tmp/dist-test-taskZhIiPz/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714778688451250-400-0/minicluster-data/ts-0-root/wals/d0c4430fc56644cb9f87deffe4fe9407
I20240503 23:26:39.952646 3648 catalog_manager.cc:4830] TS 83ffd7184ec94302bd1ee74b0fab3e17 (127.0.100.1:32909): tablet d0c4430fc56644cb9f87deffe4fe9407 (table test-workload [id=1345b2a336c84ec2b1de62853e7fa3f6]) successfully deleted
I20240503 23:26:40.038571 3837 raft_consensus.cc:1261] T a0e18d0975a54e3dbeb0a13cb226fefd P a2f9bf2e5d9b4fc19f804797bca4a3ff [term 3 FOLLOWER]: Refusing update from remote peer 692da0319a2d4a81ad81f5d8f9f4c86b: Log matching property violated. Preceding OpId in replica: term: 2 index: 30. Preceding OpId from leader: term: 3 index: 31. (index mismatch)
I20240503 23:26:40.039304 4592 consensus_queue.cc:1035] T a0e18d0975a54e3dbeb0a13cb226fefd P 692da0319a2d4a81ad81f5d8f9f4c86b [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2f9bf2e5d9b4fc19f804797bca4a3ff" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 46261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 31, Last known committed idx: 30, Time since last communication: 0.000s
I20240503 23:26:40.043473 3760 raft_consensus.cc:1261] T a0e18d0975a54e3dbeb0a13cb226fefd P 83ffd7184ec94302bd1ee74b0fab3e17 [term 3 FOLLOWER]: Refusing update from remote peer 692da0319a2d4a81ad81f5d8f9f4c86b: Log matching property violated. Preceding OpId in replica: term: 2 index: 30. Preceding OpId from leader: term: 3 index: 31. (index mismatch)
I20240503 23:26:40.044127 4592 consensus_queue.cc:1035] T a0e18d0975a54e3dbeb0a13cb226fefd P 692da0319a2d4a81ad81f5d8f9f4c86b [LEADER]: Connected to new peer: Peer: permanent_uuid: "83ffd7184ec94302bd1ee74b0fab3e17" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 32909 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 31, Last known committed idx: 30, Time since last communication: 0.000s
W20240503 23:26:40.624848 3710 auto_rebalancer.cc:249] could not retrieve auto-rebalancing replica moves: Not found: table 1345b2a336c84ec2b1de62853e7fa3f6: could not find any suitable replica to move from server a2f9bf2e5d9b4fc19f804797bca4a3ff to server e4086c099d7f4447b2b4733b6ca364df
I20240503 23:26:41.256511 3715 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240503 23:26:41.259701 3837 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "6039230222ff4e57929bed49cd56caf5"
dest_uuid: "a2f9bf2e5d9b4fc19f804797bca4a3ff"
mode: GRACEFUL
new_leader_uuid: "a3eecbaec43d4954b1096bc3121e824c"
from {username='slave'} at 127.0.0.1:55472
I20240503 23:26:41.260000 3837 raft_consensus.cc:595] T 6039230222ff4e57929bed49cd56caf5 P a2f9bf2e5d9b4fc19f804797bca4a3ff [term 3 LEADER]: Received request to transfer leadership to TS a3eecbaec43d4954b1096bc3121e824c
I20240503 23:26:41.261369 3837 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "d0c4430fc56644cb9f87deffe4fe9407"
dest_uuid: "a2f9bf2e5d9b4fc19f804797bca4a3ff"
mode: GRACEFUL
new_leader_uuid: "4d1d4e261584467fbbce4c72fa18045a"
from {username='slave'} at 127.0.0.1:55472
I20240503 23:26:41.261588 3837 raft_consensus.cc:595] T d0c4430fc56644cb9f87deffe4fe9407 P a2f9bf2e5d9b4fc19f804797bca4a3ff [term 5 LEADER]: Received request to transfer leadership to TS 4d1d4e261584467fbbce4c72fa18045a
I20240503 23:26:41.262039 3715 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 2
I20240503 23:26:41.262266 3715 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240503 23:26:41.368458 4697 raft_consensus.cc:979] T 6039230222ff4e57929bed49cd56caf5 P a2f9bf2e5d9b4fc19f804797bca4a3ff: : Instructing follower a3eecbaec43d4954b1096bc3121e824c to start an election
I20240503 23:26:41.368707 4683 raft_consensus.cc:1067] T 6039230222ff4e57929bed49cd56caf5 P a2f9bf2e5d9b4fc19f804797bca4a3ff [term 3 LEADER]: Signalling peer a3eecbaec43d4954b1096bc3121e824c to start an election
I20240503 23:26:41.369380 3908 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "6039230222ff4e57929bed49cd56caf5"
dest_uuid: "a3eecbaec43d4954b1096bc3121e824c"
from {username='slave'} at 127.0.0.1:40636
I20240503 23:26:41.369565 3908 raft_consensus.cc:482] T 6039230222ff4e57929bed49cd56caf5 P a3eecbaec43d4954b1096bc3121e824c [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 23:26:41.369683 3908 raft_consensus.cc:3037] T 6039230222ff4e57929bed49cd56caf5 P a3eecbaec43d4954b1096bc3121e824c [term 3 FOLLOWER]: Advancing to term 4
I20240503 23:26:41.371855 3908 raft_consensus.cc:504] T 6039230222ff4e57929bed49cd56caf5 P a3eecbaec43d4954b1096bc3121e824c [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "a3eecbaec43d4954b1096bc3121e824c" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 40649 } } peers { permanent_uuid: "a2f9bf2e5d9b4fc19f804797bca4a3ff" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 46261 } } peers { permanent_uuid: "692da0319a2d4a81ad81f5d8f9f4c86b" member_type: VOTER last_known_addr { host: "127.0.100.5" port: 38561 } attrs { promote: false } }
I20240503 23:26:41.372785 3908 leader_election.cc:288] T 6039230222ff4e57929bed49cd56caf5 P a3eecbaec43d4954b1096bc3121e824c [CANDIDATE]: Term 4 election: Requested vote from peers a2f9bf2e5d9b4fc19f804797bca4a3ff (127.0.100.2:46261), 692da0319a2d4a81ad81f5d8f9f4c86b (127.0.100.5:38561)
I20240503 23:26:41.373358 4293 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6039230222ff4e57929bed49cd56caf5" candidate_uuid: "a3eecbaec43d4954b1096bc3121e824c" candidate_term: 4 candidate_status { last_received { term: 3 index: 30 } } ignore_live_leader: true dest_uuid: "692da0319a2d4a81ad81f5d8f9f4c86b"
I20240503 23:26:41.373358 3837 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6039230222ff4e57929bed49cd56caf5" candidate_uuid: "a3eecbaec43d4954b1096bc3121e824c" candidate_term: 4 candidate_status { last_received { term: 3 index: 30 } } ignore_live_leader: true dest_uuid: "a2f9bf2e5d9b4fc19f804797bca4a3ff"
I20240503 23:26:41.373633 3837 raft_consensus.cc:3032] T 6039230222ff4e57929bed49cd56caf5 P a2f9bf2e5d9b4fc19f804797bca4a3ff [term 3 LEADER]: Stepping down as leader of term 3
I20240503 23:26:41.373633 4293 raft_consensus.cc:3037] T 6039230222ff4e57929bed49cd56caf5 P 692da0319a2d4a81ad81f5d8f9f4c86b [term 3 FOLLOWER]: Advancing to term 4
I20240503 23:26:41.373739 3837 raft_consensus.cc:729] T 6039230222ff4e57929bed49cd56caf5 P a2f9bf2e5d9b4fc19f804797bca4a3ff [term 3 LEADER]: Becoming Follower/Learner. State: Replica: a2f9bf2e5d9b4fc19f804797bca4a3ff, State: Running, Role: LEADER
I20240503 23:26:41.374044 3837 consensus_queue.cc:260] T 6039230222ff4e57929bed49cd56caf5 P a2f9bf2e5d9b4fc19f804797bca4a3ff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 30, Committed index: 30, Last appended: 3.30, Last appended by leader: 30, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "a3eecbaec43d4954b1096bc3121e824c" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 40649 } } peers { permanent_uuid: "a2f9bf2e5d9b4fc19f804797bca4a3ff" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 46261 } } peers { permanent_uuid: "692da0319a2d4a81ad81f5d8f9f4c86b" member_type: VOTER last_known_addr { host: "127.0.100.5" port: 38561 } attrs { promote: false } }
I20240503 23:26:41.374447 3837 raft_consensus.cc:3037] T 6039230222ff4e57929bed49cd56caf5 P a2f9bf2e5d9b4fc19f804797bca4a3ff [term 3 FOLLOWER]: Advancing to term 4
I20240503 23:26:41.376848 4293 raft_consensus.cc:2454] T 6039230222ff4e57929bed49cd56caf5 P 692da0319a2d4a81ad81f5d8f9f4c86b [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a3eecbaec43d4954b1096bc3121e824c in term 4.
I20240503 23:26:41.377068 3837 raft_consensus.cc:2454] T 6039230222ff4e57929bed49cd56caf5 P a2f9bf2e5d9b4fc19f804797bca4a3ff [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a3eecbaec43d4954b1096bc3121e824c in term 4.
I20240503 23:26:41.377449 3876 leader_election.cc:302] T 6039230222ff4e57929bed49cd56caf5 P a3eecbaec43d4954b1096bc3121e824c [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 692da0319a2d4a81ad81f5d8f9f4c86b, a2f9bf2e5d9b4fc19f804797bca4a3ff, a3eecbaec43d4954b1096bc3121e824c; no voters:
I20240503 23:26:41.377725 4687 raft_consensus.cc:2781] T 6039230222ff4e57929bed49cd56caf5 P a3eecbaec43d4954b1096bc3121e824c [term 4 FOLLOWER]: Leader election won for term 4
I20240503 23:26:41.378196 4687 raft_consensus.cc:686] T 6039230222ff4e57929bed49cd56caf5 P a3eecbaec43d4954b1096bc3121e824c [term 4 LEADER]: Becoming Leader. State: Replica: a3eecbaec43d4954b1096bc3121e824c, State: Running, Role: LEADER
I20240503 23:26:41.378458 4687 consensus_queue.cc:237] T 6039230222ff4e57929bed49cd56caf5 P a3eecbaec43d4954b1096bc3121e824c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 30, Committed index: 30, Last appended: 3.30, Last appended by leader: 30, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "a3eecbaec43d4954b1096bc3121e824c" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 40649 } } peers { permanent_uuid: "a2f9bf2e5d9b4fc19f804797bca4a3ff" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 46261 } } peers { permanent_uuid: "692da0319a2d4a81ad81f5d8f9f4c86b" member_type: VOTER last_known_addr { host: "127.0.100.5" port: 38561 } attrs { promote: false } }
I20240503 23:26:41.381407 3659 catalog_manager.cc:5484] T 6039230222ff4e57929bed49cd56caf5 P a3eecbaec43d4954b1096bc3121e824c reported cstate change: term changed from 3 to 4, leader changed from a2f9bf2e5d9b4fc19f804797bca4a3ff (127.0.100.2) to a3eecbaec43d4954b1096bc3121e824c (127.0.100.3). New cstate: current_term: 4 leader_uuid: "a3eecbaec43d4954b1096bc3121e824c" committed_config { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "a3eecbaec43d4954b1096bc3121e824c" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 40649 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a2f9bf2e5d9b4fc19f804797bca4a3ff" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 46261 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "692da0319a2d4a81ad81f5d8f9f4c86b" member_type: VOTER last_known_addr { host: "127.0.100.5" port: 38561 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240503 23:26:41.499125 4697 raft_consensus.cc:979] T d0c4430fc56644cb9f87deffe4fe9407 P a2f9bf2e5d9b4fc19f804797bca4a3ff: : Instructing follower 4d1d4e261584467fbbce4c72fa18045a to start an election
I20240503 23:26:41.499336 4697 raft_consensus.cc:1067] T d0c4430fc56644cb9f87deffe4fe9407 P a2f9bf2e5d9b4fc19f804797bca4a3ff [term 5 LEADER]: Signalling peer 4d1d4e261584467fbbce4c72fa18045a to start an election
I20240503 23:26:41.500034 4367 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "d0c4430fc56644cb9f87deffe4fe9407"
dest_uuid: "4d1d4e261584467fbbce4c72fa18045a"
from {username='slave'} at 127.0.0.1:47568
I20240503 23:26:41.500262 4367 raft_consensus.cc:482] T d0c4430fc56644cb9f87deffe4fe9407 P 4d1d4e261584467fbbce4c72fa18045a [term 5 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 23:26:41.500365 4367 raft_consensus.cc:3037] T d0c4430fc56644cb9f87deffe4fe9407 P 4d1d4e261584467fbbce4c72fa18045a [term 5 FOLLOWER]: Advancing to term 6
I20240503 23:26:41.502552 4367 raft_consensus.cc:504] T d0c4430fc56644cb9f87deffe4fe9407 P 4d1d4e261584467fbbce4c72fa18045a [term 6 FOLLOWER]: Starting forced leader election with config: opid_index: 33 OBSOLETE_local: false peers { permanent_uuid: "a2f9bf2e5d9b4fc19f804797bca4a3ff" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 46261 } } peers { permanent_uuid: "a3eecbaec43d4954b1096bc3121e824c" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 40649 } } peers { permanent_uuid: "4d1d4e261584467fbbce4c72fa18045a" member_type: VOTER last_known_addr { host: "127.0.100.6" port: 35349 } attrs { promote: false } }
I20240503 23:26:41.503386 4367 leader_election.cc:288] T d0c4430fc56644cb9f87deffe4fe9407 P 4d1d4e261584467fbbce4c72fa18045a [CANDIDATE]: Term 6 election: Requested vote from peers a2f9bf2e5d9b4fc19f804797bca4a3ff (127.0.100.2:46261), a3eecbaec43d4954b1096bc3121e824c (127.0.100.3:40649)
I20240503 23:26:41.503885 3908 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0c4430fc56644cb9f87deffe4fe9407" candidate_uuid: "4d1d4e261584467fbbce4c72fa18045a" candidate_term: 6 candidate_status { last_received { term: 5 index: 33 } } ignore_live_leader: true dest_uuid: "a3eecbaec43d4954b1096bc3121e824c"
I20240503 23:26:41.503890 3837 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0c4430fc56644cb9f87deffe4fe9407" candidate_uuid: "4d1d4e261584467fbbce4c72fa18045a" candidate_term: 6 candidate_status { last_received { term: 5 index: 33 } } ignore_live_leader: true dest_uuid: "a2f9bf2e5d9b4fc19f804797bca4a3ff"
I20240503 23:26:41.504112 3908 raft_consensus.cc:3037] T d0c4430fc56644cb9f87deffe4fe9407 P a3eecbaec43d4954b1096bc3121e824c [term 5 FOLLOWER]: Advancing to term 6
I20240503 23:26:41.504173 3837 raft_consensus.cc:3032] T d0c4430fc56644cb9f87deffe4fe9407 P a2f9bf2e5d9b4fc19f804797bca4a3ff [term 5 LEADER]: Stepping down as leader of term 5
I20240503 23:26:41.504252 3837 raft_consensus.cc:729] T d0c4430fc56644cb9f87deffe4fe9407 P a2f9bf2e5d9b4fc19f804797bca4a3ff [term 5 LEADER]: Becoming Follower/Learner. State: Replica: a2f9bf2e5d9b4fc19f804797bca4a3ff, State: Running, Role: LEADER
I20240503 23:26:41.504514 3837 consensus_queue.cc:260] T d0c4430fc56644cb9f87deffe4fe9407 P a2f9bf2e5d9b4fc19f804797bca4a3ff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 33, Committed index: 33, Last appended: 5.33, Last appended by leader: 33, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 33 OBSOLETE_local: false peers { permanent_uuid: "a2f9bf2e5d9b4fc19f804797bca4a3ff" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 46261 } } peers { permanent_uuid: "a3eecbaec43d4954b1096bc3121e824c" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 40649 } } peers { permanent_uuid: "4d1d4e261584467fbbce4c72fa18045a" member_type: VOTER last_known_addr { host: "127.0.100.6" port: 35349 } attrs { promote: false } }
I20240503 23:26:41.505172 3837 raft_consensus.cc:3037] T d0c4430fc56644cb9f87deffe4fe9407 P a2f9bf2e5d9b4fc19f804797bca4a3ff [term 5 FOLLOWER]: Advancing to term 6
I20240503 23:26:41.506481 3908 raft_consensus.cc:2454] T d0c4430fc56644cb9f87deffe4fe9407 P a3eecbaec43d4954b1096bc3121e824c [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d1d4e261584467fbbce4c72fa18045a in term 6.
I20240503 23:26:41.507030 4330 leader_election.cc:302] T d0c4430fc56644cb9f87deffe4fe9407 P 4d1d4e261584467fbbce4c72fa18045a [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d1d4e261584467fbbce4c72fa18045a, a3eecbaec43d4954b1096bc3121e824c; no voters:
I20240503 23:26:41.507331 4606 raft_consensus.cc:2781] T d0c4430fc56644cb9f87deffe4fe9407 P 4d1d4e261584467fbbce4c72fa18045a [term 6 FOLLOWER]: Leader election won for term 6
I20240503 23:26:41.507735 3837 raft_consensus.cc:2454] T d0c4430fc56644cb9f87deffe4fe9407 P a2f9bf2e5d9b4fc19f804797bca4a3ff [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d1d4e261584467fbbce4c72fa18045a in term 6.
I20240503 23:26:41.507814 4606 raft_consensus.cc:686] T d0c4430fc56644cb9f87deffe4fe9407 P 4d1d4e261584467fbbce4c72fa18045a [term 6 LEADER]: Becoming Leader. State: Replica: 4d1d4e261584467fbbce4c72fa18045a, State: Running, Role: LEADER
I20240503 23:26:41.508213 4606 consensus_queue.cc:237] T d0c4430fc56644cb9f87deffe4fe9407 P 4d1d4e261584467fbbce4c72fa18045a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 33, Committed index: 33, Last appended: 5.33, Last appended by leader: 33, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 33 OBSOLETE_local: false peers { permanent_uuid: "a2f9bf2e5d9b4fc19f804797bca4a3ff" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 46261 } } peers { permanent_uuid: "a3eecbaec43d4954b1096bc3121e824c" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 40649 } } peers { permanent_uuid: "4d1d4e261584467fbbce4c72fa18045a" member_type: VOTER last_known_addr { host: "127.0.100.6" port: 35349 } attrs { promote: false } }
I20240503 23:26:41.512367 3659 catalog_manager.cc:5484] T d0c4430fc56644cb9f87deffe4fe9407 P 4d1d4e261584467fbbce4c72fa18045a reported cstate change: term changed from 5 to 6, leader changed from a2f9bf2e5d9b4fc19f804797bca4a3ff (127.0.100.2) to 4d1d4e261584467fbbce4c72fa18045a (127.0.100.6). New cstate: current_term: 6 leader_uuid: "4d1d4e261584467fbbce4c72fa18045a" committed_config { opid_index: 33 OBSOLETE_local: false peers { permanent_uuid: "a2f9bf2e5d9b4fc19f804797bca4a3ff" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 46261 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a3eecbaec43d4954b1096bc3121e824c" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 40649 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4d1d4e261584467fbbce4c72fa18045a" member_type: VOTER last_known_addr { host: "127.0.100.6" port: 35349 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 23:26:41.632228 3837 consensus_queue.cc:237] T 059f6233ea514fe69b8c6504efabeff0 P a2f9bf2e5d9b4fc19f804797bca4a3ff [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 27, Committed index: 27, Last appended: 2.27, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "83ffd7184ec94302bd1ee74b0fab3e17" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 32909 } attrs { replace: true } } peers { permanent_uuid: "a2f9bf2e5d9b4fc19f804797bca4a3ff" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 46261 } } peers { permanent_uuid: "a3eecbaec43d4954b1096bc3121e824c" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 40649 } } peers { permanent_uuid: "4d1d4e261584467fbbce4c72fa18045a" member_type: NON_VOTER last_known_addr { host: "127.0.100.6" port: 35349 } attrs { promote: true } }
I20240503 23:26:41.634967 3908 raft_consensus.cc:1261] T 059f6233ea514fe69b8c6504efabeff0 P a3eecbaec43d4954b1096bc3121e824c [term 2 FOLLOWER]: Refusing update from remote peer a2f9bf2e5d9b4fc19f804797bca4a3ff: Log matching property violated. Preceding OpId in replica: term: 2 index: 27. Preceding OpId from leader: term: 2 index: 28. (index mismatch)
I20240503 23:26:41.635001 3760 raft_consensus.cc:1261] T 059f6233ea514fe69b8c6504efabeff0 P 83ffd7184ec94302bd1ee74b0fab3e17 [term 2 FOLLOWER]: Refusing update from remote peer a2f9bf2e5d9b4fc19f804797bca4a3ff: Log matching property violated. Preceding OpId in replica: term: 2 index: 27. Preceding OpId from leader: term: 2 index: 28. (index mismatch)
W20240503 23:26:41.635203 3803 consensus_peers.cc:485] T 059f6233ea514fe69b8c6504efabeff0 P a2f9bf2e5d9b4fc19f804797bca4a3ff -> Peer 4d1d4e261584467fbbce4c72fa18045a (127.0.100.6:35349): Couldn't send request to peer 4d1d4e261584467fbbce4c72fa18045a. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 059f6233ea514fe69b8c6504efabeff0. This is attempt 1: this message will repeat every 5th retry.
I20240503 23:26:41.635602 4707 consensus_queue.cc:1035] T 059f6233ea514fe69b8c6504efabeff0 P a2f9bf2e5d9b4fc19f804797bca4a3ff [LEADER]: Connected to new peer: Peer: permanent_uuid: "a3eecbaec43d4954b1096bc3121e824c" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 40649 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 28, Last known committed idx: 27, Time since last communication: 0.000s
I20240503 23:26:41.635936 4683 consensus_queue.cc:1035] T 059f6233ea514fe69b8c6504efabeff0 P a2f9bf2e5d9b4fc19f804797bca4a3ff [LEADER]: Connected to new peer: Peer: permanent_uuid: "83ffd7184ec94302bd1ee74b0fab3e17" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 32909 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 28, Last known committed idx: 27, Time since last communication: 0.000s
I20240503 23:26:41.639150 4707 raft_consensus.cc:2932] T 059f6233ea514fe69b8c6504efabeff0 P a2f9bf2e5d9b4fc19f804797bca4a3ff [term 2 LEADER]: Committing config change with OpId 2.28: config changed from index -1 to 28, NON_VOTER 4d1d4e261584467fbbce4c72fa18045a (127.0.100.6) added. New config: { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "83ffd7184ec94302bd1ee74b0fab3e17" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 32909 } attrs { replace: true } } peers { permanent_uuid: "a2f9bf2e5d9b4fc19f804797bca4a3ff" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 46261 } } peers { permanent_uuid: "a3eecbaec43d4954b1096bc3121e824c" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 40649 } } peers { permanent_uuid: "4d1d4e261584467fbbce4c72fa18045a" member_type: NON_VOTER last_known_addr { host: "127.0.100.6" port: 35349 } attrs { promote: true } } }
I20240503 23:26:41.639958 3760 raft_consensus.cc:2932] T 059f6233ea514fe69b8c6504efabeff0 P 83ffd7184ec94302bd1ee74b0fab3e17 [term 2 FOLLOWER]: Committing config change with OpId 2.28: config changed from index -1 to 28, NON_VOTER 4d1d4e261584467fbbce4c72fa18045a (127.0.100.6) added. New config: { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "83ffd7184ec94302bd1ee74b0fab3e17" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 32909 } attrs { replace: true } } peers { permanent_uuid: "a2f9bf2e5d9b4fc19f804797bca4a3ff" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 46261 } } peers { permanent_uuid: "a3eecbaec43d4954b1096bc3121e824c" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 40649 } } peers { permanent_uuid: "4d1d4e261584467fbbce4c72fa18045a" member_type: NON_VOTER last_known_addr { host: "127.0.100.6" port: 35349 } attrs { promote: true } } }
I20240503 23:26:41.640085 3908 raft_consensus.cc:2932] T 059f6233ea514fe69b8c6504efabeff0 P a3eecbaec43d4954b1096bc3121e824c [term 2 FOLLOWER]: Committing config change with OpId 2.28: config changed from index -1 to 28, NON_VOTER 4d1d4e261584467fbbce4c72fa18045a (127.0.100.6) added. New config: { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "83ffd7184ec94302bd1ee74b0fab3e17" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 32909 } attrs { replace: true } } peers { permanent_uuid: "a2f9bf2e5d9b4fc19f804797bca4a3ff" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 46261 } } peers { permanent_uuid: "a3eecbaec43d4954b1096bc3121e824c" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 40649 } } peers { permanent_uuid: "4d1d4e261584467fbbce4c72fa18045a" member_type: NON_VOTER last_known_addr { host: "127.0.100.6" port: 35349 } attrs { promote: true } } }
I20240503 23:26:41.645896 3655 catalog_manager.cc:5484] T 059f6233ea514fe69b8c6504efabeff0 P 83ffd7184ec94302bd1ee74b0fab3e17 reported cstate change: config changed from index -1 to 28, NON_VOTER 4d1d4e261584467fbbce4c72fa18045a (127.0.100.6) added. New cstate: current_term: 2 leader_uuid: "a2f9bf2e5d9b4fc19f804797bca4a3ff" committed_config { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "83ffd7184ec94302bd1ee74b0fab3e17" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 32909 } attrs { replace: true } } peers { permanent_uuid: "a2f9bf2e5d9b4fc19f804797bca4a3ff" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 46261 } } peers { permanent_uuid: "a3eecbaec43d4954b1096bc3121e824c" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 40649 } } peers { permanent_uuid: "4d1d4e261584467fbbce4c72fa18045a" member_type: NON_VOTER last_known_addr { host: "127.0.100.6" port: 35349 } attrs { promote: true } } }
I20240503 23:26:41.661370 4293 consensus_queue.cc:237] T a0e18d0975a54e3dbeb0a13cb226fefd P 692da0319a2d4a81ad81f5d8f9f4c86b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 31, Committed index: 31, Last appended: 3.31, Last appended by leader: 30, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "83ffd7184ec94302bd1ee74b0fab3e17" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 32909 } } peers { permanent_uuid: "a2f9bf2e5d9b4fc19f804797bca4a3ff" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 46261 } attrs { replace: true } } peers { permanent_uuid: "692da0319a2d4a81ad81f5d8f9f4c86b" member_type: VOTER last_known_addr { host: "127.0.100.5" port: 38561 } attrs { promote: false } } peers { permanent_uuid: "e4086c099d7f4447b2b4733b6ca364df" member_type: NON_VOTER last_known_addr { host: "127.0.100.4" port: 35011 } attrs { promote: true } }
I20240503 23:26:41.665064 3760 raft_consensus.cc:1261] T a0e18d0975a54e3dbeb0a13cb226fefd P 83ffd7184ec94302bd1ee74b0fab3e17 [term 3 FOLLOWER]: Refusing update from remote peer 692da0319a2d4a81ad81f5d8f9f4c86b: Log matching property violated. Preceding OpId in replica: term: 3 index: 31. Preceding OpId from leader: term: 3 index: 32. (index mismatch)
I20240503 23:26:41.665488 3837 raft_consensus.cc:1261] T a0e18d0975a54e3dbeb0a13cb226fefd P a2f9bf2e5d9b4fc19f804797bca4a3ff [term 3 FOLLOWER]: Refusing update from remote peer 692da0319a2d4a81ad81f5d8f9f4c86b: Log matching property violated. Preceding OpId in replica: term: 3 index: 31. Preceding OpId from leader: term: 3 index: 32. (index mismatch)
I20240503 23:26:41.665879 4714 consensus_queue.cc:1035] T a0e18d0975a54e3dbeb0a13cb226fefd P 692da0319a2d4a81ad81f5d8f9f4c86b [LEADER]: Connected to new peer: Peer: permanent_uuid: "83ffd7184ec94302bd1ee74b0fab3e17" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 32909 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 32, Last known committed idx: 31, Time since last communication: 0.000s
I20240503 23:26:41.666420 4713 consensus_queue.cc:1035] T a0e18d0975a54e3dbeb0a13cb226fefd P 692da0319a2d4a81ad81f5d8f9f4c86b [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2f9bf2e5d9b4fc19f804797bca4a3ff" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 46261 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 32, Last known committed idx: 31, Time since last communication: 0.000s
I20240503 23:26:41.669678 4714 raft_consensus.cc:2932] T a0e18d0975a54e3dbeb0a13cb226fefd P 692da0319a2d4a81ad81f5d8f9f4c86b [term 3 LEADER]: Committing config change with OpId 3.32: config changed from index 30 to 32, NON_VOTER e4086c099d7f4447b2b4733b6ca364df (127.0.100.4) added. New config: { opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "83ffd7184ec94302bd1ee74b0fab3e17" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 32909 } } peers { permanent_uuid: "a2f9bf2e5d9b4fc19f804797bca4a3ff" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 46261 } attrs { replace: true } } peers { permanent_uuid: "692da0319a2d4a81ad81f5d8f9f4c86b" member_type: VOTER last_known_addr { host: "127.0.100.5" port: 38561 } attrs { promote: false } } peers { permanent_uuid: "e4086c099d7f4447b2b4733b6ca364df" member_type: NON_VOTER last_known_addr { host: "127.0.100.4" port: 35011 } attrs { promote: true } } }
I20240503 23:26:41.670644 3760 raft_consensus.cc:2932] T a0e18d0975a54e3dbeb0a13cb226fefd P 83ffd7184ec94302bd1ee74b0fab3e17 [term 3 FOLLOWER]: Committing config change with OpId 3.32: config changed from index 30 to 32, NON_VOTER e4086c099d7f4447b2b4733b6ca364df (127.0.100.4) added. New config: { opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "83ffd7184ec94302bd1ee74b0fab3e17" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 32909 } } peers { permanent_uuid: "a2f9bf2e5d9b4fc19f804797bca4a3ff" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 46261 } attrs { replace: true } } peers { permanent_uuid: "692da0319a2d4a81ad81f5d8f9f4c86b" member_type: VOTER last_known_addr { host: "127.0.100.5" port: 38561 } attrs { promote: false } } peers { permanent_uuid: "e4086c099d7f4447b2b4733b6ca364df" member_type: NON_VOTER last_known_addr { host: "127.0.100.4" port: 35011 } attrs { promote: true } } }
W20240503 23:26:41.672847 4256 consensus_peers.cc:485] T a0e18d0975a54e3dbeb0a13cb226fefd P 692da0319a2d4a81ad81f5d8f9f4c86b -> Peer e4086c099d7f4447b2b4733b6ca364df (127.0.100.4:35011): Couldn't send request to peer e4086c099d7f4447b2b4733b6ca364df. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: a0e18d0975a54e3dbeb0a13cb226fefd. This is attempt 1: this message will repeat every 5th retry.
I20240503 23:26:41.672856 3837 raft_consensus.cc:2932] T a0e18d0975a54e3dbeb0a13cb226fefd P a2f9bf2e5d9b4fc19f804797bca4a3ff [term 3 FOLLOWER]: Committing config change with OpId 3.32: config changed from index 30 to 32, NON_VOTER e4086c099d7f4447b2b4733b6ca364df (127.0.100.4) added. New config: { opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "83ffd7184ec94302bd1ee74b0fab3e17" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 32909 } } peers { permanent_uuid: "a2f9bf2e5d9b4fc19f804797bca4a3ff" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 46261 } attrs { replace: true } } peers { permanent_uuid: "692da0319a2d4a81ad81f5d8f9f4c86b" member_type: VOTER last_known_addr { host: "127.0.100.5" port: 38561 } attrs { promote: false } } peers { permanent_uuid: "e4086c099d7f4447b2b4733b6ca364df" member_type: NON_VOTER last_known_addr { host: "127.0.100.4" port: 35011 } attrs { promote: true } } }
I20240503 23:26:41.676646 3662 catalog_manager.cc:5484] T a0e18d0975a54e3dbeb0a13cb226fefd P 83ffd7184ec94302bd1ee74b0fab3e17 reported cstate change: config changed from index 30 to 32, NON_VOTER e4086c099d7f4447b2b4733b6ca364df (127.0.100.4) added. New cstate: current_term: 3 leader_uuid: "692da0319a2d4a81ad81f5d8f9f4c86b" committed_config { opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "83ffd7184ec94302bd1ee74b0fab3e17" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 32909 } } peers { permanent_uuid: "a2f9bf2e5d9b4fc19f804797bca4a3ff" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 46261 } attrs { replace: true } } peers { permanent_uuid: "692da0319a2d4a81ad81f5d8f9f4c86b" member_type: VOTER last_known_addr { host: "127.0.100.5" port: 38561 } attrs { promote: false } } peers { permanent_uuid: "e4086c099d7f4447b2b4733b6ca364df" member_type: NON_VOTER last_known_addr { host: "127.0.100.4" port: 35011 } attrs { promote: true } } }
I20240503 23:26:41.678900 3760 consensus_queue.cc:237] T 481cd0f8c8d147adac217915a53ff6d9 P 83ffd7184ec94302bd1ee74b0fab3e17 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 27, Committed index: 27, Last appended: 2.27, Last appended by leader: 26, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "83ffd7184ec94302bd1ee74b0fab3e17" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 32909 } } peers { permanent_uuid: "a2f9bf2e5d9b4fc19f804797bca4a3ff" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 46261 } } peers { permanent_uuid: "a3eecbaec43d4954b1096bc3121e824c" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 40649 } attrs { replace: true } } peers { permanent_uuid: "692da0319a2d4a81ad81f5d8f9f4c86b" member_type: NON_VOTER last_known_addr { host: "127.0.100.5" port: 38561 } attrs { promote: true } }
I20240503 23:26:41.682817 3837 raft_consensus.cc:1261] T 481cd0f8c8d147adac217915a53ff6d9 P a2f9bf2e5d9b4fc19f804797bca4a3ff [term 2 FOLLOWER]: Refusing update from remote peer 83ffd7184ec94302bd1ee74b0fab3e17: Log matching property violated. Preceding OpId in replica: term: 2 index: 27. Preceding OpId from leader: term: 2 index: 28. (index mismatch)
W20240503 23:26:41.683214 3727 consensus_peers.cc:485] T 481cd0f8c8d147adac217915a53ff6d9 P 83ffd7184ec94302bd1ee74b0fab3e17 -> Peer 692da0319a2d4a81ad81f5d8f9f4c86b (127.0.100.5:38561): Couldn't send request to peer 692da0319a2d4a81ad81f5d8f9f4c86b. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 481cd0f8c8d147adac217915a53ff6d9. This is attempt 1: this message will repeat every 5th retry.
I20240503 23:26:41.683539 4613 consensus_queue.cc:1035] T 481cd0f8c8d147adac217915a53ff6d9 P 83ffd7184ec94302bd1ee74b0fab3e17 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2f9bf2e5d9b4fc19f804797bca4a3ff" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 46261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 28, Last known committed idx: 27, Time since last communication: 0.000s
I20240503 23:26:41.684916 3908 raft_consensus.cc:1261] T 481cd0f8c8d147adac217915a53ff6d9 P a3eecbaec43d4954b1096bc3121e824c [term 2 FOLLOWER]: Refusing update from remote peer 83ffd7184ec94302bd1ee74b0fab3e17: Log matching property violated. Preceding OpId in replica: term: 2 index: 27. Preceding OpId from leader: term: 2 index: 28. (index mismatch)
I20240503 23:26:41.685498 4720 consensus_queue.cc:1035] T 481cd0f8c8d147adac217915a53ff6d9 P 83ffd7184ec94302bd1ee74b0fab3e17 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a3eecbaec43d4954b1096bc3121e824c" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 40649 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 28, Last known committed idx: 27, Time since last communication: 0.000s
I20240503 23:26:41.686816 4613 raft_consensus.cc:2932] T 481cd0f8c8d147adac217915a53ff6d9 P 83ffd7184ec94302bd1ee74b0fab3e17 [term 2 LEADER]: Committing config change with OpId 2.28: config changed from index -1 to 28, NON_VOTER 692da0319a2d4a81ad81f5d8f9f4c86b (127.0.100.5) added. New config: { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "83ffd7184ec94302bd1ee74b0fab3e17" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 32909 } } peers { permanent_uuid: "a2f9bf2e5d9b4fc19f804797bca4a3ff" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 46261 } } peers { permanent_uuid: "a3eecbaec43d4954b1096bc3121e824c" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 40649 } attrs { replace: true } } peers { permanent_uuid: "692da0319a2d4a81ad81f5d8f9f4c86b" member_type: NON_VOTER last_known_addr { host: "127.0.100.5" port: 38561 } attrs { promote: true } } }
I20240503 23:26:41.687534 3837 raft_consensus.cc:2932] T 481cd0f8c8d147adac217915a53ff6d9 P a2f9bf2e5d9b4fc19f804797bca4a3ff [term 2 FOLLOWER]: Committing config change with OpId 2.28: config changed from index -1 to 28, NON_VOTER 692da0319a2d4a81ad81f5d8f9f4c86b (127.0.100.5) added. New config: { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "83ffd7184ec94302bd1ee74b0fab3e17" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 32909 } } peers { permanent_uuid: "a2f9bf2e5d9b4fc19f804797bca4a3ff" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 46261 } } peers { permanent_uuid: "a3eecbaec43d4954b1096bc3121e824c" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 40649 } attrs { replace: true } } peers { permanent_uuid: "692da0319a2d4a81ad81f5d8f9f4c86b" member_type: NON_VOTER last_known_addr { host: "127.0.100.5" port: 38561 } attrs { promote: true } } }
I20240503 23:26:41.689302 3908 raft_consensus.cc:2932] T 481cd0f8c8d147adac217915a53ff6d9 P a3eecbaec43d4954b1096bc3121e824c [term 2 FOLLOWER]: Committing config change with OpId 2.28: config changed from index -1 to 28, NON_VOTER 692da0319a2d4a81ad81f5d8f9f4c86b (127.0.100.5) added. New config: { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "83ffd7184ec94302bd1ee74b0fab3e17" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 32909 } } peers { permanent_uuid: "a2f9bf2e5d9b4fc19f804797bca4a3ff" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 46261 } } peers { permanent_uuid: "a3eecbaec43d4954b1096bc3121e824c" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 40649 } attrs { replace: true } } peers { permanent_uuid: "692da0319a2d4a81ad81f5d8f9f4c86b" member_type: NON_VOTER last_known_addr { host: "127.0.100.5" port: 38561 } attrs { promote: true } } }
I20240503 23:26:41.694478 3659 catalog_manager.cc:5484] T 481cd0f8c8d147adac217915a53ff6d9 P a2f9bf2e5d9b4fc19f804797bca4a3ff reported cstate change: config changed from index -1 to 28, NON_VOTER 692da0319a2d4a81ad81f5d8f9f4c86b (127.0.100.5) added. New cstate: current_term: 2 leader_uuid: "83ffd7184ec94302bd1ee74b0fab3e17" committed_config { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "83ffd7184ec94302bd1ee74b0fab3e17" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 32909 } } peers { permanent_uuid: "a2f9bf2e5d9b4fc19f804797bca4a3ff" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 46261 } } peers { permanent_uuid: "a3eecbaec43d4954b1096bc3121e824c" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 40649 } attrs { replace: true } } peers { permanent_uuid: "692da0319a2d4a81ad81f5d8f9f4c86b" member_type: NON_VOTER last_known_addr { host: "127.0.100.5" port: 38561 } attrs { promote: true } } }
I20240503 23:26:41.694703 4217 consensus_queue.cc:237] T 55cf664d5e874c0d9b9adf8bdc9c43ed P e4086c099d7f4447b2b4733b6ca364df [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 33, Committed index: 33, Last appended: 2.33, Last appended by leader: 30, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 34 OBSOLETE_local: false peers { permanent_uuid: "a2f9bf2e5d9b4fc19f804797bca4a3ff" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 46261 } attrs { replace: true } } peers { permanent_uuid: "e4086c099d7f4447b2b4733b6ca364df" member_type: VOTER last_known_addr { host: "127.0.100.4" port: 35011 } attrs { promote: false } } peers { permanent_uuid: "4d1d4e261584467fbbce4c72fa18045a" member_type: VOTER last_known_addr { host: "127.0.100.6" port: 35349 } attrs { promote: false } } peers { permanent_uuid: "692da0319a2d4a81ad81f5d8f9f4c86b" member_type: NON_VOTER last_known_addr { host: "127.0.100.5" port: 38561 } attrs { promote: true } }
I20240503 23:26:41.705000 4367 raft_consensus.cc:1261] T 55cf664d5e874c0d9b9adf8bdc9c43ed P 4d1d4e261584467fbbce4c72fa18045a [term 2 FOLLOWER]: Refusing update from remote peer e4086c099d7f4447b2b4733b6ca364df: Log matching property violated. Preceding OpId in replica: term: 2 index: 33. Preceding OpId from leader: term: 2 index: 34. (index mismatch)
I20240503 23:26:41.705974 4696 consensus_queue.cc:1035] T 55cf664d5e874c0d9b9adf8bdc9c43ed P e4086c099d7f4447b2b4733b6ca364df [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d1d4e261584467fbbce4c72fa18045a" member_type: VOTER last_known_addr { host: "127.0.100.6" port: 35349 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 34, Last known committed idx: 33, Time since last communication: 0.000s
I20240503 23:26:41.709729 3837 raft_consensus.cc:1261] T 55cf664d5e874c0d9b9adf8bdc9c43ed P a2f9bf2e5d9b4fc19f804797bca4a3ff [term 2 FOLLOWER]: Refusing update from remote peer e4086c099d7f4447b2b4733b6ca364df: Log matching property violated. Preceding OpId in replica: term: 2 index: 33. Preceding OpId from leader: term: 2 index: 34. (index mismatch)
I20240503 23:26:41.710459 4696 consensus_queue.cc:1035] T 55cf664d5e874c0d9b9adf8bdc9c43ed P e4086c099d7f4447b2b4733b6ca364df [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2f9bf2e5d9b4fc19f804797bca4a3ff" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 46261 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 34, Last known committed idx: 33, Time since last communication: 0.000s
I20240503 23:26:41.715621 4696 raft_consensus.cc:2932] T 55cf664d5e874c0d9b9adf8bdc9c43ed P e4086c099d7f4447b2b4733b6ca364df [term 2 LEADER]: Committing config change with OpId 2.34: config changed from index 33 to 34, NON_VOTER 692da0319a2d4a81ad81f5d8f9f4c86b (127.0.100.5) added. New config: { opid_index: 34 OBSOLETE_local: false peers { permanent_uuid: "a2f9bf2e5d9b4fc19f804797bca4a3ff" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 46261 } attrs { replace: true } } peers { permanent_uuid: "e4086c099d7f4447b2b4733b6ca364df" member_type: VOTER last_known_addr { host: "127.0.100.4" port: 35011 } attrs { promote: false } } peers { permanent_uuid: "4d1d4e261584467fbbce4c72fa18045a" member_type: VOTER last_known_addr { host: "127.0.100.6" port: 35349 } attrs { promote: false } } peers { permanent_uuid: "692da0319a2d4a81ad81f5d8f9f4c86b" member_type: NON_VOTER last_known_addr { host: "127.0.100.5" port: 38561 } attrs { promote: true } } }
I20240503 23:26:41.721678 3837 raft_consensus.cc:2932] T 55cf664d5e874c0d9b9adf8bdc9c43ed P a2f9bf2e5d9b4fc19f804797bca4a3ff [term 2 FOLLOWER]: Committing config change with OpId 2.34: config changed from index 33 to 34, NON_VOTER 692da0319a2d4a81ad81f5d8f9f4c86b (127.0.100.5) added. New config: { opid_index: 34 OBSOLETE_local: false peers { permanent_uuid: "a2f9bf2e5d9b4fc19f804797bca4a3ff" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 46261 } attrs { replace: true } } peers { permanent_uuid: "e4086c099d7f4447b2b4733b6ca364df" member_type: VOTER last_known_addr { host: "127.0.100.4" port: 35011 } attrs { promote: false } } peers { permanent_uuid: "4d1d4e261584467fbbce4c72fa18045a" member_type: VOTER last_known_addr { host: "127.0.100.6" port: 35349 } attrs { promote: false } } peers { permanent_uuid: "692da0319a2d4a81ad81f5d8f9f4c86b" member_type: NON_VOTER last_known_addr { host: "127.0.100.5" port: 38561 } attrs { promote: true } } }
I20240503 23:26:41.722071 3908 consensus_queue.cc:237] T 60267cb4849d4823a0afb2380305905f P a3eecbaec43d4954b1096bc3121e824c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 28, Committed index: 28, Last appended: 1.28, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "83ffd7184ec94302bd1ee74b0fab3e17" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 32909 } attrs { replace: true } } peers { permanent_uuid: "a3eecbaec43d4954b1096bc3121e824c" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 40649 } } peers { permanent_uuid: "e4086c099d7f4447b2b4733b6ca364df" member_type: VOTER last_known_addr { host: "127.0.100.4" port: 35011 } attrs { promote: false } } peers { permanent_uuid: "4d1d4e261584467fbbce4c72fa18045a" member_type: NON_VOTER last_known_addr { host: "127.0.100.6" port: 35349 } attrs { promote: true } }
I20240503 23:26:41.723734 4367 raft_consensus.cc:2932] T 55cf664d5e874c0d9b9adf8bdc9c43ed P 4d1d4e261584467fbbce4c72fa18045a [term 2 FOLLOWER]: Committing config change with OpId 2.34: config changed from index 33 to 34, NON_VOTER 692da0319a2d4a81ad81f5d8f9f4c86b (127.0.100.5) added. New config: { opid_index: 34 OBSOLETE_local: false peers { permanent_uuid: "a2f9bf2e5d9b4fc19f804797bca4a3ff" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 46261 } attrs { replace: true } } peers { permanent_uuid: "e4086c099d7f4447b2b4733b6ca364df" member_type: VOTER last_known_addr { host: "127.0.100.4" port: 35011 } attrs { promote: false } } peers { permanent_uuid: "4d1d4e261584467fbbce4c72fa18045a" member_type: VOTER last_known_addr { host: "127.0.100.6" port: 35349 } attrs { promote: false } } peers { permanent_uuid: "692da0319a2d4a81ad81f5d8f9f4c86b" member_type: NON_VOTER last_known_addr { host: "127.0.100.5" port: 38561 } attrs { promote: true } } }
W20240503 23:26:41.724676 4181 consensus_peers.cc:485] T 55cf664d5e874c0d9b9adf8bdc9c43ed P e4086c099d7f4447b2b4733b6ca364df -> Peer 692da0319a2d4a81ad81f5d8f9f4c86b (127.0.100.5:38561): Couldn't send request to peer 692da0319a2d4a81ad81f5d8f9f4c86b. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 55cf664d5e874c0d9b9adf8bdc9c43ed. This is attempt 1: this message will repeat every 5th retry.
I20240503 23:26:41.726505 4217 raft_consensus.cc:1261] T 60267cb4849d4823a0afb2380305905f P e4086c099d7f4447b2b4733b6ca364df [term 1 FOLLOWER]: Refusing update from remote peer a3eecbaec43d4954b1096bc3121e824c: Log matching property violated. Preceding OpId in replica: term: 1 index: 28. Preceding OpId from leader: term: 1 index: 29. (index mismatch)
I20240503 23:26:41.727149 3659 catalog_manager.cc:5484] T 55cf664d5e874c0d9b9adf8bdc9c43ed P e4086c099d7f4447b2b4733b6ca364df reported cstate change: config changed from index 33 to 34, NON_VOTER 692da0319a2d4a81ad81f5d8f9f4c86b (127.0.100.5) added. New cstate: current_term: 2 leader_uuid: "e4086c099d7f4447b2b4733b6ca364df" committed_config { opid_index: 34 OBSOLETE_local: false peers { permanent_uuid: "a2f9bf2e5d9b4fc19f804797bca4a3ff" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 46261 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e4086c099d7f4447b2b4733b6ca364df" member_type: VOTER last_known_addr { host: "127.0.100.4" port: 35011 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4d1d4e261584467fbbce4c72fa18045a" member_type: VOTER last_known_addr { host: "127.0.100.6" port: 35349 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "692da0319a2d4a81ad81f5d8f9f4c86b" member_type: NON_VOTER last_known_addr { host: "127.0.100.5" port: 38561 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 23:26:41.727902 4698 consensus_queue.cc:1035] T 60267cb4849d4823a0afb2380305905f P a3eecbaec43d4954b1096bc3121e824c [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4086c099d7f4447b2b4733b6ca364df" member_type: VOTER last_known_addr { host: "127.0.100.4" port: 35011 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.000s
I20240503 23:26:41.727185 3760 raft_consensus.cc:1261] T 60267cb4849d4823a0afb2380305905f P 83ffd7184ec94302bd1ee74b0fab3e17 [term 1 FOLLOWER]: Refusing update from remote peer a3eecbaec43d4954b1096bc3121e824c: Log matching property violated. Preceding OpId in replica: term: 1 index: 28. Preceding OpId from leader: term: 1 index: 29. (index mismatch)
W20240503 23:26:41.729686 3877 consensus_peers.cc:485] T 60267cb4849d4823a0afb2380305905f P a3eecbaec43d4954b1096bc3121e824c -> Peer 4d1d4e261584467fbbce4c72fa18045a (127.0.100.6:35349): Couldn't send request to peer 4d1d4e261584467fbbce4c72fa18045a. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 60267cb4849d4823a0afb2380305905f. This is attempt 1: this message will repeat every 5th retry.
I20240503 23:26:41.731305 4698 consensus_queue.cc:1035] T 60267cb4849d4823a0afb2380305905f P a3eecbaec43d4954b1096bc3121e824c [LEADER]: Connected to new peer: Peer: permanent_uuid: "83ffd7184ec94302bd1ee74b0fab3e17" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 32909 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.000s
I20240503 23:26:41.752883 4687 raft_consensus.cc:2932] T 60267cb4849d4823a0afb2380305905f P a3eecbaec43d4954b1096bc3121e824c [term 1 LEADER]: Committing config change with OpId 1.29: config changed from index 28 to 29, NON_VOTER 4d1d4e261584467fbbce4c72fa18045a (127.0.100.6) added. New config: { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "83ffd7184ec94302bd1ee74b0fab3e17" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 32909 } attrs { replace: true } } peers { permanent_uuid: "a3eecbaec43d4954b1096bc3121e824c" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 40649 } } peers { permanent_uuid: "e4086c099d7f4447b2b4733b6ca364df" member_type: VOTER last_known_addr { host: "127.0.100.4" port: 35011 } attrs { promote: false } } peers { permanent_uuid: "4d1d4e261584467fbbce4c72fa18045a" member_type: NON_VOTER last_known_addr { host: "127.0.100.6" port: 35349 } attrs { promote: true } } }
I20240503 23:26:41.753894 3760 raft_consensus.cc:2932] T 60267cb4849d4823a0afb2380305905f P 83ffd7184ec94302bd1ee74b0fab3e17 [term 1 FOLLOWER]: Committing config change with OpId 1.29: config changed from index 28 to 29, NON_VOTER 4d1d4e261584467fbbce4c72fa18045a (127.0.100.6) added. New config: { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "83ffd7184ec94302bd1ee74b0fab3e17" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 32909 } attrs { replace: true } } peers { permanent_uuid: "a3eecbaec43d4954b1096bc3121e824c" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 40649 } } peers { permanent_uuid: "e4086c099d7f4447b2b4733b6ca364df" member_type: VOTER last_known_addr { host: "127.0.100.4" port: 35011 } attrs { promote: false } } peers { permanent_uuid: "4d1d4e261584467fbbce4c72fa18045a" member_type: NON_VOTER last_known_addr { host: "127.0.100.6" port: 35349 } attrs { promote: true } } }
I20240503 23:26:41.758497 4217 raft_consensus.cc:2932] T 60267cb4849d4823a0afb2380305905f P e4086c099d7f4447b2b4733b6ca364df [term 1 FOLLOWER]: Committing config change with OpId 1.29: config changed from index 28 to 29, NON_VOTER 4d1d4e261584467fbbce4c72fa18045a (127.0.100.6) added. New config: { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "83ffd7184ec94302bd1ee74b0fab3e17" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 32909 } attrs { replace: true } } peers { permanent_uuid: "a3eecbaec43d4954b1096bc3121e824c" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 40649 } } peers { permanent_uuid: "e4086c099d7f4447b2b4733b6ca364df" member_type: VOTER last_known_addr { host: "127.0.100.4" port: 35011 } attrs { promote: false } } peers { permanent_uuid: "4d1d4e261584467fbbce4c72fa18045a" member_type: NON_VOTER last_known_addr { host: "127.0.100.6" port: 35349 } attrs { promote: true } } }
I20240503 23:26:41.760933 3659 catalog_manager.cc:5484] T 60267cb4849d4823a0afb2380305905f P a3eecbaec43d4954b1096bc3121e824c reported cstate change: config changed from index 28 to 29, NON_VOTER 4d1d4e261584467fbbce4c72fa18045a (127.0.100.6) added. New cstate: current_term: 1 leader_uuid: "a3eecbaec43d4954b1096bc3121e824c" committed_config { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "83ffd7184ec94302bd1ee74b0fab3e17" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 32909 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a3eecbaec43d4954b1096bc3121e824c" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 40649 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e4086c099d7f4447b2b4733b6ca364df" member_type: VOTER last_known_addr { host: "127.0.100.4" port: 35011 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4d1d4e261584467fbbce4c72fa18045a" member_type: NON_VOTER last_known_addr { host: "127.0.100.6" port: 35349 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 23:26:41.779297 4367 consensus_queue.cc:237] T a14a12115b8d4fec947c6e223f1ef486 P 4d1d4e261584467fbbce4c72fa18045a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 30, Committed index: 30, Last appended: 2.30, Last appended by leader: 29, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "83ffd7184ec94302bd1ee74b0fab3e17" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 32909 } } peers { permanent_uuid: "a3eecbaec43d4954b1096bc3121e824c" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 40649 } attrs { replace: true } } peers { permanent_uuid: "4d1d4e261584467fbbce4c72fa18045a" member_type: VOTER last_known_addr { host: "127.0.100.6" port: 35349 } attrs { promote: false } } peers { permanent_uuid: "e4086c099d7f4447b2b4733b6ca364df" member_type: NON_VOTER last_known_addr { host: "127.0.100.4" port: 35011 } attrs { promote: true } }
I20240503 23:26:41.782433 3908 raft_consensus.cc:1261] T a14a12115b8d4fec947c6e223f1ef486 P a3eecbaec43d4954b1096bc3121e824c [term 2 FOLLOWER]: Refusing update from remote peer 4d1d4e261584467fbbce4c72fa18045a: Log matching property violated. Preceding OpId in replica: term: 2 index: 30. Preceding OpId from leader: term: 2 index: 31. (index mismatch)
W20240503 23:26:41.782582 4330 consensus_peers.cc:485] T a14a12115b8d4fec947c6e223f1ef486 P 4d1d4e261584467fbbce4c72fa18045a -> Peer e4086c099d7f4447b2b4733b6ca364df (127.0.100.4:35011): Couldn't send request to peer e4086c099d7f4447b2b4733b6ca364df. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: a14a12115b8d4fec947c6e223f1ef486. This is attempt 1: this message will repeat every 5th retry.
I20240503 23:26:41.782447 3760 raft_consensus.cc:1261] T a14a12115b8d4fec947c6e223f1ef486 P 83ffd7184ec94302bd1ee74b0fab3e17 [term 2 FOLLOWER]: Refusing update from remote peer 4d1d4e261584467fbbce4c72fa18045a: Log matching property violated. Preceding OpId in replica: term: 2 index: 30. Preceding OpId from leader: term: 2 index: 31. (index mismatch)
I20240503 23:26:41.783113 4704 consensus_queue.cc:1035] T a14a12115b8d4fec947c6e223f1ef486 P 4d1d4e261584467fbbce4c72fa18045a [LEADER]: Connected to new peer: Peer: permanent_uuid: "a3eecbaec43d4954b1096bc3121e824c" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 40649 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 31, Last known committed idx: 30, Time since last communication: 0.000s
I20240503 23:26:41.783468 4606 consensus_queue.cc:1035] T a14a12115b8d4fec947c6e223f1ef486 P 4d1d4e261584467fbbce4c72fa18045a [LEADER]: Connected to new peer: Peer: permanent_uuid: "83ffd7184ec94302bd1ee74b0fab3e17" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 32909 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 31, Last known committed idx: 30, Time since last communication: 0.000s
I20240503 23:26:41.787784 4704 raft_consensus.cc:2932] T a14a12115b8d4fec947c6e223f1ef486 P 4d1d4e261584467fbbce4c72fa18045a [term 2 LEADER]: Committing config change with OpId 2.31: config changed from index 29 to 31, NON_VOTER e4086c099d7f4447b2b4733b6ca364df (127.0.100.4) added. New config: { opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "83ffd7184ec94302bd1ee74b0fab3e17" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 32909 } } peers { permanent_uuid: "a3eecbaec43d4954b1096bc3121e824c" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 40649 } attrs { replace: true } } peers { permanent_uuid: "4d1d4e261584467fbbce4c72fa18045a" member_type: VOTER last_known_addr { host: "127.0.100.6" port: 35349 } attrs { promote: false } } peers { permanent_uuid: "e4086c099d7f4447b2b4733b6ca364df" member_type: NON_VOTER last_known_addr { host: "127.0.100.4" port: 35011 } attrs { promote: true } } }
I20240503 23:26:41.788821 3760 raft_consensus.cc:2932] T a14a12115b8d4fec947c6e223f1ef486 P 83ffd7184ec94302bd1ee74b0fab3e17 [term 2 FOLLOWER]: Committing config change with OpId 2.31: config changed from index 29 to 31, NON_VOTER e4086c099d7f4447b2b4733b6ca364df (127.0.100.4) added. New config: { opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "83ffd7184ec94302bd1ee74b0fab3e17" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 32909 } } peers { permanent_uuid: "a3eecbaec43d4954b1096bc3121e824c" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 40649 } attrs { replace: true } } peers { permanent_uuid: "4d1d4e261584467fbbce4c72fa18045a" member_type: VOTER last_known_addr { host: "127.0.100.6" port: 35349 } attrs { promote: false } } peers { permanent_uuid: "e4086c099d7f4447b2b4733b6ca364df" member_type: NON_VOTER last_known_addr { host: "127.0.100.4" port: 35011 } attrs { promote: true } } }
I20240503 23:26:41.788772 3908 raft_consensus.cc:2932] T a14a12115b8d4fec947c6e223f1ef486 P a3eecbaec43d4954b1096bc3121e824c [term 2 FOLLOWER]: Committing config change with OpId 2.31: config changed from index 29 to 31, NON_VOTER e4086c099d7f4447b2b4733b6ca364df (127.0.100.4) added. New config: { opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "83ffd7184ec94302bd1ee74b0fab3e17" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 32909 } } peers { permanent_uuid: "a3eecbaec43d4954b1096bc3121e824c" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 40649 } attrs { replace: true } } peers { permanent_uuid: "4d1d4e261584467fbbce4c72fa18045a" member_type: VOTER last_known_addr { host: "127.0.100.6" port: 35349 } attrs { promote: false } } peers { permanent_uuid: "e4086c099d7f4447b2b4733b6ca364df" member_type: NON_VOTER last_known_addr { host: "127.0.100.4" port: 35011 } attrs { promote: true } } }
I20240503 23:26:41.794068 3659 catalog_manager.cc:5484] T a14a12115b8d4fec947c6e223f1ef486 P 4d1d4e261584467fbbce4c72fa18045a reported cstate change: config changed from index 29 to 31, NON_VOTER e4086c099d7f4447b2b4733b6ca364df (127.0.100.4) added. New cstate: current_term: 2 leader_uuid: "4d1d4e261584467fbbce4c72fa18045a" committed_config { opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "83ffd7184ec94302bd1ee74b0fab3e17" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 32909 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a3eecbaec43d4954b1096bc3121e824c" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 40649 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4d1d4e261584467fbbce4c72fa18045a" member_type: VOTER last_known_addr { host: "127.0.100.6" port: 35349 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e4086c099d7f4447b2b4733b6ca364df" member_type: NON_VOTER last_known_addr { host: "127.0.100.4" port: 35011 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 23:26:41.871373 3837 raft_consensus.cc:1261] T 6039230222ff4e57929bed49cd56caf5 P a2f9bf2e5d9b4fc19f804797bca4a3ff [term 4 FOLLOWER]: Refusing update from remote peer a3eecbaec43d4954b1096bc3121e824c: Log matching property violated. Preceding OpId in replica: term: 3 index: 30. Preceding OpId from leader: term: 4 index: 31. (index mismatch)
I20240503 23:26:41.872000 4687 consensus_queue.cc:1035] T 6039230222ff4e57929bed49cd56caf5 P a3eecbaec43d4954b1096bc3121e824c [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2f9bf2e5d9b4fc19f804797bca4a3ff" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 46261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 31, Last known committed idx: 30, Time since last communication: 0.000s
I20240503 23:26:41.892134 4293 raft_consensus.cc:1261] T 6039230222ff4e57929bed49cd56caf5 P 692da0319a2d4a81ad81f5d8f9f4c86b [term 4 FOLLOWER]: Refusing update from remote peer a3eecbaec43d4954b1096bc3121e824c: Log matching property violated. Preceding OpId in replica: term: 3 index: 30. Preceding OpId from leader: term: 4 index: 31. (index mismatch)
I20240503 23:26:41.893191 4698 consensus_queue.cc:1035] T 6039230222ff4e57929bed49cd56caf5 P a3eecbaec43d4954b1096bc3121e824c [LEADER]: Connected to new peer: Peer: permanent_uuid: "692da0319a2d4a81ad81f5d8f9f4c86b" member_type: VOTER last_known_addr { host: "127.0.100.5" port: 38561 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 31, Last known committed idx: 30, Time since last communication: 0.000s
I20240503 23:26:42.049322 3907 raft_consensus.cc:1261] T d0c4430fc56644cb9f87deffe4fe9407 P a3eecbaec43d4954b1096bc3121e824c [term 6 FOLLOWER]: Refusing update from remote peer 4d1d4e261584467fbbce4c72fa18045a: Log matching property violated. Preceding OpId in replica: term: 5 index: 33. Preceding OpId from leader: term: 6 index: 34. (index mismatch)
I20240503 23:26:42.050067 4738 consensus_queue.cc:1035] T d0c4430fc56644cb9f87deffe4fe9407 P 4d1d4e261584467fbbce4c72fa18045a [LEADER]: Connected to new peer: Peer: permanent_uuid: "a3eecbaec43d4954b1096bc3121e824c" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 40649 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 34, Last known committed idx: 33, Time since last communication: 0.000s
I20240503 23:26:42.055544 3838 raft_consensus.cc:1261] T d0c4430fc56644cb9f87deffe4fe9407 P a2f9bf2e5d9b4fc19f804797bca4a3ff [term 6 FOLLOWER]: Refusing update from remote peer 4d1d4e261584467fbbce4c72fa18045a: Log matching property violated. Preceding OpId in replica: term: 5 index: 33. Preceding OpId from leader: term: 6 index: 34. (index mismatch)
I20240503 23:26:42.056134 4738 consensus_queue.cc:1035] T d0c4430fc56644cb9f87deffe4fe9407 P 4d1d4e261584467fbbce4c72fa18045a [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2f9bf2e5d9b4fc19f804797bca4a3ff" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 46261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 34, Last known committed idx: 33, Time since last communication: 0.000s
I20240503 23:26:42.095271 4749 ts_tablet_manager.cc:902] T a0e18d0975a54e3dbeb0a13cb226fefd P e4086c099d7f4447b2b4733b6ca364df: Initiating tablet copy from peer 692da0319a2d4a81ad81f5d8f9f4c86b (127.0.100.5:38561)
I20240503 23:26:42.096246 4749 tablet_copy_client.cc:316] T a0e18d0975a54e3dbeb0a13cb226fefd P e4086c099d7f4447b2b4733b6ca364df: tablet copy: Beginning tablet copy session from remote peer at address 127.0.100.5:38561
I20240503 23:26:42.096983 4303 tablet_copy_service.cc:140] P 692da0319a2d4a81ad81f5d8f9f4c86b: Received BeginTabletCopySession request for tablet a0e18d0975a54e3dbeb0a13cb226fefd from peer e4086c099d7f4447b2b4733b6ca364df ({username='slave'} at 127.0.0.1:46128)
I20240503 23:26:42.097234 4303 tablet_copy_service.cc:161] P 692da0319a2d4a81ad81f5d8f9f4c86b: Beginning new tablet copy session on tablet a0e18d0975a54e3dbeb0a13cb226fefd from peer e4086c099d7f4447b2b4733b6ca364df at {username='slave'} at 127.0.0.1:46128: session id = e4086c099d7f4447b2b4733b6ca364df-a0e18d0975a54e3dbeb0a13cb226fefd
I20240503 23:26:42.100425 4303 tablet_copy_source_session.cc:205] T a0e18d0975a54e3dbeb0a13cb226fefd P 692da0319a2d4a81ad81f5d8f9f4c86b: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 23:26:42.102459 4749 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a0e18d0975a54e3dbeb0a13cb226fefd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 23:26:42.109556 4749 tablet_copy_client.cc:791] T a0e18d0975a54e3dbeb0a13cb226fefd P e4086c099d7f4447b2b4733b6ca364df: tablet copy: Starting download of 0 data blocks...
I20240503 23:26:42.109938 4749 tablet_copy_client.cc:655] T a0e18d0975a54e3dbeb0a13cb226fefd P e4086c099d7f4447b2b4733b6ca364df: tablet copy: Starting download of 1 WAL segments...
I20240503 23:26:42.111784 4750 tablet_copy_client.cc:1010] T a0e18d0975a54e3dbeb0a13cb226fefd P e4086c099d7f4447b2b4733b6ca364df: tablet copy: Injecting latency into file download: 2000
I20240503 23:26:42.122222 4751 ts_tablet_manager.cc:902] T 059f6233ea514fe69b8c6504efabeff0 P 4d1d4e261584467fbbce4c72fa18045a: Initiating tablet copy from peer a2f9bf2e5d9b4fc19f804797bca4a3ff (127.0.100.2:46261)
I20240503 23:26:42.124142 4751 tablet_copy_client.cc:316] T 059f6233ea514fe69b8c6504efabeff0 P 4d1d4e261584467fbbce4c72fa18045a: tablet copy: Beginning tablet copy session from remote peer at address 127.0.100.2:46261
I20240503 23:26:42.124966 3848 tablet_copy_service.cc:140] P a2f9bf2e5d9b4fc19f804797bca4a3ff: Received BeginTabletCopySession request for tablet 059f6233ea514fe69b8c6504efabeff0 from peer 4d1d4e261584467fbbce4c72fa18045a ({username='slave'} at 127.0.0.1:55484)
I20240503 23:26:42.125180 3848 tablet_copy_service.cc:161] P a2f9bf2e5d9b4fc19f804797bca4a3ff: Beginning new tablet copy session on tablet 059f6233ea514fe69b8c6504efabeff0 from peer 4d1d4e261584467fbbce4c72fa18045a at {username='slave'} at 127.0.0.1:55484: session id = 4d1d4e261584467fbbce4c72fa18045a-059f6233ea514fe69b8c6504efabeff0
I20240503 23:26:42.127724 3848 tablet_copy_source_session.cc:205] T 059f6233ea514fe69b8c6504efabeff0 P a2f9bf2e5d9b4fc19f804797bca4a3ff: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 23:26:42.129376 4751 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 059f6233ea514fe69b8c6504efabeff0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 23:26:42.136245 4751 tablet_copy_client.cc:791] T 059f6233ea514fe69b8c6504efabeff0 P 4d1d4e261584467fbbce4c72fa18045a: tablet copy: Starting download of 0 data blocks...
I20240503 23:26:42.136617 4751 tablet_copy_client.cc:655] T 059f6233ea514fe69b8c6504efabeff0 P 4d1d4e261584467fbbce4c72fa18045a: tablet copy: Starting download of 1 WAL segments...
I20240503 23:26:42.138942 4752 tablet_copy_client.cc:1010] T 059f6233ea514fe69b8c6504efabeff0 P 4d1d4e261584467fbbce4c72fa18045a: tablet copy: Injecting latency into file download: 2000
I20240503 23:26:42.260939 4753 ts_tablet_manager.cc:902] T 481cd0f8c8d147adac217915a53ff6d9 P 692da0319a2d4a81ad81f5d8f9f4c86b: Initiating tablet copy from peer 83ffd7184ec94302bd1ee74b0fab3e17 (127.0.100.1:32909)
I20240503 23:26:42.261631 4753 tablet_copy_client.cc:316] T 481cd0f8c8d147adac217915a53ff6d9 P 692da0319a2d4a81ad81f5d8f9f4c86b: tablet copy: Beginning tablet copy session from remote peer at address 127.0.100.1:32909
I20240503 23:26:42.262223 3774 tablet_copy_service.cc:140] P 83ffd7184ec94302bd1ee74b0fab3e17: Received BeginTabletCopySession request for tablet 481cd0f8c8d147adac217915a53ff6d9 from peer 692da0319a2d4a81ad81f5d8f9f4c86b ({username='slave'} at 127.0.0.1:57952)
I20240503 23:26:42.262347 3774 tablet_copy_service.cc:161] P 83ffd7184ec94302bd1ee74b0fab3e17: Beginning new tablet copy session on tablet 481cd0f8c8d147adac217915a53ff6d9 from peer 692da0319a2d4a81ad81f5d8f9f4c86b at {username='slave'} at 127.0.0.1:57952: session id = 692da0319a2d4a81ad81f5d8f9f4c86b-481cd0f8c8d147adac217915a53ff6d9
I20240503 23:26:42.265105 3774 tablet_copy_source_session.cc:205] T 481cd0f8c8d147adac217915a53ff6d9 P 83ffd7184ec94302bd1ee74b0fab3e17: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 23:26:42.266484 4753 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 481cd0f8c8d147adac217915a53ff6d9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 23:26:42.272737 4753 tablet_copy_client.cc:791] T 481cd0f8c8d147adac217915a53ff6d9 P 692da0319a2d4a81ad81f5d8f9f4c86b: tablet copy: Starting download of 0 data blocks...
I20240503 23:26:42.273066 4753 tablet_copy_client.cc:655] T 481cd0f8c8d147adac217915a53ff6d9 P 692da0319a2d4a81ad81f5d8f9f4c86b: tablet copy: Starting download of 1 WAL segments...
I20240503 23:26:42.274502 4754 tablet_copy_client.cc:1010] T 481cd0f8c8d147adac217915a53ff6d9 P 692da0319a2d4a81ad81f5d8f9f4c86b: tablet copy: Injecting latency into file download: 2000
I20240503 23:26:42.275508 4755 ts_tablet_manager.cc:902] T a14a12115b8d4fec947c6e223f1ef486 P e4086c099d7f4447b2b4733b6ca364df: Initiating tablet copy from peer 4d1d4e261584467fbbce4c72fa18045a (127.0.100.6:35349)
I20240503 23:26:42.276201 4755 tablet_copy_client.cc:316] T a14a12115b8d4fec947c6e223f1ef486 P e4086c099d7f4447b2b4733b6ca364df: tablet copy: Beginning tablet copy session from remote peer at address 127.0.100.6:35349
I20240503 23:26:42.276958 4377 tablet_copy_service.cc:140] P 4d1d4e261584467fbbce4c72fa18045a: Received BeginTabletCopySession request for tablet a14a12115b8d4fec947c6e223f1ef486 from peer e4086c099d7f4447b2b4733b6ca364df ({username='slave'} at 127.0.0.1:47592)
I20240503 23:26:42.277117 4377 tablet_copy_service.cc:161] P 4d1d4e261584467fbbce4c72fa18045a: Beginning new tablet copy session on tablet a14a12115b8d4fec947c6e223f1ef486 from peer e4086c099d7f4447b2b4733b6ca364df at {username='slave'} at 127.0.0.1:47592: session id = e4086c099d7f4447b2b4733b6ca364df-a14a12115b8d4fec947c6e223f1ef486
I20240503 23:26:42.279562 4377 tablet_copy_source_session.cc:205] T a14a12115b8d4fec947c6e223f1ef486 P 4d1d4e261584467fbbce4c72fa18045a: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 23:26:42.280864 4755 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a14a12115b8d4fec947c6e223f1ef486. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 23:26:42.286897 4755 tablet_copy_client.cc:791] T a14a12115b8d4fec947c6e223f1ef486 P e4086c099d7f4447b2b4733b6ca364df: tablet copy: Starting download of 0 data blocks...
I20240503 23:26:42.287204 4755 tablet_copy_client.cc:655] T a14a12115b8d4fec947c6e223f1ef486 P e4086c099d7f4447b2b4733b6ca364df: tablet copy: Starting download of 1 WAL segments...
I20240503 23:26:42.289023 4756 tablet_copy_client.cc:1010] T a14a12115b8d4fec947c6e223f1ef486 P e4086c099d7f4447b2b4733b6ca364df: tablet copy: Injecting latency into file download: 2000
I20240503 23:26:42.311379 4757 ts_tablet_manager.cc:902] T 55cf664d5e874c0d9b9adf8bdc9c43ed P 692da0319a2d4a81ad81f5d8f9f4c86b: Initiating tablet copy from peer e4086c099d7f4447b2b4733b6ca364df (127.0.100.4:35011)
I20240503 23:26:42.312193 4757 tablet_copy_client.cc:316] T 55cf664d5e874c0d9b9adf8bdc9c43ed P 692da0319a2d4a81ad81f5d8f9f4c86b: tablet copy: Beginning tablet copy session from remote peer at address 127.0.100.4:35011
I20240503 23:26:42.313046 4228 tablet_copy_service.cc:140] P e4086c099d7f4447b2b4733b6ca364df: Received BeginTabletCopySession request for tablet 55cf664d5e874c0d9b9adf8bdc9c43ed from peer 692da0319a2d4a81ad81f5d8f9f4c86b ({username='slave'} at 127.0.0.1:36794)
I20240503 23:26:42.313189 4228 tablet_copy_service.cc:161] P e4086c099d7f4447b2b4733b6ca364df: Beginning new tablet copy session on tablet 55cf664d5e874c0d9b9adf8bdc9c43ed from peer 692da0319a2d4a81ad81f5d8f9f4c86b at {username='slave'} at 127.0.0.1:36794: session id = 692da0319a2d4a81ad81f5d8f9f4c86b-55cf664d5e874c0d9b9adf8bdc9c43ed
I20240503 23:26:42.316002 4228 tablet_copy_source_session.cc:205] T 55cf664d5e874c0d9b9adf8bdc9c43ed P e4086c099d7f4447b2b4733b6ca364df: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 23:26:42.317430 4757 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 55cf664d5e874c0d9b9adf8bdc9c43ed. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 23:26:42.323980 4757 tablet_copy_client.cc:791] T 55cf664d5e874c0d9b9adf8bdc9c43ed P 692da0319a2d4a81ad81f5d8f9f4c86b: tablet copy: Starting download of 0 data blocks...
I20240503 23:26:42.324370 4757 tablet_copy_client.cc:655] T 55cf664d5e874c0d9b9adf8bdc9c43ed P 692da0319a2d4a81ad81f5d8f9f4c86b: tablet copy: Starting download of 1 WAL segments...
I20240503 23:26:42.326346 4758 tablet_copy_client.cc:1010] T 55cf664d5e874c0d9b9adf8bdc9c43ed P 692da0319a2d4a81ad81f5d8f9f4c86b: tablet copy: Injecting latency into file download: 2000
I20240503 23:26:42.347306 4759 ts_tablet_manager.cc:902] T 60267cb4849d4823a0afb2380305905f P 4d1d4e261584467fbbce4c72fa18045a: Initiating tablet copy from peer a3eecbaec43d4954b1096bc3121e824c (127.0.100.3:40649)
I20240503 23:26:42.348117 4759 tablet_copy_client.cc:316] T 60267cb4849d4823a0afb2380305905f P 4d1d4e261584467fbbce4c72fa18045a: tablet copy: Beginning tablet copy session from remote peer at address 127.0.100.3:40649
I20240503 23:26:42.348861 3923 tablet_copy_service.cc:140] P a3eecbaec43d4954b1096bc3121e824c: Received BeginTabletCopySession request for tablet 60267cb4849d4823a0afb2380305905f from peer 4d1d4e261584467fbbce4c72fa18045a ({username='slave'} at 127.0.0.1:39462)
I20240503 23:26:42.349005 3923 tablet_copy_service.cc:161] P a3eecbaec43d4954b1096bc3121e824c: Beginning new tablet copy session on tablet 60267cb4849d4823a0afb2380305905f from peer 4d1d4e261584467fbbce4c72fa18045a at {username='slave'} at 127.0.0.1:39462: session id = 4d1d4e261584467fbbce4c72fa18045a-60267cb4849d4823a0afb2380305905f
I20240503 23:26:42.351929 3923 tablet_copy_source_session.cc:205] T 60267cb4849d4823a0afb2380305905f P a3eecbaec43d4954b1096bc3121e824c: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 23:26:42.353785 4759 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 60267cb4849d4823a0afb2380305905f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 23:26:42.360488 4759 tablet_copy_client.cc:791] T 60267cb4849d4823a0afb2380305905f P 4d1d4e261584467fbbce4c72fa18045a: tablet copy: Starting download of 0 data blocks...
I20240503 23:26:42.360899 4759 tablet_copy_client.cc:655] T 60267cb4849d4823a0afb2380305905f P 4d1d4e261584467fbbce4c72fa18045a: tablet copy: Starting download of 1 WAL segments...
I20240503 23:26:42.362609 4760 tablet_copy_client.cc:1010] T 60267cb4849d4823a0afb2380305905f P 4d1d4e261584467fbbce4c72fa18045a: tablet copy: Injecting latency into file download: 2000
I20240503 23:26:43.262650 3715 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240503 23:26:43.266273 3908 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "7181e4e8c0324a8eb095dde23651a1ab"
dest_uuid: "a3eecbaec43d4954b1096bc3121e824c"
mode: GRACEFUL
new_leader_uuid: "83ffd7184ec94302bd1ee74b0fab3e17"
from {username='slave'} at 127.0.0.1:40650
I20240503 23:26:43.266578 3908 raft_consensus.cc:595] T 7181e4e8c0324a8eb095dde23651a1ab P a3eecbaec43d4954b1096bc3121e824c [term 2 LEADER]: Received request to transfer leadership to TS 83ffd7184ec94302bd1ee74b0fab3e17
I20240503 23:26:43.267227 3715 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 1
I20240503 23:26:43.267491 3715 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240503 23:26:43.833096 4779 raft_consensus.cc:979] T 7181e4e8c0324a8eb095dde23651a1ab P a3eecbaec43d4954b1096bc3121e824c: : Instructing follower 83ffd7184ec94302bd1ee74b0fab3e17 to start an election
I20240503 23:26:43.833317 4779 raft_consensus.cc:1067] T 7181e4e8c0324a8eb095dde23651a1ab P a3eecbaec43d4954b1096bc3121e824c [term 2 LEADER]: Signalling peer 83ffd7184ec94302bd1ee74b0fab3e17 to start an election
I20240503 23:26:43.833889 3760 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "7181e4e8c0324a8eb095dde23651a1ab"
dest_uuid: "83ffd7184ec94302bd1ee74b0fab3e17"
from {username='slave'} at 127.0.0.1:35592
I20240503 23:26:43.834096 3760 raft_consensus.cc:482] T 7181e4e8c0324a8eb095dde23651a1ab P 83ffd7184ec94302bd1ee74b0fab3e17 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 23:26:43.834205 3760 raft_consensus.cc:3037] T 7181e4e8c0324a8eb095dde23651a1ab P 83ffd7184ec94302bd1ee74b0fab3e17 [term 2 FOLLOWER]: Advancing to term 3
I20240503 23:26:43.837328 3760 raft_consensus.cc:504] T 7181e4e8c0324a8eb095dde23651a1ab P 83ffd7184ec94302bd1ee74b0fab3e17 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "83ffd7184ec94302bd1ee74b0fab3e17" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 32909 } } peers { permanent_uuid: "a3eecbaec43d4954b1096bc3121e824c" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 40649 } } peers { permanent_uuid: "e4086c099d7f4447b2b4733b6ca364df" member_type: VOTER last_known_addr { host: "127.0.100.4" port: 35011 } attrs { promote: false } }
I20240503 23:26:43.838028 3760 leader_election.cc:288] T 7181e4e8c0324a8eb095dde23651a1ab P 83ffd7184ec94302bd1ee74b0fab3e17 [CANDIDATE]: Term 3 election: Requested vote from peers a3eecbaec43d4954b1096bc3121e824c (127.0.100.3:40649), e4086c099d7f4447b2b4733b6ca364df (127.0.100.4:35011)
I20240503 23:26:43.838356 3908 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7181e4e8c0324a8eb095dde23651a1ab" candidate_uuid: "83ffd7184ec94302bd1ee74b0fab3e17" candidate_term: 3 candidate_status { last_received { term: 2 index: 29 } } ignore_live_leader: true dest_uuid: "a3eecbaec43d4954b1096bc3121e824c"
I20240503 23:26:43.838591 3908 raft_consensus.cc:3032] T 7181e4e8c0324a8eb095dde23651a1ab P a3eecbaec43d4954b1096bc3121e824c [term 2 LEADER]: Stepping down as leader of term 2
I20240503 23:26:43.838691 3908 raft_consensus.cc:729] T 7181e4e8c0324a8eb095dde23651a1ab P a3eecbaec43d4954b1096bc3121e824c [term 2 LEADER]: Becoming Follower/Learner. State: Replica: a3eecbaec43d4954b1096bc3121e824c, State: Running, Role: LEADER
I20240503 23:26:43.838593 4217 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7181e4e8c0324a8eb095dde23651a1ab" candidate_uuid: "83ffd7184ec94302bd1ee74b0fab3e17" candidate_term: 3 candidate_status { last_received { term: 2 index: 29 } } ignore_live_leader: true dest_uuid: "e4086c099d7f4447b2b4733b6ca364df"
I20240503 23:26:43.838800 4217 raft_consensus.cc:3037] T 7181e4e8c0324a8eb095dde23651a1ab P e4086c099d7f4447b2b4733b6ca364df [term 2 FOLLOWER]: Advancing to term 3
I20240503 23:26:43.838915 3908 consensus_queue.cc:260] T 7181e4e8c0324a8eb095dde23651a1ab P a3eecbaec43d4954b1096bc3121e824c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 29, Committed index: 29, Last appended: 2.29, Last appended by leader: 29, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "83ffd7184ec94302bd1ee74b0fab3e17" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 32909 } } peers { permanent_uuid: "a3eecbaec43d4954b1096bc3121e824c" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 40649 } } peers { permanent_uuid: "e4086c099d7f4447b2b4733b6ca364df" member_type: VOTER last_known_addr { host: "127.0.100.4" port: 35011 } attrs { promote: false } }
I20240503 23:26:43.839161 3908 raft_consensus.cc:3037] T 7181e4e8c0324a8eb095dde23651a1ab P a3eecbaec43d4954b1096bc3121e824c [term 2 FOLLOWER]: Advancing to term 3
I20240503 23:26:43.841307 4217 raft_consensus.cc:2454] T 7181e4e8c0324a8eb095dde23651a1ab P e4086c099d7f4447b2b4733b6ca364df [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83ffd7184ec94302bd1ee74b0fab3e17 in term 3.
I20240503 23:26:43.841288 3908 raft_consensus.cc:2454] T 7181e4e8c0324a8eb095dde23651a1ab P a3eecbaec43d4954b1096bc3121e824c [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83ffd7184ec94302bd1ee74b0fab3e17 in term 3.
I20240503 23:26:43.841940 3727 leader_election.cc:302] T 7181e4e8c0324a8eb095dde23651a1ab P 83ffd7184ec94302bd1ee74b0fab3e17 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 83ffd7184ec94302bd1ee74b0fab3e17, e4086c099d7f4447b2b4733b6ca364df; no voters:
I20240503 23:26:43.842515 4776 raft_consensus.cc:2781] T 7181e4e8c0324a8eb095dde23651a1ab P 83ffd7184ec94302bd1ee74b0fab3e17 [term 3 FOLLOWER]: Leader election won for term 3
I20240503 23:26:43.842689 4776 raft_consensus.cc:686] T 7181e4e8c0324a8eb095dde23651a1ab P 83ffd7184ec94302bd1ee74b0fab3e17 [term 3 LEADER]: Becoming Leader. State: Replica: 83ffd7184ec94302bd1ee74b0fab3e17, State: Running, Role: LEADER
I20240503 23:26:43.842981 4776 consensus_queue.cc:237] T 7181e4e8c0324a8eb095dde23651a1ab P 83ffd7184ec94302bd1ee74b0fab3e17 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 29, Committed index: 29, Last appended: 2.29, Last appended by leader: 29, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "83ffd7184ec94302bd1ee74b0fab3e17" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 32909 } } peers { permanent_uuid: "a3eecbaec43d4954b1096bc3121e824c" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 40649 } } peers { permanent_uuid: "e4086c099d7f4447b2b4733b6ca364df" member_type: VOTER last_known_addr { host: "127.0.100.4" port: 35011 } attrs { promote: false } }
I20240503 23:26:43.846073 3655 catalog_manager.cc:5484] T 7181e4e8c0324a8eb095dde23651a1ab P 83ffd7184ec94302bd1ee74b0fab3e17 reported cstate change: term changed from 2 to 3, leader changed from a3eecbaec43d4954b1096bc3121e824c (127.0.100.3) to 83ffd7184ec94302bd1ee74b0fab3e17 (127.0.100.1). New cstate: current_term: 3 leader_uuid: "83ffd7184ec94302bd1ee74b0fab3e17" committed_config { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "83ffd7184ec94302bd1ee74b0fab3e17" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 32909 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a3eecbaec43d4954b1096bc3121e824c" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 40649 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e4086c099d7f4447b2b4733b6ca364df" member_type: VOTER last_known_addr { host: "127.0.100.4" port: 35011 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240503 23:26:44.112260 4749 tablet_copy_client.cc:531] T a0e18d0975a54e3dbeb0a13cb226fefd P e4086c099d7f4447b2b4733b6ca364df: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 23:26:44.115880 4749 tablet_bootstrap.cc:492] T a0e18d0975a54e3dbeb0a13cb226fefd P e4086c099d7f4447b2b4733b6ca364df: Bootstrap starting.
I20240503 23:26:44.139457 4751 tablet_copy_client.cc:531] T 059f6233ea514fe69b8c6504efabeff0 P 4d1d4e261584467fbbce4c72fa18045a: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 23:26:44.143878 4751 tablet_bootstrap.cc:492] T 059f6233ea514fe69b8c6504efabeff0 P 4d1d4e261584467fbbce4c72fa18045a: Bootstrap starting.
I20240503 23:26:44.160189 4749 tablet_bootstrap.cc:492] T a0e18d0975a54e3dbeb0a13cb226fefd P e4086c099d7f4447b2b4733b6ca364df: Bootstrap replayed 1/1 log segments. Stats: ops{read=32 overwritten=0 applied=32 ignored=0} inserts{seen=94 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 23:26:44.160739 4749 tablet_bootstrap.cc:492] T a0e18d0975a54e3dbeb0a13cb226fefd P e4086c099d7f4447b2b4733b6ca364df: Bootstrap complete.
I20240503 23:26:44.161051 4749 ts_tablet_manager.cc:1367] T a0e18d0975a54e3dbeb0a13cb226fefd P e4086c099d7f4447b2b4733b6ca364df: Time spent bootstrapping tablet: real 0.045s user 0.041s sys 0.006s
I20240503 23:26:44.161967 4749 raft_consensus.cc:348] T a0e18d0975a54e3dbeb0a13cb226fefd P e4086c099d7f4447b2b4733b6ca364df [term 3 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "83ffd7184ec94302bd1ee74b0fab3e17" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 32909 } } peers { permanent_uuid: "a2f9bf2e5d9b4fc19f804797bca4a3ff" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 46261 } attrs { replace: true } } peers { permanent_uuid: "692da0319a2d4a81ad81f5d8f9f4c86b" member_type: VOTER last_known_addr { host: "127.0.100.5" port: 38561 } attrs { promote: false } } peers { permanent_uuid: "e4086c099d7f4447b2b4733b6ca364df" member_type: NON_VOTER last_known_addr { host: "127.0.100.4" port: 35011 } attrs { promote: true } }
I20240503 23:26:44.162248 4749 raft_consensus.cc:729] T a0e18d0975a54e3dbeb0a13cb226fefd P e4086c099d7f4447b2b4733b6ca364df [term 3 LEARNER]: Becoming Follower/Learner. State: Replica: e4086c099d7f4447b2b4733b6ca364df, State: Initialized, Role: LEARNER
I20240503 23:26:44.162464 4749 consensus_queue.cc:260] T a0e18d0975a54e3dbeb0a13cb226fefd P e4086c099d7f4447b2b4733b6ca364df [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 32, Last appended: 3.32, Last appended by leader: 32, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "83ffd7184ec94302bd1ee74b0fab3e17" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 32909 } } peers { permanent_uuid: "a2f9bf2e5d9b4fc19f804797bca4a3ff" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 46261 } attrs { replace: true } } peers { permanent_uuid: "692da0319a2d4a81ad81f5d8f9f4c86b" member_type: VOTER last_known_addr { host: "127.0.100.5" port: 38561 } attrs { promote: false } } peers { permanent_uuid: "e4086c099d7f4447b2b4733b6ca364df" member_type: NON_VOTER last_known_addr { host: "127.0.100.4" port: 35011 } attrs { promote: true } }
I20240503 23:26:44.163101 4749 ts_tablet_manager.cc:1398] T a0e18d0975a54e3dbeb0a13cb226fefd P e4086c099d7f4447b2b4733b6ca364df: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240503 23:26:44.163954 4303 tablet_copy_service.cc:342] P 692da0319a2d4a81ad81f5d8f9f4c86b: Request end of tablet copy session e4086c099d7f4447b2b4733b6ca364df-a0e18d0975a54e3dbeb0a13cb226fefd received from {username='slave'} at 127.0.0.1:46128
I20240503 23:26:44.164065 4303 tablet_copy_service.cc:433] P 692da0319a2d4a81ad81f5d8f9f4c86b: ending tablet copy session e4086c099d7f4447b2b4733b6ca364df-a0e18d0975a54e3dbeb0a13cb226fefd on tablet a0e18d0975a54e3dbeb0a13cb226fefd with peer e4086c099d7f4447b2b4733b6ca364df
I20240503 23:26:44.187986 4751 tablet_bootstrap.cc:492] T 059f6233ea514fe69b8c6504efabeff0 P 4d1d4e261584467fbbce4c72fa18045a: Bootstrap replayed 1/1 log segments. Stats: ops{read=28 overwritten=0 applied=28 ignored=0} inserts{seen=117 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 23:26:44.188575 4751 tablet_bootstrap.cc:492] T 059f6233ea514fe69b8c6504efabeff0 P 4d1d4e261584467fbbce4c72fa18045a: Bootstrap complete.
I20240503 23:26:44.188907 4751 ts_tablet_manager.cc:1367] T 059f6233ea514fe69b8c6504efabeff0 P 4d1d4e261584467fbbce4c72fa18045a: Time spent bootstrapping tablet: real 0.045s user 0.031s sys 0.009s
I20240503 23:26:44.189790 4751 raft_consensus.cc:348] T 059f6233ea514fe69b8c6504efabeff0 P 4d1d4e261584467fbbce4c72fa18045a [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "83ffd7184ec94302bd1ee74b0fab3e17" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 32909 } attrs { replace: true } } peers { permanent_uuid: "a2f9bf2e5d9b4fc19f804797bca4a3ff" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 46261 } } peers { permanent_uuid: "a3eecbaec43d4954b1096bc3121e824c" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 40649 } } peers { permanent_uuid: "4d1d4e261584467fbbce4c72fa18045a" member_type: NON_VOTER last_known_addr { host: "127.0.100.6" port: 35349 } attrs { promote: true } }
I20240503 23:26:44.190017 4751 raft_consensus.cc:729] T 059f6233ea514fe69b8c6504efabeff0 P 4d1d4e261584467fbbce4c72fa18045a [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: 4d1d4e261584467fbbce4c72fa18045a, State: Initialized, Role: LEARNER
I20240503 23:26:44.190213 4751 consensus_queue.cc:260] T 059f6233ea514fe69b8c6504efabeff0 P 4d1d4e261584467fbbce4c72fa18045a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 28, Last appended: 2.28, Last appended by leader: 28, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "83ffd7184ec94302bd1ee74b0fab3e17" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 32909 } attrs { replace: true } } peers { permanent_uuid: "a2f9bf2e5d9b4fc19f804797bca4a3ff" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 46261 } } peers { permanent_uuid: "a3eecbaec43d4954b1096bc3121e824c" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 40649 } } peers { permanent_uuid: "4d1d4e261584467fbbce4c72fa18045a" member_type: NON_VOTER last_known_addr { host: "127.0.100.6" port: 35349 } attrs { promote: true } }
I20240503 23:26:44.190804 4751 ts_tablet_manager.cc:1398] T 059f6233ea514fe69b8c6504efabeff0 P 4d1d4e261584467fbbce4c72fa18045a: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240503 23:26:44.191484 3848 tablet_copy_service.cc:342] P a2f9bf2e5d9b4fc19f804797bca4a3ff: Request end of tablet copy session 4d1d4e261584467fbbce4c72fa18045a-059f6233ea514fe69b8c6504efabeff0 received from {username='slave'} at 127.0.0.1:55484
I20240503 23:26:44.191599 3848 tablet_copy_service.cc:433] P a2f9bf2e5d9b4fc19f804797bca4a3ff: ending tablet copy session 4d1d4e261584467fbbce4c72fa18045a-059f6233ea514fe69b8c6504efabeff0 on tablet 059f6233ea514fe69b8c6504efabeff0 with peer 4d1d4e261584467fbbce4c72fa18045a
I20240503 23:26:44.251600 4218 raft_consensus.cc:1203] T a0e18d0975a54e3dbeb0a13cb226fefd P e4086c099d7f4447b2b4733b6ca364df [term 3 LEARNER]: Deduplicated request from leader. Original: 3.31->[3.32-3.32] Dedup: 3.32->[]
I20240503 23:26:44.275051 4753 tablet_copy_client.cc:531] T 481cd0f8c8d147adac217915a53ff6d9 P 692da0319a2d4a81ad81f5d8f9f4c86b: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 23:26:44.279114 4753 tablet_bootstrap.cc:492] T 481cd0f8c8d147adac217915a53ff6d9 P 692da0319a2d4a81ad81f5d8f9f4c86b: Bootstrap starting.
I20240503 23:26:44.289462 4755 tablet_copy_client.cc:531] T a14a12115b8d4fec947c6e223f1ef486 P e4086c099d7f4447b2b4733b6ca364df: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 23:26:44.292892 4755 tablet_bootstrap.cc:492] T a14a12115b8d4fec947c6e223f1ef486 P e4086c099d7f4447b2b4733b6ca364df: Bootstrap starting.
I20240503 23:26:44.309226 3907 raft_consensus.cc:1261] T 7181e4e8c0324a8eb095dde23651a1ab P a3eecbaec43d4954b1096bc3121e824c [term 3 FOLLOWER]: Refusing update from remote peer 83ffd7184ec94302bd1ee74b0fab3e17: Log matching property violated. Preceding OpId in replica: term: 2 index: 29. Preceding OpId from leader: term: 3 index: 30. (index mismatch)
I20240503 23:26:44.310050 4776 consensus_queue.cc:1035] T 7181e4e8c0324a8eb095dde23651a1ab P 83ffd7184ec94302bd1ee74b0fab3e17 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a3eecbaec43d4954b1096bc3121e824c" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 40649 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 30, Last known committed idx: 29, Time since last communication: 0.000s
I20240503 23:26:44.318081 4753 tablet_bootstrap.cc:492] T 481cd0f8c8d147adac217915a53ff6d9 P 692da0319a2d4a81ad81f5d8f9f4c86b: Bootstrap replayed 1/1 log segments. Stats: ops{read=28 overwritten=0 applied=28 ignored=0} inserts{seen=100 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 23:26:44.318661 4753 tablet_bootstrap.cc:492] T 481cd0f8c8d147adac217915a53ff6d9 P 692da0319a2d4a81ad81f5d8f9f4c86b: Bootstrap complete.
I20240503 23:26:44.319185 4753 ts_tablet_manager.cc:1367] T 481cd0f8c8d147adac217915a53ff6d9 P 692da0319a2d4a81ad81f5d8f9f4c86b: Time spent bootstrapping tablet: real 0.040s user 0.036s sys 0.006s
I20240503 23:26:44.320242 4753 raft_consensus.cc:348] T 481cd0f8c8d147adac217915a53ff6d9 P 692da0319a2d4a81ad81f5d8f9f4c86b [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "83ffd7184ec94302bd1ee74b0fab3e17" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 32909 } } peers { permanent_uuid: "a2f9bf2e5d9b4fc19f804797bca4a3ff" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 46261 } } peers { permanent_uuid: "a3eecbaec43d4954b1096bc3121e824c" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 40649 } attrs { replace: true } } peers { permanent_uuid: "692da0319a2d4a81ad81f5d8f9f4c86b" member_type: NON_VOTER last_known_addr { host: "127.0.100.5" port: 38561 } attrs { promote: true } }
I20240503 23:26:44.320655 4753 raft_consensus.cc:729] T 481cd0f8c8d147adac217915a53ff6d9 P 692da0319a2d4a81ad81f5d8f9f4c86b [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: 692da0319a2d4a81ad81f5d8f9f4c86b, State: Initialized, Role: LEARNER
I20240503 23:26:44.320947 4753 consensus_queue.cc:260] T 481cd0f8c8d147adac217915a53ff6d9 P 692da0319a2d4a81ad81f5d8f9f4c86b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 28, Last appended: 2.28, Last appended by leader: 28, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "83ffd7184ec94302bd1ee74b0fab3e17" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 32909 } } peers { permanent_uuid: "a2f9bf2e5d9b4fc19f804797bca4a3ff" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 46261 } } peers { permanent_uuid: "a3eecbaec43d4954b1096bc3121e824c" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 40649 } attrs { replace: true } } peers { permanent_uuid: "692da0319a2d4a81ad81f5d8f9f4c86b" member_type: NON_VOTER last_known_addr { host: "127.0.100.5" port: 38561 } attrs { promote: true } }
I20240503 23:26:44.322342 4753 ts_tablet_manager.cc:1398] T 481cd0f8c8d147adac217915a53ff6d9 P 692da0319a2d4a81ad81f5d8f9f4c86b: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240503 23:26:44.323120 3774 tablet_copy_service.cc:342] P 83ffd7184ec94302bd1ee74b0fab3e17: Request end of tablet copy session 692da0319a2d4a81ad81f5d8f9f4c86b-481cd0f8c8d147adac217915a53ff6d9 received from {username='slave'} at 127.0.0.1:57952
I20240503 23:26:44.323251 3774 tablet_copy_service.cc:433] P 83ffd7184ec94302bd1ee74b0fab3e17: ending tablet copy session 692da0319a2d4a81ad81f5d8f9f4c86b-481cd0f8c8d147adac217915a53ff6d9 on tablet 481cd0f8c8d147adac217915a53ff6d9 with peer 692da0319a2d4a81ad81f5d8f9f4c86b
I20240503 23:26:44.326754 4757 tablet_copy_client.cc:531] T 55cf664d5e874c0d9b9adf8bdc9c43ed P 692da0319a2d4a81ad81f5d8f9f4c86b: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 23:26:44.330401 4218 raft_consensus.cc:1261] T 7181e4e8c0324a8eb095dde23651a1ab P e4086c099d7f4447b2b4733b6ca364df [term 3 FOLLOWER]: Refusing update from remote peer 83ffd7184ec94302bd1ee74b0fab3e17: Log matching property violated. Preceding OpId in replica: term: 2 index: 29. Preceding OpId from leader: term: 3 index: 30. (index mismatch)
I20240503 23:26:44.331189 4757 tablet_bootstrap.cc:492] T 55cf664d5e874c0d9b9adf8bdc9c43ed P 692da0319a2d4a81ad81f5d8f9f4c86b: Bootstrap starting.
I20240503 23:26:44.332125 4775 consensus_queue.cc:1035] T 7181e4e8c0324a8eb095dde23651a1ab P 83ffd7184ec94302bd1ee74b0fab3e17 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4086c099d7f4447b2b4733b6ca364df" member_type: VOTER last_known_addr { host: "127.0.100.4" port: 35011 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 30, Last known committed idx: 29, Time since last communication: 0.000s
I20240503 23:26:44.359481 4755 tablet_bootstrap.cc:492] T a14a12115b8d4fec947c6e223f1ef486 P e4086c099d7f4447b2b4733b6ca364df: Bootstrap replayed 1/1 log segments. Stats: ops{read=31 overwritten=0 applied=31 ignored=0} inserts{seen=120 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 23:26:44.360152 4755 tablet_bootstrap.cc:492] T a14a12115b8d4fec947c6e223f1ef486 P e4086c099d7f4447b2b4733b6ca364df: Bootstrap complete.
I20240503 23:26:44.360525 4755 ts_tablet_manager.cc:1367] T a14a12115b8d4fec947c6e223f1ef486 P e4086c099d7f4447b2b4733b6ca364df: Time spent bootstrapping tablet: real 0.068s user 0.052s sys 0.000s
I20240503 23:26:44.361508 4755 raft_consensus.cc:348] T a14a12115b8d4fec947c6e223f1ef486 P e4086c099d7f4447b2b4733b6ca364df [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "83ffd7184ec94302bd1ee74b0fab3e17" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 32909 } } peers { permanent_uuid: "a3eecbaec43d4954b1096bc3121e824c" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 40649 } attrs { replace: true } } peers { permanent_uuid: "4d1d4e261584467fbbce4c72fa18045a" member_type: VOTER last_known_addr { host: "127.0.100.6" port: 35349 } attrs { promote: false } } peers { permanent_uuid: "e4086c099d7f4447b2b4733b6ca364df" member_type: NON_VOTER last_known_addr { host: "127.0.100.4" port: 35011 } attrs { promote: true } }
I20240503 23:26:44.361826 4755 raft_consensus.cc:729] T a14a12115b8d4fec947c6e223f1ef486 P e4086c099d7f4447b2b4733b6ca364df [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: e4086c099d7f4447b2b4733b6ca364df, State: Initialized, Role: LEARNER
I20240503 23:26:44.363059 4759 tablet_copy_client.cc:531] T 60267cb4849d4823a0afb2380305905f P 4d1d4e261584467fbbce4c72fa18045a: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 23:26:44.363363 4755 consensus_queue.cc:260] T a14a12115b8d4fec947c6e223f1ef486 P e4086c099d7f4447b2b4733b6ca364df [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 31, Last appended: 2.31, Last appended by leader: 31, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "83ffd7184ec94302bd1ee74b0fab3e17" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 32909 } } peers { permanent_uuid: "a3eecbaec43d4954b1096bc3121e824c" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 40649 } attrs { replace: true } } peers { permanent_uuid: "4d1d4e261584467fbbce4c72fa18045a" member_type: VOTER last_known_addr { host: "127.0.100.6" port: 35349 } attrs { promote: false } } peers { permanent_uuid: "e4086c099d7f4447b2b4733b6ca364df" member_type: NON_VOTER last_known_addr { host: "127.0.100.4" port: 35011 } attrs { promote: true } }
I20240503 23:26:44.364367 4755 ts_tablet_manager.cc:1398] T a14a12115b8d4fec947c6e223f1ef486 P e4086c099d7f4447b2b4733b6ca364df: Time spent starting tablet: real 0.004s user 0.000s sys 0.002s
I20240503 23:26:44.365154 4377 tablet_copy_service.cc:342] P 4d1d4e261584467fbbce4c72fa18045a: Request end of tablet copy session e4086c099d7f4447b2b4733b6ca364df-a14a12115b8d4fec947c6e223f1ef486 received from {username='slave'} at 127.0.0.1:47592
I20240503 23:26:44.365272 4377 tablet_copy_service.cc:433] P 4d1d4e261584467fbbce4c72fa18045a: ending tablet copy session e4086c099d7f4447b2b4733b6ca364df-a14a12115b8d4fec947c6e223f1ef486 on tablet a14a12115b8d4fec947c6e223f1ef486 with peer e4086c099d7f4447b2b4733b6ca364df
I20240503 23:26:44.366509 4759 tablet_bootstrap.cc:492] T 60267cb4849d4823a0afb2380305905f P 4d1d4e261584467fbbce4c72fa18045a: Bootstrap starting.
I20240503 23:26:44.392045 4757 tablet_bootstrap.cc:492] T 55cf664d5e874c0d9b9adf8bdc9c43ed P 692da0319a2d4a81ad81f5d8f9f4c86b: Bootstrap replayed 1/1 log segments. Stats: ops{read=34 overwritten=0 applied=34 ignored=0} inserts{seen=116 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 23:26:44.392647 4757 tablet_bootstrap.cc:492] T 55cf664d5e874c0d9b9adf8bdc9c43ed P 692da0319a2d4a81ad81f5d8f9f4c86b: Bootstrap complete.
I20240503 23:26:44.393013 4757 ts_tablet_manager.cc:1367] T 55cf664d5e874c0d9b9adf8bdc9c43ed P 692da0319a2d4a81ad81f5d8f9f4c86b: Time spent bootstrapping tablet: real 0.062s user 0.049s sys 0.007s
I20240503 23:26:44.393872 4757 raft_consensus.cc:348] T 55cf664d5e874c0d9b9adf8bdc9c43ed P 692da0319a2d4a81ad81f5d8f9f4c86b [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 34 OBSOLETE_local: false peers { permanent_uuid: "a2f9bf2e5d9b4fc19f804797bca4a3ff" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 46261 } attrs { replace: true } } peers { permanent_uuid: "e4086c099d7f4447b2b4733b6ca364df" member_type: VOTER last_known_addr { host: "127.0.100.4" port: 35011 } attrs { promote: false } } peers { permanent_uuid: "4d1d4e261584467fbbce4c72fa18045a" member_type: VOTER last_known_addr { host: "127.0.100.6" port: 35349 } attrs { promote: false } } peers { permanent_uuid: "692da0319a2d4a81ad81f5d8f9f4c86b" member_type: NON_VOTER last_known_addr { host: "127.0.100.5" port: 38561 } attrs { promote: true } }
I20240503 23:26:44.394179 4757 raft_consensus.cc:729] T 55cf664d5e874c0d9b9adf8bdc9c43ed P 692da0319a2d4a81ad81f5d8f9f4c86b [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: 692da0319a2d4a81ad81f5d8f9f4c86b, State: Initialized, Role: LEARNER
I20240503 23:26:44.394377 4757 consensus_queue.cc:260] T 55cf664d5e874c0d9b9adf8bdc9c43ed P 692da0319a2d4a81ad81f5d8f9f4c86b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 34, Last appended: 2.34, Last appended by leader: 34, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 34 OBSOLETE_local: false peers { permanent_uuid: "a2f9bf2e5d9b4fc19f804797bca4a3ff" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 46261 } attrs { replace: true } } peers { permanent_uuid: "e4086c099d7f4447b2b4733b6ca364df" member_type: VOTER last_known_addr { host: "127.0.100.4" port: 35011 } attrs { promote: false } } peers { permanent_uuid: "4d1d4e261584467fbbce4c72fa18045a" member_type: VOTER last_known_addr { host: "127.0.100.6" port: 35349 } attrs { promote: false } } peers { permanent_uuid: "692da0319a2d4a81ad81f5d8f9f4c86b" member_type: NON_VOTER last_known_addr { host: "127.0.100.5" port: 38561 } attrs { promote: true } }
I20240503 23:26:44.394989 4757 ts_tablet_manager.cc:1398] T 55cf664d5e874c0d9b9adf8bdc9c43ed P 692da0319a2d4a81ad81f5d8f9f4c86b: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240503 23:26:44.395706 4228 tablet_copy_service.cc:342] P e4086c099d7f4447b2b4733b6ca364df: Request end of tablet copy session 692da0319a2d4a81ad81f5d8f9f4c86b-55cf664d5e874c0d9b9adf8bdc9c43ed received from {username='slave'} at 127.0.0.1:36794
I20240503 23:26:44.395851 4228 tablet_copy_service.cc:433] P e4086c099d7f4447b2b4733b6ca364df: ending tablet copy session 692da0319a2d4a81ad81f5d8f9f4c86b-55cf664d5e874c0d9b9adf8bdc9c43ed on tablet 55cf664d5e874c0d9b9adf8bdc9c43ed with peer 692da0319a2d4a81ad81f5d8f9f4c86b
I20240503 23:26:44.408118 4759 tablet_bootstrap.cc:492] T 60267cb4849d4823a0afb2380305905f P 4d1d4e261584467fbbce4c72fa18045a: Bootstrap replayed 1/1 log segments. Stats: ops{read=29 overwritten=0 applied=29 ignored=0} inserts{seen=94 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 23:26:44.408710 4759 tablet_bootstrap.cc:492] T 60267cb4849d4823a0afb2380305905f P 4d1d4e261584467fbbce4c72fa18045a: Bootstrap complete.
I20240503 23:26:44.409054 4759 ts_tablet_manager.cc:1367] T 60267cb4849d4823a0afb2380305905f P 4d1d4e261584467fbbce4c72fa18045a: Time spent bootstrapping tablet: real 0.043s user 0.025s sys 0.016s
I20240503 23:26:44.410115 4759 raft_consensus.cc:348] T 60267cb4849d4823a0afb2380305905f P 4d1d4e261584467fbbce4c72fa18045a [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "83ffd7184ec94302bd1ee74b0fab3e17" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 32909 } attrs { replace: true } } peers { permanent_uuid: "a3eecbaec43d4954b1096bc3121e824c" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 40649 } } peers { permanent_uuid: "e4086c099d7f4447b2b4733b6ca364df" member_type: VOTER last_known_addr { host: "127.0.100.4" port: 35011 } attrs { promote: false } } peers { permanent_uuid: "4d1d4e261584467fbbce4c72fa18045a" member_type: NON_VOTER last_known_addr { host: "127.0.100.6" port: 35349 } attrs { promote: true } }
I20240503 23:26:44.410461 4759 raft_consensus.cc:729] T 60267cb4849d4823a0afb2380305905f P 4d1d4e261584467fbbce4c72fa18045a [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 4d1d4e261584467fbbce4c72fa18045a, State: Initialized, Role: LEARNER
I20240503 23:26:44.410743 4759 consensus_queue.cc:260] T 60267cb4849d4823a0afb2380305905f P 4d1d4e261584467fbbce4c72fa18045a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 29, Last appended: 1.29, Last appended by leader: 29, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "83ffd7184ec94302bd1ee74b0fab3e17" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 32909 } attrs { replace: true } } peers { permanent_uuid: "a3eecbaec43d4954b1096bc3121e824c" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 40649 } } peers { permanent_uuid: "e4086c099d7f4447b2b4733b6ca364df" member_type: VOTER last_known_addr { host: "127.0.100.4" port: 35011 } attrs { promote: false } } peers { permanent_uuid: "4d1d4e261584467fbbce4c72fa18045a" member_type: NON_VOTER last_known_addr { host: "127.0.100.6" port: 35349 } attrs { promote: true } }
I20240503 23:26:44.411543 4759 ts_tablet_manager.cc:1398] T 60267cb4849d4823a0afb2380305905f P 4d1d4e261584467fbbce4c72fa18045a: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240503 23:26:44.412734 3923 tablet_copy_service.cc:342] P a3eecbaec43d4954b1096bc3121e824c: Request end of tablet copy session 4d1d4e261584467fbbce4c72fa18045a-60267cb4849d4823a0afb2380305905f received from {username='slave'} at 127.0.0.1:39462
I20240503 23:26:44.412911 3923 tablet_copy_service.cc:433] P a3eecbaec43d4954b1096bc3121e824c: ending tablet copy session 4d1d4e261584467fbbce4c72fa18045a-60267cb4849d4823a0afb2380305905f on tablet 60267cb4849d4823a0afb2380305905f with peer 4d1d4e261584467fbbce4c72fa18045a
I20240503 23:26:44.485810 4293 raft_consensus.cc:1203] T 55cf664d5e874c0d9b9adf8bdc9c43ed P 692da0319a2d4a81ad81f5d8f9f4c86b [term 2 LEARNER]: Deduplicated request from leader. Original: 2.33->[2.34-2.34] Dedup: 2.34->[]
I20240503 23:26:44.537438 4366 raft_consensus.cc:1203] T 059f6233ea514fe69b8c6504efabeff0 P 4d1d4e261584467fbbce4c72fa18045a [term 2 LEARNER]: Deduplicated request from leader. Original: 2.27->[2.28-2.28] Dedup: 2.28->[]
I20240503 23:26:44.705564 4293 raft_consensus.cc:1203] T 481cd0f8c8d147adac217915a53ff6d9 P 692da0319a2d4a81ad81f5d8f9f4c86b [term 2 LEARNER]: Deduplicated request from leader. Original: 2.27->[2.28-2.28] Dedup: 2.28->[]
I20240503 23:26:44.773888 4218 raft_consensus.cc:1203] T a14a12115b8d4fec947c6e223f1ef486 P e4086c099d7f4447b2b4733b6ca364df [term 2 LEARNER]: Deduplicated request from leader. Original: 2.30->[2.31-2.31] Dedup: 2.31->[]
I20240503 23:26:44.788684 4800 raft_consensus.cc:1050] T a0e18d0975a54e3dbeb0a13cb226fefd P 692da0319a2d4a81ad81f5d8f9f4c86b: attempting to promote NON_VOTER e4086c099d7f4447b2b4733b6ca364df to VOTER
I20240503 23:26:44.789673 4800 consensus_queue.cc:237] T a0e18d0975a54e3dbeb0a13cb226fefd P 692da0319a2d4a81ad81f5d8f9f4c86b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 32, Committed index: 32, Last appended: 3.32, Last appended by leader: 30, Current term: 3, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 33 OBSOLETE_local: false peers { permanent_uuid: "83ffd7184ec94302bd1ee74b0fab3e17" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 32909 } } peers { permanent_uuid: "a2f9bf2e5d9b4fc19f804797bca4a3ff" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 46261 } attrs { replace: true } } peers { permanent_uuid: "692da0319a2d4a81ad81f5d8f9f4c86b" member_type: VOTER last_known_addr { host: "127.0.100.5" port: 38561 } attrs { promote: false } } peers { permanent_uuid: "e4086c099d7f4447b2b4733b6ca364df" member_type: VOTER last_known_addr { host: "127.0.100.4" port: 35011 } attrs { promote: false } }
I20240503 23:26:44.792716 4217 raft_consensus.cc:1261] T a0e18d0975a54e3dbeb0a13cb226fefd P e4086c099d7f4447b2b4733b6ca364df [term 3 LEARNER]: Refusing update from remote peer 692da0319a2d4a81ad81f5d8f9f4c86b: Log matching property violated. Preceding OpId in replica: term: 3 index: 32. Preceding OpId from leader: term: 3 index: 33. (index mismatch)
I20240503 23:26:44.792963 3837 raft_consensus.cc:1261] T a0e18d0975a54e3dbeb0a13cb226fefd P a2f9bf2e5d9b4fc19f804797bca4a3ff [term 3 FOLLOWER]: Refusing update from remote peer 692da0319a2d4a81ad81f5d8f9f4c86b: Log matching property violated. Preceding OpId in replica: term: 3 index: 32. Preceding OpId from leader: term: 3 index: 33. (index mismatch)
I20240503 23:26:44.794010 4801 consensus_queue.cc:1035] T a0e18d0975a54e3dbeb0a13cb226fefd P 692da0319a2d4a81ad81f5d8f9f4c86b [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4086c099d7f4447b2b4733b6ca364df" member_type: VOTER last_known_addr { host: "127.0.100.4" port: 35011 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 33, Last known committed idx: 32, Time since last communication: 0.000s
I20240503 23:26:44.794395 3762 raft_consensus.cc:1261] T a0e18d0975a54e3dbeb0a13cb226fefd P 83ffd7184ec94302bd1ee74b0fab3e17 [term 3 FOLLOWER]: Refusing update from remote peer 692da0319a2d4a81ad81f5d8f9f4c86b: Log matching property violated. Preceding OpId in replica: term: 3 index: 32. Preceding OpId from leader: term: 3 index: 33. (index mismatch)
I20240503 23:26:44.794991 4774 consensus_queue.cc:1035] T a0e18d0975a54e3dbeb0a13cb226fefd P 692da0319a2d4a81ad81f5d8f9f4c86b [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2f9bf2e5d9b4fc19f804797bca4a3ff" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 46261 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 33, Last known committed idx: 32, Time since last communication: 0.000s
I20240503 23:26:44.795800 4801 consensus_queue.cc:1035] T a0e18d0975a54e3dbeb0a13cb226fefd P 692da0319a2d4a81ad81f5d8f9f4c86b [LEADER]: Connected to new peer: Peer: permanent_uuid: "83ffd7184ec94302bd1ee74b0fab3e17" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 32909 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 33, Last known committed idx: 32, Time since last communication: 0.000s
I20240503 23:26:44.803282 4774 raft_consensus.cc:2932] T a0e18d0975a54e3dbeb0a13cb226fefd P 692da0319a2d4a81ad81f5d8f9f4c86b [term 3 LEADER]: Committing config change with OpId 3.33: config changed from index 32 to 33, e4086c099d7f4447b2b4733b6ca364df (127.0.100.4) changed from NON_VOTER to VOTER. New config: { opid_index: 33 OBSOLETE_local: false peers { permanent_uuid: "83ffd7184ec94302bd1ee74b0fab3e17" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 32909 } } peers { permanent_uuid: "a2f9bf2e5d9b4fc19f804797bca4a3ff" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 46261 } attrs { replace: true } } peers { permanent_uuid: "692da0319a2d4a81ad81f5d8f9f4c86b" member_type: VOTER last_known_addr { host: "127.0.100.5" port: 38561 } attrs { promote: false } } peers { permanent_uuid: "e4086c099d7f4447b2b4733b6ca364df" member_type: VOTER last_known_addr { host: "127.0.100.4" port: 35011 } attrs { promote: false } } }
I20240503 23:26:44.803635 4217 raft_consensus.cc:2932] T a0e18d0975a54e3dbeb0a13cb226fefd P e4086c099d7f4447b2b4733b6ca364df [term 3 FOLLOWER]: Committing config change with OpId 3.33: config changed from index 32 to 33, e4086c099d7f4447b2b4733b6ca364df (127.0.100.4) changed from NON_VOTER to VOTER. New config: { opid_index: 33 OBSOLETE_local: false peers { permanent_uuid: "83ffd7184ec94302bd1ee74b0fab3e17" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 32909 } } peers { permanent_uuid: "a2f9bf2e5d9b4fc19f804797bca4a3ff" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 46261 } attrs { replace: true } } peers { permanent_uuid: "692da0319a2d4a81ad81f5d8f9f4c86b" member_type: VOTER last_known_addr { host: "127.0.100.5" port: 38561 } attrs { promote: false } } peers { permanent_uuid: "e4086c099d7f4447b2b4733b6ca364df" member_type: VOTER last_known_addr { host: "127.0.100.4" port: 35011 } attrs { promote: false } } }
I20240503 23:26:44.807884 3762 raft_consensus.cc:2932] T a0e18d0975a54e3dbeb0a13cb226fefd P 83ffd7184ec94302bd1ee74b0fab3e17 [term 3 FOLLOWER]: Committing config change with OpId 3.33: config changed from index 32 to 33, e4086c099d7f4447b2b4733b6ca364df (127.0.100.4) changed from NON_VOTER to VOTER. New config: { opid_index: 33 OBSOLETE_local: false peers { permanent_uuid: "83ffd7184ec94302bd1ee74b0fab3e17" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 32909 } } peers { permanent_uuid: "a2f9bf2e5d9b4fc19f804797bca4a3ff" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 46261 } attrs { replace: true } } peers { permanent_uuid: "692da0319a2d4a81ad81f5d8f9f4c86b" member_type: VOTER last_known_addr { host: "127.0.100.5" port: 38561 } attrs { promote: false } } peers { permanent_uuid: "e4086c099d7f4447b2b4733b6ca364df" member_type: VOTER last_known_addr { host: "127.0.100.4" port: 35011 } attrs { promote: false } } }
I20240503 23:26:44.809504 3655 catalog_manager.cc:5484] T a0e18d0975a54e3dbeb0a13cb226fefd P e4086c099d7f4447b2b4733b6ca364df reported cstate change: config changed from index 32 to 33, e4086c099d7f4447b2b4733b6ca364df (127.0.100.4) changed from NON_VOTER to VOTER. New cstate: current_term: 3 leader_uuid: "692da0319a2d4a81ad81f5d8f9f4c86b" committed_config { opid_index: 33 OBSOLETE_local: false peers { permanent_uuid: "83ffd7184ec94302bd1ee74b0fab3e17" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 32909 } } peers { permanent_uuid: "a2f9bf2e5d9b4fc19f804797bca4a3ff" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 46261 } attrs { replace: true } } peers { permanent_uuid: "692da0319a2d4a81ad81f5d8f9f4c86b" member_type: VOTER last_known_addr { host: "127.0.100.5" port: 38561 } attrs { promote: false } } peers { permanent_uuid: "e4086c099d7f4447b2b4733b6ca364df" member_type: VOTER last_known_addr { host: "127.0.100.4" port: 35011 } attrs { promote: false } } }
I20240503 23:26:44.813026 3837 raft_consensus.cc:2932] T a0e18d0975a54e3dbeb0a13cb226fefd P a2f9bf2e5d9b4fc19f804797bca4a3ff [term 3 FOLLOWER]: Committing config change with OpId 3.33: config changed from index 32 to 33, e4086c099d7f4447b2b4733b6ca364df (127.0.100.4) changed from NON_VOTER to VOTER. New config: { opid_index: 33 OBSOLETE_local: false peers { permanent_uuid: "83ffd7184ec94302bd1ee74b0fab3e17" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 32909 } } peers { permanent_uuid: "a2f9bf2e5d9b4fc19f804797bca4a3ff" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 46261 } attrs { replace: true } } peers { permanent_uuid: "692da0319a2d4a81ad81f5d8f9f4c86b" member_type: VOTER last_known_addr { host: "127.0.100.5" port: 38561 } attrs { promote: false } } peers { permanent_uuid: "e4086c099d7f4447b2b4733b6ca364df" member_type: VOTER last_known_addr { host: "127.0.100.4" port: 35011 } attrs { promote: false } } }
I20240503 23:26:44.819128 4293 consensus_queue.cc:237] T a0e18d0975a54e3dbeb0a13cb226fefd P 692da0319a2d4a81ad81f5d8f9f4c86b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 33, Committed index: 33, Last appended: 3.33, Last appended by leader: 30, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 34 OBSOLETE_local: false peers { permanent_uuid: "83ffd7184ec94302bd1ee74b0fab3e17" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 32909 } } peers { permanent_uuid: "692da0319a2d4a81ad81f5d8f9f4c86b" member_type: VOTER last_known_addr { host: "127.0.100.5" port: 38561 } attrs { promote: false } } peers { permanent_uuid: "e4086c099d7f4447b2b4733b6ca364df" member_type: VOTER last_known_addr { host: "127.0.100.4" port: 35011 } attrs { promote: false } }
I20240503 23:26:44.821089 4217 raft_consensus.cc:1261] T a0e18d0975a54e3dbeb0a13cb226fefd P e4086c099d7f4447b2b4733b6ca364df [term 3 FOLLOWER]: Refusing update from remote peer 692da0319a2d4a81ad81f5d8f9f4c86b: Log matching property violated. Preceding OpId in replica: term: 3 index: 33. Preceding OpId from leader: term: 3 index: 34. (index mismatch)
I20240503 23:26:44.822356 4774 consensus_queue.cc:1035] T a0e18d0975a54e3dbeb0a13cb226fefd P 692da0319a2d4a81ad81f5d8f9f4c86b [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4086c099d7f4447b2b4733b6ca364df" member_type: VOTER last_known_addr { host: "127.0.100.4" port: 35011 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 34, Last known committed idx: 33, Time since last communication: 0.000s
I20240503 23:26:44.823812 3762 raft_consensus.cc:1261] T a0e18d0975a54e3dbeb0a13cb226fefd P 83ffd7184ec94302bd1ee74b0fab3e17 [term 3 FOLLOWER]: Refusing update from remote peer 692da0319a2d4a81ad81f5d8f9f4c86b: Log matching property violated. Preceding OpId in replica: term: 3 index: 33. Preceding OpId from leader: term: 3 index: 34. (index mismatch)
I20240503 23:26:44.824304 4800 consensus_queue.cc:1035] T a0e18d0975a54e3dbeb0a13cb226fefd P 692da0319a2d4a81ad81f5d8f9f4c86b [LEADER]: Connected to new peer: Peer: permanent_uuid: "83ffd7184ec94302bd1ee74b0fab3e17" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 32909 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 34, Last known committed idx: 33, Time since last communication: 0.000s
I20240503 23:26:44.825379 4774 raft_consensus.cc:2932] T a0e18d0975a54e3dbeb0a13cb226fefd P 692da0319a2d4a81ad81f5d8f9f4c86b [term 3 LEADER]: Committing config change with OpId 3.34: config changed from index 33 to 34, VOTER a2f9bf2e5d9b4fc19f804797bca4a3ff (127.0.100.2) evicted. New config: { opid_index: 34 OBSOLETE_local: false peers { permanent_uuid: "83ffd7184ec94302bd1ee74b0fab3e17" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 32909 } } peers { permanent_uuid: "692da0319a2d4a81ad81f5d8f9f4c86b" member_type: VOTER last_known_addr { host: "127.0.100.5" port: 38561 } attrs { promote: false } } peers { permanent_uuid: "e4086c099d7f4447b2b4733b6ca364df" member_type: VOTER last_known_addr { host: "127.0.100.4" port: 35011 } attrs { promote: false } } }
I20240503 23:26:44.826045 4216 raft_consensus.cc:2932] T a0e18d0975a54e3dbeb0a13cb226fefd P e4086c099d7f4447b2b4733b6ca364df [term 3 FOLLOWER]: Committing config change with OpId 3.34: config changed from index 33 to 34, VOTER a2f9bf2e5d9b4fc19f804797bca4a3ff (127.0.100.2) evicted. New config: { opid_index: 34 OBSOLETE_local: false peers { permanent_uuid: "83ffd7184ec94302bd1ee74b0fab3e17" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 32909 } } peers { permanent_uuid: "692da0319a2d4a81ad81f5d8f9f4c86b" member_type: VOTER last_known_addr { host: "127.0.100.5" port: 38561 } attrs { promote: false } } peers { permanent_uuid: "e4086c099d7f4447b2b4733b6ca364df" member_type: VOTER last_known_addr { host: "127.0.100.4" port: 35011 } attrs { promote: false } } }
I20240503 23:26:44.830497 3704 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet a0e18d0975a54e3dbeb0a13cb226fefd with cas_config_opid_index 33: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 23:26:44.830585 3659 catalog_manager.cc:5484] T a0e18d0975a54e3dbeb0a13cb226fefd P e4086c099d7f4447b2b4733b6ca364df reported cstate change: config changed from index 33 to 34, VOTER a2f9bf2e5d9b4fc19f804797bca4a3ff (127.0.100.2) evicted. New cstate: current_term: 3 leader_uuid: "692da0319a2d4a81ad81f5d8f9f4c86b" committed_config { opid_index: 34 OBSOLETE_local: false peers { permanent_uuid: "83ffd7184ec94302bd1ee74b0fab3e17" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 32909 } } peers { permanent_uuid: "692da0319a2d4a81ad81f5d8f9f4c86b" member_type: VOTER last_known_addr { host: "127.0.100.5" port: 38561 } attrs { promote: false } } peers { permanent_uuid: "e4086c099d7f4447b2b4733b6ca364df" member_type: VOTER last_known_addr { host: "127.0.100.4" port: 35011 } attrs { promote: false } } }
I20240503 23:26:44.833526 3762 raft_consensus.cc:2932] T a0e18d0975a54e3dbeb0a13cb226fefd P 83ffd7184ec94302bd1ee74b0fab3e17 [term 3 FOLLOWER]: Committing config change with OpId 3.34: config changed from index 33 to 34, VOTER a2f9bf2e5d9b4fc19f804797bca4a3ff (127.0.100.2) evicted. New config: { opid_index: 34 OBSOLETE_local: false peers { permanent_uuid: "83ffd7184ec94302bd1ee74b0fab3e17" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 32909 } } peers { permanent_uuid: "692da0319a2d4a81ad81f5d8f9f4c86b" member_type: VOTER last_known_addr { host: "127.0.100.5" port: 38561 } attrs { promote: false } } peers { permanent_uuid: "e4086c099d7f4447b2b4733b6ca364df" member_type: VOTER last_known_addr { host: "127.0.100.4" port: 35011 } attrs { promote: false } } }
I20240503 23:26:44.835726 3824 tablet_service.cc:1514] Processing DeleteTablet for tablet a0e18d0975a54e3dbeb0a13cb226fefd with delete_type TABLET_DATA_TOMBSTONED (TS a2f9bf2e5d9b4fc19f804797bca4a3ff not found in new config with opid_index 34) from {username='slave'} at 127.0.0.1:45354
I20240503 23:26:44.836463 4810 tablet_replica.cc:330] T a0e18d0975a54e3dbeb0a13cb226fefd P a2f9bf2e5d9b4fc19f804797bca4a3ff: stopping tablet replica
I20240503 23:26:44.836910 4810 raft_consensus.cc:2229] T a0e18d0975a54e3dbeb0a13cb226fefd P a2f9bf2e5d9b4fc19f804797bca4a3ff [term 3 FOLLOWER]: Raft consensus shutting down.
I20240503 23:26:44.837288 4810 raft_consensus.cc:2258] T a0e18d0975a54e3dbeb0a13cb226fefd P a2f9bf2e5d9b4fc19f804797bca4a3ff [term 3 FOLLOWER]: Raft consensus is shut down!
I20240503 23:26:44.839035 4810 ts_tablet_manager.cc:1875] T a0e18d0975a54e3dbeb0a13cb226fefd P a2f9bf2e5d9b4fc19f804797bca4a3ff: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 23:26:44.846611 4810 ts_tablet_manager.cc:1888] T a0e18d0975a54e3dbeb0a13cb226fefd P a2f9bf2e5d9b4fc19f804797bca4a3ff: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 3.33
I20240503 23:26:44.846801 4810 log.cc:1196] T a0e18d0975a54e3dbeb0a13cb226fefd P a2f9bf2e5d9b4fc19f804797bca4a3ff: Deleting WAL directory at /tmp/dist-test-taskZhIiPz/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714778688451250-400-0/minicluster-data/ts-1-root/wals/a0e18d0975a54e3dbeb0a13cb226fefd
I20240503 23:26:44.847695 3649 catalog_manager.cc:4830] TS a2f9bf2e5d9b4fc19f804797bca4a3ff (127.0.100.2:46261): tablet a0e18d0975a54e3dbeb0a13cb226fefd (table test-workload [id=1345b2a336c84ec2b1de62853e7fa3f6]) successfully deleted
I20240503 23:26:44.894361 4812 raft_consensus.cc:1050] T 55cf664d5e874c0d9b9adf8bdc9c43ed P e4086c099d7f4447b2b4733b6ca364df: attempting to promote NON_VOTER 692da0319a2d4a81ad81f5d8f9f4c86b to VOTER
I20240503 23:26:44.895264 4812 consensus_queue.cc:237] T 55cf664d5e874c0d9b9adf8bdc9c43ed P e4086c099d7f4447b2b4733b6ca364df [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 34, Committed index: 34, Last appended: 2.34, Last appended by leader: 30, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 35 OBSOLETE_local: false peers { permanent_uuid: "a2f9bf2e5d9b4fc19f804797bca4a3ff" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 46261 } attrs { replace: true } } peers { permanent_uuid: "e4086c099d7f4447b2b4733b6ca364df" member_type: VOTER last_known_addr { host: "127.0.100.4" port: 35011 } attrs { promote: false } } peers { permanent_uuid: "4d1d4e261584467fbbce4c72fa18045a" member_type: VOTER last_known_addr { host: "127.0.100.6" port: 35349 } attrs { promote: false } } peers { permanent_uuid: "692da0319a2d4a81ad81f5d8f9f4c86b" member_type: VOTER last_known_addr { host: "127.0.100.5" port: 38561 } attrs { promote: false } }
I20240503 23:26:44.898238 3837 raft_consensus.cc:1261] T 55cf664d5e874c0d9b9adf8bdc9c43ed P a2f9bf2e5d9b4fc19f804797bca4a3ff [term 2 FOLLOWER]: Refusing update from remote peer e4086c099d7f4447b2b4733b6ca364df: Log matching property violated. Preceding OpId in replica: term: 2 index: 34. Preceding OpId from leader: term: 2 index: 35. (index mismatch)
I20240503 23:26:44.898339 4293 raft_consensus.cc:1261] T 55cf664d5e874c0d9b9adf8bdc9c43ed P 692da0319a2d4a81ad81f5d8f9f4c86b [term 2 LEARNER]: Refusing update from remote peer e4086c099d7f4447b2b4733b6ca364df: Log matching property violated. Preceding OpId in replica: term: 2 index: 34. Preceding OpId from leader: term: 2 index: 35. (index mismatch)
I20240503 23:26:44.898371 4366 raft_consensus.cc:1261] T 55cf664d5e874c0d9b9adf8bdc9c43ed P 4d1d4e261584467fbbce4c72fa18045a [term 2 FOLLOWER]: Refusing update from remote peer e4086c099d7f4447b2b4733b6ca364df: Log matching property violated. Preceding OpId in replica: term: 2 index: 34. Preceding OpId from leader: term: 2 index: 35. (index mismatch)
I20240503 23:26:44.899690 4778 consensus_queue.cc:1035] T 55cf664d5e874c0d9b9adf8bdc9c43ed P e4086c099d7f4447b2b4733b6ca364df [LEADER]: Connected to new peer: Peer: permanent_uuid: "692da0319a2d4a81ad81f5d8f9f4c86b" member_type: VOTER last_known_addr { host: "127.0.100.5" port: 38561 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 35, Last known committed idx: 34, Time since last communication: 0.000s
I20240503 23:26:44.900080 4811 consensus_queue.cc:1035] T 55cf664d5e874c0d9b9adf8bdc9c43ed P e4086c099d7f4447b2b4733b6ca364df [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2f9bf2e5d9b4fc19f804797bca4a3ff" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 46261 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 35, Last known committed idx: 34, Time since last communication: 0.000s
I20240503 23:26:44.900599 4811 consensus_queue.cc:1035] T 55cf664d5e874c0d9b9adf8bdc9c43ed P e4086c099d7f4447b2b4733b6ca364df [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d1d4e261584467fbbce4c72fa18045a" member_type: VOTER last_known_addr { host: "127.0.100.6" port: 35349 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 35, Last known committed idx: 34, Time since last communication: 0.000s
I20240503 23:26:44.904960 4778 raft_consensus.cc:2932] T 55cf664d5e874c0d9b9adf8bdc9c43ed P e4086c099d7f4447b2b4733b6ca364df [term 2 LEADER]: Committing config change with OpId 2.35: config changed from index 34 to 35, 692da0319a2d4a81ad81f5d8f9f4c86b (127.0.100.5) changed from NON_VOTER to VOTER. New config: { opid_index: 35 OBSOLETE_local: false peers { permanent_uuid: "a2f9bf2e5d9b4fc19f804797bca4a3ff" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 46261 } attrs { replace: true } } peers { permanent_uuid: "e4086c099d7f4447b2b4733b6ca364df" member_type: VOTER last_known_addr { host: "127.0.100.4" port: 35011 } attrs { promote: false } } peers { permanent_uuid: "4d1d4e261584467fbbce4c72fa18045a" member_type: VOTER last_known_addr { host: "127.0.100.6" port: 35349 } attrs { promote: false } } peers { permanent_uuid: "692da0319a2d4a81ad81f5d8f9f4c86b" member_type: VOTER last_known_addr { host: "127.0.100.5" port: 38561 } attrs { promote: false } } }
I20240503 23:26:44.905550 4366 raft_consensus.cc:2932] T 55cf664d5e874c0d9b9adf8bdc9c43ed P 4d1d4e261584467fbbce4c72fa18045a [term 2 FOLLOWER]: Committing config change with OpId 2.35: config changed from index 34 to 35, 692da0319a2d4a81ad81f5d8f9f4c86b (127.0.100.5) changed from NON_VOTER to VOTER. New config: { opid_index: 35 OBSOLETE_local: false peers { permanent_uuid: "a2f9bf2e5d9b4fc19f804797bca4a3ff" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 46261 } attrs { replace: true } } peers { permanent_uuid: "e4086c099d7f4447b2b4733b6ca364df" member_type: VOTER last_known_addr { host: "127.0.100.4" port: 35011 } attrs { promote: false } } peers { permanent_uuid: "4d1d4e261584467fbbce4c72fa18045a" member_type: VOTER last_known_addr { host: "127.0.100.6" port: 35349 } attrs { promote: false } } peers { permanent_uuid: "692da0319a2d4a81ad81f5d8f9f4c86b" member_type: VOTER last_known_addr { host: "127.0.100.5" port: 38561 } attrs { promote: false } } }
I20240503 23:26:44.909396 4293 raft_consensus.cc:2932] T 55cf664d5e874c0d9b9adf8bdc9c43ed P 692da0319a2d4a81ad81f5d8f9f4c86b [term 2 FOLLOWER]: Committing config change with OpId 2.35: config changed from index 34 to 35, 692da0319a2d4a81ad81f5d8f9f4c86b (127.0.100.5) changed from NON_VOTER to VOTER. New config: { opid_index: 35 OBSOLETE_local: false peers { permanent_uuid: "a2f9bf2e5d9b4fc19f804797bca4a3ff" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 46261 } attrs { replace: true } } peers { permanent_uuid: "e4086c099d7f4447b2b4733b6ca364df" member_type: VOTER last_known_addr { host: "127.0.100.4" port: 35011 } attrs { promote: false } } peers { permanent_uuid: "4d1d4e261584467fbbce4c72fa18045a" member_type: VOTER last_known_addr { host: "127.0.100.6" port: 35349 } attrs { promote: false } } peers { permanent_uuid: "692da0319a2d4a81ad81f5d8f9f4c86b" member_type: VOTER last_known_addr { host: "127.0.100.5" port: 38561 } attrs { promote: false } } }
I20240503 23:26:44.913612 3837 raft_consensus.cc:2932] T 55cf664d5e874c0d9b9adf8bdc9c43ed P a2f9bf2e5d9b4fc19f804797bca4a3ff [term 2 FOLLOWER]: Committing config change with OpId 2.35: config changed from index 34 to 35, 692da0319a2d4a81ad81f5d8f9f4c86b (127.0.100.5) changed from NON_VOTER to VOTER. New config: { opid_index: 35 OBSOLETE_local: false peers { permanent_uuid: "a2f9bf2e5d9b4fc19f804797bca4a3ff" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 46261 } attrs { replace: true } } peers { permanent_uuid: "e4086c099d7f4447b2b4733b6ca364df" member_type: VOTER last_known_addr { host: "127.0.100.4" port: 35011 } attrs { promote: false } } peers { permanent_uuid: "4d1d4e261584467fbbce4c72fa18045a" member_type: VOTER last_known_addr { host: "127.0.100.6" port: 35349 } attrs { promote: false } } peers { permanent_uuid: "692da0319a2d4a81ad81f5d8f9f4c86b" member_type: VOTER last_known_addr { host: "127.0.100.5" port: 38561 } attrs { promote: false } } }
I20240503 23:26:44.914055 3662 catalog_manager.cc:5484] T 55cf664d5e874c0d9b9adf8bdc9c43ed P e4086c099d7f4447b2b4733b6ca364df reported cstate change: config changed from index 34 to 35, 692da0319a2d4a81ad81f5d8f9f4c86b (127.0.100.5) changed from NON_VOTER to VOTER. New cstate: current_term: 2 leader_uuid: "e4086c099d7f4447b2b4733b6ca364df" committed_config { opid_index: 35 OBSOLETE_local: false peers { permanent_uuid: "a2f9bf2e5d9b4fc19f804797bca4a3ff" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 46261 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e4086c099d7f4447b2b4733b6ca364df" member_type: VOTER last_known_addr { host: "127.0.100.4" port: 35011 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4d1d4e261584467fbbce4c72fa18045a" member_type: VOTER last_known_addr { host: "127.0.100.6" port: 35349 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "692da0319a2d4a81ad81f5d8f9f4c86b" member_type: VOTER last_known_addr { host: "127.0.100.5" port: 38561 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 23:26:44.919315 4216 consensus_queue.cc:237] T 55cf664d5e874c0d9b9adf8bdc9c43ed P e4086c099d7f4447b2b4733b6ca364df [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 35, Committed index: 35, Last appended: 2.35, Last appended by leader: 30, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 36 OBSOLETE_local: false peers { permanent_uuid: "e4086c099d7f4447b2b4733b6ca364df" member_type: VOTER last_known_addr { host: "127.0.100.4" port: 35011 } attrs { promote: false } } peers { permanent_uuid: "4d1d4e261584467fbbce4c72fa18045a" member_type: VOTER last_known_addr { host: "127.0.100.6" port: 35349 } attrs { promote: false } } peers { permanent_uuid: "692da0319a2d4a81ad81f5d8f9f4c86b" member_type: VOTER last_known_addr { host: "127.0.100.5" port: 38561 } attrs { promote: false } }
I20240503 23:26:44.921298 4293 raft_consensus.cc:1261] T 55cf664d5e874c0d9b9adf8bdc9c43ed P 692da0319a2d4a81ad81f5d8f9f4c86b [term 2 FOLLOWER]: Refusing update from remote peer e4086c099d7f4447b2b4733b6ca364df: Log matching property violated. Preceding OpId in replica: term: 2 index: 35. Preceding OpId from leader: term: 2 index: 36. (index mismatch)
I20240503 23:26:44.921909 4811 consensus_queue.cc:1035] T 55cf664d5e874c0d9b9adf8bdc9c43ed P e4086c099d7f4447b2b4733b6ca364df [LEADER]: Connected to new peer: Peer: permanent_uuid: "692da0319a2d4a81ad81f5d8f9f4c86b" member_type: VOTER last_known_addr { host: "127.0.100.5" port: 38561 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 36, Last known committed idx: 35, Time since last communication: 0.000s
I20240503 23:26:44.922935 4367 raft_consensus.cc:1261] T 55cf664d5e874c0d9b9adf8bdc9c43ed P 4d1d4e261584467fbbce4c72fa18045a [term 2 FOLLOWER]: Refusing update from remote peer e4086c099d7f4447b2b4733b6ca364df: Log matching property violated. Preceding OpId in replica: term: 2 index: 35. Preceding OpId from leader: term: 2 index: 36. (index mismatch)
I20240503 23:26:44.923544 4812 consensus_queue.cc:1035] T 55cf664d5e874c0d9b9adf8bdc9c43ed P e4086c099d7f4447b2b4733b6ca364df [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d1d4e261584467fbbce4c72fa18045a" member_type: VOTER last_known_addr { host: "127.0.100.6" port: 35349 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 36, Last known committed idx: 35, Time since last communication: 0.000s
I20240503 23:26:44.926653 4812 raft_consensus.cc:2932] T 55cf664d5e874c0d9b9adf8bdc9c43ed P e4086c099d7f4447b2b4733b6ca364df [term 2 LEADER]: Committing config change with OpId 2.36: config changed from index 35 to 36, VOTER a2f9bf2e5d9b4fc19f804797bca4a3ff (127.0.100.2) evicted. New config: { opid_index: 36 OBSOLETE_local: false peers { permanent_uuid: "e4086c099d7f4447b2b4733b6ca364df" member_type: VOTER last_known_addr { host: "127.0.100.4" port: 35011 } attrs { promote: false } } peers { permanent_uuid: "4d1d4e261584467fbbce4c72fa18045a" member_type: VOTER last_known_addr { host: "127.0.100.6" port: 35349 } attrs { promote: false } } peers { permanent_uuid: "692da0319a2d4a81ad81f5d8f9f4c86b" member_type: VOTER last_known_addr { host: "127.0.100.5" port: 38561 } attrs { promote: false } } }
I20240503 23:26:44.927326 4367 raft_consensus.cc:2932] T 55cf664d5e874c0d9b9adf8bdc9c43ed P 4d1d4e261584467fbbce4c72fa18045a [term 2 FOLLOWER]: Committing config change with OpId 2.36: config changed from index 35 to 36, VOTER a2f9bf2e5d9b4fc19f804797bca4a3ff (127.0.100.2) evicted. New config: { opid_index: 36 OBSOLETE_local: false peers { permanent_uuid: "e4086c099d7f4447b2b4733b6ca364df" member_type: VOTER last_known_addr { host: "127.0.100.4" port: 35011 } attrs { promote: false } } peers { permanent_uuid: "4d1d4e261584467fbbce4c72fa18045a" member_type: VOTER last_known_addr { host: "127.0.100.6" port: 35349 } attrs { promote: false } } peers { permanent_uuid: "692da0319a2d4a81ad81f5d8f9f4c86b" member_type: VOTER last_known_addr { host: "127.0.100.5" port: 38561 } attrs { promote: false } } }
I20240503 23:26:44.927359 4293 raft_consensus.cc:2932] T 55cf664d5e874c0d9b9adf8bdc9c43ed P 692da0319a2d4a81ad81f5d8f9f4c86b [term 2 FOLLOWER]: Committing config change with OpId 2.36: config changed from index 35 to 36, VOTER a2f9bf2e5d9b4fc19f804797bca4a3ff (127.0.100.2) evicted. New config: { opid_index: 36 OBSOLETE_local: false peers { permanent_uuid: "e4086c099d7f4447b2b4733b6ca364df" member_type: VOTER last_known_addr { host: "127.0.100.4" port: 35011 } attrs { promote: false } } peers { permanent_uuid: "4d1d4e261584467fbbce4c72fa18045a" member_type: VOTER last_known_addr { host: "127.0.100.6" port: 35349 } attrs { promote: false } } peers { permanent_uuid: "692da0319a2d4a81ad81f5d8f9f4c86b" member_type: VOTER last_known_addr { host: "127.0.100.5" port: 38561 } attrs { promote: false } } }
I20240503 23:26:44.930402 3704 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 55cf664d5e874c0d9b9adf8bdc9c43ed with cas_config_opid_index 35: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 23:26:44.932672 3659 catalog_manager.cc:5484] T 55cf664d5e874c0d9b9adf8bdc9c43ed P 4d1d4e261584467fbbce4c72fa18045a reported cstate change: config changed from index 35 to 36, VOTER a2f9bf2e5d9b4fc19f804797bca4a3ff (127.0.100.2) evicted. New cstate: current_term: 2 leader_uuid: "e4086c099d7f4447b2b4733b6ca364df" committed_config { opid_index: 36 OBSOLETE_local: false peers { permanent_uuid: "e4086c099d7f4447b2b4733b6ca364df" member_type: VOTER last_known_addr { host: "127.0.100.4" port: 35011 } attrs { promote: false } } peers { permanent_uuid: "4d1d4e261584467fbbce4c72fa18045a" member_type: VOTER last_known_addr { host: "127.0.100.6" port: 35349 } attrs { promote: false } } peers { permanent_uuid: "692da0319a2d4a81ad81f5d8f9f4c86b" member_type: VOTER last_known_addr { host: "127.0.100.5" port: 38561 } attrs { promote: false } } }
I20240503 23:26:44.937044 3824 tablet_service.cc:1514] Processing DeleteTablet for tablet 55cf664d5e874c0d9b9adf8bdc9c43ed with delete_type TABLET_DATA_TOMBSTONED (TS a2f9bf2e5d9b4fc19f804797bca4a3ff not found in new config with opid_index 36) from {username='slave'} at 127.0.0.1:45354
I20240503 23:26:44.938076 4810 tablet_replica.cc:330] T 55cf664d5e874c0d9b9adf8bdc9c43ed P a2f9bf2e5d9b4fc19f804797bca4a3ff: stopping tablet replica
I20240503 23:26:44.938040 4367 raft_consensus.cc:1203] T 60267cb4849d4823a0afb2380305905f P 4d1d4e261584467fbbce4c72fa18045a [term 1 LEARNER]: Deduplicated request from leader. Original: 1.28->[1.29-1.29] Dedup: 1.29->[]
I20240503 23:26:44.938390 4810 raft_consensus.cc:2229] T 55cf664d5e874c0d9b9adf8bdc9c43ed P a2f9bf2e5d9b4fc19f804797bca4a3ff [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 23:26:44.938822 4810 raft_consensus.cc:2258] T 55cf664d5e874c0d9b9adf8bdc9c43ed P a2f9bf2e5d9b4fc19f804797bca4a3ff [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 23:26:44.940207 4810 ts_tablet_manager.cc:1875] T 55cf664d5e874c0d9b9adf8bdc9c43ed P a2f9bf2e5d9b4fc19f804797bca4a3ff: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 23:26:44.947000 4810 ts_tablet_manager.cc:1888] T 55cf664d5e874c0d9b9adf8bdc9c43ed P a2f9bf2e5d9b4fc19f804797bca4a3ff: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.35
I20240503 23:26:44.947216 4810 log.cc:1196] T 55cf664d5e874c0d9b9adf8bdc9c43ed P a2f9bf2e5d9b4fc19f804797bca4a3ff: Deleting WAL directory at /tmp/dist-test-taskZhIiPz/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714778688451250-400-0/minicluster-data/ts-1-root/wals/55cf664d5e874c0d9b9adf8bdc9c43ed
I20240503 23:26:44.948019 3649 catalog_manager.cc:4830] TS a2f9bf2e5d9b4fc19f804797bca4a3ff (127.0.100.2:46261): tablet 55cf664d5e874c0d9b9adf8bdc9c43ed (table test-workload [id=1345b2a336c84ec2b1de62853e7fa3f6]) successfully deleted
I20240503 23:26:45.052470 4818 raft_consensus.cc:1050] T 059f6233ea514fe69b8c6504efabeff0 P a2f9bf2e5d9b4fc19f804797bca4a3ff: attempting to promote NON_VOTER 4d1d4e261584467fbbce4c72fa18045a to VOTER
I20240503 23:26:45.053701 4818 consensus_queue.cc:237] T 059f6233ea514fe69b8c6504efabeff0 P a2f9bf2e5d9b4fc19f804797bca4a3ff [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 28, Committed index: 28, Last appended: 2.28, Last appended by leader: 1, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "83ffd7184ec94302bd1ee74b0fab3e17" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 32909 } attrs { replace: true } } peers { permanent_uuid: "a2f9bf2e5d9b4fc19f804797bca4a3ff" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 46261 } } peers { permanent_uuid: "a3eecbaec43d4954b1096bc3121e824c" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 40649 } } peers { permanent_uuid: "4d1d4e261584467fbbce4c72fa18045a" member_type: VOTER last_known_addr { host: "127.0.100.6" port: 35349 } attrs { promote: false } }
I20240503 23:26:45.056612 4367 raft_consensus.cc:1261] T 059f6233ea514fe69b8c6504efabeff0 P 4d1d4e261584467fbbce4c72fa18045a [term 2 LEARNER]: Refusing update from remote peer a2f9bf2e5d9b4fc19f804797bca4a3ff: Log matching property violated. Preceding OpId in replica: term: 2 index: 28. Preceding OpId from leader: term: 2 index: 29. (index mismatch)
I20240503 23:26:45.056900 3907 raft_consensus.cc:1261] T 059f6233ea514fe69b8c6504efabeff0 P a3eecbaec43d4954b1096bc3121e824c [term 2 FOLLOWER]: Refusing update from remote peer a2f9bf2e5d9b4fc19f804797bca4a3ff: Log matching property violated. Preceding OpId in replica: term: 2 index: 28. Preceding OpId from leader: term: 2 index: 29. (index mismatch)
I20240503 23:26:45.057139 3762 raft_consensus.cc:1261] T 059f6233ea514fe69b8c6504efabeff0 P 83ffd7184ec94302bd1ee74b0fab3e17 [term 2 FOLLOWER]: Refusing update from remote peer a2f9bf2e5d9b4fc19f804797bca4a3ff: Log matching property violated. Preceding OpId in replica: term: 2 index: 28. Preceding OpId from leader: term: 2 index: 29. (index mismatch)
I20240503 23:26:45.057189 4764 consensus_queue.cc:1035] T 059f6233ea514fe69b8c6504efabeff0 P a2f9bf2e5d9b4fc19f804797bca4a3ff [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d1d4e261584467fbbce4c72fa18045a" member_type: VOTER last_known_addr { host: "127.0.100.6" port: 35349 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.000s
I20240503 23:26:45.057533 4817 consensus_queue.cc:1035] T 059f6233ea514fe69b8c6504efabeff0 P a2f9bf2e5d9b4fc19f804797bca4a3ff [LEADER]: Connected to new peer: Peer: permanent_uuid: "a3eecbaec43d4954b1096bc3121e824c" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 40649 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.000s
I20240503 23:26:45.058172 4764 consensus_queue.cc:1035] T 059f6233ea514fe69b8c6504efabeff0 P a2f9bf2e5d9b4fc19f804797bca4a3ff [LEADER]: Connected to new peer: Peer: permanent_uuid: "83ffd7184ec94302bd1ee74b0fab3e17" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 32909 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.000s
I20240503 23:26:45.062484 4818 raft_consensus.cc:2932] T 059f6233ea514fe69b8c6504efabeff0 P a2f9bf2e5d9b4fc19f804797bca4a3ff [term 2 LEADER]: Committing config change with OpId 2.29: config changed from index 28 to 29, 4d1d4e261584467fbbce4c72fa18045a (127.0.100.6) changed from NON_VOTER to VOTER. New config: { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "83ffd7184ec94302bd1ee74b0fab3e17" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 32909 } attrs { replace: true } } peers { permanent_uuid: "a2f9bf2e5d9b4fc19f804797bca4a3ff" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 46261 } } peers { permanent_uuid: "a3eecbaec43d4954b1096bc3121e824c" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 40649 } } peers { permanent_uuid: "4d1d4e261584467fbbce4c72fa18045a" member_type: VOTER last_known_addr { host: "127.0.100.6" port: 35349 } attrs { promote: false } } }
I20240503 23:26:45.063242 3907 raft_consensus.cc:2932] T 059f6233ea514fe69b8c6504efabeff0 P a3eecbaec43d4954b1096bc3121e824c [term 2 FOLLOWER]: Committing config change with OpId 2.29: config changed from index 28 to 29, 4d1d4e261584467fbbce4c72fa18045a (127.0.100.6) changed from NON_VOTER to VOTER. New config: { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "83ffd7184ec94302bd1ee74b0fab3e17" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 32909 } attrs { replace: true } } peers { permanent_uuid: "a2f9bf2e5d9b4fc19f804797bca4a3ff" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 46261 } } peers { permanent_uuid: "a3eecbaec43d4954b1096bc3121e824c" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 40649 } } peers { permanent_uuid: "4d1d4e261584467fbbce4c72fa18045a" member_type: VOTER last_known_addr { host: "127.0.100.6" port: 35349 } attrs { promote: false } } }
I20240503 23:26:45.069259 3655 catalog_manager.cc:5484] T 059f6233ea514fe69b8c6504efabeff0 P a2f9bf2e5d9b4fc19f804797bca4a3ff reported cstate change: config changed from index 28 to 29, 4d1d4e261584467fbbce4c72fa18045a (127.0.100.6) changed from NON_VOTER to VOTER. New cstate: current_term: 2 leader_uuid: "a2f9bf2e5d9b4fc19f804797bca4a3ff" committed_config { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "83ffd7184ec94302bd1ee74b0fab3e17" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 32909 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a2f9bf2e5d9b4fc19f804797bca4a3ff" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 46261 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a3eecbaec43d4954b1096bc3121e824c" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 40649 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4d1d4e261584467fbbce4c72fa18045a" member_type: VOTER last_known_addr { host: "127.0.100.6" port: 35349 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240503 23:26:45.069008 3760 raft_consensus.cc:2932] T 059f6233ea514fe69b8c6504efabeff0 P 83ffd7184ec94302bd1ee74b0fab3e17 [term 2 FOLLOWER]: Committing config change with OpId 2.29: config changed from index 28 to 29, 4d1d4e261584467fbbce4c72fa18045a (127.0.100.6) changed from NON_VOTER to VOTER. New config: { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "83ffd7184ec94302bd1ee74b0fab3e17" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 32909 } attrs { replace: true } } peers { permanent_uuid: "a2f9bf2e5d9b4fc19f804797bca4a3ff" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 46261 } } peers { permanent_uuid: "a3eecbaec43d4954b1096bc3121e824c" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 40649 } } peers { permanent_uuid: "4d1d4e261584467fbbce4c72fa18045a" member_type: VOTER last_known_addr { host: "127.0.100.6" port: 35349 } attrs { promote: false } } }
I20240503 23:26:45.080226 4367 raft_consensus.cc:2932] T 059f6233ea514fe69b8c6504efabeff0 P 4d1d4e261584467fbbce4c72fa18045a [term 2 FOLLOWER]: Committing config change with OpId 2.29: config changed from index 28 to 29, 4d1d4e261584467fbbce4c72fa18045a (127.0.100.6) changed from NON_VOTER to VOTER. New config: { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "83ffd7184ec94302bd1ee74b0fab3e17" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 32909 } attrs { replace: true } } peers { permanent_uuid: "a2f9bf2e5d9b4fc19f804797bca4a3ff" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 46261 } } peers { permanent_uuid: "a3eecbaec43d4954b1096bc3121e824c" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 40649 } } peers { permanent_uuid: "4d1d4e261584467fbbce4c72fa18045a" member_type: VOTER last_known_addr { host: "127.0.100.6" port: 35349 } attrs { promote: false } } }
I20240503 23:26:45.083629 3837 consensus_queue.cc:237] T 059f6233ea514fe69b8c6504efabeff0 P a2f9bf2e5d9b4fc19f804797bca4a3ff [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 29, Committed index: 29, Last appended: 2.29, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "a2f9bf2e5d9b4fc19f804797bca4a3ff" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 46261 } } peers { permanent_uuid: "a3eecbaec43d4954b1096bc3121e824c" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 40649 } } peers { permanent_uuid: "4d1d4e261584467fbbce4c72fa18045a" member_type: VOTER last_known_addr { host: "127.0.100.6" port: 35349 } attrs { promote: false } }
I20240503 23:26:45.085922 3907 raft_consensus.cc:1261] T 059f6233ea514fe69b8c6504efabeff0 P a3eecbaec43d4954b1096bc3121e824c [term 2 FOLLOWER]: Refusing update from remote peer a2f9bf2e5d9b4fc19f804797bca4a3ff: Log matching property violated. Preceding OpId in replica: term: 2 index: 29. Preceding OpId from leader: term: 2 index: 30. (index mismatch)
I20240503 23:26:45.086706 4818 consensus_queue.cc:1035] T 059f6233ea514fe69b8c6504efabeff0 P a2f9bf2e5d9b4fc19f804797bca4a3ff [LEADER]: Connected to new peer: Peer: permanent_uuid: "a3eecbaec43d4954b1096bc3121e824c" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 40649 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 30, Last known committed idx: 29, Time since last communication: 0.000s
I20240503 23:26:45.087977 4367 raft_consensus.cc:1261] T 059f6233ea514fe69b8c6504efabeff0 P 4d1d4e261584467fbbce4c72fa18045a [term 2 FOLLOWER]: Refusing update from remote peer a2f9bf2e5d9b4fc19f804797bca4a3ff: Log matching property violated. Preceding OpId in replica: term: 2 index: 29. Preceding OpId from leader: term: 2 index: 30. (index mismatch)
I20240503 23:26:45.088609 4818 consensus_queue.cc:1035] T 059f6233ea514fe69b8c6504efabeff0 P a2f9bf2e5d9b4fc19f804797bca4a3ff [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d1d4e261584467fbbce4c72fa18045a" member_type: VOTER last_known_addr { host: "127.0.100.6" port: 35349 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 30, Last known committed idx: 29, Time since last communication: 0.000s
I20240503 23:26:45.089453 4820 raft_consensus.cc:2932] T 059f6233ea514fe69b8c6504efabeff0 P a2f9bf2e5d9b4fc19f804797bca4a3ff [term 2 LEADER]: Committing config change with OpId 2.30: config changed from index 29 to 30, VOTER 83ffd7184ec94302bd1ee74b0fab3e17 (127.0.100.1) evicted. New config: { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "a2f9bf2e5d9b4fc19f804797bca4a3ff" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 46261 } } peers { permanent_uuid: "a3eecbaec43d4954b1096bc3121e824c" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 40649 } } peers { permanent_uuid: "4d1d4e261584467fbbce4c72fa18045a" member_type: VOTER last_known_addr { host: "127.0.100.6" port: 35349 } attrs { promote: false } } }
I20240503 23:26:45.090323 3907 raft_consensus.cc:2932] T 059f6233ea514fe69b8c6504efabeff0 P a3eecbaec43d4954b1096bc3121e824c [term 2 FOLLOWER]: Committing config change with OpId 2.30: config changed from index 29 to 30, VOTER 83ffd7184ec94302bd1ee74b0fab3e17 (127.0.100.1) evicted. New config: { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "a2f9bf2e5d9b4fc19f804797bca4a3ff" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 46261 } } peers { permanent_uuid: "a3eecbaec43d4954b1096bc3121e824c" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 40649 } } peers { permanent_uuid: "4d1d4e261584467fbbce4c72fa18045a" member_type: VOTER last_known_addr { host: "127.0.100.6" port: 35349 } attrs { promote: false } } }
I20240503 23:26:45.091894 4366 raft_consensus.cc:2932] T 059f6233ea514fe69b8c6504efabeff0 P 4d1d4e261584467fbbce4c72fa18045a [term 2 FOLLOWER]: Committing config change with OpId 2.30: config changed from index 29 to 30, VOTER 83ffd7184ec94302bd1ee74b0fab3e17 (127.0.100.1) evicted. New config: { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "a2f9bf2e5d9b4fc19f804797bca4a3ff" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 46261 } } peers { permanent_uuid: "a3eecbaec43d4954b1096bc3121e824c" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 40649 } } peers { permanent_uuid: "4d1d4e261584467fbbce4c72fa18045a" member_type: VOTER last_known_addr { host: "127.0.100.6" port: 35349 } attrs { promote: false } } }
I20240503 23:26:45.094277 3649 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 059f6233ea514fe69b8c6504efabeff0 with cas_config_opid_index 29: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 23:26:45.095893 3655 catalog_manager.cc:5484] T 059f6233ea514fe69b8c6504efabeff0 P a2f9bf2e5d9b4fc19f804797bca4a3ff reported cstate change: config changed from index 29 to 30, VOTER 83ffd7184ec94302bd1ee74b0fab3e17 (127.0.100.1) evicted. New cstate: current_term: 2 leader_uuid: "a2f9bf2e5d9b4fc19f804797bca4a3ff" committed_config { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "a2f9bf2e5d9b4fc19f804797bca4a3ff" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 46261 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a3eecbaec43d4954b1096bc3121e824c" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 40649 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4d1d4e261584467fbbce4c72fa18045a" member_type: VOTER last_known_addr { host: "127.0.100.6" port: 35349 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 23:26:45.100768 3747 tablet_service.cc:1514] Processing DeleteTablet for tablet 059f6233ea514fe69b8c6504efabeff0 with delete_type TABLET_DATA_TOMBSTONED (TS 83ffd7184ec94302bd1ee74b0fab3e17 not found in new config with opid_index 30) from {username='slave'} at 127.0.0.1:35576
I20240503 23:26:45.101661 4825 tablet_replica.cc:330] T 059f6233ea514fe69b8c6504efabeff0 P 83ffd7184ec94302bd1ee74b0fab3e17: stopping tablet replica
I20240503 23:26:45.101984 4825 raft_consensus.cc:2229] T 059f6233ea514fe69b8c6504efabeff0 P 83ffd7184ec94302bd1ee74b0fab3e17 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 23:26:45.102289 4825 raft_consensus.cc:2258] T 059f6233ea514fe69b8c6504efabeff0 P 83ffd7184ec94302bd1ee74b0fab3e17 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 23:26:45.105118 4825 ts_tablet_manager.cc:1875] T 059f6233ea514fe69b8c6504efabeff0 P 83ffd7184ec94302bd1ee74b0fab3e17: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 23:26:45.113261 4825 ts_tablet_manager.cc:1888] T 059f6233ea514fe69b8c6504efabeff0 P 83ffd7184ec94302bd1ee74b0fab3e17: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.29
I20240503 23:26:45.113443 4825 log.cc:1196] T 059f6233ea514fe69b8c6504efabeff0 P 83ffd7184ec94302bd1ee74b0fab3e17: Deleting WAL directory at /tmp/dist-test-taskZhIiPz/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714778688451250-400-0/minicluster-data/ts-0-root/wals/059f6233ea514fe69b8c6504efabeff0
I20240503 23:26:45.114140 3648 catalog_manager.cc:4830] TS 83ffd7184ec94302bd1ee74b0fab3e17 (127.0.100.1:32909): tablet 059f6233ea514fe69b8c6504efabeff0 (table test-workload [id=1345b2a336c84ec2b1de62853e7fa3f6]) successfully deleted
I20240503 23:26:45.188730 4775 raft_consensus.cc:1050] T 481cd0f8c8d147adac217915a53ff6d9 P 83ffd7184ec94302bd1ee74b0fab3e17: attempting to promote NON_VOTER 692da0319a2d4a81ad81f5d8f9f4c86b to VOTER
I20240503 23:26:45.189661 4775 consensus_queue.cc:237] T 481cd0f8c8d147adac217915a53ff6d9 P 83ffd7184ec94302bd1ee74b0fab3e17 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 28, Committed index: 28, Last appended: 2.28, Last appended by leader: 26, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "83ffd7184ec94302bd1ee74b0fab3e17" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 32909 } } peers { permanent_uuid: "a2f9bf2e5d9b4fc19f804797bca4a3ff" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 46261 } } peers { permanent_uuid: "a3eecbaec43d4954b1096bc3121e824c" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 40649 } attrs { replace: true } } peers { permanent_uuid: "692da0319a2d4a81ad81f5d8f9f4c86b" member_type: VOTER last_known_addr { host: "127.0.100.5" port: 38561 } attrs { promote: false } }
I20240503 23:26:45.192518 3837 raft_consensus.cc:1261] T 481cd0f8c8d147adac217915a53ff6d9 P a2f9bf2e5d9b4fc19f804797bca4a3ff [term 2 FOLLOWER]: Refusing update from remote peer 83ffd7184ec94302bd1ee74b0fab3e17: Log matching property violated. Preceding OpId in replica: term: 2 index: 28. Preceding OpId from leader: term: 2 index: 29. (index mismatch)
I20240503 23:26:45.192637 3907 raft_consensus.cc:1261] T 481cd0f8c8d147adac217915a53ff6d9 P a3eecbaec43d4954b1096bc3121e824c [term 2 FOLLOWER]: Refusing update from remote peer 83ffd7184ec94302bd1ee74b0fab3e17: Log matching property violated. Preceding OpId in replica: term: 2 index: 28. Preceding OpId from leader: term: 2 index: 29. (index mismatch)
I20240503 23:26:45.192782 4293 raft_consensus.cc:1261] T 481cd0f8c8d147adac217915a53ff6d9 P 692da0319a2d4a81ad81f5d8f9f4c86b [term 2 LEARNER]: Refusing update from remote peer 83ffd7184ec94302bd1ee74b0fab3e17: Log matching property violated. Preceding OpId in replica: term: 2 index: 28. Preceding OpId from leader: term: 2 index: 29. (index mismatch)
I20240503 23:26:45.193420 4775 consensus_queue.cc:1035] T 481cd0f8c8d147adac217915a53ff6d9 P 83ffd7184ec94302bd1ee74b0fab3e17 [LEADER]: Connected to new peer: Peer: permanent_uuid: "692da0319a2d4a81ad81f5d8f9f4c86b" member_type: VOTER last_known_addr { host: "127.0.100.5" port: 38561 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.000s
I20240503 23:26:45.194108 4827 consensus_queue.cc:1035] T 481cd0f8c8d147adac217915a53ff6d9 P 83ffd7184ec94302bd1ee74b0fab3e17 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2f9bf2e5d9b4fc19f804797bca4a3ff" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 46261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.000s
I20240503 23:26:45.194438 4795 consensus_queue.cc:1035] T 481cd0f8c8d147adac217915a53ff6d9 P 83ffd7184ec94302bd1ee74b0fab3e17 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a3eecbaec43d4954b1096bc3121e824c" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 40649 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.000s
I20240503 23:26:45.201165 4775 raft_consensus.cc:2932] T 481cd0f8c8d147adac217915a53ff6d9 P 83ffd7184ec94302bd1ee74b0fab3e17 [term 2 LEADER]: Committing config change with OpId 2.29: config changed from index 28 to 29, 692da0319a2d4a81ad81f5d8f9f4c86b (127.0.100.5) changed from NON_VOTER to VOTER. New config: { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "83ffd7184ec94302bd1ee74b0fab3e17" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 32909 } } peers { permanent_uuid: "a2f9bf2e5d9b4fc19f804797bca4a3ff" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 46261 } } peers { permanent_uuid: "a3eecbaec43d4954b1096bc3121e824c" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 40649 } attrs { replace: true } } peers { permanent_uuid: "692da0319a2d4a81ad81f5d8f9f4c86b" member_type: VOTER last_known_addr { host: "127.0.100.5" port: 38561 } attrs { promote: false } } }
I20240503 23:26:45.201716 3837 raft_consensus.cc:2932] T 481cd0f8c8d147adac217915a53ff6d9 P a2f9bf2e5d9b4fc19f804797bca4a3ff [term 2 FOLLOWER]: Committing config change with OpId 2.29: config changed from index 28 to 29, 692da0319a2d4a81ad81f5d8f9f4c86b (127.0.100.5) changed from NON_VOTER to VOTER. New config: { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "83ffd7184ec94302bd1ee74b0fab3e17" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 32909 } } peers { permanent_uuid: "a2f9bf2e5d9b4fc19f804797bca4a3ff" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 46261 } } peers { permanent_uuid: "a3eecbaec43d4954b1096bc3121e824c" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 40649 } attrs { replace: true } } peers { permanent_uuid: "692da0319a2d4a81ad81f5d8f9f4c86b" member_type: VOTER last_known_addr { host: "127.0.100.5" port: 38561 } attrs { promote: false } } }
I20240503 23:26:45.202548 4293 raft_consensus.cc:2932] T 481cd0f8c8d147adac217915a53ff6d9 P 692da0319a2d4a81ad81f5d8f9f4c86b [term 2 FOLLOWER]: Committing config change with OpId 2.29: config changed from index 28 to 29, 692da0319a2d4a81ad81f5d8f9f4c86b (127.0.100.5) changed from NON_VOTER to VOTER. New config: { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "83ffd7184ec94302bd1ee74b0fab3e17" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 32909 } } peers { permanent_uuid: "a2f9bf2e5d9b4fc19f804797bca4a3ff" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 46261 } } peers { permanent_uuid: "a3eecbaec43d4954b1096bc3121e824c" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 40649 } attrs { replace: true } } peers { permanent_uuid: "692da0319a2d4a81ad81f5d8f9f4c86b" member_type: VOTER last_known_addr { host: "127.0.100.5" port: 38561 } attrs { promote: false } } }
I20240503 23:26:45.206995 3907 raft_consensus.cc:2932] T 481cd0f8c8d147adac217915a53ff6d9 P a3eecbaec43d4954b1096bc3121e824c [term 2 FOLLOWER]: Committing config change with OpId 2.29: config changed from index 28 to 29, 692da0319a2d4a81ad81f5d8f9f4c86b (127.0.100.5) changed from NON_VOTER to VOTER. New config: { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "83ffd7184ec94302bd1ee74b0fab3e17" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 32909 } } peers { permanent_uuid: "a2f9bf2e5d9b4fc19f804797bca4a3ff" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 46261 } } peers { permanent_uuid: "a3eecbaec43d4954b1096bc3121e824c" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 40649 } attrs { replace: true } } peers { permanent_uuid: "692da0319a2d4a81ad81f5d8f9f4c86b" member_type: VOTER last_known_addr { host: "127.0.100.5" port: 38561 } attrs { promote: false } } }
I20240503 23:26:45.207553 3662 catalog_manager.cc:5484] T 481cd0f8c8d147adac217915a53ff6d9 P 83ffd7184ec94302bd1ee74b0fab3e17 reported cstate change: config changed from index 28 to 29, 692da0319a2d4a81ad81f5d8f9f4c86b (127.0.100.5) changed from NON_VOTER to VOTER. New cstate: current_term: 2 leader_uuid: "83ffd7184ec94302bd1ee74b0fab3e17" committed_config { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "83ffd7184ec94302bd1ee74b0fab3e17" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 32909 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a2f9bf2e5d9b4fc19f804797bca4a3ff" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 46261 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a3eecbaec43d4954b1096bc3121e824c" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 40649 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "692da0319a2d4a81ad81f5d8f9f4c86b" member_type: VOTER last_known_addr { host: "127.0.100.5" port: 38561 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 23:26:45.214172 3762 consensus_queue.cc:237] T 481cd0f8c8d147adac217915a53ff6d9 P 83ffd7184ec94302bd1ee74b0fab3e17 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 29, Committed index: 29, Last appended: 2.29, Last appended by leader: 26, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "83ffd7184ec94302bd1ee74b0fab3e17" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 32909 } } peers { permanent_uuid: "a2f9bf2e5d9b4fc19f804797bca4a3ff" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 46261 } } peers { permanent_uuid: "692da0319a2d4a81ad81f5d8f9f4c86b" member_type: VOTER last_known_addr { host: "127.0.100.5" port: 38561 } attrs { promote: false } }
W20240503 23:26:45.215445 4795 consensus_queue.cc:1159] T 481cd0f8c8d147adac217915a53ff6d9 P 83ffd7184ec94302bd1ee74b0fab3e17 [LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "a3eecbaec43d4954b1096bc3121e824c" responder_term: 2 status { last_received { term: 2 index: 29 } last_committed_idx: 29 last_received_current_leader { term: 2 index: 29 } }
I20240503 23:26:45.216856 3837 raft_consensus.cc:1261] T 481cd0f8c8d147adac217915a53ff6d9 P a2f9bf2e5d9b4fc19f804797bca4a3ff [term 2 FOLLOWER]: Refusing update from remote peer 83ffd7184ec94302bd1ee74b0fab3e17: Log matching property violated. Preceding OpId in replica: term: 2 index: 29. Preceding OpId from leader: term: 2 index: 30. (index mismatch)
I20240503 23:26:45.216856 4293 raft_consensus.cc:1261] T 481cd0f8c8d147adac217915a53ff6d9 P 692da0319a2d4a81ad81f5d8f9f4c86b [term 2 FOLLOWER]: Refusing update from remote peer 83ffd7184ec94302bd1ee74b0fab3e17: Log matching property violated. Preceding OpId in replica: term: 2 index: 29. Preceding OpId from leader: term: 2 index: 30. (index mismatch)
I20240503 23:26:45.217537 4794 consensus_queue.cc:1035] T 481cd0f8c8d147adac217915a53ff6d9 P 83ffd7184ec94302bd1ee74b0fab3e17 [LEADER]: Connected to new peer: Peer: permanent_uuid: "692da0319a2d4a81ad81f5d8f9f4c86b" member_type: VOTER last_known_addr { host: "127.0.100.5" port: 38561 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 30, Last known committed idx: 29, Time since last communication: 0.000s
I20240503 23:26:45.217927 4775 consensus_queue.cc:1035] T 481cd0f8c8d147adac217915a53ff6d9 P 83ffd7184ec94302bd1ee74b0fab3e17 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2f9bf2e5d9b4fc19f804797bca4a3ff" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 46261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 30, Last known committed idx: 29, Time since last communication: 0.000s
I20240503 23:26:45.220638 4795 raft_consensus.cc:2932] T 481cd0f8c8d147adac217915a53ff6d9 P 83ffd7184ec94302bd1ee74b0fab3e17 [term 2 LEADER]: Committing config change with OpId 2.30: config changed from index 29 to 30, VOTER a3eecbaec43d4954b1096bc3121e824c (127.0.100.3) evicted. New config: { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "83ffd7184ec94302bd1ee74b0fab3e17" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 32909 } } peers { permanent_uuid: "a2f9bf2e5d9b4fc19f804797bca4a3ff" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 46261 } } peers { permanent_uuid: "692da0319a2d4a81ad81f5d8f9f4c86b" member_type: VOTER last_known_addr { host: "127.0.100.5" port: 38561 } attrs { promote: false } } }
I20240503 23:26:45.224205 3648 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 481cd0f8c8d147adac217915a53ff6d9 with cas_config_opid_index 29: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 23:26:45.225049 3837 raft_consensus.cc:2932] T 481cd0f8c8d147adac217915a53ff6d9 P a2f9bf2e5d9b4fc19f804797bca4a3ff [term 2 FOLLOWER]: Committing config change with OpId 2.30: config changed from index 29 to 30, VOTER a3eecbaec43d4954b1096bc3121e824c (127.0.100.3) evicted. New config: { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "83ffd7184ec94302bd1ee74b0fab3e17" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 32909 } } peers { permanent_uuid: "a2f9bf2e5d9b4fc19f804797bca4a3ff" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 46261 } } peers { permanent_uuid: "692da0319a2d4a81ad81f5d8f9f4c86b" member_type: VOTER last_known_addr { host: "127.0.100.5" port: 38561 } attrs { promote: false } } }
I20240503 23:26:45.226203 4293 raft_consensus.cc:2932] T 481cd0f8c8d147adac217915a53ff6d9 P 692da0319a2d4a81ad81f5d8f9f4c86b [term 2 FOLLOWER]: Committing config change with OpId 2.30: config changed from index 29 to 30, VOTER a3eecbaec43d4954b1096bc3121e824c (127.0.100.3) evicted. New config: { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "83ffd7184ec94302bd1ee74b0fab3e17" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 32909 } } peers { permanent_uuid: "a2f9bf2e5d9b4fc19f804797bca4a3ff" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 46261 } } peers { permanent_uuid: "692da0319a2d4a81ad81f5d8f9f4c86b" member_type: VOTER last_known_addr { host: "127.0.100.5" port: 38561 } attrs { promote: false } } }
I20240503 23:26:45.226457 3659 catalog_manager.cc:5484] T 481cd0f8c8d147adac217915a53ff6d9 P 83ffd7184ec94302bd1ee74b0fab3e17 reported cstate change: config changed from index 29 to 30, VOTER a3eecbaec43d4954b1096bc3121e824c (127.0.100.3) evicted. New cstate: current_term: 2 leader_uuid: "83ffd7184ec94302bd1ee74b0fab3e17" committed_config { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "83ffd7184ec94302bd1ee74b0fab3e17" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 32909 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a2f9bf2e5d9b4fc19f804797bca4a3ff" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 46261 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "692da0319a2d4a81ad81f5d8f9f4c86b" member_type: VOTER last_known_addr { host: "127.0.100.5" port: 38561 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240503 23:26:45.231693 3902 tablet_service.cc:1514] Processing DeleteTablet for tablet 481cd0f8c8d147adac217915a53ff6d9 with delete_type TABLET_DATA_TOMBSTONED (TS a3eecbaec43d4954b1096bc3121e824c not found in new config with opid_index 30) from {username='slave'} at 127.0.0.1:40632
I20240503 23:26:45.233034 4831 tablet_replica.cc:330] T 481cd0f8c8d147adac217915a53ff6d9 P a3eecbaec43d4954b1096bc3121e824c: stopping tablet replica
I20240503 23:26:45.233335 4831 raft_consensus.cc:2229] T 481cd0f8c8d147adac217915a53ff6d9 P a3eecbaec43d4954b1096bc3121e824c [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 23:26:45.234097 4831 raft_consensus.cc:2258] T 481cd0f8c8d147adac217915a53ff6d9 P a3eecbaec43d4954b1096bc3121e824c [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 23:26:45.235184 4831 ts_tablet_manager.cc:1875] T 481cd0f8c8d147adac217915a53ff6d9 P a3eecbaec43d4954b1096bc3121e824c: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 23:26:45.239965 4832 raft_consensus.cc:1050] T a14a12115b8d4fec947c6e223f1ef486 P 4d1d4e261584467fbbce4c72fa18045a: attempting to promote NON_VOTER e4086c099d7f4447b2b4733b6ca364df to VOTER
I20240503 23:26:45.240882 4832 consensus_queue.cc:237] T a14a12115b8d4fec947c6e223f1ef486 P 4d1d4e261584467fbbce4c72fa18045a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 31, Committed index: 31, Last appended: 2.31, Last appended by leader: 29, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "83ffd7184ec94302bd1ee74b0fab3e17" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 32909 } } peers { permanent_uuid: "a3eecbaec43d4954b1096bc3121e824c" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 40649 } attrs { replace: true } } peers { permanent_uuid: "4d1d4e261584467fbbce4c72fa18045a" member_type: VOTER last_known_addr { host: "127.0.100.6" port: 35349 } attrs { promote: false } } peers { permanent_uuid: "e4086c099d7f4447b2b4733b6ca364df" member_type: VOTER last_known_addr { host: "127.0.100.4" port: 35011 } attrs { promote: false } }
I20240503 23:26:45.242861 4831 ts_tablet_manager.cc:1888] T 481cd0f8c8d147adac217915a53ff6d9 P a3eecbaec43d4954b1096bc3121e824c: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.29
I20240503 23:26:45.243161 4831 log.cc:1196] T 481cd0f8c8d147adac217915a53ff6d9 P a3eecbaec43d4954b1096bc3121e824c: Deleting WAL directory at /tmp/dist-test-taskZhIiPz/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714778688451250-400-0/minicluster-data/ts-2-root/wals/481cd0f8c8d147adac217915a53ff6d9
I20240503 23:26:45.243520 4216 raft_consensus.cc:1261] T a14a12115b8d4fec947c6e223f1ef486 P e4086c099d7f4447b2b4733b6ca364df [term 2 LEARNER]: Refusing update from remote peer 4d1d4e261584467fbbce4c72fa18045a: Log matching property violated. Preceding OpId in replica: term: 2 index: 31. Preceding OpId from leader: term: 2 index: 32. (index mismatch)
I20240503 23:26:45.244210 3646 catalog_manager.cc:4830] TS a3eecbaec43d4954b1096bc3121e824c (127.0.100.3:40649): tablet 481cd0f8c8d147adac217915a53ff6d9 (table test-workload [id=1345b2a336c84ec2b1de62853e7fa3f6]) successfully deleted
I20240503 23:26:45.244298 4765 consensus_queue.cc:1035] T a14a12115b8d4fec947c6e223f1ef486 P 4d1d4e261584467fbbce4c72fa18045a [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4086c099d7f4447b2b4733b6ca364df" member_type: VOTER last_known_addr { host: "127.0.100.4" port: 35011 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 32, Last known committed idx: 31, Time since last communication: 0.000s
I20240503 23:26:45.244650 3908 raft_consensus.cc:1261] T a14a12115b8d4fec947c6e223f1ef486 P a3eecbaec43d4954b1096bc3121e824c [term 2 FOLLOWER]: Refusing update from remote peer 4d1d4e261584467fbbce4c72fa18045a: Log matching property violated. Preceding OpId in replica: term: 2 index: 31. Preceding OpId from leader: term: 2 index: 32. (index mismatch)
I20240503 23:26:45.245213 4765 consensus_queue.cc:1035] T a14a12115b8d4fec947c6e223f1ef486 P 4d1d4e261584467fbbce4c72fa18045a [LEADER]: Connected to new peer: Peer: permanent_uuid: "a3eecbaec43d4954b1096bc3121e824c" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 40649 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 32, Last known committed idx: 31, Time since last communication: 0.000s
I20240503 23:26:45.246589 3762 raft_consensus.cc:1261] T a14a12115b8d4fec947c6e223f1ef486 P 83ffd7184ec94302bd1ee74b0fab3e17 [term 2 FOLLOWER]: Refusing update from remote peer 4d1d4e261584467fbbce4c72fa18045a: Log matching property violated. Preceding OpId in replica: term: 2 index: 31. Preceding OpId from leader: term: 2 index: 32. (index mismatch)
I20240503 23:26:45.248106 4765 consensus_queue.cc:1035] T a14a12115b8d4fec947c6e223f1ef486 P 4d1d4e261584467fbbce4c72fa18045a [LEADER]: Connected to new peer: Peer: permanent_uuid: "83ffd7184ec94302bd1ee74b0fab3e17" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 32909 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 32, Last known committed idx: 31, Time since last communication: 0.000s
I20240503 23:26:45.253638 4832 raft_consensus.cc:2932] T a14a12115b8d4fec947c6e223f1ef486 P 4d1d4e261584467fbbce4c72fa18045a [term 2 LEADER]: Committing config change with OpId 2.32: config changed from index 31 to 32, e4086c099d7f4447b2b4733b6ca364df (127.0.100.4) changed from NON_VOTER to VOTER. New config: { opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "83ffd7184ec94302bd1ee74b0fab3e17" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 32909 } } peers { permanent_uuid: "a3eecbaec43d4954b1096bc3121e824c" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 40649 } attrs { replace: true } } peers { permanent_uuid: "4d1d4e261584467fbbce4c72fa18045a" member_type: VOTER last_known_addr { host: "127.0.100.6" port: 35349 } attrs { promote: false } } peers { permanent_uuid: "e4086c099d7f4447b2b4733b6ca364df" member_type: VOTER last_known_addr { host: "127.0.100.4" port: 35011 } attrs { promote: false } } }
I20240503 23:26:45.254103 3762 raft_consensus.cc:2932] T a14a12115b8d4fec947c6e223f1ef486 P 83ffd7184ec94302bd1ee74b0fab3e17 [term 2 FOLLOWER]: Committing config change with OpId 2.32: config changed from index 31 to 32, e4086c099d7f4447b2b4733b6ca364df (127.0.100.4) changed from NON_VOTER to VOTER. New config: { opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "83ffd7184ec94302bd1ee74b0fab3e17" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 32909 } } peers { permanent_uuid: "a3eecbaec43d4954b1096bc3121e824c" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 40649 } attrs { replace: true } } peers { permanent_uuid: "4d1d4e261584467fbbce4c72fa18045a" member_type: VOTER last_known_addr { host: "127.0.100.6" port: 35349 } attrs { promote: false } } peers { permanent_uuid: "e4086c099d7f4447b2b4733b6ca364df" member_type: VOTER last_known_addr { host: "127.0.100.4" port: 35011 } attrs { promote: false } } }
I20240503 23:26:45.256906 3907 raft_consensus.cc:2932] T a14a12115b8d4fec947c6e223f1ef486 P a3eecbaec43d4954b1096bc3121e824c [term 2 FOLLOWER]: Committing config change with OpId 2.32: config changed from index 31 to 32, e4086c099d7f4447b2b4733b6ca364df (127.0.100.4) changed from NON_VOTER to VOTER. New config: { opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "83ffd7184ec94302bd1ee74b0fab3e17" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 32909 } } peers { permanent_uuid: "a3eecbaec43d4954b1096bc3121e824c" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 40649 } attrs { replace: true } } peers { permanent_uuid: "4d1d4e261584467fbbce4c72fa18045a" member_type: VOTER last_known_addr { host: "127.0.100.6" port: 35349 } attrs { promote: false } } peers { permanent_uuid: "e4086c099d7f4447b2b4733b6ca364df" member_type: VOTER last_known_addr { host: "127.0.100.4" port: 35011 } attrs { promote: false } } }
I20240503 23:26:45.259310 4216 raft_consensus.cc:2932] T a14a12115b8d4fec947c6e223f1ef486 P e4086c099d7f4447b2b4733b6ca364df [term 2 FOLLOWER]: Committing config change with OpId 2.32: config changed from index 31 to 32, e4086c099d7f4447b2b4733b6ca364df (127.0.100.4) changed from NON_VOTER to VOTER. New config: { opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "83ffd7184ec94302bd1ee74b0fab3e17" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 32909 } } peers { permanent_uuid: "a3eecbaec43d4954b1096bc3121e824c" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 40649 } attrs { replace: true } } peers { permanent_uuid: "4d1d4e261584467fbbce4c72fa18045a" member_type: VOTER last_known_addr { host: "127.0.100.6" port: 35349 } attrs { promote: false } } peers { permanent_uuid: "e4086c099d7f4447b2b4733b6ca364df" member_type: VOTER last_known_addr { host: "127.0.100.4" port: 35011 } attrs { promote: false } } }
I20240503 23:26:45.260113 3659 catalog_manager.cc:5484] T a14a12115b8d4fec947c6e223f1ef486 P 4d1d4e261584467fbbce4c72fa18045a reported cstate change: config changed from index 31 to 32, e4086c099d7f4447b2b4733b6ca364df (127.0.100.4) changed from NON_VOTER to VOTER. New cstate: current_term: 2 leader_uuid: "4d1d4e261584467fbbce4c72fa18045a" committed_config { opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "83ffd7184ec94302bd1ee74b0fab3e17" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 32909 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a3eecbaec43d4954b1096bc3121e824c" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 40649 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4d1d4e261584467fbbce4c72fa18045a" member_type: VOTER last_known_addr { host: "127.0.100.6" port: 35349 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e4086c099d7f4447b2b4733b6ca364df" member_type: VOTER last_known_addr { host: "127.0.100.4" port: 35011 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 23:26:45.267830 3715 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240503 23:26:45.270433 4367 consensus_queue.cc:237] T a14a12115b8d4fec947c6e223f1ef486 P 4d1d4e261584467fbbce4c72fa18045a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 32, Committed index: 32, Last appended: 2.32, Last appended by leader: 29, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 33 OBSOLETE_local: false peers { permanent_uuid: "83ffd7184ec94302bd1ee74b0fab3e17" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 32909 } } peers { permanent_uuid: "4d1d4e261584467fbbce4c72fa18045a" member_type: VOTER last_known_addr { host: "127.0.100.6" port: 35349 } attrs { promote: false } } peers { permanent_uuid: "e4086c099d7f4447b2b4733b6ca364df" member_type: VOTER last_known_addr { host: "127.0.100.4" port: 35011 } attrs { promote: false } }
I20240503 23:26:45.271533 3762 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "481cd0f8c8d147adac217915a53ff6d9"
dest_uuid: "83ffd7184ec94302bd1ee74b0fab3e17"
mode: GRACEFUL
new_leader_uuid: "a2f9bf2e5d9b4fc19f804797bca4a3ff"
from {username='slave'} at 127.0.0.1:57936
I20240503 23:26:45.271860 3762 raft_consensus.cc:595] T 481cd0f8c8d147adac217915a53ff6d9 P 83ffd7184ec94302bd1ee74b0fab3e17 [term 2 LEADER]: Received request to transfer leadership to TS a2f9bf2e5d9b4fc19f804797bca4a3ff
I20240503 23:26:45.272440 3760 raft_consensus.cc:1261] T a14a12115b8d4fec947c6e223f1ef486 P 83ffd7184ec94302bd1ee74b0fab3e17 [term 2 FOLLOWER]: Refusing update from remote peer 4d1d4e261584467fbbce4c72fa18045a: Log matching property violated. Preceding OpId in replica: term: 2 index: 32. Preceding OpId from leader: term: 2 index: 33. (index mismatch)
I20240503 23:26:45.272663 4216 raft_consensus.cc:1261] T a14a12115b8d4fec947c6e223f1ef486 P e4086c099d7f4447b2b4733b6ca364df [term 2 FOLLOWER]: Refusing update from remote peer 4d1d4e261584467fbbce4c72fa18045a: Log matching property violated. Preceding OpId in replica: term: 2 index: 32. Preceding OpId from leader: term: 2 index: 33. (index mismatch)
I20240503 23:26:45.272938 3715 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 1
I20240503 23:26:45.273265 3715 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240503 23:26:45.273231 4824 consensus_queue.cc:1035] T a14a12115b8d4fec947c6e223f1ef486 P 4d1d4e261584467fbbce4c72fa18045a [LEADER]: Connected to new peer: Peer: permanent_uuid: "83ffd7184ec94302bd1ee74b0fab3e17" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 32909 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 33, Last known committed idx: 32, Time since last communication: 0.000s
I20240503 23:26:45.273865 4765 consensus_queue.cc:1035] T a14a12115b8d4fec947c6e223f1ef486 P 4d1d4e261584467fbbce4c72fa18045a [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4086c099d7f4447b2b4733b6ca364df" member_type: VOTER last_known_addr { host: "127.0.100.4" port: 35011 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 33, Last known committed idx: 32, Time since last communication: 0.000s
I20240503 23:26:45.275945 4765 raft_consensus.cc:2932] T a14a12115b8d4fec947c6e223f1ef486 P 4d1d4e261584467fbbce4c72fa18045a [term 2 LEADER]: Committing config change with OpId 2.33: config changed from index 32 to 33, VOTER a3eecbaec43d4954b1096bc3121e824c (127.0.100.3) evicted. New config: { opid_index: 33 OBSOLETE_local: false peers { permanent_uuid: "83ffd7184ec94302bd1ee74b0fab3e17" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 32909 } } peers { permanent_uuid: "4d1d4e261584467fbbce4c72fa18045a" member_type: VOTER last_known_addr { host: "127.0.100.6" port: 35349 } attrs { promote: false } } peers { permanent_uuid: "e4086c099d7f4447b2b4733b6ca364df" member_type: VOTER last_known_addr { host: "127.0.100.4" port: 35011 } attrs { promote: false } } }
I20240503 23:26:45.276358 3760 raft_consensus.cc:2932] T a14a12115b8d4fec947c6e223f1ef486 P 83ffd7184ec94302bd1ee74b0fab3e17 [term 2 FOLLOWER]: Committing config change with OpId 2.33: config changed from index 32 to 33, VOTER a3eecbaec43d4954b1096bc3121e824c (127.0.100.3) evicted. New config: { opid_index: 33 OBSOLETE_local: false peers { permanent_uuid: "83ffd7184ec94302bd1ee74b0fab3e17" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 32909 } } peers { permanent_uuid: "4d1d4e261584467fbbce4c72fa18045a" member_type: VOTER last_known_addr { host: "127.0.100.6" port: 35349 } attrs { promote: false } } peers { permanent_uuid: "e4086c099d7f4447b2b4733b6ca364df" member_type: VOTER last_known_addr { host: "127.0.100.4" port: 35011 } attrs { promote: false } } }
I20240503 23:26:45.281080 3707 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet a14a12115b8d4fec947c6e223f1ef486 with cas_config_opid_index 32: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 23:26:45.281445 4216 raft_consensus.cc:2932] T a14a12115b8d4fec947c6e223f1ef486 P e4086c099d7f4447b2b4733b6ca364df [term 2 FOLLOWER]: Committing config change with OpId 2.33: config changed from index 32 to 33, VOTER a3eecbaec43d4954b1096bc3121e824c (127.0.100.3) evicted. New config: { opid_index: 33 OBSOLETE_local: false peers { permanent_uuid: "83ffd7184ec94302bd1ee74b0fab3e17" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 32909 } } peers { permanent_uuid: "4d1d4e261584467fbbce4c72fa18045a" member_type: VOTER last_known_addr { host: "127.0.100.6" port: 35349 } attrs { promote: false } } peers { permanent_uuid: "e4086c099d7f4447b2b4733b6ca364df" member_type: VOTER last_known_addr { host: "127.0.100.4" port: 35011 } attrs { promote: false } } }
I20240503 23:26:45.282470 3662 catalog_manager.cc:5484] T a14a12115b8d4fec947c6e223f1ef486 P 83ffd7184ec94302bd1ee74b0fab3e17 reported cstate change: config changed from index 32 to 33, VOTER a3eecbaec43d4954b1096bc3121e824c (127.0.100.3) evicted. New cstate: current_term: 2 leader_uuid: "4d1d4e261584467fbbce4c72fa18045a" committed_config { opid_index: 33 OBSOLETE_local: false peers { permanent_uuid: "83ffd7184ec94302bd1ee74b0fab3e17" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 32909 } } peers { permanent_uuid: "4d1d4e261584467fbbce4c72fa18045a" member_type: VOTER last_known_addr { host: "127.0.100.6" port: 35349 } attrs { promote: false } } peers { permanent_uuid: "e4086c099d7f4447b2b4733b6ca364df" member_type: VOTER last_known_addr { host: "127.0.100.4" port: 35011 } attrs { promote: false } } }
I20240503 23:26:45.287323 3902 tablet_service.cc:1514] Processing DeleteTablet for tablet a14a12115b8d4fec947c6e223f1ef486 with delete_type TABLET_DATA_TOMBSTONED (TS a3eecbaec43d4954b1096bc3121e824c not found in new config with opid_index 33) from {username='slave'} at 127.0.0.1:40632
I20240503 23:26:45.287739 4831 tablet_replica.cc:330] T a14a12115b8d4fec947c6e223f1ef486 P a3eecbaec43d4954b1096bc3121e824c: stopping tablet replica
I20240503 23:26:45.287986 4831 raft_consensus.cc:2229] T a14a12115b8d4fec947c6e223f1ef486 P a3eecbaec43d4954b1096bc3121e824c [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 23:26:45.288421 4831 raft_consensus.cc:2258] T a14a12115b8d4fec947c6e223f1ef486 P a3eecbaec43d4954b1096bc3121e824c [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 23:26:45.289573 4831 ts_tablet_manager.cc:1875] T a14a12115b8d4fec947c6e223f1ef486 P a3eecbaec43d4954b1096bc3121e824c: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 23:26:45.295341 4831 ts_tablet_manager.cc:1888] T a14a12115b8d4fec947c6e223f1ef486 P a3eecbaec43d4954b1096bc3121e824c: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.32
I20240503 23:26:45.295523 4831 log.cc:1196] T a14a12115b8d4fec947c6e223f1ef486 P a3eecbaec43d4954b1096bc3121e824c: Deleting WAL directory at /tmp/dist-test-taskZhIiPz/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714778688451250-400-0/minicluster-data/ts-2-root/wals/a14a12115b8d4fec947c6e223f1ef486
I20240503 23:26:45.296299 3646 catalog_manager.cc:4830] TS a3eecbaec43d4954b1096bc3121e824c (127.0.100.3:40649): tablet a14a12115b8d4fec947c6e223f1ef486 (table test-workload [id=1345b2a336c84ec2b1de62853e7fa3f6]) successfully deleted
I20240503 23:26:45.466259 4838 raft_consensus.cc:1050] T 60267cb4849d4823a0afb2380305905f P a3eecbaec43d4954b1096bc3121e824c: attempting to promote NON_VOTER 4d1d4e261584467fbbce4c72fa18045a to VOTER
I20240503 23:26:45.467352 4838 consensus_queue.cc:237] T 60267cb4849d4823a0afb2380305905f P a3eecbaec43d4954b1096bc3121e824c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 29, Committed index: 29, Last appended: 1.29, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "83ffd7184ec94302bd1ee74b0fab3e17" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 32909 } attrs { replace: true } } peers { permanent_uuid: "a3eecbaec43d4954b1096bc3121e824c" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 40649 } } peers { permanent_uuid: "e4086c099d7f4447b2b4733b6ca364df" member_type: VOTER last_known_addr { host: "127.0.100.4" port: 35011 } attrs { promote: false } } peers { permanent_uuid: "4d1d4e261584467fbbce4c72fa18045a" member_type: VOTER last_known_addr { host: "127.0.100.6" port: 35349 } attrs { promote: false } }
I20240503 23:26:45.470067 4216 raft_consensus.cc:1261] T 60267cb4849d4823a0afb2380305905f P e4086c099d7f4447b2b4733b6ca364df [term 1 FOLLOWER]: Refusing update from remote peer a3eecbaec43d4954b1096bc3121e824c: Log matching property violated. Preceding OpId in replica: term: 1 index: 29. Preceding OpId from leader: term: 1 index: 30. (index mismatch)
I20240503 23:26:45.470341 3760 raft_consensus.cc:1261] T 60267cb4849d4823a0afb2380305905f P 83ffd7184ec94302bd1ee74b0fab3e17 [term 1 FOLLOWER]: Refusing update from remote peer a3eecbaec43d4954b1096bc3121e824c: Log matching property violated. Preceding OpId in replica: term: 1 index: 29. Preceding OpId from leader: term: 1 index: 30. (index mismatch)
I20240503 23:26:45.470341 4367 raft_consensus.cc:1261] T 60267cb4849d4823a0afb2380305905f P 4d1d4e261584467fbbce4c72fa18045a [term 1 LEARNER]: Refusing update from remote peer a3eecbaec43d4954b1096bc3121e824c: Log matching property violated. Preceding OpId in replica: term: 1 index: 29. Preceding OpId from leader: term: 1 index: 30. (index mismatch)
I20240503 23:26:45.470705 4839 consensus_queue.cc:1035] T 60267cb4849d4823a0afb2380305905f P a3eecbaec43d4954b1096bc3121e824c [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4086c099d7f4447b2b4733b6ca364df" member_type: VOTER last_known_addr { host: "127.0.100.4" port: 35011 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 30, Last known committed idx: 29, Time since last communication: 0.000s
I20240503 23:26:45.471305 4838 consensus_queue.cc:1035] T 60267cb4849d4823a0afb2380305905f P a3eecbaec43d4954b1096bc3121e824c [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d1d4e261584467fbbce4c72fa18045a" member_type: VOTER last_known_addr { host: "127.0.100.6" port: 35349 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 30, Last known committed idx: 29, Time since last communication: 0.000s
I20240503 23:26:45.471590 4779 consensus_queue.cc:1035] T 60267cb4849d4823a0afb2380305905f P a3eecbaec43d4954b1096bc3121e824c [LEADER]: Connected to new peer: Peer: permanent_uuid: "83ffd7184ec94302bd1ee74b0fab3e17" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 32909 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 30, Last known committed idx: 29, Time since last communication: 0.000s
I20240503 23:26:45.475420 4839 raft_consensus.cc:2932] T 60267cb4849d4823a0afb2380305905f P a3eecbaec43d4954b1096bc3121e824c [term 1 LEADER]: Committing config change with OpId 1.30: config changed from index 29 to 30, 4d1d4e261584467fbbce4c72fa18045a (127.0.100.6) changed from NON_VOTER to VOTER. New config: { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "83ffd7184ec94302bd1ee74b0fab3e17" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 32909 } attrs { replace: true } } peers { permanent_uuid: "a3eecbaec43d4954b1096bc3121e824c" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 40649 } } peers { permanent_uuid: "e4086c099d7f4447b2b4733b6ca364df" member_type: VOTER last_known_addr { host: "127.0.100.4" port: 35011 } attrs { promote: false } } peers { permanent_uuid: "4d1d4e261584467fbbce4c72fa18045a" member_type: VOTER last_known_addr { host: "127.0.100.6" port: 35349 } attrs { promote: false } } }
I20240503 23:26:45.476290 4367 raft_consensus.cc:2932] T 60267cb4849d4823a0afb2380305905f P 4d1d4e261584467fbbce4c72fa18045a [term 1 FOLLOWER]: Committing config change with OpId 1.30: config changed from index 29 to 30, 4d1d4e261584467fbbce4c72fa18045a (127.0.100.6) changed from NON_VOTER to VOTER. New config: { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "83ffd7184ec94302bd1ee74b0fab3e17" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 32909 } attrs { replace: true } } peers { permanent_uuid: "a3eecbaec43d4954b1096bc3121e824c" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 40649 } } peers { permanent_uuid: "e4086c099d7f4447b2b4733b6ca364df" member_type: VOTER last_known_addr { host: "127.0.100.4" port: 35011 } attrs { promote: false } } peers { permanent_uuid: "4d1d4e261584467fbbce4c72fa18045a" member_type: VOTER last_known_addr { host: "127.0.100.6" port: 35349 } attrs { promote: false } } }
I20240503 23:26:45.479996 4216 raft_consensus.cc:2932] T 60267cb4849d4823a0afb2380305905f P e4086c099d7f4447b2b4733b6ca364df [term 1 FOLLOWER]: Committing config change with OpId 1.30: config changed from index 29 to 30, 4d1d4e261584467fbbce4c72fa18045a (127.0.100.6) changed from NON_VOTER to VOTER. New config: { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "83ffd7184ec94302bd1ee74b0fab3e17" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 32909 } attrs { replace: true } } peers { permanent_uuid: "a3eecbaec43d4954b1096bc3121e824c" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 40649 } } peers { permanent_uuid: "e4086c099d7f4447b2b4733b6ca364df" member_type: VOTER last_known_addr { host: "127.0.100.4" port: 35011 } attrs { promote: false } } peers { permanent_uuid: "4d1d4e261584467fbbce4c72fa18045a" member_type: VOTER last_known_addr { host: "127.0.100.6" port: 35349 } attrs { promote: false } } }
I20240503 23:26:45.484630 3662 catalog_manager.cc:5484] T 60267cb4849d4823a0afb2380305905f P a3eecbaec43d4954b1096bc3121e824c reported cstate change: config changed from index 29 to 30, 4d1d4e261584467fbbce4c72fa18045a (127.0.100.6) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "a3eecbaec43d4954b1096bc3121e824c" committed_config { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "83ffd7184ec94302bd1ee74b0fab3e17" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 32909 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a3eecbaec43d4954b1096bc3121e824c" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 40649 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e4086c099d7f4447b2b4733b6ca364df" member_type: VOTER last_known_addr { host: "127.0.100.4" port: 35011 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4d1d4e261584467fbbce4c72fa18045a" member_type: VOTER last_known_addr { host: "127.0.100.6" port: 35349 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 23:26:45.487704 3760 raft_consensus.cc:2932] T 60267cb4849d4823a0afb2380305905f P 83ffd7184ec94302bd1ee74b0fab3e17 [term 1 FOLLOWER]: Committing config change with OpId 1.30: config changed from index 29 to 30, 4d1d4e261584467fbbce4c72fa18045a (127.0.100.6) changed from NON_VOTER to VOTER. New config: { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "83ffd7184ec94302bd1ee74b0fab3e17" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 32909 } attrs { replace: true } } peers { permanent_uuid: "a3eecbaec43d4954b1096bc3121e824c" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 40649 } } peers { permanent_uuid: "e4086c099d7f4447b2b4733b6ca364df" member_type: VOTER last_known_addr { host: "127.0.100.4" port: 35011 } attrs { promote: false } } peers { permanent_uuid: "4d1d4e261584467fbbce4c72fa18045a" member_type: VOTER last_known_addr { host: "127.0.100.6" port: 35349 } attrs { promote: false } } }
I20240503 23:26:45.489842 3907 consensus_queue.cc:237] T 60267cb4849d4823a0afb2380305905f P a3eecbaec43d4954b1096bc3121e824c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 30, Committed index: 30, Last appended: 1.30, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "a3eecbaec43d4954b1096bc3121e824c" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 40649 } } peers { permanent_uuid: "e4086c099d7f4447b2b4733b6ca364df" member_type: VOTER last_known_addr { host: "127.0.100.4" port: 35011 } attrs { promote: false } } peers { permanent_uuid: "4d1d4e261584467fbbce4c72fa18045a" member_type: VOTER last_known_addr { host: "127.0.100.6" port: 35349 } attrs { promote: false } }
I20240503 23:26:45.491828 4216 raft_consensus.cc:1261] T 60267cb4849d4823a0afb2380305905f P e4086c099d7f4447b2b4733b6ca364df [term 1 FOLLOWER]: Refusing update from remote peer a3eecbaec43d4954b1096bc3121e824c: Log matching property violated. Preceding OpId in replica: term: 1 index: 30. Preceding OpId from leader: term: 1 index: 31. (index mismatch)
I20240503 23:26:45.492048 4367 raft_consensus.cc:1261] T 60267cb4849d4823a0afb2380305905f P 4d1d4e261584467fbbce4c72fa18045a [term 1 FOLLOWER]: Refusing update from remote peer a3eecbaec43d4954b1096bc3121e824c: Log matching property violated. Preceding OpId in replica: term: 1 index: 30. Preceding OpId from leader: term: 1 index: 31. (index mismatch)
I20240503 23:26:45.492635 4779 consensus_queue.cc:1035] T 60267cb4849d4823a0afb2380305905f P a3eecbaec43d4954b1096bc3121e824c [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d1d4e261584467fbbce4c72fa18045a" member_type: VOTER last_known_addr { host: "127.0.100.6" port: 35349 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 31, Last known committed idx: 30, Time since last communication: 0.000s
I20240503 23:26:45.493577 4779 consensus_queue.cc:1035] T 60267cb4849d4823a0afb2380305905f P a3eecbaec43d4954b1096bc3121e824c [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4086c099d7f4447b2b4733b6ca364df" member_type: VOTER last_known_addr { host: "127.0.100.4" port: 35011 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 31, Last known committed idx: 30, Time since last communication: 0.000s
I20240503 23:26:45.495867 4838 raft_consensus.cc:2932] T 60267cb4849d4823a0afb2380305905f P a3eecbaec43d4954b1096bc3121e824c [term 1 LEADER]: Committing config change with OpId 1.31: config changed from index 30 to 31, VOTER 83ffd7184ec94302bd1ee74b0fab3e17 (127.0.100.1) evicted. New config: { opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "a3eecbaec43d4954b1096bc3121e824c" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 40649 } } peers { permanent_uuid: "e4086c099d7f4447b2b4733b6ca364df" member_type: VOTER last_known_addr { host: "127.0.100.4" port: 35011 } attrs { promote: false } } peers { permanent_uuid: "4d1d4e261584467fbbce4c72fa18045a" member_type: VOTER last_known_addr { host: "127.0.100.6" port: 35349 } attrs { promote: false } } }
I20240503 23:26:45.496511 4367 raft_consensus.cc:2932] T 60267cb4849d4823a0afb2380305905f P 4d1d4e261584467fbbce4c72fa18045a [term 1 FOLLOWER]: Committing config change with OpId 1.31: config changed from index 30 to 31, VOTER 83ffd7184ec94302bd1ee74b0fab3e17 (127.0.100.1) evicted. New config: { opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "a3eecbaec43d4954b1096bc3121e824c" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 40649 } } peers { permanent_uuid: "e4086c099d7f4447b2b4733b6ca364df" member_type: VOTER last_known_addr { host: "127.0.100.4" port: 35011 } attrs { promote: false } } peers { permanent_uuid: "4d1d4e261584467fbbce4c72fa18045a" member_type: VOTER last_known_addr { host: "127.0.100.6" port: 35349 } attrs { promote: false } } }
I20240503 23:26:45.496763 4216 raft_consensus.cc:2932] T 60267cb4849d4823a0afb2380305905f P e4086c099d7f4447b2b4733b6ca364df [term 1 FOLLOWER]: Committing config change with OpId 1.31: config changed from index 30 to 31, VOTER 83ffd7184ec94302bd1ee74b0fab3e17 (127.0.100.1) evicted. New config: { opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "a3eecbaec43d4954b1096bc3121e824c" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 40649 } } peers { permanent_uuid: "e4086c099d7f4447b2b4733b6ca364df" member_type: VOTER last_known_addr { host: "127.0.100.4" port: 35011 } attrs { promote: false } } peers { permanent_uuid: "4d1d4e261584467fbbce4c72fa18045a" member_type: VOTER last_known_addr { host: "127.0.100.6" port: 35349 } attrs { promote: false } } }
I20240503 23:26:45.501528 3646 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 60267cb4849d4823a0afb2380305905f with cas_config_opid_index 30: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 23:26:45.501809 3655 catalog_manager.cc:5484] T 60267cb4849d4823a0afb2380305905f P e4086c099d7f4447b2b4733b6ca364df reported cstate change: config changed from index 30 to 31, VOTER 83ffd7184ec94302bd1ee74b0fab3e17 (127.0.100.1) evicted. New cstate: current_term: 1 leader_uuid: "a3eecbaec43d4954b1096bc3121e824c" committed_config { opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "a3eecbaec43d4954b1096bc3121e824c" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 40649 } } peers { permanent_uuid: "e4086c099d7f4447b2b4733b6ca364df" member_type: VOTER last_known_addr { host: "127.0.100.4" port: 35011 } attrs { promote: false } } peers { permanent_uuid: "4d1d4e261584467fbbce4c72fa18045a" member_type: VOTER last_known_addr { host: "127.0.100.6" port: 35349 } attrs { promote: false } } }
I20240503 23:26:45.505856 3747 tablet_service.cc:1514] Processing DeleteTablet for tablet 60267cb4849d4823a0afb2380305905f with delete_type TABLET_DATA_TOMBSTONED (TS 83ffd7184ec94302bd1ee74b0fab3e17 not found in new config with opid_index 31) from {username='slave'} at 127.0.0.1:35576
I20240503 23:26:45.506557 4825 tablet_replica.cc:330] T 60267cb4849d4823a0afb2380305905f P 83ffd7184ec94302bd1ee74b0fab3e17: stopping tablet replica
I20240503 23:26:45.506808 4825 raft_consensus.cc:2229] T 60267cb4849d4823a0afb2380305905f P 83ffd7184ec94302bd1ee74b0fab3e17 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 23:26:45.507331 4825 raft_consensus.cc:2258] T 60267cb4849d4823a0afb2380305905f P 83ffd7184ec94302bd1ee74b0fab3e17 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 23:26:45.508416 4825 ts_tablet_manager.cc:1875] T 60267cb4849d4823a0afb2380305905f P 83ffd7184ec94302bd1ee74b0fab3e17: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 23:26:45.513545 4825 ts_tablet_manager.cc:1888] T 60267cb4849d4823a0afb2380305905f P 83ffd7184ec94302bd1ee74b0fab3e17: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.30
I20240503 23:26:45.513777 4825 log.cc:1196] T 60267cb4849d4823a0afb2380305905f P 83ffd7184ec94302bd1ee74b0fab3e17: Deleting WAL directory at /tmp/dist-test-taskZhIiPz/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714778688451250-400-0/minicluster-data/ts-0-root/wals/60267cb4849d4823a0afb2380305905f
I20240503 23:26:45.515645 3648 catalog_manager.cc:4830] TS 83ffd7184ec94302bd1ee74b0fab3e17 (127.0.100.1:32909): tablet 60267cb4849d4823a0afb2380305905f (table test-workload [id=1345b2a336c84ec2b1de62853e7fa3f6]) successfully deleted
I20240503 23:26:45.671743 4775 raft_consensus.cc:979] T 481cd0f8c8d147adac217915a53ff6d9 P 83ffd7184ec94302bd1ee74b0fab3e17: : Instructing follower a2f9bf2e5d9b4fc19f804797bca4a3ff to start an election
I20240503 23:26:45.671983 4775 raft_consensus.cc:1067] T 481cd0f8c8d147adac217915a53ff6d9 P 83ffd7184ec94302bd1ee74b0fab3e17 [term 2 LEADER]: Signalling peer a2f9bf2e5d9b4fc19f804797bca4a3ff to start an election
I20240503 23:26:45.673106 3837 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "481cd0f8c8d147adac217915a53ff6d9"
dest_uuid: "a2f9bf2e5d9b4fc19f804797bca4a3ff"
from {username='slave'} at 127.0.0.1:45370
I20240503 23:26:45.673403 3837 raft_consensus.cc:482] T 481cd0f8c8d147adac217915a53ff6d9 P a2f9bf2e5d9b4fc19f804797bca4a3ff [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 23:26:45.673542 3837 raft_consensus.cc:3037] T 481cd0f8c8d147adac217915a53ff6d9 P a2f9bf2e5d9b4fc19f804797bca4a3ff [term 2 FOLLOWER]: Advancing to term 3
I20240503 23:26:45.675841 3837 raft_consensus.cc:504] T 481cd0f8c8d147adac217915a53ff6d9 P a2f9bf2e5d9b4fc19f804797bca4a3ff [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "83ffd7184ec94302bd1ee74b0fab3e17" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 32909 } } peers { permanent_uuid: "a2f9bf2e5d9b4fc19f804797bca4a3ff" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 46261 } } peers { permanent_uuid: "692da0319a2d4a81ad81f5d8f9f4c86b" member_type: VOTER last_known_addr { host: "127.0.100.5" port: 38561 } attrs { promote: false } }
I20240503 23:26:45.676743 3837 leader_election.cc:288] T 481cd0f8c8d147adac217915a53ff6d9 P a2f9bf2e5d9b4fc19f804797bca4a3ff [CANDIDATE]: Term 3 election: Requested vote from peers 83ffd7184ec94302bd1ee74b0fab3e17 (127.0.100.1:32909), 692da0319a2d4a81ad81f5d8f9f4c86b (127.0.100.5:38561)
I20240503 23:26:45.677335 4293 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "481cd0f8c8d147adac217915a53ff6d9" candidate_uuid: "a2f9bf2e5d9b4fc19f804797bca4a3ff" candidate_term: 3 candidate_status { last_received { term: 2 index: 30 } } ignore_live_leader: true dest_uuid: "692da0319a2d4a81ad81f5d8f9f4c86b"
I20240503 23:26:45.677335 3760 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "481cd0f8c8d147adac217915a53ff6d9" candidate_uuid: "a2f9bf2e5d9b4fc19f804797bca4a3ff" candidate_term: 3 candidate_status { last_received { term: 2 index: 30 } } ignore_live_leader: true dest_uuid: "83ffd7184ec94302bd1ee74b0fab3e17"
I20240503 23:26:45.677644 3760 raft_consensus.cc:3032] T 481cd0f8c8d147adac217915a53ff6d9 P 83ffd7184ec94302bd1ee74b0fab3e17 [term 2 LEADER]: Stepping down as leader of term 2
I20240503 23:26:45.677647 4293 raft_consensus.cc:3037] T 481cd0f8c8d147adac217915a53ff6d9 P 692da0319a2d4a81ad81f5d8f9f4c86b [term 2 FOLLOWER]: Advancing to term 3
I20240503 23:26:45.677773 3760 raft_consensus.cc:729] T 481cd0f8c8d147adac217915a53ff6d9 P 83ffd7184ec94302bd1ee74b0fab3e17 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 83ffd7184ec94302bd1ee74b0fab3e17, State: Running, Role: LEADER
I20240503 23:26:45.678158 3760 consensus_queue.cc:260] T 481cd0f8c8d147adac217915a53ff6d9 P 83ffd7184ec94302bd1ee74b0fab3e17 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 30, Committed index: 30, Last appended: 2.30, Last appended by leader: 30, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "83ffd7184ec94302bd1ee74b0fab3e17" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 32909 } } peers { permanent_uuid: "a2f9bf2e5d9b4fc19f804797bca4a3ff" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 46261 } } peers { permanent_uuid: "692da0319a2d4a81ad81f5d8f9f4c86b" member_type: VOTER last_known_addr { host: "127.0.100.5" port: 38561 } attrs { promote: false } }
I20240503 23:26:45.678603 3760 raft_consensus.cc:3037] T 481cd0f8c8d147adac217915a53ff6d9 P 83ffd7184ec94302bd1ee74b0fab3e17 [term 2 FOLLOWER]: Advancing to term 3
I20240503 23:26:45.681072 4293 raft_consensus.cc:2454] T 481cd0f8c8d147adac217915a53ff6d9 P 692da0319a2d4a81ad81f5d8f9f4c86b [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2f9bf2e5d9b4fc19f804797bca4a3ff in term 3.
I20240503 23:26:45.681375 3760 raft_consensus.cc:2454] T 481cd0f8c8d147adac217915a53ff6d9 P 83ffd7184ec94302bd1ee74b0fab3e17 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2f9bf2e5d9b4fc19f804797bca4a3ff in term 3.
I20240503 23:26:45.681811 3802 leader_election.cc:302] T 481cd0f8c8d147adac217915a53ff6d9 P a2f9bf2e5d9b4fc19f804797bca4a3ff [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 692da0319a2d4a81ad81f5d8f9f4c86b, a2f9bf2e5d9b4fc19f804797bca4a3ff; no voters:
I20240503 23:26:45.682183 4817 raft_consensus.cc:2781] T 481cd0f8c8d147adac217915a53ff6d9 P a2f9bf2e5d9b4fc19f804797bca4a3ff [term 3 FOLLOWER]: Leader election won for term 3
I20240503 23:26:45.682667 4817 raft_consensus.cc:686] T 481cd0f8c8d147adac217915a53ff6d9 P a2f9bf2e5d9b4fc19f804797bca4a3ff [term 3 LEADER]: Becoming Leader. State: Replica: a2f9bf2e5d9b4fc19f804797bca4a3ff, State: Running, Role: LEADER
I20240503 23:26:45.683049 4817 consensus_queue.cc:237] T 481cd0f8c8d147adac217915a53ff6d9 P a2f9bf2e5d9b4fc19f804797bca4a3ff [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 30, Committed index: 30, Last appended: 2.30, Last appended by leader: 30, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "83ffd7184ec94302bd1ee74b0fab3e17" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 32909 } } peers { permanent_uuid: "a2f9bf2e5d9b4fc19f804797bca4a3ff" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 46261 } } peers { permanent_uuid: "692da0319a2d4a81ad81f5d8f9f4c86b" member_type: VOTER last_known_addr { host: "127.0.100.5" port: 38561 } attrs { promote: false } }
I20240503 23:26:45.685671 3662 catalog_manager.cc:5484] T 481cd0f8c8d147adac217915a53ff6d9 P a2f9bf2e5d9b4fc19f804797bca4a3ff reported cstate change: term changed from 2 to 3, leader changed from 83ffd7184ec94302bd1ee74b0fab3e17 (127.0.100.1) to a2f9bf2e5d9b4fc19f804797bca4a3ff (127.0.100.2). New cstate: current_term: 3 leader_uuid: "a2f9bf2e5d9b4fc19f804797bca4a3ff" committed_config { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "83ffd7184ec94302bd1ee74b0fab3e17" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 32909 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a2f9bf2e5d9b4fc19f804797bca4a3ff" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 46261 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "692da0319a2d4a81ad81f5d8f9f4c86b" member_type: VOTER last_known_addr { host: "127.0.100.5" port: 38561 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240503 23:26:46.107784 3762 raft_consensus.cc:1261] T 481cd0f8c8d147adac217915a53ff6d9 P 83ffd7184ec94302bd1ee74b0fab3e17 [term 3 FOLLOWER]: Refusing update from remote peer a2f9bf2e5d9b4fc19f804797bca4a3ff: Log matching property violated. Preceding OpId in replica: term: 2 index: 30. Preceding OpId from leader: term: 3 index: 31. (index mismatch)
I20240503 23:26:46.108418 4817 consensus_queue.cc:1035] T 481cd0f8c8d147adac217915a53ff6d9 P a2f9bf2e5d9b4fc19f804797bca4a3ff [LEADER]: Connected to new peer: Peer: permanent_uuid: "83ffd7184ec94302bd1ee74b0fab3e17" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 32909 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 31, Last known committed idx: 30, Time since last communication: 0.000s
I20240503 23:26:46.112092 4293 raft_consensus.cc:1261] T 481cd0f8c8d147adac217915a53ff6d9 P 692da0319a2d4a81ad81f5d8f9f4c86b [term 3 FOLLOWER]: Refusing update from remote peer a2f9bf2e5d9b4fc19f804797bca4a3ff: Log matching property violated. Preceding OpId in replica: term: 2 index: 30. Preceding OpId from leader: term: 3 index: 31. (index mismatch)
I20240503 23:26:46.112696 4844 consensus_queue.cc:1035] T 481cd0f8c8d147adac217915a53ff6d9 P a2f9bf2e5d9b4fc19f804797bca4a3ff [LEADER]: Connected to new peer: Peer: permanent_uuid: "692da0319a2d4a81ad81f5d8f9f4c86b" member_type: VOTER last_known_addr { host: "127.0.100.5" port: 38561 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 31, Last known committed idx: 30, Time since last communication: 0.000s
I20240503 23:26:47.273890 3715 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240503 23:26:47.277087 3837 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "487ac5b9bf2d4de2b80a1b13387a2a77"
dest_uuid: "a2f9bf2e5d9b4fc19f804797bca4a3ff"
mode: GRACEFUL
new_leader_uuid: "83ffd7184ec94302bd1ee74b0fab3e17"
from {username='slave'} at 127.0.0.1:55472
I20240503 23:26:47.277379 3837 raft_consensus.cc:595] T 487ac5b9bf2d4de2b80a1b13387a2a77 P a2f9bf2e5d9b4fc19f804797bca4a3ff [term 2 LEADER]: Received request to transfer leadership to TS 83ffd7184ec94302bd1ee74b0fab3e17
I20240503 23:26:47.278074 3715 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 1
I20240503 23:26:47.278334 3715 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240503 23:26:47.592418 4848 raft_consensus.cc:979] T 487ac5b9bf2d4de2b80a1b13387a2a77 P a2f9bf2e5d9b4fc19f804797bca4a3ff: : Instructing follower 83ffd7184ec94302bd1ee74b0fab3e17 to start an election
I20240503 23:26:47.592617 4848 raft_consensus.cc:1067] T 487ac5b9bf2d4de2b80a1b13387a2a77 P a2f9bf2e5d9b4fc19f804797bca4a3ff [term 2 LEADER]: Signalling peer 83ffd7184ec94302bd1ee74b0fab3e17 to start an election
I20240503 23:26:47.593457 3762 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "487ac5b9bf2d4de2b80a1b13387a2a77"
dest_uuid: "83ffd7184ec94302bd1ee74b0fab3e17"
from {username='slave'} at 127.0.0.1:35586
I20240503 23:26:47.593706 3762 raft_consensus.cc:482] T 487ac5b9bf2d4de2b80a1b13387a2a77 P 83ffd7184ec94302bd1ee74b0fab3e17 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 23:26:47.593814 3762 raft_consensus.cc:3037] T 487ac5b9bf2d4de2b80a1b13387a2a77 P 83ffd7184ec94302bd1ee74b0fab3e17 [term 2 FOLLOWER]: Advancing to term 3
I20240503 23:26:47.596136 3762 raft_consensus.cc:504] T 487ac5b9bf2d4de2b80a1b13387a2a77 P 83ffd7184ec94302bd1ee74b0fab3e17 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "83ffd7184ec94302bd1ee74b0fab3e17" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 32909 } } peers { permanent_uuid: "a2f9bf2e5d9b4fc19f804797bca4a3ff" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 46261 } } peers { permanent_uuid: "692da0319a2d4a81ad81f5d8f9f4c86b" member_type: VOTER last_known_addr { host: "127.0.100.5" port: 38561 } attrs { promote: false } }
I20240503 23:26:47.596958 3762 leader_election.cc:288] T 487ac5b9bf2d4de2b80a1b13387a2a77 P 83ffd7184ec94302bd1ee74b0fab3e17 [CANDIDATE]: Term 3 election: Requested vote from peers a2f9bf2e5d9b4fc19f804797bca4a3ff (127.0.100.2:46261), 692da0319a2d4a81ad81f5d8f9f4c86b (127.0.100.5:38561)
I20240503 23:26:47.597568 3837 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "487ac5b9bf2d4de2b80a1b13387a2a77" candidate_uuid: "83ffd7184ec94302bd1ee74b0fab3e17" candidate_term: 3 candidate_status { last_received { term: 2 index: 30 } } ignore_live_leader: true dest_uuid: "a2f9bf2e5d9b4fc19f804797bca4a3ff"
I20240503 23:26:47.597568 4293 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "487ac5b9bf2d4de2b80a1b13387a2a77" candidate_uuid: "83ffd7184ec94302bd1ee74b0fab3e17" candidate_term: 3 candidate_status { last_received { term: 2 index: 30 } } ignore_live_leader: true dest_uuid: "692da0319a2d4a81ad81f5d8f9f4c86b"
I20240503 23:26:47.597780 3837 raft_consensus.cc:3032] T 487ac5b9bf2d4de2b80a1b13387a2a77 P a2f9bf2e5d9b4fc19f804797bca4a3ff [term 2 LEADER]: Stepping down as leader of term 2
I20240503 23:26:47.597812 4293 raft_consensus.cc:3037] T 487ac5b9bf2d4de2b80a1b13387a2a77 P 692da0319a2d4a81ad81f5d8f9f4c86b [term 2 FOLLOWER]: Advancing to term 3
I20240503 23:26:47.597853 3837 raft_consensus.cc:729] T 487ac5b9bf2d4de2b80a1b13387a2a77 P a2f9bf2e5d9b4fc19f804797bca4a3ff [term 2 LEADER]: Becoming Follower/Learner. State: Replica: a2f9bf2e5d9b4fc19f804797bca4a3ff, State: Running, Role: LEADER
I20240503 23:26:47.598202 3837 consensus_queue.cc:260] T 487ac5b9bf2d4de2b80a1b13387a2a77 P a2f9bf2e5d9b4fc19f804797bca4a3ff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 30, Committed index: 30, Last appended: 2.30, Last appended by leader: 30, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "83ffd7184ec94302bd1ee74b0fab3e17" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 32909 } } peers { permanent_uuid: "a2f9bf2e5d9b4fc19f804797bca4a3ff" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 46261 } } peers { permanent_uuid: "692da0319a2d4a81ad81f5d8f9f4c86b" member_type: VOTER last_known_addr { host: "127.0.100.5" port: 38561 } attrs { promote: false } }
I20240503 23:26:47.598577 3837 raft_consensus.cc:3037] T 487ac5b9bf2d4de2b80a1b13387a2a77 P a2f9bf2e5d9b4fc19f804797bca4a3ff [term 2 FOLLOWER]: Advancing to term 3
I20240503 23:26:47.600831 4293 raft_consensus.cc:2454] T 487ac5b9bf2d4de2b80a1b13387a2a77 P 692da0319a2d4a81ad81f5d8f9f4c86b [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83ffd7184ec94302bd1ee74b0fab3e17 in term 3.
I20240503 23:26:47.600831 3837 raft_consensus.cc:2454] T 487ac5b9bf2d4de2b80a1b13387a2a77 P a2f9bf2e5d9b4fc19f804797bca4a3ff [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83ffd7184ec94302bd1ee74b0fab3e17 in term 3.
I20240503 23:26:47.601429 3730 leader_election.cc:302] T 487ac5b9bf2d4de2b80a1b13387a2a77 P 83ffd7184ec94302bd1ee74b0fab3e17 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 83ffd7184ec94302bd1ee74b0fab3e17, a2f9bf2e5d9b4fc19f804797bca4a3ff; no voters:
I20240503 23:26:47.601773 4775 raft_consensus.cc:2781] T 487ac5b9bf2d4de2b80a1b13387a2a77 P 83ffd7184ec94302bd1ee74b0fab3e17 [term 3 FOLLOWER]: Leader election won for term 3
I20240503 23:26:47.602160 4775 raft_consensus.cc:686] T 487ac5b9bf2d4de2b80a1b13387a2a77 P 83ffd7184ec94302bd1ee74b0fab3e17 [term 3 LEADER]: Becoming Leader. State: Replica: 83ffd7184ec94302bd1ee74b0fab3e17, State: Running, Role: LEADER
I20240503 23:26:47.602490 4775 consensus_queue.cc:237] T 487ac5b9bf2d4de2b80a1b13387a2a77 P 83ffd7184ec94302bd1ee74b0fab3e17 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 30, Committed index: 30, Last appended: 2.30, Last appended by leader: 30, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "83ffd7184ec94302bd1ee74b0fab3e17" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 32909 } } peers { permanent_uuid: "a2f9bf2e5d9b4fc19f804797bca4a3ff" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 46261 } } peers { permanent_uuid: "692da0319a2d4a81ad81f5d8f9f4c86b" member_type: VOTER last_known_addr { host: "127.0.100.5" port: 38561 } attrs { promote: false } }
I20240503 23:26:47.605540 3659 catalog_manager.cc:5484] T 487ac5b9bf2d4de2b80a1b13387a2a77 P 83ffd7184ec94302bd1ee74b0fab3e17 reported cstate change: term changed from 2 to 3, leader changed from a2f9bf2e5d9b4fc19f804797bca4a3ff (127.0.100.2) to 83ffd7184ec94302bd1ee74b0fab3e17 (127.0.100.1). New cstate: current_term: 3 leader_uuid: "83ffd7184ec94302bd1ee74b0fab3e17" committed_config { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "83ffd7184ec94302bd1ee74b0fab3e17" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 32909 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a2f9bf2e5d9b4fc19f804797bca4a3ff" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 46261 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "692da0319a2d4a81ad81f5d8f9f4c86b" member_type: VOTER last_known_addr { host: "127.0.100.5" port: 38561 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240503 23:26:48.119786 3837 raft_consensus.cc:1261] T 487ac5b9bf2d4de2b80a1b13387a2a77 P a2f9bf2e5d9b4fc19f804797bca4a3ff [term 3 FOLLOWER]: Refusing update from remote peer 83ffd7184ec94302bd1ee74b0fab3e17: Log matching property violated. Preceding OpId in replica: term: 2 index: 30. Preceding OpId from leader: term: 3 index: 31. (index mismatch)
I20240503 23:26:48.120720 4775 consensus_queue.cc:1035] T 487ac5b9bf2d4de2b80a1b13387a2a77 P 83ffd7184ec94302bd1ee74b0fab3e17 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2f9bf2e5d9b4fc19f804797bca4a3ff" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 46261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 31, Last known committed idx: 30, Time since last communication: 0.000s
I20240503 23:26:48.123922 4293 raft_consensus.cc:1261] T 487ac5b9bf2d4de2b80a1b13387a2a77 P 692da0319a2d4a81ad81f5d8f9f4c86b [term 3 FOLLOWER]: Refusing update from remote peer 83ffd7184ec94302bd1ee74b0fab3e17: Log matching property violated. Preceding OpId in replica: term: 2 index: 30. Preceding OpId from leader: term: 3 index: 31. (index mismatch)
I20240503 23:26:48.125735 4859 consensus_queue.cc:1035] T 487ac5b9bf2d4de2b80a1b13387a2a77 P 83ffd7184ec94302bd1ee74b0fab3e17 [LEADER]: Connected to new peer: Peer: permanent_uuid: "692da0319a2d4a81ad81f5d8f9f4c86b" member_type: VOTER last_known_addr { host: "127.0.100.5" port: 38561 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 31, Last known committed idx: 30, Time since last communication: 0.000s
I20240503 23:26:49.278846 3715 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240503 23:26:49.282307 3762 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "487ac5b9bf2d4de2b80a1b13387a2a77"
dest_uuid: "83ffd7184ec94302bd1ee74b0fab3e17"
mode: GRACEFUL
new_leader_uuid: "a2f9bf2e5d9b4fc19f804797bca4a3ff"
from {username='slave'} at 127.0.0.1:57936
I20240503 23:26:49.282604 3762 raft_consensus.cc:595] T 487ac5b9bf2d4de2b80a1b13387a2a77 P 83ffd7184ec94302bd1ee74b0fab3e17 [term 3 LEADER]: Received request to transfer leadership to TS a2f9bf2e5d9b4fc19f804797bca4a3ff
I20240503 23:26:49.283303 3715 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 1
I20240503 23:26:49.283591 3715 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240503 23:26:49.680081 4866 raft_consensus.cc:979] T 487ac5b9bf2d4de2b80a1b13387a2a77 P 83ffd7184ec94302bd1ee74b0fab3e17: : Instructing follower a2f9bf2e5d9b4fc19f804797bca4a3ff to start an election
I20240503 23:26:49.680253 4866 raft_consensus.cc:1067] T 487ac5b9bf2d4de2b80a1b13387a2a77 P 83ffd7184ec94302bd1ee74b0fab3e17 [term 3 LEADER]: Signalling peer a2f9bf2e5d9b4fc19f804797bca4a3ff to start an election
I20240503 23:26:49.680999 3837 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "487ac5b9bf2d4de2b80a1b13387a2a77"
dest_uuid: "a2f9bf2e5d9b4fc19f804797bca4a3ff"
from {username='slave'} at 127.0.0.1:45370
I20240503 23:26:49.681217 3837 raft_consensus.cc:482] T 487ac5b9bf2d4de2b80a1b13387a2a77 P a2f9bf2e5d9b4fc19f804797bca4a3ff [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 23:26:49.681301 3837 raft_consensus.cc:3037] T 487ac5b9bf2d4de2b80a1b13387a2a77 P a2f9bf2e5d9b4fc19f804797bca4a3ff [term 3 FOLLOWER]: Advancing to term 4
I20240503 23:26:49.683815 3837 raft_consensus.cc:504] T 487ac5b9bf2d4de2b80a1b13387a2a77 P a2f9bf2e5d9b4fc19f804797bca4a3ff [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "83ffd7184ec94302bd1ee74b0fab3e17" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 32909 } } peers { permanent_uuid: "a2f9bf2e5d9b4fc19f804797bca4a3ff" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 46261 } } peers { permanent_uuid: "692da0319a2d4a81ad81f5d8f9f4c86b" member_type: VOTER last_known_addr { host: "127.0.100.5" port: 38561 } attrs { promote: false } }
I20240503 23:26:49.684661 3837 leader_election.cc:288] T 487ac5b9bf2d4de2b80a1b13387a2a77 P a2f9bf2e5d9b4fc19f804797bca4a3ff [CANDIDATE]: Term 4 election: Requested vote from peers 83ffd7184ec94302bd1ee74b0fab3e17 (127.0.100.1:32909), 692da0319a2d4a81ad81f5d8f9f4c86b (127.0.100.5:38561)
I20240503 23:26:49.685004 3762 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "487ac5b9bf2d4de2b80a1b13387a2a77" candidate_uuid: "a2f9bf2e5d9b4fc19f804797bca4a3ff" candidate_term: 4 candidate_status { last_received { term: 3 index: 31 } } ignore_live_leader: true dest_uuid: "83ffd7184ec94302bd1ee74b0fab3e17"
I20240503 23:26:49.685127 4293 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "487ac5b9bf2d4de2b80a1b13387a2a77" candidate_uuid: "a2f9bf2e5d9b4fc19f804797bca4a3ff" candidate_term: 4 candidate_status { last_received { term: 3 index: 31 } } ignore_live_leader: true dest_uuid: "692da0319a2d4a81ad81f5d8f9f4c86b"
I20240503 23:26:49.685215 3762 raft_consensus.cc:3032] T 487ac5b9bf2d4de2b80a1b13387a2a77 P 83ffd7184ec94302bd1ee74b0fab3e17 [term 3 LEADER]: Stepping down as leader of term 3
I20240503 23:26:49.685295 3762 raft_consensus.cc:729] T 487ac5b9bf2d4de2b80a1b13387a2a77 P 83ffd7184ec94302bd1ee74b0fab3e17 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 83ffd7184ec94302bd1ee74b0fab3e17, State: Running, Role: LEADER
I20240503 23:26:49.685302 4293 raft_consensus.cc:3037] T 487ac5b9bf2d4de2b80a1b13387a2a77 P 692da0319a2d4a81ad81f5d8f9f4c86b [term 3 FOLLOWER]: Advancing to term 4
I20240503 23:26:49.685642 3762 consensus_queue.cc:260] T 487ac5b9bf2d4de2b80a1b13387a2a77 P 83ffd7184ec94302bd1ee74b0fab3e17 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 31, Committed index: 31, Last appended: 3.31, Last appended by leader: 31, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "83ffd7184ec94302bd1ee74b0fab3e17" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 32909 } } peers { permanent_uuid: "a2f9bf2e5d9b4fc19f804797bca4a3ff" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 46261 } } peers { permanent_uuid: "692da0319a2d4a81ad81f5d8f9f4c86b" member_type: VOTER last_known_addr { host: "127.0.100.5" port: 38561 } attrs { promote: false } }
I20240503 23:26:49.685966 3762 raft_consensus.cc:3037] T 487ac5b9bf2d4de2b80a1b13387a2a77 P 83ffd7184ec94302bd1ee74b0fab3e17 [term 3 FOLLOWER]: Advancing to term 4
I20240503 23:26:49.687636 4293 raft_consensus.cc:2454] T 487ac5b9bf2d4de2b80a1b13387a2a77 P 692da0319a2d4a81ad81f5d8f9f4c86b [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2f9bf2e5d9b4fc19f804797bca4a3ff in term 4.
I20240503 23:26:49.688124 3762 raft_consensus.cc:2454] T 487ac5b9bf2d4de2b80a1b13387a2a77 P 83ffd7184ec94302bd1ee74b0fab3e17 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2f9bf2e5d9b4fc19f804797bca4a3ff in term 4.
I20240503 23:26:49.688297 3802 leader_election.cc:302] T 487ac5b9bf2d4de2b80a1b13387a2a77 P a2f9bf2e5d9b4fc19f804797bca4a3ff [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 692da0319a2d4a81ad81f5d8f9f4c86b, a2f9bf2e5d9b4fc19f804797bca4a3ff; no voters:
I20240503 23:26:49.688643 4864 raft_consensus.cc:2781] T 487ac5b9bf2d4de2b80a1b13387a2a77 P a2f9bf2e5d9b4fc19f804797bca4a3ff [term 4 FOLLOWER]: Leader election won for term 4
I20240503 23:26:49.689126 4864 raft_consensus.cc:686] T 487ac5b9bf2d4de2b80a1b13387a2a77 P a2f9bf2e5d9b4fc19f804797bca4a3ff [term 4 LEADER]: Becoming Leader. State: Replica: a2f9bf2e5d9b4fc19f804797bca4a3ff, State: Running, Role: LEADER
I20240503 23:26:49.689415 4864 consensus_queue.cc:237] T 487ac5b9bf2d4de2b80a1b13387a2a77 P a2f9bf2e5d9b4fc19f804797bca4a3ff [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 31, Committed index: 31, Last appended: 3.31, Last appended by leader: 31, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "83ffd7184ec94302bd1ee74b0fab3e17" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 32909 } } peers { permanent_uuid: "a2f9bf2e5d9b4fc19f804797bca4a3ff" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 46261 } } peers { permanent_uuid: "692da0319a2d4a81ad81f5d8f9f4c86b" member_type: VOTER last_known_addr { host: "127.0.100.5" port: 38561 } attrs { promote: false } }
I20240503 23:26:49.692768 3655 catalog_manager.cc:5484] T 487ac5b9bf2d4de2b80a1b13387a2a77 P a2f9bf2e5d9b4fc19f804797bca4a3ff reported cstate change: term changed from 3 to 4, leader changed from 83ffd7184ec94302bd1ee74b0fab3e17 (127.0.100.1) to a2f9bf2e5d9b4fc19f804797bca4a3ff (127.0.100.2). New cstate: current_term: 4 leader_uuid: "a2f9bf2e5d9b4fc19f804797bca4a3ff" committed_config { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "83ffd7184ec94302bd1ee74b0fab3e17" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 32909 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a2f9bf2e5d9b4fc19f804797bca4a3ff" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 46261 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "692da0319a2d4a81ad81f5d8f9f4c86b" member_type: VOTER last_known_addr { host: "127.0.100.5" port: 38561 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240503 23:26:50.068261 3762 raft_consensus.cc:1261] T 487ac5b9bf2d4de2b80a1b13387a2a77 P 83ffd7184ec94302bd1ee74b0fab3e17 [term 4 FOLLOWER]: Refusing update from remote peer a2f9bf2e5d9b4fc19f804797bca4a3ff: Log matching property violated. Preceding OpId in replica: term: 3 index: 31. Preceding OpId from leader: term: 4 index: 32. (index mismatch)
I20240503 23:26:50.069173 4864 consensus_queue.cc:1035] T 487ac5b9bf2d4de2b80a1b13387a2a77 P a2f9bf2e5d9b4fc19f804797bca4a3ff [LEADER]: Connected to new peer: Peer: permanent_uuid: "83ffd7184ec94302bd1ee74b0fab3e17" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 32909 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 32, Last known committed idx: 31, Time since last communication: 0.000s
I20240503 23:26:50.076462 4293 raft_consensus.cc:1261] T 487ac5b9bf2d4de2b80a1b13387a2a77 P 692da0319a2d4a81ad81f5d8f9f4c86b [term 4 FOLLOWER]: Refusing update from remote peer a2f9bf2e5d9b4fc19f804797bca4a3ff: Log matching property violated. Preceding OpId in replica: term: 3 index: 31. Preceding OpId from leader: term: 4 index: 32. (index mismatch)
I20240503 23:26:50.077289 4876 consensus_queue.cc:1035] T 487ac5b9bf2d4de2b80a1b13387a2a77 P a2f9bf2e5d9b4fc19f804797bca4a3ff [LEADER]: Connected to new peer: Peer: permanent_uuid: "692da0319a2d4a81ad81f5d8f9f4c86b" member_type: VOTER last_known_addr { host: "127.0.100.5" port: 38561 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 32, Last known committed idx: 31, Time since last communication: 0.000s
I20240503 23:26:51.284036 3715 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240503 23:26:51.287235 3837 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "481cd0f8c8d147adac217915a53ff6d9"
dest_uuid: "a2f9bf2e5d9b4fc19f804797bca4a3ff"
mode: GRACEFUL
new_leader_uuid: "83ffd7184ec94302bd1ee74b0fab3e17"
from {username='slave'} at 127.0.0.1:55472
I20240503 23:26:51.287497 3837 raft_consensus.cc:595] T 481cd0f8c8d147adac217915a53ff6d9 P a2f9bf2e5d9b4fc19f804797bca4a3ff [term 3 LEADER]: Received request to transfer leadership to TS 83ffd7184ec94302bd1ee74b0fab3e17
I20240503 23:26:51.288060 3715 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 1
I20240503 23:26:51.288336 3715 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240503 23:26:51.401253 4876 raft_consensus.cc:979] T 481cd0f8c8d147adac217915a53ff6d9 P a2f9bf2e5d9b4fc19f804797bca4a3ff: : Instructing follower 83ffd7184ec94302bd1ee74b0fab3e17 to start an election
I20240503 23:26:51.401444 4876 raft_consensus.cc:1067] T 481cd0f8c8d147adac217915a53ff6d9 P a2f9bf2e5d9b4fc19f804797bca4a3ff [term 3 LEADER]: Signalling peer 83ffd7184ec94302bd1ee74b0fab3e17 to start an election
I20240503 23:26:51.402169 3762 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "481cd0f8c8d147adac217915a53ff6d9"
dest_uuid: "83ffd7184ec94302bd1ee74b0fab3e17"
from {username='slave'} at 127.0.0.1:35586
I20240503 23:26:51.402400 3762 raft_consensus.cc:482] T 481cd0f8c8d147adac217915a53ff6d9 P 83ffd7184ec94302bd1ee74b0fab3e17 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 23:26:51.402493 3762 raft_consensus.cc:3037] T 481cd0f8c8d147adac217915a53ff6d9 P 83ffd7184ec94302bd1ee74b0fab3e17 [term 3 FOLLOWER]: Advancing to term 4
I20240503 23:26:51.404750 3762 raft_consensus.cc:504] T 481cd0f8c8d147adac217915a53ff6d9 P 83ffd7184ec94302bd1ee74b0fab3e17 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "83ffd7184ec94302bd1ee74b0fab3e17" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 32909 } } peers { permanent_uuid: "a2f9bf2e5d9b4fc19f804797bca4a3ff" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 46261 } } peers { permanent_uuid: "692da0319a2d4a81ad81f5d8f9f4c86b" member_type: VOTER last_known_addr { host: "127.0.100.5" port: 38561 } attrs { promote: false } }
I20240503 23:26:51.405689 3762 leader_election.cc:288] T 481cd0f8c8d147adac217915a53ff6d9 P 83ffd7184ec94302bd1ee74b0fab3e17 [CANDIDATE]: Term 4 election: Requested vote from peers a2f9bf2e5d9b4fc19f804797bca4a3ff (127.0.100.2:46261), 692da0319a2d4a81ad81f5d8f9f4c86b (127.0.100.5:38561)
I20240503 23:26:51.406200 3837 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "481cd0f8c8d147adac217915a53ff6d9" candidate_uuid: "83ffd7184ec94302bd1ee74b0fab3e17" candidate_term: 4 candidate_status { last_received { term: 3 index: 31 } } ignore_live_leader: true dest_uuid: "a2f9bf2e5d9b4fc19f804797bca4a3ff"
I20240503 23:26:51.406313 4293 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "481cd0f8c8d147adac217915a53ff6d9" candidate_uuid: "83ffd7184ec94302bd1ee74b0fab3e17" candidate_term: 4 candidate_status { last_received { term: 3 index: 31 } } ignore_live_leader: true dest_uuid: "692da0319a2d4a81ad81f5d8f9f4c86b"
I20240503 23:26:51.406463 4293 raft_consensus.cc:3037] T 481cd0f8c8d147adac217915a53ff6d9 P 692da0319a2d4a81ad81f5d8f9f4c86b [term 3 FOLLOWER]: Advancing to term 4
I20240503 23:26:51.406447 3837 raft_consensus.cc:3032] T 481cd0f8c8d147adac217915a53ff6d9 P a2f9bf2e5d9b4fc19f804797bca4a3ff [term 3 LEADER]: Stepping down as leader of term 3
I20240503 23:26:51.408743 3837 raft_consensus.cc:729] T 481cd0f8c8d147adac217915a53ff6d9 P a2f9bf2e5d9b4fc19f804797bca4a3ff [term 3 LEADER]: Becoming Follower/Learner. State: Replica: a2f9bf2e5d9b4fc19f804797bca4a3ff, State: Running, Role: LEADER
I20240503 23:26:51.408758 4293 raft_consensus.cc:2454] T 481cd0f8c8d147adac217915a53ff6d9 P 692da0319a2d4a81ad81f5d8f9f4c86b [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83ffd7184ec94302bd1ee74b0fab3e17 in term 4.
I20240503 23:26:51.409101 3837 consensus_queue.cc:260] T 481cd0f8c8d147adac217915a53ff6d9 P a2f9bf2e5d9b4fc19f804797bca4a3ff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 31, Committed index: 31, Last appended: 3.31, Last appended by leader: 31, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "83ffd7184ec94302bd1ee74b0fab3e17" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 32909 } } peers { permanent_uuid: "a2f9bf2e5d9b4fc19f804797bca4a3ff" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 46261 } } peers { permanent_uuid: "692da0319a2d4a81ad81f5d8f9f4c86b" member_type: VOTER last_known_addr { host: "127.0.100.5" port: 38561 } attrs { promote: false } }
I20240503 23:26:51.409427 3727 leader_election.cc:302] T 481cd0f8c8d147adac217915a53ff6d9 P 83ffd7184ec94302bd1ee74b0fab3e17 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 692da0319a2d4a81ad81f5d8f9f4c86b, 83ffd7184ec94302bd1ee74b0fab3e17; no voters:
I20240503 23:26:51.409559 3837 raft_consensus.cc:3037] T 481cd0f8c8d147adac217915a53ff6d9 P a2f9bf2e5d9b4fc19f804797bca4a3ff [term 3 FOLLOWER]: Advancing to term 4
I20240503 23:26:51.409746 4866 raft_consensus.cc:2781] T 481cd0f8c8d147adac217915a53ff6d9 P 83ffd7184ec94302bd1ee74b0fab3e17 [term 4 FOLLOWER]: Leader election won for term 4
I20240503 23:26:51.410293 4866 raft_consensus.cc:686] T 481cd0f8c8d147adac217915a53ff6d9 P 83ffd7184ec94302bd1ee74b0fab3e17 [term 4 LEADER]: Becoming Leader. State: Replica: 83ffd7184ec94302bd1ee74b0fab3e17, State: Running, Role: LEADER
I20240503 23:26:51.410714 4866 consensus_queue.cc:237] T 481cd0f8c8d147adac217915a53ff6d9 P 83ffd7184ec94302bd1ee74b0fab3e17 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 31, Committed index: 31, Last appended: 3.31, Last appended by leader: 31, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "83ffd7184ec94302bd1ee74b0fab3e17" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 32909 } } peers { permanent_uuid: "a2f9bf2e5d9b4fc19f804797bca4a3ff" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 46261 } } peers { permanent_uuid: "692da0319a2d4a81ad81f5d8f9f4c86b" member_type: VOTER last_known_addr { host: "127.0.100.5" port: 38561 } attrs { promote: false } }
I20240503 23:26:51.412523 3837 raft_consensus.cc:2454] T 481cd0f8c8d147adac217915a53ff6d9 P a2f9bf2e5d9b4fc19f804797bca4a3ff [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83ffd7184ec94302bd1ee74b0fab3e17 in term 4.
I20240503 23:26:51.414005 3659 catalog_manager.cc:5484] T 481cd0f8c8d147adac217915a53ff6d9 P 83ffd7184ec94302bd1ee74b0fab3e17 reported cstate change: term changed from 3 to 4, leader changed from a2f9bf2e5d9b4fc19f804797bca4a3ff (127.0.100.2) to 83ffd7184ec94302bd1ee74b0fab3e17 (127.0.100.1). New cstate: current_term: 4 leader_uuid: "83ffd7184ec94302bd1ee74b0fab3e17" committed_config { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "83ffd7184ec94302bd1ee74b0fab3e17" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 32909 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a2f9bf2e5d9b4fc19f804797bca4a3ff" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 46261 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "692da0319a2d4a81ad81f5d8f9f4c86b" member_type: VOTER last_known_addr { host: "127.0.100.5" port: 38561 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240503 23:26:52.017793 3837 raft_consensus.cc:1261] T 481cd0f8c8d147adac217915a53ff6d9 P a2f9bf2e5d9b4fc19f804797bca4a3ff [term 4 FOLLOWER]: Refusing update from remote peer 83ffd7184ec94302bd1ee74b0fab3e17: Log matching property violated. Preceding OpId in replica: term: 3 index: 31. Preceding OpId from leader: term: 4 index: 32. (index mismatch)
I20240503 23:26:52.018435 4866 consensus_queue.cc:1035] T 481cd0f8c8d147adac217915a53ff6d9 P 83ffd7184ec94302bd1ee74b0fab3e17 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2f9bf2e5d9b4fc19f804797bca4a3ff" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 46261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 32, Last known committed idx: 31, Time since last communication: 0.000s
I20240503 23:26:52.019753 4293 raft_consensus.cc:1261] T 481cd0f8c8d147adac217915a53ff6d9 P 692da0319a2d4a81ad81f5d8f9f4c86b [term 4 FOLLOWER]: Refusing update from remote peer 83ffd7184ec94302bd1ee74b0fab3e17: Log matching property violated. Preceding OpId in replica: term: 3 index: 31. Preceding OpId from leader: term: 4 index: 32. (index mismatch)
I20240503 23:26:52.020339 4866 consensus_queue.cc:1035] T 481cd0f8c8d147adac217915a53ff6d9 P 83ffd7184ec94302bd1ee74b0fab3e17 [LEADER]: Connected to new peer: Peer: permanent_uuid: "692da0319a2d4a81ad81f5d8f9f4c86b" member_type: VOTER last_known_addr { host: "127.0.100.5" port: 38561 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 32, Last known committed idx: 31, Time since last communication: 0.000s
I20240503 23:26:53.288936 3715 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240503 23:26:53.292277 3762 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "481cd0f8c8d147adac217915a53ff6d9"
dest_uuid: "83ffd7184ec94302bd1ee74b0fab3e17"
mode: GRACEFUL
new_leader_uuid: "a2f9bf2e5d9b4fc19f804797bca4a3ff"
from {username='slave'} at 127.0.0.1:57936
I20240503 23:26:53.292593 3762 raft_consensus.cc:595] T 481cd0f8c8d147adac217915a53ff6d9 P 83ffd7184ec94302bd1ee74b0fab3e17 [term 4 LEADER]: Received request to transfer leadership to TS a2f9bf2e5d9b4fc19f804797bca4a3ff
I20240503 23:26:53.293175 3715 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 1
I20240503 23:26:53.293417 3715 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240503 23:26:53.434430 4893 raft_consensus.cc:979] T 481cd0f8c8d147adac217915a53ff6d9 P 83ffd7184ec94302bd1ee74b0fab3e17: : Instructing follower a2f9bf2e5d9b4fc19f804797bca4a3ff to start an election
I20240503 23:26:53.434648 4893 raft_consensus.cc:1067] T 481cd0f8c8d147adac217915a53ff6d9 P 83ffd7184ec94302bd1ee74b0fab3e17 [term 4 LEADER]: Signalling peer a2f9bf2e5d9b4fc19f804797bca4a3ff to start an election
I20240503 23:26:53.435432 3837 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "481cd0f8c8d147adac217915a53ff6d9"
dest_uuid: "a2f9bf2e5d9b4fc19f804797bca4a3ff"
from {username='slave'} at 127.0.0.1:45370
I20240503 23:26:53.435659 3837 raft_consensus.cc:482] T 481cd0f8c8d147adac217915a53ff6d9 P a2f9bf2e5d9b4fc19f804797bca4a3ff [term 4 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 23:26:53.435760 3837 raft_consensus.cc:3037] T 481cd0f8c8d147adac217915a53ff6d9 P a2f9bf2e5d9b4fc19f804797bca4a3ff [term 4 FOLLOWER]: Advancing to term 5
I20240503 23:26:53.438057 3837 raft_consensus.cc:504] T 481cd0f8c8d147adac217915a53ff6d9 P a2f9bf2e5d9b4fc19f804797bca4a3ff [term 5 FOLLOWER]: Starting forced leader election with config: opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "83ffd7184ec94302bd1ee74b0fab3e17" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 32909 } } peers { permanent_uuid: "a2f9bf2e5d9b4fc19f804797bca4a3ff" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 46261 } } peers { permanent_uuid: "692da0319a2d4a81ad81f5d8f9f4c86b" member_type: VOTER last_known_addr { host: "127.0.100.5" port: 38561 } attrs { promote: false } }
I20240503 23:26:53.438784 3837 leader_election.cc:288] T 481cd0f8c8d147adac217915a53ff6d9 P a2f9bf2e5d9b4fc19f804797bca4a3ff [CANDIDATE]: Term 5 election: Requested vote from peers 83ffd7184ec94302bd1ee74b0fab3e17 (127.0.100.1:32909), 692da0319a2d4a81ad81f5d8f9f4c86b (127.0.100.5:38561)
I20240503 23:26:53.439440 4293 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "481cd0f8c8d147adac217915a53ff6d9" candidate_uuid: "a2f9bf2e5d9b4fc19f804797bca4a3ff" candidate_term: 5 candidate_status { last_received { term: 4 index: 32 } } ignore_live_leader: true dest_uuid: "692da0319a2d4a81ad81f5d8f9f4c86b"
I20240503 23:26:53.439440 3762 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "481cd0f8c8d147adac217915a53ff6d9" candidate_uuid: "a2f9bf2e5d9b4fc19f804797bca4a3ff" candidate_term: 5 candidate_status { last_received { term: 4 index: 32 } } ignore_live_leader: true dest_uuid: "83ffd7184ec94302bd1ee74b0fab3e17"
I20240503 23:26:53.439700 3762 raft_consensus.cc:3032] T 481cd0f8c8d147adac217915a53ff6d9 P 83ffd7184ec94302bd1ee74b0fab3e17 [term 4 LEADER]: Stepping down as leader of term 4
I20240503 23:26:53.439700 4293 raft_consensus.cc:3037] T 481cd0f8c8d147adac217915a53ff6d9 P 692da0319a2d4a81ad81f5d8f9f4c86b [term 4 FOLLOWER]: Advancing to term 5
I20240503 23:26:53.439810 3762 raft_consensus.cc:729] T 481cd0f8c8d147adac217915a53ff6d9 P 83ffd7184ec94302bd1ee74b0fab3e17 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 83ffd7184ec94302bd1ee74b0fab3e17, State: Running, Role: LEADER
I20240503 23:26:53.440111 3762 consensus_queue.cc:260] T 481cd0f8c8d147adac217915a53ff6d9 P 83ffd7184ec94302bd1ee74b0fab3e17 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 32, Committed index: 32, Last appended: 4.32, Last appended by leader: 32, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "83ffd7184ec94302bd1ee74b0fab3e17" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 32909 } } peers { permanent_uuid: "a2f9bf2e5d9b4fc19f804797bca4a3ff" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 46261 } } peers { permanent_uuid: "692da0319a2d4a81ad81f5d8f9f4c86b" member_type: VOTER last_known_addr { host: "127.0.100.5" port: 38561 } attrs { promote: false } }
I20240503 23:26:53.440526 3762 raft_consensus.cc:3037] T 481cd0f8c8d147adac217915a53ff6d9 P 83ffd7184ec94302bd1ee74b0fab3e17 [term 4 FOLLOWER]: Advancing to term 5
I20240503 23:26:53.442845 4293 raft_consensus.cc:2454] T 481cd0f8c8d147adac217915a53ff6d9 P 692da0319a2d4a81ad81f5d8f9f4c86b [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2f9bf2e5d9b4fc19f804797bca4a3ff in term 5.
I20240503 23:26:53.443048 3762 raft_consensus.cc:2454] T 481cd0f8c8d147adac217915a53ff6d9 P 83ffd7184ec94302bd1ee74b0fab3e17 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2f9bf2e5d9b4fc19f804797bca4a3ff in term 5.
I20240503 23:26:53.443526 3804 leader_election.cc:302] T 481cd0f8c8d147adac217915a53ff6d9 P a2f9bf2e5d9b4fc19f804797bca4a3ff [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 83ffd7184ec94302bd1ee74b0fab3e17, a2f9bf2e5d9b4fc19f804797bca4a3ff; no voters:
I20240503 23:26:53.443897 4876 raft_consensus.cc:2781] T 481cd0f8c8d147adac217915a53ff6d9 P a2f9bf2e5d9b4fc19f804797bca4a3ff [term 5 FOLLOWER]: Leader election won for term 5
I20240503 23:26:53.444396 4876 raft_consensus.cc:686] T 481cd0f8c8d147adac217915a53ff6d9 P a2f9bf2e5d9b4fc19f804797bca4a3ff [term 5 LEADER]: Becoming Leader. State: Replica: a2f9bf2e5d9b4fc19f804797bca4a3ff, State: Running, Role: LEADER
I20240503 23:26:53.444749 4876 consensus_queue.cc:237] T 481cd0f8c8d147adac217915a53ff6d9 P a2f9bf2e5d9b4fc19f804797bca4a3ff [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 32, Committed index: 32, Last appended: 4.32, Last appended by leader: 32, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "83ffd7184ec94302bd1ee74b0fab3e17" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 32909 } } peers { permanent_uuid: "a2f9bf2e5d9b4fc19f804797bca4a3ff" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 46261 } } peers { permanent_uuid: "692da0319a2d4a81ad81f5d8f9f4c86b" member_type: VOTER last_known_addr { host: "127.0.100.5" port: 38561 } attrs { promote: false } }
I20240503 23:26:53.447679 3659 catalog_manager.cc:5484] T 481cd0f8c8d147adac217915a53ff6d9 P a2f9bf2e5d9b4fc19f804797bca4a3ff reported cstate change: term changed from 4 to 5, leader changed from 83ffd7184ec94302bd1ee74b0fab3e17 (127.0.100.1) to a2f9bf2e5d9b4fc19f804797bca4a3ff (127.0.100.2). New cstate: current_term: 5 leader_uuid: "a2f9bf2e5d9b4fc19f804797bca4a3ff" committed_config { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "83ffd7184ec94302bd1ee74b0fab3e17" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 32909 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a2f9bf2e5d9b4fc19f804797bca4a3ff" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 46261 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "692da0319a2d4a81ad81f5d8f9f4c86b" member_type: VOTER last_known_addr { host: "127.0.100.5" port: 38561 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240503 23:26:53.853861 4293 raft_consensus.cc:1261] T 481cd0f8c8d147adac217915a53ff6d9 P 692da0319a2d4a81ad81f5d8f9f4c86b [term 5 FOLLOWER]: Refusing update from remote peer a2f9bf2e5d9b4fc19f804797bca4a3ff: Log matching property violated. Preceding OpId in replica: term: 4 index: 32. Preceding OpId from leader: term: 5 index: 33. (index mismatch)
I20240503 23:26:53.854610 4876 consensus_queue.cc:1035] T 481cd0f8c8d147adac217915a53ff6d9 P a2f9bf2e5d9b4fc19f804797bca4a3ff [LEADER]: Connected to new peer: Peer: permanent_uuid: "692da0319a2d4a81ad81f5d8f9f4c86b" member_type: VOTER last_known_addr { host: "127.0.100.5" port: 38561 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 33, Last known committed idx: 32, Time since last communication: 0.000s
I20240503 23:26:53.861969 3762 raft_consensus.cc:1261] T 481cd0f8c8d147adac217915a53ff6d9 P 83ffd7184ec94302bd1ee74b0fab3e17 [term 5 FOLLOWER]: Refusing update from remote peer a2f9bf2e5d9b4fc19f804797bca4a3ff: Log matching property violated. Preceding OpId in replica: term: 4 index: 32. Preceding OpId from leader: term: 5 index: 33. (index mismatch)
I20240503 23:26:53.862668 4903 consensus_queue.cc:1035] T 481cd0f8c8d147adac217915a53ff6d9 P a2f9bf2e5d9b4fc19f804797bca4a3ff [LEADER]: Connected to new peer: Peer: permanent_uuid: "83ffd7184ec94302bd1ee74b0fab3e17" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 32909 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 33, Last known committed idx: 32, Time since last communication: 0.000s
I20240503 23:26:55.293983 3715 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240503 23:26:55.297226 3838 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "059f6233ea514fe69b8c6504efabeff0"
dest_uuid: "a2f9bf2e5d9b4fc19f804797bca4a3ff"
mode: GRACEFUL
new_leader_uuid: "a3eecbaec43d4954b1096bc3121e824c"
from {username='slave'} at 127.0.0.1:55472
I20240503 23:26:55.297582 3838 raft_consensus.cc:595] T 059f6233ea514fe69b8c6504efabeff0 P a2f9bf2e5d9b4fc19f804797bca4a3ff [term 2 LEADER]: Received request to transfer leadership to TS a3eecbaec43d4954b1096bc3121e824c
I20240503 23:26:55.298207 3715 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 1
I20240503 23:26:55.298529 3715 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240503 23:26:55.429919 4903 raft_consensus.cc:979] T 059f6233ea514fe69b8c6504efabeff0 P a2f9bf2e5d9b4fc19f804797bca4a3ff: : Instructing follower a3eecbaec43d4954b1096bc3121e824c to start an election
I20240503 23:26:55.430158 4903 raft_consensus.cc:1067] T 059f6233ea514fe69b8c6504efabeff0 P a2f9bf2e5d9b4fc19f804797bca4a3ff [term 2 LEADER]: Signalling peer a3eecbaec43d4954b1096bc3121e824c to start an election
I20240503 23:26:55.431061 3907 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "059f6233ea514fe69b8c6504efabeff0"
dest_uuid: "a3eecbaec43d4954b1096bc3121e824c"
from {username='slave'} at 127.0.0.1:40636
I20240503 23:26:55.431300 3907 raft_consensus.cc:482] T 059f6233ea514fe69b8c6504efabeff0 P a3eecbaec43d4954b1096bc3121e824c [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 23:26:55.431402 3907 raft_consensus.cc:3037] T 059f6233ea514fe69b8c6504efabeff0 P a3eecbaec43d4954b1096bc3121e824c [term 2 FOLLOWER]: Advancing to term 3
I20240503 23:26:55.433651 3907 raft_consensus.cc:504] T 059f6233ea514fe69b8c6504efabeff0 P a3eecbaec43d4954b1096bc3121e824c [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "a2f9bf2e5d9b4fc19f804797bca4a3ff" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 46261 } } peers { permanent_uuid: "a3eecbaec43d4954b1096bc3121e824c" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 40649 } } peers { permanent_uuid: "4d1d4e261584467fbbce4c72fa18045a" member_type: VOTER last_known_addr { host: "127.0.100.6" port: 35349 } attrs { promote: false } }
I20240503 23:26:55.434542 3907 leader_election.cc:288] T 059f6233ea514fe69b8c6504efabeff0 P a3eecbaec43d4954b1096bc3121e824c [CANDIDATE]: Term 3 election: Requested vote from peers a2f9bf2e5d9b4fc19f804797bca4a3ff (127.0.100.2:46261), 4d1d4e261584467fbbce4c72fa18045a (127.0.100.6:35349)
I20240503 23:26:55.435209 4366 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "059f6233ea514fe69b8c6504efabeff0" candidate_uuid: "a3eecbaec43d4954b1096bc3121e824c" candidate_term: 3 candidate_status { last_received { term: 2 index: 30 } } ignore_live_leader: true dest_uuid: "4d1d4e261584467fbbce4c72fa18045a"
I20240503 23:26:55.435207 3838 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "059f6233ea514fe69b8c6504efabeff0" candidate_uuid: "a3eecbaec43d4954b1096bc3121e824c" candidate_term: 3 candidate_status { last_received { term: 2 index: 30 } } ignore_live_leader: true dest_uuid: "a2f9bf2e5d9b4fc19f804797bca4a3ff"
I20240503 23:26:55.435496 4366 raft_consensus.cc:3037] T 059f6233ea514fe69b8c6504efabeff0 P 4d1d4e261584467fbbce4c72fa18045a [term 2 FOLLOWER]: Advancing to term 3
I20240503 23:26:55.435500 3838 raft_consensus.cc:3032] T 059f6233ea514fe69b8c6504efabeff0 P a2f9bf2e5d9b4fc19f804797bca4a3ff [term 2 LEADER]: Stepping down as leader of term 2
I20240503 23:26:55.435616 3838 raft_consensus.cc:729] T 059f6233ea514fe69b8c6504efabeff0 P a2f9bf2e5d9b4fc19f804797bca4a3ff [term 2 LEADER]: Becoming Follower/Learner. State: Replica: a2f9bf2e5d9b4fc19f804797bca4a3ff, State: Running, Role: LEADER
I20240503 23:26:55.435956 3838 consensus_queue.cc:260] T 059f6233ea514fe69b8c6504efabeff0 P a2f9bf2e5d9b4fc19f804797bca4a3ff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 30, Committed index: 30, Last appended: 2.30, Last appended by leader: 30, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "a2f9bf2e5d9b4fc19f804797bca4a3ff" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 46261 } } peers { permanent_uuid: "a3eecbaec43d4954b1096bc3121e824c" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 40649 } } peers { permanent_uuid: "4d1d4e261584467fbbce4c72fa18045a" member_type: VOTER last_known_addr { host: "127.0.100.6" port: 35349 } attrs { promote: false } }
I20240503 23:26:55.436391 3838 raft_consensus.cc:3037] T 059f6233ea514fe69b8c6504efabeff0 P a2f9bf2e5d9b4fc19f804797bca4a3ff [term 2 FOLLOWER]: Advancing to term 3
I20240503 23:26:55.439049 4366 raft_consensus.cc:2454] T 059f6233ea514fe69b8c6504efabeff0 P 4d1d4e261584467fbbce4c72fa18045a [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a3eecbaec43d4954b1096bc3121e824c in term 3.
I20240503 23:26:55.439384 3838 raft_consensus.cc:2454] T 059f6233ea514fe69b8c6504efabeff0 P a2f9bf2e5d9b4fc19f804797bca4a3ff [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a3eecbaec43d4954b1096bc3121e824c in term 3.
I20240503 23:26:55.439891 3877 leader_election.cc:302] T 059f6233ea514fe69b8c6504efabeff0 P a3eecbaec43d4954b1096bc3121e824c [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d1d4e261584467fbbce4c72fa18045a, a3eecbaec43d4954b1096bc3121e824c; no voters:
I20240503 23:26:55.440260 4838 raft_consensus.cc:2781] T 059f6233ea514fe69b8c6504efabeff0 P a3eecbaec43d4954b1096bc3121e824c [term 3 FOLLOWER]: Leader election won for term 3
I20240503 23:26:55.440810 4838 raft_consensus.cc:686] T 059f6233ea514fe69b8c6504efabeff0 P a3eecbaec43d4954b1096bc3121e824c [term 3 LEADER]: Becoming Leader. State: Replica: a3eecbaec43d4954b1096bc3121e824c, State: Running, Role: LEADER
I20240503 23:26:55.441170 4838 consensus_queue.cc:237] T 059f6233ea514fe69b8c6504efabeff0 P a3eecbaec43d4954b1096bc3121e824c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 30, Committed index: 30, Last appended: 2.30, Last appended by leader: 30, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "a2f9bf2e5d9b4fc19f804797bca4a3ff" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 46261 } } peers { permanent_uuid: "a3eecbaec43d4954b1096bc3121e824c" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 40649 } } peers { permanent_uuid: "4d1d4e261584467fbbce4c72fa18045a" member_type: VOTER last_known_addr { host: "127.0.100.6" port: 35349 } attrs { promote: false } }
I20240503 23:26:55.444293 3655 catalog_manager.cc:5484] T 059f6233ea514fe69b8c6504efabeff0 P a3eecbaec43d4954b1096bc3121e824c reported cstate change: term changed from 2 to 3, leader changed from a2f9bf2e5d9b4fc19f804797bca4a3ff (127.0.100.2) to a3eecbaec43d4954b1096bc3121e824c (127.0.100.3). New cstate: current_term: 3 leader_uuid: "a3eecbaec43d4954b1096bc3121e824c" committed_config { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "a2f9bf2e5d9b4fc19f804797bca4a3ff" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 46261 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a3eecbaec43d4954b1096bc3121e824c" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 40649 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4d1d4e261584467fbbce4c72fa18045a" member_type: VOTER last_known_addr { host: "127.0.100.6" port: 35349 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240503 23:26:55.997339 4366 raft_consensus.cc:1261] T 059f6233ea514fe69b8c6504efabeff0 P 4d1d4e261584467fbbce4c72fa18045a [term 3 FOLLOWER]: Refusing update from remote peer a3eecbaec43d4954b1096bc3121e824c: Log matching property violated. Preceding OpId in replica: term: 2 index: 30. Preceding OpId from leader: term: 3 index: 31. (index mismatch)
I20240503 23:26:55.998059 4838 consensus_queue.cc:1035] T 059f6233ea514fe69b8c6504efabeff0 P a3eecbaec43d4954b1096bc3121e824c [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d1d4e261584467fbbce4c72fa18045a" member_type: VOTER last_known_addr { host: "127.0.100.6" port: 35349 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 31, Last known committed idx: 30, Time since last communication: 0.000s
I20240503 23:26:56.002744 3838 raft_consensus.cc:1261] T 059f6233ea514fe69b8c6504efabeff0 P a2f9bf2e5d9b4fc19f804797bca4a3ff [term 3 FOLLOWER]: Refusing update from remote peer a3eecbaec43d4954b1096bc3121e824c: Log matching property violated. Preceding OpId in replica: term: 2 index: 30. Preceding OpId from leader: term: 3 index: 31. (index mismatch)
I20240503 23:26:56.003376 4915 consensus_queue.cc:1035] T 059f6233ea514fe69b8c6504efabeff0 P a3eecbaec43d4954b1096bc3121e824c [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2f9bf2e5d9b4fc19f804797bca4a3ff" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 46261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 31, Last known committed idx: 30, Time since last communication: 0.000s
I20240503 23:26:57.299177 3715 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240503 23:26:57.302786 3907 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "059f6233ea514fe69b8c6504efabeff0"
dest_uuid: "a3eecbaec43d4954b1096bc3121e824c"
mode: GRACEFUL
new_leader_uuid: "a2f9bf2e5d9b4fc19f804797bca4a3ff"
from {username='slave'} at 127.0.0.1:40650
I20240503 23:26:57.303114 3907 raft_consensus.cc:595] T 059f6233ea514fe69b8c6504efabeff0 P a3eecbaec43d4954b1096bc3121e824c [term 3 LEADER]: Received request to transfer leadership to TS a2f9bf2e5d9b4fc19f804797bca4a3ff
I20240503 23:26:57.303766 3715 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 1
I20240503 23:26:57.304073 3715 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240503 23:26:57.350791 4923 raft_consensus.cc:979] T 059f6233ea514fe69b8c6504efabeff0 P a3eecbaec43d4954b1096bc3121e824c: : Instructing follower a2f9bf2e5d9b4fc19f804797bca4a3ff to start an election
I20240503 23:26:57.351061 4923 raft_consensus.cc:1067] T 059f6233ea514fe69b8c6504efabeff0 P a3eecbaec43d4954b1096bc3121e824c [term 3 LEADER]: Signalling peer a2f9bf2e5d9b4fc19f804797bca4a3ff to start an election
I20240503 23:26:57.351985 3838 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "059f6233ea514fe69b8c6504efabeff0"
dest_uuid: "a2f9bf2e5d9b4fc19f804797bca4a3ff"
from {username='slave'} at 127.0.0.1:45358
I20240503 23:26:57.352252 3838 raft_consensus.cc:482] T 059f6233ea514fe69b8c6504efabeff0 P a2f9bf2e5d9b4fc19f804797bca4a3ff [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 23:26:57.352380 3838 raft_consensus.cc:3037] T 059f6233ea514fe69b8c6504efabeff0 P a2f9bf2e5d9b4fc19f804797bca4a3ff [term 3 FOLLOWER]: Advancing to term 4
I20240503 23:26:57.354833 3838 raft_consensus.cc:504] T 059f6233ea514fe69b8c6504efabeff0 P a2f9bf2e5d9b4fc19f804797bca4a3ff [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "a2f9bf2e5d9b4fc19f804797bca4a3ff" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 46261 } } peers { permanent_uuid: "a3eecbaec43d4954b1096bc3121e824c" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 40649 } } peers { permanent_uuid: "4d1d4e261584467fbbce4c72fa18045a" member_type: VOTER last_known_addr { host: "127.0.100.6" port: 35349 } attrs { promote: false } }
I20240503 23:26:57.355851 3838 leader_election.cc:288] T 059f6233ea514fe69b8c6504efabeff0 P a2f9bf2e5d9b4fc19f804797bca4a3ff [CANDIDATE]: Term 4 election: Requested vote from peers a3eecbaec43d4954b1096bc3121e824c (127.0.100.3:40649), 4d1d4e261584467fbbce4c72fa18045a (127.0.100.6:35349)
I20240503 23:26:57.356488 4366 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "059f6233ea514fe69b8c6504efabeff0" candidate_uuid: "a2f9bf2e5d9b4fc19f804797bca4a3ff" candidate_term: 4 candidate_status { last_received { term: 3 index: 31 } } ignore_live_leader: true dest_uuid: "4d1d4e261584467fbbce4c72fa18045a"
I20240503 23:26:57.356496 3907 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "059f6233ea514fe69b8c6504efabeff0" candidate_uuid: "a2f9bf2e5d9b4fc19f804797bca4a3ff" candidate_term: 4 candidate_status { last_received { term: 3 index: 31 } } ignore_live_leader: true dest_uuid: "a3eecbaec43d4954b1096bc3121e824c"
I20240503 23:26:57.356737 4366 raft_consensus.cc:3037] T 059f6233ea514fe69b8c6504efabeff0 P 4d1d4e261584467fbbce4c72fa18045a [term 3 FOLLOWER]: Advancing to term 4
I20240503 23:26:57.356770 3907 raft_consensus.cc:3032] T 059f6233ea514fe69b8c6504efabeff0 P a3eecbaec43d4954b1096bc3121e824c [term 3 LEADER]: Stepping down as leader of term 3
I20240503 23:26:57.356863 3907 raft_consensus.cc:729] T 059f6233ea514fe69b8c6504efabeff0 P a3eecbaec43d4954b1096bc3121e824c [term 3 LEADER]: Becoming Follower/Learner. State: Replica: a3eecbaec43d4954b1096bc3121e824c, State: Running, Role: LEADER
I20240503 23:26:57.357316 3907 consensus_queue.cc:260] T 059f6233ea514fe69b8c6504efabeff0 P a3eecbaec43d4954b1096bc3121e824c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 31, Committed index: 31, Last appended: 3.31, Last appended by leader: 31, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "a2f9bf2e5d9b4fc19f804797bca4a3ff" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 46261 } } peers { permanent_uuid: "a3eecbaec43d4954b1096bc3121e824c" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 40649 } } peers { permanent_uuid: "4d1d4e261584467fbbce4c72fa18045a" member_type: VOTER last_known_addr { host: "127.0.100.6" port: 35349 } attrs { promote: false } }
I20240503 23:26:57.357748 3907 raft_consensus.cc:3037] T 059f6233ea514fe69b8c6504efabeff0 P a3eecbaec43d4954b1096bc3121e824c [term 3 FOLLOWER]: Advancing to term 4
I20240503 23:26:57.359304 4366 raft_consensus.cc:2454] T 059f6233ea514fe69b8c6504efabeff0 P 4d1d4e261584467fbbce4c72fa18045a [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2f9bf2e5d9b4fc19f804797bca4a3ff in term 4.
I20240503 23:26:57.360216 3907 raft_consensus.cc:2454] T 059f6233ea514fe69b8c6504efabeff0 P a3eecbaec43d4954b1096bc3121e824c [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2f9bf2e5d9b4fc19f804797bca4a3ff in term 4.
I20240503 23:26:57.360165 3803 leader_election.cc:302] T 059f6233ea514fe69b8c6504efabeff0 P a2f9bf2e5d9b4fc19f804797bca4a3ff [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d1d4e261584467fbbce4c72fa18045a, a2f9bf2e5d9b4fc19f804797bca4a3ff; no voters:
I20240503 23:26:57.360680 4903 raft_consensus.cc:2781] T 059f6233ea514fe69b8c6504efabeff0 P a2f9bf2e5d9b4fc19f804797bca4a3ff [term 4 FOLLOWER]: Leader election won for term 4
I20240503 23:26:57.361235 4903 raft_consensus.cc:686] T 059f6233ea514fe69b8c6504efabeff0 P a2f9bf2e5d9b4fc19f804797bca4a3ff [term 4 LEADER]: Becoming Leader. State: Replica: a2f9bf2e5d9b4fc19f804797bca4a3ff, State: Running, Role: LEADER
I20240503 23:26:57.361572 4903 consensus_queue.cc:237] T 059f6233ea514fe69b8c6504efabeff0 P a2f9bf2e5d9b4fc19f804797bca4a3ff [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 31, Committed index: 31, Last appended: 3.31, Last appended by leader: 31, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "a2f9bf2e5d9b4fc19f804797bca4a3ff" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 46261 } } peers { permanent_uuid: "a3eecbaec43d4954b1096bc3121e824c" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 40649 } } peers { permanent_uuid: "4d1d4e261584467fbbce4c72fa18045a" member_type: VOTER last_known_addr { host: "127.0.100.6" port: 35349 } attrs { promote: false } }
I20240503 23:26:57.364748 3662 catalog_manager.cc:5484] T 059f6233ea514fe69b8c6504efabeff0 P a2f9bf2e5d9b4fc19f804797bca4a3ff reported cstate change: term changed from 3 to 4, leader changed from a3eecbaec43d4954b1096bc3121e824c (127.0.100.3) to a2f9bf2e5d9b4fc19f804797bca4a3ff (127.0.100.2). New cstate: current_term: 4 leader_uuid: "a2f9bf2e5d9b4fc19f804797bca4a3ff" committed_config { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "a2f9bf2e5d9b4fc19f804797bca4a3ff" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 46261 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a3eecbaec43d4954b1096bc3121e824c" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 40649 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4d1d4e261584467fbbce4c72fa18045a" member_type: VOTER last_known_addr { host: "127.0.100.6" port: 35349 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240503 23:26:57.815284 3907 raft_consensus.cc:1261] T 059f6233ea514fe69b8c6504efabeff0 P a3eecbaec43d4954b1096bc3121e824c [term 4 FOLLOWER]: Refusing update from remote peer a2f9bf2e5d9b4fc19f804797bca4a3ff: Log matching property violated. Preceding OpId in replica: term: 3 index: 31. Preceding OpId from leader: term: 4 index: 32. (index mismatch)
I20240503 23:26:57.815953 4903 consensus_queue.cc:1035] T 059f6233ea514fe69b8c6504efabeff0 P a2f9bf2e5d9b4fc19f804797bca4a3ff [LEADER]: Connected to new peer: Peer: permanent_uuid: "a3eecbaec43d4954b1096bc3121e824c" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 40649 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 32, Last known committed idx: 31, Time since last communication: 0.000s
I20240503 23:26:57.822365 4366 raft_consensus.cc:1261] T 059f6233ea514fe69b8c6504efabeff0 P 4d1d4e261584467fbbce4c72fa18045a [term 4 FOLLOWER]: Refusing update from remote peer a2f9bf2e5d9b4fc19f804797bca4a3ff: Log matching property violated. Preceding OpId in replica: term: 3 index: 31. Preceding OpId from leader: term: 4 index: 32. (index mismatch)
I20240503 23:26:57.823088 4903 consensus_queue.cc:1035] T 059f6233ea514fe69b8c6504efabeff0 P a2f9bf2e5d9b4fc19f804797bca4a3ff [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d1d4e261584467fbbce4c72fa18045a" member_type: VOTER last_known_addr { host: "127.0.100.6" port: 35349 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 32, Last known committed idx: 31, Time since last communication: 0.000s
I20240503 23:26:59.304771 3715 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240503 23:26:59.307973 3838 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "059f6233ea514fe69b8c6504efabeff0"
dest_uuid: "a2f9bf2e5d9b4fc19f804797bca4a3ff"
mode: GRACEFUL
new_leader_uuid: "a3eecbaec43d4954b1096bc3121e824c"
from {username='slave'} at 127.0.0.1:55472
I20240503 23:26:59.308307 3838 raft_consensus.cc:595] T 059f6233ea514fe69b8c6504efabeff0 P a2f9bf2e5d9b4fc19f804797bca4a3ff [term 4 LEADER]: Received request to transfer leadership to TS a3eecbaec43d4954b1096bc3121e824c
I20240503 23:26:59.308991 3715 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 1
I20240503 23:26:59.309266 3715 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240503 23:26:59.456010 4936 raft_consensus.cc:979] T 059f6233ea514fe69b8c6504efabeff0 P a2f9bf2e5d9b4fc19f804797bca4a3ff: : Instructing follower a3eecbaec43d4954b1096bc3121e824c to start an election
I20240503 23:26:59.456254 4936 raft_consensus.cc:1067] T 059f6233ea514fe69b8c6504efabeff0 P a2f9bf2e5d9b4fc19f804797bca4a3ff [term 4 LEADER]: Signalling peer a3eecbaec43d4954b1096bc3121e824c to start an election
I20240503 23:26:59.457046 3907 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "059f6233ea514fe69b8c6504efabeff0"
dest_uuid: "a3eecbaec43d4954b1096bc3121e824c"
from {username='slave'} at 127.0.0.1:40636
I20240503 23:26:59.457278 3907 raft_consensus.cc:482] T 059f6233ea514fe69b8c6504efabeff0 P a3eecbaec43d4954b1096bc3121e824c [term 4 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 23:26:59.457403 3907 raft_consensus.cc:3037] T 059f6233ea514fe69b8c6504efabeff0 P a3eecbaec43d4954b1096bc3121e824c [term 4 FOLLOWER]: Advancing to term 5
I20240503 23:26:59.459906 3907 raft_consensus.cc:504] T 059f6233ea514fe69b8c6504efabeff0 P a3eecbaec43d4954b1096bc3121e824c [term 5 FOLLOWER]: Starting forced leader election with config: opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "a2f9bf2e5d9b4fc19f804797bca4a3ff" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 46261 } } peers { permanent_uuid: "a3eecbaec43d4954b1096bc3121e824c" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 40649 } } peers { permanent_uuid: "4d1d4e261584467fbbce4c72fa18045a" member_type: VOTER last_known_addr { host: "127.0.100.6" port: 35349 } attrs { promote: false } }
I20240503 23:26:59.460842 3907 leader_election.cc:288] T 059f6233ea514fe69b8c6504efabeff0 P a3eecbaec43d4954b1096bc3121e824c [CANDIDATE]: Term 5 election: Requested vote from peers a2f9bf2e5d9b4fc19f804797bca4a3ff (127.0.100.2:46261), 4d1d4e261584467fbbce4c72fa18045a (127.0.100.6:35349)
I20240503 23:26:59.461354 4366 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "059f6233ea514fe69b8c6504efabeff0" candidate_uuid: "a3eecbaec43d4954b1096bc3121e824c" candidate_term: 5 candidate_status { last_received { term: 4 index: 32 } } ignore_live_leader: true dest_uuid: "4d1d4e261584467fbbce4c72fa18045a"
I20240503 23:26:59.461372 3838 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "059f6233ea514fe69b8c6504efabeff0" candidate_uuid: "a3eecbaec43d4954b1096bc3121e824c" candidate_term: 5 candidate_status { last_received { term: 4 index: 32 } } ignore_live_leader: true dest_uuid: "a2f9bf2e5d9b4fc19f804797bca4a3ff"
I20240503 23:26:59.461575 4366 raft_consensus.cc:3037] T 059f6233ea514fe69b8c6504efabeff0 P 4d1d4e261584467fbbce4c72fa18045a [term 4 FOLLOWER]: Advancing to term 5
I20240503 23:26:59.461635 3838 raft_consensus.cc:3032] T 059f6233ea514fe69b8c6504efabeff0 P a2f9bf2e5d9b4fc19f804797bca4a3ff [term 4 LEADER]: Stepping down as leader of term 4
I20240503 23:26:59.461745 3838 raft_consensus.cc:729] T 059f6233ea514fe69b8c6504efabeff0 P a2f9bf2e5d9b4fc19f804797bca4a3ff [term 4 LEADER]: Becoming Follower/Learner. State: Replica: a2f9bf2e5d9b4fc19f804797bca4a3ff, State: Running, Role: LEADER
I20240503 23:26:59.462056 3838 consensus_queue.cc:260] T 059f6233ea514fe69b8c6504efabeff0 P a2f9bf2e5d9b4fc19f804797bca4a3ff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 32, Committed index: 32, Last appended: 4.32, Last appended by leader: 32, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "a2f9bf2e5d9b4fc19f804797bca4a3ff" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 46261 } } peers { permanent_uuid: "a3eecbaec43d4954b1096bc3121e824c" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 40649 } } peers { permanent_uuid: "4d1d4e261584467fbbce4c72fa18045a" member_type: VOTER last_known_addr { host: "127.0.100.6" port: 35349 } attrs { promote: false } }
I20240503 23:26:59.462399 3838 raft_consensus.cc:3037] T 059f6233ea514fe69b8c6504efabeff0 P a2f9bf2e5d9b4fc19f804797bca4a3ff [term 4 FOLLOWER]: Advancing to term 5
I20240503 23:26:59.463932 4366 raft_consensus.cc:2454] T 059f6233ea514fe69b8c6504efabeff0 P 4d1d4e261584467fbbce4c72fa18045a [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a3eecbaec43d4954b1096bc3121e824c in term 5.
I20240503 23:26:59.464346 3838 raft_consensus.cc:2454] T 059f6233ea514fe69b8c6504efabeff0 P a2f9bf2e5d9b4fc19f804797bca4a3ff [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a3eecbaec43d4954b1096bc3121e824c in term 5.
I20240503 23:26:59.464522 3877 leader_election.cc:302] T 059f6233ea514fe69b8c6504efabeff0 P a3eecbaec43d4954b1096bc3121e824c [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d1d4e261584467fbbce4c72fa18045a, a3eecbaec43d4954b1096bc3121e824c; no voters:
I20240503 23:26:59.464946 4923 raft_consensus.cc:2781] T 059f6233ea514fe69b8c6504efabeff0 P a3eecbaec43d4954b1096bc3121e824c [term 5 FOLLOWER]: Leader election won for term 5
I20240503 23:26:59.465407 4923 raft_consensus.cc:686] T 059f6233ea514fe69b8c6504efabeff0 P a3eecbaec43d4954b1096bc3121e824c [term 5 LEADER]: Becoming Leader. State: Replica: a3eecbaec43d4954b1096bc3121e824c, State: Running, Role: LEADER
I20240503 23:26:59.465691 4923 consensus_queue.cc:237] T 059f6233ea514fe69b8c6504efabeff0 P a3eecbaec43d4954b1096bc3121e824c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 32, Committed index: 32, Last appended: 4.32, Last appended by leader: 32, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "a2f9bf2e5d9b4fc19f804797bca4a3ff" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 46261 } } peers { permanent_uuid: "a3eecbaec43d4954b1096bc3121e824c" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 40649 } } peers { permanent_uuid: "4d1d4e261584467fbbce4c72fa18045a" member_type: VOTER last_known_addr { host: "127.0.100.6" port: 35349 } attrs { promote: false } }
I20240503 23:26:59.468490 3655 catalog_manager.cc:5484] T 059f6233ea514fe69b8c6504efabeff0 P a3eecbaec43d4954b1096bc3121e824c reported cstate change: term changed from 4 to 5, leader changed from a2f9bf2e5d9b4fc19f804797bca4a3ff (127.0.100.2) to a3eecbaec43d4954b1096bc3121e824c (127.0.100.3). New cstate: current_term: 5 leader_uuid: "a3eecbaec43d4954b1096bc3121e824c" committed_config { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "a2f9bf2e5d9b4fc19f804797bca4a3ff" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 46261 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a3eecbaec43d4954b1096bc3121e824c" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 40649 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4d1d4e261584467fbbce4c72fa18045a" member_type: VOTER last_known_addr { host: "127.0.100.6" port: 35349 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240503 23:26:59.908859 4366 raft_consensus.cc:1261] T 059f6233ea514fe69b8c6504efabeff0 P 4d1d4e261584467fbbce4c72fa18045a [term 5 FOLLOWER]: Refusing update from remote peer a3eecbaec43d4954b1096bc3121e824c: Log matching property violated. Preceding OpId in replica: term: 4 index: 32. Preceding OpId from leader: term: 5 index: 33. (index mismatch)
I20240503 23:26:59.909579 4923 consensus_queue.cc:1035] T 059f6233ea514fe69b8c6504efabeff0 P a3eecbaec43d4954b1096bc3121e824c [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d1d4e261584467fbbce4c72fa18045a" member_type: VOTER last_known_addr { host: "127.0.100.6" port: 35349 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 33, Last known committed idx: 32, Time since last communication: 0.000s
I20240503 23:26:59.914220 3838 raft_consensus.cc:1261] T 059f6233ea514fe69b8c6504efabeff0 P a2f9bf2e5d9b4fc19f804797bca4a3ff [term 5 FOLLOWER]: Refusing update from remote peer a3eecbaec43d4954b1096bc3121e824c: Log matching property violated. Preceding OpId in replica: term: 4 index: 32. Preceding OpId from leader: term: 5 index: 33. (index mismatch)
I20240503 23:26:59.914827 4923 consensus_queue.cc:1035] T 059f6233ea514fe69b8c6504efabeff0 P a3eecbaec43d4954b1096bc3121e824c [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2f9bf2e5d9b4fc19f804797bca4a3ff" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 46261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 33, Last known committed idx: 32, Time since last communication: 0.000s
I20240503 23:27:01.309825 3715 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240503 23:27:01.312963 3907 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "059f6233ea514fe69b8c6504efabeff0"
dest_uuid: "a3eecbaec43d4954b1096bc3121e824c"
mode: GRACEFUL
new_leader_uuid: "a2f9bf2e5d9b4fc19f804797bca4a3ff"
from {username='slave'} at 127.0.0.1:40650
I20240503 23:27:01.313285 3907 raft_consensus.cc:595] T 059f6233ea514fe69b8c6504efabeff0 P a3eecbaec43d4954b1096bc3121e824c [term 5 LEADER]: Received request to transfer leadership to TS a2f9bf2e5d9b4fc19f804797bca4a3ff
I20240503 23:27:01.313932 3715 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 1
I20240503 23:27:01.314208 3715 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240503 23:27:01.532495 4951 raft_consensus.cc:979] T 059f6233ea514fe69b8c6504efabeff0 P a3eecbaec43d4954b1096bc3121e824c: : Instructing follower a2f9bf2e5d9b4fc19f804797bca4a3ff to start an election
I20240503 23:27:01.532719 4951 raft_consensus.cc:1067] T 059f6233ea514fe69b8c6504efabeff0 P a3eecbaec43d4954b1096bc3121e824c [term 5 LEADER]: Signalling peer a2f9bf2e5d9b4fc19f804797bca4a3ff to start an election
I20240503 23:27:01.533641 3838 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "059f6233ea514fe69b8c6504efabeff0"
dest_uuid: "a2f9bf2e5d9b4fc19f804797bca4a3ff"
from {username='slave'} at 127.0.0.1:45358
I20240503 23:27:01.533883 3838 raft_consensus.cc:482] T 059f6233ea514fe69b8c6504efabeff0 P a2f9bf2e5d9b4fc19f804797bca4a3ff [term 5 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 23:27:01.533996 3838 raft_consensus.cc:3037] T 059f6233ea514fe69b8c6504efabeff0 P a2f9bf2e5d9b4fc19f804797bca4a3ff [term 5 FOLLOWER]: Advancing to term 6
I20240503 23:27:01.536300 3838 raft_consensus.cc:504] T 059f6233ea514fe69b8c6504efabeff0 P a2f9bf2e5d9b4fc19f804797bca4a3ff [term 6 FOLLOWER]: Starting forced leader election with config: opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "a2f9bf2e5d9b4fc19f804797bca4a3ff" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 46261 } } peers { permanent_uuid: "a3eecbaec43d4954b1096bc3121e824c" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 40649 } } peers { permanent_uuid: "4d1d4e261584467fbbce4c72fa18045a" member_type: VOTER last_known_addr { host: "127.0.100.6" port: 35349 } attrs { promote: false } }
I20240503 23:27:01.537091 3838 leader_election.cc:288] T 059f6233ea514fe69b8c6504efabeff0 P a2f9bf2e5d9b4fc19f804797bca4a3ff [CANDIDATE]: Term 6 election: Requested vote from peers a3eecbaec43d4954b1096bc3121e824c (127.0.100.3:40649), 4d1d4e261584467fbbce4c72fa18045a (127.0.100.6:35349)
I20240503 23:27:01.537628 4366 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "059f6233ea514fe69b8c6504efabeff0" candidate_uuid: "a2f9bf2e5d9b4fc19f804797bca4a3ff" candidate_term: 6 candidate_status { last_received { term: 5 index: 33 } } ignore_live_leader: true dest_uuid: "4d1d4e261584467fbbce4c72fa18045a"
I20240503 23:27:01.537633 3907 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "059f6233ea514fe69b8c6504efabeff0" candidate_uuid: "a2f9bf2e5d9b4fc19f804797bca4a3ff" candidate_term: 6 candidate_status { last_received { term: 5 index: 33 } } ignore_live_leader: true dest_uuid: "a3eecbaec43d4954b1096bc3121e824c"
I20240503 23:27:01.537866 3907 raft_consensus.cc:3032] T 059f6233ea514fe69b8c6504efabeff0 P a3eecbaec43d4954b1096bc3121e824c [term 5 LEADER]: Stepping down as leader of term 5
I20240503 23:27:01.537868 4366 raft_consensus.cc:3037] T 059f6233ea514fe69b8c6504efabeff0 P 4d1d4e261584467fbbce4c72fa18045a [term 5 FOLLOWER]: Advancing to term 6
I20240503 23:27:01.537969 3907 raft_consensus.cc:729] T 059f6233ea514fe69b8c6504efabeff0 P a3eecbaec43d4954b1096bc3121e824c [term 5 LEADER]: Becoming Follower/Learner. State: Replica: a3eecbaec43d4954b1096bc3121e824c, State: Running, Role: LEADER
I20240503 23:27:01.538259 3907 consensus_queue.cc:260] T 059f6233ea514fe69b8c6504efabeff0 P a3eecbaec43d4954b1096bc3121e824c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 33, Committed index: 33, Last appended: 5.33, Last appended by leader: 33, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "a2f9bf2e5d9b4fc19f804797bca4a3ff" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 46261 } } peers { permanent_uuid: "a3eecbaec43d4954b1096bc3121e824c" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 40649 } } peers { permanent_uuid: "4d1d4e261584467fbbce4c72fa18045a" member_type: VOTER last_known_addr { host: "127.0.100.6" port: 35349 } attrs { promote: false } }
I20240503 23:27:01.538650 3907 raft_consensus.cc:3037] T 059f6233ea514fe69b8c6504efabeff0 P a3eecbaec43d4954b1096bc3121e824c [term 5 FOLLOWER]: Advancing to term 6
I20240503 23:27:01.541232 3907 raft_consensus.cc:2454] T 059f6233ea514fe69b8c6504efabeff0 P a3eecbaec43d4954b1096bc3121e824c [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2f9bf2e5d9b4fc19f804797bca4a3ff in term 6.
I20240503 23:27:01.541222 4366 raft_consensus.cc:2454] T 059f6233ea514fe69b8c6504efabeff0 P 4d1d4e261584467fbbce4c72fa18045a [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2f9bf2e5d9b4fc19f804797bca4a3ff in term 6.
I20240503 23:27:01.541996 3802 leader_election.cc:302] T 059f6233ea514fe69b8c6504efabeff0 P a2f9bf2e5d9b4fc19f804797bca4a3ff [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a2f9bf2e5d9b4fc19f804797bca4a3ff, a3eecbaec43d4954b1096bc3121e824c; no voters:
I20240503 23:27:01.542376 4936 raft_consensus.cc:2781] T 059f6233ea514fe69b8c6504efabeff0 P a2f9bf2e5d9b4fc19f804797bca4a3ff [term 6 FOLLOWER]: Leader election won for term 6
I20240503 23:27:01.542822 4936 raft_consensus.cc:686] T 059f6233ea514fe69b8c6504efabeff0 P a2f9bf2e5d9b4fc19f804797bca4a3ff [term 6 LEADER]: Becoming Leader. State: Replica: a2f9bf2e5d9b4fc19f804797bca4a3ff, State: Running, Role: LEADER
I20240503 23:27:01.543150 4936 consensus_queue.cc:237] T 059f6233ea514fe69b8c6504efabeff0 P a2f9bf2e5d9b4fc19f804797bca4a3ff [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 33, Committed index: 33, Last appended: 5.33, Last appended by leader: 33, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "a2f9bf2e5d9b4fc19f804797bca4a3ff" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 46261 } } peers { permanent_uuid: "a3eecbaec43d4954b1096bc3121e824c" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 40649 } } peers { permanent_uuid: "4d1d4e261584467fbbce4c72fa18045a" member_type: VOTER last_known_addr { host: "127.0.100.6" port: 35349 } attrs { promote: false } }
I20240503 23:27:01.546828 3655 catalog_manager.cc:5484] T 059f6233ea514fe69b8c6504efabeff0 P a2f9bf2e5d9b4fc19f804797bca4a3ff reported cstate change: term changed from 5 to 6, leader changed from a3eecbaec43d4954b1096bc3121e824c (127.0.100.3) to a2f9bf2e5d9b4fc19f804797bca4a3ff (127.0.100.2). New cstate: current_term: 6 leader_uuid: "a2f9bf2e5d9b4fc19f804797bca4a3ff" committed_config { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "a2f9bf2e5d9b4fc19f804797bca4a3ff" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 46261 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a3eecbaec43d4954b1096bc3121e824c" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 40649 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4d1d4e261584467fbbce4c72fa18045a" member_type: VOTER last_known_addr { host: "127.0.100.6" port: 35349 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240503 23:27:02.024049 3907 raft_consensus.cc:1261] T 059f6233ea514fe69b8c6504efabeff0 P a3eecbaec43d4954b1096bc3121e824c [term 6 FOLLOWER]: Refusing update from remote peer a2f9bf2e5d9b4fc19f804797bca4a3ff: Log matching property violated. Preceding OpId in replica: term: 5 index: 33. Preceding OpId from leader: term: 6 index: 34. (index mismatch)
I20240503 23:27:02.024657 4952 consensus_queue.cc:1035] T 059f6233ea514fe69b8c6504efabeff0 P a2f9bf2e5d9b4fc19f804797bca4a3ff [LEADER]: Connected to new peer: Peer: permanent_uuid: "a3eecbaec43d4954b1096bc3121e824c" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 40649 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 34, Last known committed idx: 33, Time since last communication: 0.000s
I20240503 23:27:02.029740 4366 raft_consensus.cc:1261] T 059f6233ea514fe69b8c6504efabeff0 P 4d1d4e261584467fbbce4c72fa18045a [term 6 FOLLOWER]: Refusing update from remote peer a2f9bf2e5d9b4fc19f804797bca4a3ff: Log matching property violated. Preceding OpId in replica: term: 5 index: 33. Preceding OpId from leader: term: 6 index: 34. (index mismatch)
I20240503 23:27:02.030294 4936 consensus_queue.cc:1035] T 059f6233ea514fe69b8c6504efabeff0 P a2f9bf2e5d9b4fc19f804797bca4a3ff [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d1d4e261584467fbbce4c72fa18045a" member_type: VOTER last_known_addr { host: "127.0.100.6" port: 35349 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 34, Last known committed idx: 33, Time since last communication: 0.000s
/home/jenkins-slave/workspace/build_and_test_flaky/src/kudu/master/auto_rebalancer-test.cc:195: Failure
Expected equality of these values:
0
NumMovesScheduled(leader_idx, BalanceThreadType::LEADER_REBALANCE)
Which is: 1
/home/jenkins-slave/workspace/build_and_test_flaky/src/kudu/util/test_util.cc:395: Failure
Failed
Timed out waiting for assertion to pass.
/home/jenkins-slave/workspace/build_and_test_flaky/src/kudu/master/auto_rebalancer-test.cc:575: Failure
Expected: CheckNoLeaderMovesScheduled() doesn't generate new fatal failures in the current thread.
Actual: it does.
I20240503 23:27:03.040059 400 tablet_server.cc:178] TabletServer@127.0.100.1:0 shutting down...
I20240503 23:27:03.053547 400 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240503 23:27:03.053925 400 tablet_replica.cc:330] T a0e18d0975a54e3dbeb0a13cb226fefd P 83ffd7184ec94302bd1ee74b0fab3e17: stopping tablet replica
I20240503 23:27:03.054145 400 raft_consensus.cc:2229] T a0e18d0975a54e3dbeb0a13cb226fefd P 83ffd7184ec94302bd1ee74b0fab3e17 [term 3 FOLLOWER]: Raft consensus shutting down.
I20240503 23:27:03.054417 400 raft_consensus.cc:2258] T a0e18d0975a54e3dbeb0a13cb226fefd P 83ffd7184ec94302bd1ee74b0fab3e17 [term 3 FOLLOWER]: Raft consensus is shut down!
I20240503 23:27:03.055142 400 tablet_replica.cc:330] T 98c9500acb43423b86f54ce7a4d98a72 P 83ffd7184ec94302bd1ee74b0fab3e17: stopping tablet replica
I20240503 23:27:03.055267 400 raft_consensus.cc:2229] T 98c9500acb43423b86f54ce7a4d98a72 P 83ffd7184ec94302bd1ee74b0fab3e17 [term 1 LEADER]: Raft consensus shutting down.
I20240503 23:27:03.055570 400 raft_consensus.cc:2258] T 98c9500acb43423b86f54ce7a4d98a72 P 83ffd7184ec94302bd1ee74b0fab3e17 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 23:27:03.056082 400 tablet_replica.cc:330] T 481cd0f8c8d147adac217915a53ff6d9 P 83ffd7184ec94302bd1ee74b0fab3e17: stopping tablet replica
I20240503 23:27:03.056195 400 raft_consensus.cc:2229] T 481cd0f8c8d147adac217915a53ff6d9 P 83ffd7184ec94302bd1ee74b0fab3e17 [term 5 FOLLOWER]: Raft consensus shutting down.
I20240503 23:27:03.056305 400 raft_consensus.cc:2258] T 481cd0f8c8d147adac217915a53ff6d9 P 83ffd7184ec94302bd1ee74b0fab3e17 [term 5 FOLLOWER]: Raft consensus is shut down!
I20240503 23:27:03.056823 400 tablet_replica.cc:330] T 7181e4e8c0324a8eb095dde23651a1ab P 83ffd7184ec94302bd1ee74b0fab3e17: stopping tablet replica
I20240503 23:27:03.056936 400 raft_consensus.cc:2229] T 7181e4e8c0324a8eb095dde23651a1ab P 83ffd7184ec94302bd1ee74b0fab3e17 [term 3 LEADER]: Raft consensus shutting down.
I20240503 23:27:03.057251 400 raft_consensus.cc:2258] T 7181e4e8c0324a8eb095dde23651a1ab P 83ffd7184ec94302bd1ee74b0fab3e17 [term 3 FOLLOWER]: Raft consensus is shut down!
I20240503 23:27:03.057683 400 tablet_replica.cc:330] T 487ac5b9bf2d4de2b80a1b13387a2a77 P 83ffd7184ec94302bd1ee74b0fab3e17: stopping tablet replica
I20240503 23:27:03.057786 400 raft_consensus.cc:2229] T 487ac5b9bf2d4de2b80a1b13387a2a77 P 83ffd7184ec94302bd1ee74b0fab3e17 [term 4 FOLLOWER]: Raft consensus shutting down.
I20240503 23:27:03.057889 400 raft_consensus.cc:2258] T 487ac5b9bf2d4de2b80a1b13387a2a77 P 83ffd7184ec94302bd1ee74b0fab3e17 [term 4 FOLLOWER]: Raft consensus is shut down!
I20240503 23:27:03.058298 400 tablet_replica.cc:330] T a14a12115b8d4fec947c6e223f1ef486 P 83ffd7184ec94302bd1ee74b0fab3e17: stopping tablet replica
I20240503 23:27:03.058413 400 raft_consensus.cc:2229] T a14a12115b8d4fec947c6e223f1ef486 P 83ffd7184ec94302bd1ee74b0fab3e17 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 23:27:03.058646 400 raft_consensus.cc:2258] T a14a12115b8d4fec947c6e223f1ef486 P 83ffd7184ec94302bd1ee74b0fab3e17 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 23:27:03.077277 400 tablet_server.cc:194] TabletServer@127.0.100.1:0 shutdown complete.
I20240503 23:27:03.084108 400 tablet_server.cc:178] TabletServer@127.0.100.2:0 shutting down...
I20240503 23:27:03.117394 400 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240503 23:27:03.117743 400 tablet_replica.cc:330] T 481cd0f8c8d147adac217915a53ff6d9 P a2f9bf2e5d9b4fc19f804797bca4a3ff: stopping tablet replica
I20240503 23:27:03.117964 400 raft_consensus.cc:2229] T 481cd0f8c8d147adac217915a53ff6d9 P a2f9bf2e5d9b4fc19f804797bca4a3ff [term 5 LEADER]: Raft consensus shutting down.
I20240503 23:27:03.118335 400 raft_consensus.cc:2258] T 481cd0f8c8d147adac217915a53ff6d9 P a2f9bf2e5d9b4fc19f804797bca4a3ff [term 5 FOLLOWER]: Raft consensus is shut down!
I20240503 23:27:03.119040 400 tablet_replica.cc:330] T 6039230222ff4e57929bed49cd56caf5 P a2f9bf2e5d9b4fc19f804797bca4a3ff: stopping tablet replica
I20240503 23:27:03.119168 400 raft_consensus.cc:2229] T 6039230222ff4e57929bed49cd56caf5 P a2f9bf2e5d9b4fc19f804797bca4a3ff [term 4 FOLLOWER]: Raft consensus shutting down.
I20240503 23:27:03.119282 400 raft_consensus.cc:2258] T 6039230222ff4e57929bed49cd56caf5 P a2f9bf2e5d9b4fc19f804797bca4a3ff [term 4 FOLLOWER]: Raft consensus is shut down!
I20240503 23:27:03.119794 400 tablet_replica.cc:330] T 487ac5b9bf2d4de2b80a1b13387a2a77 P a2f9bf2e5d9b4fc19f804797bca4a3ff: stopping tablet replica
I20240503 23:27:03.119922 400 raft_consensus.cc:2229] T 487ac5b9bf2d4de2b80a1b13387a2a77 P a2f9bf2e5d9b4fc19f804797bca4a3ff [term 4 LEADER]: Raft consensus shutting down.
I20240503 23:27:03.120215 400 raft_consensus.cc:2258] T 487ac5b9bf2d4de2b80a1b13387a2a77 P a2f9bf2e5d9b4fc19f804797bca4a3ff [term 4 FOLLOWER]: Raft consensus is shut down!
I20240503 23:27:03.120752 400 tablet_replica.cc:330] T 1b9566be6c644cfca852ec6507fab893 P a2f9bf2e5d9b4fc19f804797bca4a3ff: stopping tablet replica
I20240503 23:27:03.120899 400 raft_consensus.cc:2229] T 1b9566be6c644cfca852ec6507fab893 P a2f9bf2e5d9b4fc19f804797bca4a3ff [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 23:27:03.121021 400 raft_consensus.cc:2258] T 1b9566be6c644cfca852ec6507fab893 P a2f9bf2e5d9b4fc19f804797bca4a3ff [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 23:27:03.121441 400 tablet_replica.cc:330] T d0c4430fc56644cb9f87deffe4fe9407 P a2f9bf2e5d9b4fc19f804797bca4a3ff: stopping tablet replica
I20240503 23:27:03.121543 400 raft_consensus.cc:2229] T d0c4430fc56644cb9f87deffe4fe9407 P a2f9bf2e5d9b4fc19f804797bca4a3ff [term 6 FOLLOWER]: Raft consensus shutting down.
I20240503 23:27:03.121650 400 raft_consensus.cc:2258] T d0c4430fc56644cb9f87deffe4fe9407 P a2f9bf2e5d9b4fc19f804797bca4a3ff [term 6 FOLLOWER]: Raft consensus is shut down!
I20240503 23:27:03.122061 400 tablet_replica.cc:330] T 059f6233ea514fe69b8c6504efabeff0 P a2f9bf2e5d9b4fc19f804797bca4a3ff: stopping tablet replica
I20240503 23:27:03.122159 400 raft_consensus.cc:2229] T 059f6233ea514fe69b8c6504efabeff0 P a2f9bf2e5d9b4fc19f804797bca4a3ff [term 6 LEADER]: Raft consensus shutting down.
I20240503 23:27:03.122398 400 raft_consensus.cc:2258] T 059f6233ea514fe69b8c6504efabeff0 P a2f9bf2e5d9b4fc19f804797bca4a3ff [term 6 FOLLOWER]: Raft consensus is shut down!
I20240503 23:27:03.140028 400 tablet_server.cc:194] TabletServer@127.0.100.2:0 shutdown complete.
I20240503 23:27:03.146629 400 tablet_server.cc:178] TabletServer@127.0.100.3:0 shutting down...
I20240503 23:27:03.157063 400 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240503 23:27:03.157450 400 tablet_replica.cc:330] T 60267cb4849d4823a0afb2380305905f P a3eecbaec43d4954b1096bc3121e824c: stopping tablet replica
I20240503 23:27:03.157632 400 raft_consensus.cc:2229] T 60267cb4849d4823a0afb2380305905f P a3eecbaec43d4954b1096bc3121e824c [term 1 LEADER]: Raft consensus shutting down.
I20240503 23:27:03.158026 400 raft_consensus.cc:2258] T 60267cb4849d4823a0afb2380305905f P a3eecbaec43d4954b1096bc3121e824c [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 23:27:03.158818 400 tablet_replica.cc:330] T 7181e4e8c0324a8eb095dde23651a1ab P a3eecbaec43d4954b1096bc3121e824c: stopping tablet replica
I20240503 23:27:03.159005 400 raft_consensus.cc:2229] T 7181e4e8c0324a8eb095dde23651a1ab P a3eecbaec43d4954b1096bc3121e824c [term 3 FOLLOWER]: Raft consensus shutting down.
I20240503 23:27:03.159147 400 raft_consensus.cc:2258] T 7181e4e8c0324a8eb095dde23651a1ab P a3eecbaec43d4954b1096bc3121e824c [term 3 FOLLOWER]: Raft consensus is shut down!
I20240503 23:27:03.159679 400 tablet_replica.cc:330] T 6039230222ff4e57929bed49cd56caf5 P a3eecbaec43d4954b1096bc3121e824c: stopping tablet replica
I20240503 23:27:03.159801 400 raft_consensus.cc:2229] T 6039230222ff4e57929bed49cd56caf5 P a3eecbaec43d4954b1096bc3121e824c [term 4 LEADER]: Raft consensus shutting down.
I20240503 23:27:03.160066 400 raft_consensus.cc:2258] T 6039230222ff4e57929bed49cd56caf5 P a3eecbaec43d4954b1096bc3121e824c [term 4 FOLLOWER]: Raft consensus is shut down!
I20240503 23:27:03.160523 400 tablet_replica.cc:330] T 059f6233ea514fe69b8c6504efabeff0 P a3eecbaec43d4954b1096bc3121e824c: stopping tablet replica
I20240503 23:27:03.160847 400 raft_consensus.cc:2229] T 059f6233ea514fe69b8c6504efabeff0 P a3eecbaec43d4954b1096bc3121e824c [term 6 FOLLOWER]: Raft consensus shutting down.
I20240503 23:27:03.160975 400 raft_consensus.cc:2258] T 059f6233ea514fe69b8c6504efabeff0 P a3eecbaec43d4954b1096bc3121e824c [term 6 FOLLOWER]: Raft consensus is shut down!
I20240503 23:27:03.161423 400 tablet_replica.cc:330] T d0c4430fc56644cb9f87deffe4fe9407 P a3eecbaec43d4954b1096bc3121e824c: stopping tablet replica
I20240503 23:27:03.161541 400 raft_consensus.cc:2229] T d0c4430fc56644cb9f87deffe4fe9407 P a3eecbaec43d4954b1096bc3121e824c [term 6 FOLLOWER]: Raft consensus shutting down.
I20240503 23:27:03.161772 400 raft_consensus.cc:2258] T d0c4430fc56644cb9f87deffe4fe9407 P a3eecbaec43d4954b1096bc3121e824c [term 6 FOLLOWER]: Raft consensus is shut down!
I20240503 23:27:03.162214 400 tablet_replica.cc:330] T 1b9566be6c644cfca852ec6507fab893 P a3eecbaec43d4954b1096bc3121e824c: stopping tablet replica
I20240503 23:27:03.162334 400 raft_consensus.cc:2229] T 1b9566be6c644cfca852ec6507fab893 P a3eecbaec43d4954b1096bc3121e824c [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 23:27:03.162515 400 raft_consensus.cc:2258] T 1b9566be6c644cfca852ec6507fab893 P a3eecbaec43d4954b1096bc3121e824c [term 2 FOLLOWER]: Raft consensus is shut down!
W20240503 23:27:03.177228 4256 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.100.3:40649: connect: Connection refused (error 111)
W20240503 23:27:03.178539 4256 consensus_peers.cc:485] T 1b9566be6c644cfca852ec6507fab893 P 692da0319a2d4a81ad81f5d8f9f4c86b -> Peer a3eecbaec43d4954b1096bc3121e824c (127.0.100.3:40649): Couldn't send request to peer a3eecbaec43d4954b1096bc3121e824c. Status: Network error: Client connection negotiation failed: client connection to 127.0.100.3:40649: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 23:27:03.180780 400 tablet_server.cc:194] TabletServer@127.0.100.3:0 shutdown complete.
W20240503 23:27:03.202962 4333 consensus_peers.cc:485] T d0c4430fc56644cb9f87deffe4fe9407 P 4d1d4e261584467fbbce4c72fa18045a -> Peer a2f9bf2e5d9b4fc19f804797bca4a3ff (127.0.100.2:46261): Couldn't send request to peer a2f9bf2e5d9b4fc19f804797bca4a3ff. Status: Network error: Client connection negotiation failed: client connection to 127.0.100.2:46261: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 23:27:03.205950 400 tablet_server.cc:178] TabletServer@127.0.100.4:0 shutting down...
I20240503 23:27:03.216638 400 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240503 23:27:03.216987 400 tablet_replica.cc:330] T a14a12115b8d4fec947c6e223f1ef486 P e4086c099d7f4447b2b4733b6ca364df: stopping tablet replica
I20240503 23:27:03.217193 400 raft_consensus.cc:2229] T a14a12115b8d4fec947c6e223f1ef486 P e4086c099d7f4447b2b4733b6ca364df [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 23:27:03.217482 400 raft_consensus.cc:2258] T a14a12115b8d4fec947c6e223f1ef486 P e4086c099d7f4447b2b4733b6ca364df [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 23:27:03.218221 400 tablet_replica.cc:330] T 98c9500acb43423b86f54ce7a4d98a72 P e4086c099d7f4447b2b4733b6ca364df: stopping tablet replica
I20240503 23:27:03.218390 400 raft_consensus.cc:2229] T 98c9500acb43423b86f54ce7a4d98a72 P e4086c099d7f4447b2b4733b6ca364df [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 23:27:03.218597 400 raft_consensus.cc:2258] T 98c9500acb43423b86f54ce7a4d98a72 P e4086c099d7f4447b2b4733b6ca364df [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 23:27:03.219122 400 tablet_replica.cc:330] T 60267cb4849d4823a0afb2380305905f P e4086c099d7f4447b2b4733b6ca364df: stopping tablet replica
I20240503 23:27:03.219265 400 raft_consensus.cc:2229] T 60267cb4849d4823a0afb2380305905f P e4086c099d7f4447b2b4733b6ca364df [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 23:27:03.219480 400 raft_consensus.cc:2258] T 60267cb4849d4823a0afb2380305905f P e4086c099d7f4447b2b4733b6ca364df [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 23:27:03.219935 400 tablet_replica.cc:330] T 55cf664d5e874c0d9b9adf8bdc9c43ed P e4086c099d7f4447b2b4733b6ca364df: stopping tablet replica
I20240503 23:27:03.220073 400 raft_consensus.cc:2229] T 55cf664d5e874c0d9b9adf8bdc9c43ed P e4086c099d7f4447b2b4733b6ca364df [term 2 LEADER]: Raft consensus shutting down.
I20240503 23:27:03.220378 400 raft_consensus.cc:2258] T 55cf664d5e874c0d9b9adf8bdc9c43ed P e4086c099d7f4447b2b4733b6ca364df [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 23:27:03.220873 400 tablet_replica.cc:330] T a0e18d0975a54e3dbeb0a13cb226fefd P e4086c099d7f4447b2b4733b6ca364df: stopping tablet replica
I20240503 23:27:03.221010 400 raft_consensus.cc:2229] T a0e18d0975a54e3dbeb0a13cb226fefd P e4086c099d7f4447b2b4733b6ca364df [term 3 FOLLOWER]: Raft consensus shutting down.
I20240503 23:27:03.221192 400 raft_consensus.cc:2258] T a0e18d0975a54e3dbeb0a13cb226fefd P e4086c099d7f4447b2b4733b6ca364df [term 3 FOLLOWER]: Raft consensus is shut down!
I20240503 23:27:03.221637 400 tablet_replica.cc:330] T 7181e4e8c0324a8eb095dde23651a1ab P e4086c099d7f4447b2b4733b6ca364df: stopping tablet replica
I20240503 23:27:03.221765 400 raft_consensus.cc:2229] T 7181e4e8c0324a8eb095dde23651a1ab P e4086c099d7f4447b2b4733b6ca364df [term 3 FOLLOWER]: Raft consensus shutting down.
I20240503 23:27:03.221890 400 raft_consensus.cc:2258] T 7181e4e8c0324a8eb095dde23651a1ab P e4086c099d7f4447b2b4733b6ca364df [term 3 FOLLOWER]: Raft consensus is shut down!
I20240503 23:27:03.239035 400 tablet_server.cc:194] TabletServer@127.0.100.4:0 shutdown complete.
I20240503 23:27:03.244172 400 tablet_server.cc:178] TabletServer@127.0.100.5:0 shutting down...
I20240503 23:27:03.255491 400 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240503 23:27:03.255853 400 tablet_replica.cc:330] T 487ac5b9bf2d4de2b80a1b13387a2a77 P 692da0319a2d4a81ad81f5d8f9f4c86b: stopping tablet replica
I20240503 23:27:03.256053 400 raft_consensus.cc:2229] T 487ac5b9bf2d4de2b80a1b13387a2a77 P 692da0319a2d4a81ad81f5d8f9f4c86b [term 4 FOLLOWER]: Raft consensus shutting down.
I20240503 23:27:03.256249 400 raft_consensus.cc:2258] T 487ac5b9bf2d4de2b80a1b13387a2a77 P 692da0319a2d4a81ad81f5d8f9f4c86b [term 4 FOLLOWER]: Raft consensus is shut down!
I20240503 23:27:03.257114 400 tablet_replica.cc:330] T 481cd0f8c8d147adac217915a53ff6d9 P 692da0319a2d4a81ad81f5d8f9f4c86b: stopping tablet replica
I20240503 23:27:03.257292 400 raft_consensus.cc:2229] T 481cd0f8c8d147adac217915a53ff6d9 P 692da0319a2d4a81ad81f5d8f9f4c86b [term 5 FOLLOWER]: Raft consensus shutting down.
I20240503 23:27:03.257470 400 raft_consensus.cc:2258] T 481cd0f8c8d147adac217915a53ff6d9 P 692da0319a2d4a81ad81f5d8f9f4c86b [term 5 FOLLOWER]: Raft consensus is shut down!
I20240503 23:27:03.258035 400 tablet_replica.cc:330] T a0e18d0975a54e3dbeb0a13cb226fefd P 692da0319a2d4a81ad81f5d8f9f4c86b: stopping tablet replica
I20240503 23:27:03.258175 400 raft_consensus.cc:2229] T a0e18d0975a54e3dbeb0a13cb226fefd P 692da0319a2d4a81ad81f5d8f9f4c86b [term 3 LEADER]: Raft consensus shutting down.
I20240503 23:27:03.258472 400 raft_consensus.cc:2258] T a0e18d0975a54e3dbeb0a13cb226fefd P 692da0319a2d4a81ad81f5d8f9f4c86b [term 3 FOLLOWER]: Raft consensus is shut down!
I20240503 23:27:03.258957 400 tablet_replica.cc:330] T 1b9566be6c644cfca852ec6507fab893 P 692da0319a2d4a81ad81f5d8f9f4c86b: stopping tablet replica
I20240503 23:27:03.259073 400 raft_consensus.cc:2229] T 1b9566be6c644cfca852ec6507fab893 P 692da0319a2d4a81ad81f5d8f9f4c86b [term 2 LEADER]: Raft consensus shutting down.
I20240503 23:27:03.259310 400 raft_consensus.cc:2258] T 1b9566be6c644cfca852ec6507fab893 P 692da0319a2d4a81ad81f5d8f9f4c86b [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 23:27:03.259753 400 tablet_replica.cc:330] T 55cf664d5e874c0d9b9adf8bdc9c43ed P 692da0319a2d4a81ad81f5d8f9f4c86b: stopping tablet replica
I20240503 23:27:03.259855 400 raft_consensus.cc:2229] T 55cf664d5e874c0d9b9adf8bdc9c43ed P 692da0319a2d4a81ad81f5d8f9f4c86b [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 23:27:03.260023 400 raft_consensus.cc:2258] T 55cf664d5e874c0d9b9adf8bdc9c43ed P 692da0319a2d4a81ad81f5d8f9f4c86b [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 23:27:03.260417 400 tablet_replica.cc:330] T 6039230222ff4e57929bed49cd56caf5 P 692da0319a2d4a81ad81f5d8f9f4c86b: stopping tablet replica
I20240503 23:27:03.260515 400 raft_consensus.cc:2229] T 6039230222ff4e57929bed49cd56caf5 P 692da0319a2d4a81ad81f5d8f9f4c86b [term 4 FOLLOWER]: Raft consensus shutting down.
I20240503 23:27:03.260679 400 raft_consensus.cc:2258] T 6039230222ff4e57929bed49cd56caf5 P 692da0319a2d4a81ad81f5d8f9f4c86b [term 4 FOLLOWER]: Raft consensus is shut down!
W20240503 23:27:03.276424 4332 consensus_peers.cc:485] T a14a12115b8d4fec947c6e223f1ef486 P 4d1d4e261584467fbbce4c72fa18045a -> Peer 83ffd7184ec94302bd1ee74b0fab3e17 (127.0.100.1:32909): Couldn't send request to peer 83ffd7184ec94302bd1ee74b0fab3e17. Status: Network error: Client connection negotiation failed: client connection to 127.0.100.1:32909: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 23:27:03.277062 400 tablet_server.cc:194] TabletServer@127.0.100.5:0 shutdown complete.
I20240503 23:27:03.281986 400 tablet_server.cc:178] TabletServer@127.0.100.6:0 shutting down...
I20240503 23:27:03.311834 400 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240503 23:27:03.312449 400 tablet_replica.cc:330] T 60267cb4849d4823a0afb2380305905f P 4d1d4e261584467fbbce4c72fa18045a: stopping tablet replica
I20240503 23:27:03.312739 400 raft_consensus.cc:2229] T 60267cb4849d4823a0afb2380305905f P 4d1d4e261584467fbbce4c72fa18045a [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 23:27:03.313078 400 raft_consensus.cc:2258] T 60267cb4849d4823a0afb2380305905f P 4d1d4e261584467fbbce4c72fa18045a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 23:27:03.313855 400 tablet_replica.cc:330] T 55cf664d5e874c0d9b9adf8bdc9c43ed P 4d1d4e261584467fbbce4c72fa18045a: stopping tablet replica
I20240503 23:27:03.314033 400 raft_consensus.cc:2229] T 55cf664d5e874c0d9b9adf8bdc9c43ed P 4d1d4e261584467fbbce4c72fa18045a [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 23:27:03.314271 400 raft_consensus.cc:2258] T 55cf664d5e874c0d9b9adf8bdc9c43ed P 4d1d4e261584467fbbce4c72fa18045a [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 23:27:03.314683 3715 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240503 23:27:03.314875 400 tablet_replica.cc:330] T 059f6233ea514fe69b8c6504efabeff0 P 4d1d4e261584467fbbce4c72fa18045a: stopping tablet replica
I20240503 23:27:03.315032 400 raft_consensus.cc:2229] T 059f6233ea514fe69b8c6504efabeff0 P 4d1d4e261584467fbbce4c72fa18045a [term 6 FOLLOWER]: Raft consensus shutting down.
I20240503 23:27:03.315164 400 raft_consensus.cc:2258] T 059f6233ea514fe69b8c6504efabeff0 P 4d1d4e261584467fbbce4c72fa18045a [term 6 FOLLOWER]: Raft consensus is shut down!
I20240503 23:27:03.315665 400 tablet_replica.cc:330] T d0c4430fc56644cb9f87deffe4fe9407 P 4d1d4e261584467fbbce4c72fa18045a: stopping tablet replica
I20240503 23:27:03.315830 400 raft_consensus.cc:2229] T d0c4430fc56644cb9f87deffe4fe9407 P 4d1d4e261584467fbbce4c72fa18045a [term 6 LEADER]: Raft consensus shutting down.
I20240503 23:27:03.316152 400 raft_consensus.cc:2258] T d0c4430fc56644cb9f87deffe4fe9407 P 4d1d4e261584467fbbce4c72fa18045a [term 6 FOLLOWER]: Raft consensus is shut down!
I20240503 23:27:03.316685 400 tablet_replica.cc:330] T 98c9500acb43423b86f54ce7a4d98a72 P 4d1d4e261584467fbbce4c72fa18045a: stopping tablet replica
I20240503 23:27:03.316855 400 raft_consensus.cc:2229] T 98c9500acb43423b86f54ce7a4d98a72 P 4d1d4e261584467fbbce4c72fa18045a [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 23:27:03.317165 400 raft_consensus.cc:2258] T 98c9500acb43423b86f54ce7a4d98a72 P 4d1d4e261584467fbbce4c72fa18045a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 23:27:03.318078 400 tablet_replica.cc:330] T a14a12115b8d4fec947c6e223f1ef486 P 4d1d4e261584467fbbce4c72fa18045a: stopping tablet replica
I20240503 23:27:03.318241 400 raft_consensus.cc:2229] T a14a12115b8d4fec947c6e223f1ef486 P 4d1d4e261584467fbbce4c72fa18045a [term 2 LEADER]: Raft consensus shutting down.
I20240503 23:27:03.318673 400 raft_consensus.cc:2258] T a14a12115b8d4fec947c6e223f1ef486 P 4d1d4e261584467fbbce4c72fa18045a [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 23:27:03.319177 3715 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240503 23:27:03.334977 400 tablet_server.cc:194] TabletServer@127.0.100.6:0 shutdown complete.
I20240503 23:27:03.340394 400 master.cc:537] Master@127.0.100.62:38775 shutting down...
I20240503 23:27:03.349216 400 raft_consensus.cc:2229] T 00000000000000000000000000000000 P bd1b1f48d92849bcad7225c6235d0873 [term 1 LEADER]: Raft consensus shutting down.
I20240503 23:27:03.349519 400 raft_consensus.cc:2258] T 00000000000000000000000000000000 P bd1b1f48d92849bcad7225c6235d0873 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 23:27:03.349606 400 tablet_replica.cc:330] T 00000000000000000000000000000000 P bd1b1f48d92849bcad7225c6235d0873: stopping tablet replica
I20240503 23:27:03.364833 400 master.cc:552] Master@127.0.100.62:38775 shutdown complete.
I20240503 23:27:03.374166 400 test_util.cc:175] -----------------------------------------------
I20240503 23:27:03.374287 400 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-taskZhIiPz/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714778688451250-400-0
[ FAILED ] AutoRebalancerTest.TestMaxMovesPerServer (40346 ms)
[ RUN ] AutoRebalancerTest.AutoRebalancingUnstableCluster
I20240503 23:27:03.376430 400 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.0.100.62:39883
I20240503 23:27:03.377100 400 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 23:27:03.380960 4970 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 23:27:03.381419 4969 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 23:27:03.382278 4972 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 23:27:03.383160 400 server_base.cc:1034] running on GCE node
I20240503 23:27:03.383442 400 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240503 23:27:03.383517 400 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240503 23:27:03.383580 400 hybrid_clock.cc:648] HybridClock initialized: now 1714778823383572 us; error 0 us; skew 500 ppm
I20240503 23:27:03.383790 400 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 23:27:03.385452 400 webserver.cc:460] Webserver started at http://127.0.100.62:46345/ using document root <none> and password file <none>
I20240503 23:27:03.385744 400 fs_manager.cc:359] Metadata directory not provided
I20240503 23:27:03.385820 400 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 23:27:03.385978 400 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 23:27:03.386766 400 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZhIiPz/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingUnstableCluster.1714778688451250-400-0/minicluster-data/master-0-root/instance:
uuid: "d2c8d94c8b904af0b5807cc170c1ee96"
format_stamp: "Formatted at 2024-05-03 23:27:03 on dist-test-slave-jvbp"
I20240503 23:27:03.389847 400 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.003s
I20240503 23:27:03.391804 4977 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 23:27:03.392246 400 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.001s
I20240503 23:27:03.392366 400 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZhIiPz/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingUnstableCluster.1714778688451250-400-0/minicluster-data/master-0-root
uuid: "d2c8d94c8b904af0b5807cc170c1ee96"
format_stamp: "Formatted at 2024-05-03 23:27:03 on dist-test-slave-jvbp"
I20240503 23:27:03.392493 400 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZhIiPz/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingUnstableCluster.1714778688451250-400-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskZhIiPz/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingUnstableCluster.1714778688451250-400-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskZhIiPz/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingUnstableCluster.1714778688451250-400-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 23:27:03.413044 400 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 23:27:03.413715 400 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 23:27:03.426314 400 rpc_server.cc:307] RPC server started. Bound to: 127.0.100.62:39883
I20240503 23:27:03.426326 5028 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.100.62:39883 every 8 connection(s)
I20240503 23:27:03.427874 5029 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 23:27:03.432900 5029 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d2c8d94c8b904af0b5807cc170c1ee96: Bootstrap starting.
I20240503 23:27:03.434396 5029 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d2c8d94c8b904af0b5807cc170c1ee96: Neither blocks nor log segments found. Creating new log.
I20240503 23:27:03.436313 5029 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d2c8d94c8b904af0b5807cc170c1ee96: No bootstrap required, opened a new log
I20240503 23:27:03.437106 5029 raft_consensus.cc:348] T 00000000000000000000000000000000 P d2c8d94c8b904af0b5807cc170c1ee96 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d2c8d94c8b904af0b5807cc170c1ee96" member_type: VOTER }
I20240503 23:27:03.437270 5029 raft_consensus.cc:374] T 00000000000000000000000000000000 P d2c8d94c8b904af0b5807cc170c1ee96 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 23:27:03.437321 5029 raft_consensus.cc:729] T 00000000000000000000000000000000 P d2c8d94c8b904af0b5807cc170c1ee96 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d2c8d94c8b904af0b5807cc170c1ee96, State: Initialized, Role: FOLLOWER
I20240503 23:27:03.437543 5029 consensus_queue.cc:260] T 00000000000000000000000000000000 P d2c8d94c8b904af0b5807cc170c1ee96 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d2c8d94c8b904af0b5807cc170c1ee96" member_type: VOTER }
I20240503 23:27:03.437657 5029 raft_consensus.cc:388] T 00000000000000000000000000000000 P d2c8d94c8b904af0b5807cc170c1ee96 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 23:27:03.437722 5029 raft_consensus.cc:482] T 00000000000000000000000000000000 P d2c8d94c8b904af0b5807cc170c1ee96 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 23:27:03.437798 5029 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d2c8d94c8b904af0b5807cc170c1ee96 [term 0 FOLLOWER]: Advancing to term 1
I20240503 23:27:03.439906 5029 raft_consensus.cc:504] T 00000000000000000000000000000000 P d2c8d94c8b904af0b5807cc170c1ee96 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d2c8d94c8b904af0b5807cc170c1ee96" member_type: VOTER }
I20240503 23:27:03.440289 5029 leader_election.cc:302] T 00000000000000000000000000000000 P d2c8d94c8b904af0b5807cc170c1ee96 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d2c8d94c8b904af0b5807cc170c1ee96; no voters:
I20240503 23:27:03.440907 5029 leader_election.cc:288] T 00000000000000000000000000000000 P d2c8d94c8b904af0b5807cc170c1ee96 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240503 23:27:03.441074 5032 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d2c8d94c8b904af0b5807cc170c1ee96 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 23:27:03.441560 5032 raft_consensus.cc:686] T 00000000000000000000000000000000 P d2c8d94c8b904af0b5807cc170c1ee96 [term 1 LEADER]: Becoming Leader. State: Replica: d2c8d94c8b904af0b5807cc170c1ee96, State: Running, Role: LEADER
I20240503 23:27:03.441874 5032 consensus_queue.cc:237] T 00000000000000000000000000000000 P d2c8d94c8b904af0b5807cc170c1ee96 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d2c8d94c8b904af0b5807cc170c1ee96" member_type: VOTER }
I20240503 23:27:03.441967 5029 sys_catalog.cc:564] T 00000000000000000000000000000000 P d2c8d94c8b904af0b5807cc170c1ee96 [sys.catalog]: configured and running, proceeding with master startup.
I20240503 23:27:03.443230 5034 sys_catalog.cc:455] T 00000000000000000000000000000000 P d2c8d94c8b904af0b5807cc170c1ee96 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d2c8d94c8b904af0b5807cc170c1ee96. Latest consensus state: current_term: 1 leader_uuid: "d2c8d94c8b904af0b5807cc170c1ee96" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d2c8d94c8b904af0b5807cc170c1ee96" member_type: VOTER } }
I20240503 23:27:03.443192 5033 sys_catalog.cc:455] T 00000000000000000000000000000000 P d2c8d94c8b904af0b5807cc170c1ee96 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d2c8d94c8b904af0b5807cc170c1ee96" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d2c8d94c8b904af0b5807cc170c1ee96" member_type: VOTER } }
I20240503 23:27:03.443425 5033 sys_catalog.cc:458] T 00000000000000000000000000000000 P d2c8d94c8b904af0b5807cc170c1ee96 [sys.catalog]: This master's current role is: LEADER
I20240503 23:27:03.443934 5034 sys_catalog.cc:458] T 00000000000000000000000000000000 P d2c8d94c8b904af0b5807cc170c1ee96 [sys.catalog]: This master's current role is: LEADER
I20240503 23:27:03.444785 5038 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240503 23:27:03.446282 5038 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240503 23:27:03.448113 400 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20240503 23:27:03.449883 5038 catalog_manager.cc:1347] Generated new cluster ID: bf8ef747e22f4b4495a4ba194023c121
I20240503 23:27:03.449995 5038 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240503 23:27:03.466885 5038 catalog_manager.cc:1370] Generated new certificate authority record
I20240503 23:27:03.467842 5038 catalog_manager.cc:1504] Loading token signing keys...
I20240503 23:27:03.479771 5038 catalog_manager.cc:5857] T 00000000000000000000000000000000 P d2c8d94c8b904af0b5807cc170c1ee96: Generated new TSK 0
I20240503 23:27:03.480078 5038 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240503 23:27:03.513985 400 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 23:27:03.517556 5050 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 23:27:03.518369 5051 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 23:27:03.518790 5053 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 23:27:03.519325 400 server_base.cc:1034] running on GCE node
I20240503 23:27:03.519623 400 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240503 23:27:03.519677 400 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240503 23:27:03.519703 400 hybrid_clock.cc:648] HybridClock initialized: now 1714778823519699 us; error 0 us; skew 500 ppm
I20240503 23:27:03.519843 400 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 23:27:03.521449 400 webserver.cc:460] Webserver started at http://127.0.100.1:41383/ using document root <none> and password file <none>
I20240503 23:27:03.521688 400 fs_manager.cc:359] Metadata directory not provided
I20240503 23:27:03.521750 400 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 23:27:03.521855 400 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 23:27:03.522599 400 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZhIiPz/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingUnstableCluster.1714778688451250-400-0/minicluster-data/ts-0-root/instance:
uuid: "7642a2c0fa914384bfe0d910f91b06c2"
format_stamp: "Formatted at 2024-05-03 23:27:03 on dist-test-slave-jvbp"
I20240503 23:27:03.525748 400 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.002s
I20240503 23:27:03.527995 5058 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 23:27:03.528692 400 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20240503 23:27:03.528820 400 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZhIiPz/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingUnstableCluster.1714778688451250-400-0/minicluster-data/ts-0-root
uuid: "7642a2c0fa914384bfe0d910f91b06c2"
format_stamp: "Formatted at 2024-05-03 23:27:03 on dist-test-slave-jvbp"
I20240503 23:27:03.528985 400 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZhIiPz/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingUnstableCluster.1714778688451250-400-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskZhIiPz/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingUnstableCluster.1714778688451250-400-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskZhIiPz/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingUnstableCluster.1714778688451250-400-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 23:27:03.552856 400 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 23:27:03.553532 400 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 23:27:03.554252 400 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 23:27:03.555364 400 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 23:27:03.555433 400 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 23:27:03.555524 400 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 23:27:03.555572 400 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 23:27:03.573858 400 rpc_server.cc:307] RPC server started. Bound to: 127.0.100.1:43223
I20240503 23:27:03.573876 5120 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.100.1:43223 every 8 connection(s)
I20240503 23:27:03.576048 400 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 23:27:03.579926 5125 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 23:27:03.580325 5126 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 23:27:03.583359 5128 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 23:27:03.584026 5121 heartbeater.cc:347] Connected to a master server at 127.0.100.62:39883
I20240503 23:27:03.584204 5121 heartbeater.cc:464] Registering TS with master...
I20240503 23:27:03.584538 400 server_base.cc:1034] running on GCE node
I20240503 23:27:03.584793 5121 heartbeater.cc:511] Master 127.0.100.62:39883 requested a full tablet report, sending...
I20240503 23:27:03.584872 400 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240503 23:27:03.584935 400 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240503 23:27:03.584972 400 hybrid_clock.cc:648] HybridClock initialized: now 1714778823584966 us; error 0 us; skew 500 ppm
I20240503 23:27:03.585127 400 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 23:27:03.586015 4994 ts_manager.cc:194] Registered new tserver with Master: 7642a2c0fa914384bfe0d910f91b06c2 (127.0.100.1:43223)
I20240503 23:27:03.587160 400 webserver.cc:460] Webserver started at http://127.0.100.2:43769/ using document root <none> and password file <none>
I20240503 23:27:03.587458 400 fs_manager.cc:359] Metadata directory not provided
I20240503 23:27:03.587539 400 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 23:27:03.587762 400 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 23:27:03.587749 4994 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:41848
I20240503 23:27:03.588719 400 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZhIiPz/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingUnstableCluster.1714778688451250-400-0/minicluster-data/ts-1-root/instance:
uuid: "eed2fd17595345ef928fff404160a150"
format_stamp: "Formatted at 2024-05-03 23:27:03 on dist-test-slave-jvbp"
I20240503 23:27:03.591856 400 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20240503 23:27:03.594274 5133 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 23:27:03.594746 400 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20240503 23:27:03.594868 400 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZhIiPz/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingUnstableCluster.1714778688451250-400-0/minicluster-data/ts-1-root
uuid: "eed2fd17595345ef928fff404160a150"
format_stamp: "Formatted at 2024-05-03 23:27:03 on dist-test-slave-jvbp"
I20240503 23:27:03.595000 400 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZhIiPz/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingUnstableCluster.1714778688451250-400-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskZhIiPz/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingUnstableCluster.1714778688451250-400-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskZhIiPz/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingUnstableCluster.1714778688451250-400-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 23:27:03.608435 400 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 23:27:03.626904 400 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 23:27:03.627813 400 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 23:27:03.628866 400 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 23:27:03.628935 400 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 23:27:03.629019 400 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 23:27:03.629057 400 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 23:27:03.643692 400 rpc_server.cc:307] RPC server started. Bound to: 127.0.100.2:36927
I20240503 23:27:03.643704 5195 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.100.2:36927 every 8 connection(s)
I20240503 23:27:03.645608 400 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 23:27:03.650002 5199 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 23:27:03.650450 5200 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 23:27:03.651822 5196 heartbeater.cc:347] Connected to a master server at 127.0.100.62:39883
I20240503 23:27:03.651994 5196 heartbeater.cc:464] Registering TS with master...
W20240503 23:27:03.652096 5202 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 23:27:03.652401 5196 heartbeater.cc:511] Master 127.0.100.62:39883 requested a full tablet report, sending...
I20240503 23:27:03.652487 400 server_base.cc:1034] running on GCE node
I20240503 23:27:03.652777 400 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240503 23:27:03.652838 400 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240503 23:27:03.652880 400 hybrid_clock.cc:648] HybridClock initialized: now 1714778823652874 us; error 0 us; skew 500 ppm
I20240503 23:27:03.653026 400 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 23:27:03.653451 4994 ts_manager.cc:194] Registered new tserver with Master: eed2fd17595345ef928fff404160a150 (127.0.100.2:36927)
I20240503 23:27:03.654516 4994 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:41864
I20240503 23:27:03.654670 400 webserver.cc:460] Webserver started at http://127.0.100.3:44681/ using document root <none> and password file <none>
I20240503 23:27:03.654901 400 fs_manager.cc:359] Metadata directory not provided
I20240503 23:27:03.654976 400 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 23:27:03.655112 400 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 23:27:03.655824 400 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZhIiPz/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingUnstableCluster.1714778688451250-400-0/minicluster-data/ts-2-root/instance:
uuid: "3e0974b73a504d5cbcbab1aa6593966b"
format_stamp: "Formatted at 2024-05-03 23:27:03 on dist-test-slave-jvbp"
I20240503 23:27:03.658691 400 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.003s
I20240503 23:27:03.660585 5207 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 23:27:03.660976 400 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.001s
I20240503 23:27:03.661080 400 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZhIiPz/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingUnstableCluster.1714778688451250-400-0/minicluster-data/ts-2-root
uuid: "3e0974b73a504d5cbcbab1aa6593966b"
format_stamp: "Formatted at 2024-05-03 23:27:03 on dist-test-slave-jvbp"
I20240503 23:27:03.661208 400 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZhIiPz/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingUnstableCluster.1714778688451250-400-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-taskZhIiPz/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingUnstableCluster.1714778688451250-400-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-taskZhIiPz/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingUnstableCluster.1714778688451250-400-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 23:27:03.676769 400 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 23:27:03.677424 400 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 23:27:03.678095 400 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 23:27:03.679141 400 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 23:27:03.679209 400 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 23:27:03.679302 400 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 23:27:03.679340 400 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 23:27:03.696410 400 rpc_server.cc:307] RPC server started. Bound to: 127.0.100.3:39269
I20240503 23:27:03.696422 5269 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.100.3:39269 every 8 connection(s)
I20240503 23:27:03.704152 5270 heartbeater.cc:347] Connected to a master server at 127.0.100.62:39883
I20240503 23:27:03.704344 5270 heartbeater.cc:464] Registering TS with master...
I20240503 23:27:03.704815 5270 heartbeater.cc:511] Master 127.0.100.62:39883 requested a full tablet report, sending...
I20240503 23:27:03.705789 4994 ts_manager.cc:194] Registered new tserver with Master: 3e0974b73a504d5cbcbab1aa6593966b (127.0.100.3:39269)
I20240503 23:27:03.706648 4994 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:41874
I20240503 23:27:03.706821 400 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.009190127s
I20240503 23:27:04.589534 5121 heartbeater.cc:503] Master 127.0.100.62:39883 was elected leader, sending a full tablet report...
I20240503 23:27:04.656112 5196 heartbeater.cc:503] Master 127.0.100.62:39883 was elected leader, sending a full tablet report...
I20240503 23:27:04.708343 5270 heartbeater.cc:503] Master 127.0.100.62:39883 was elected leader, sending a full tablet report...
I20240503 23:27:04.736501 400 test_util.cc:268] Using random seed: 246906602
I20240503 23:27:04.753530 4994 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41888:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "\004\001\000\252\252\252*\004\001\000TUUU""\004\001\000\252\252\252*\004\001\000TUUU"
indirect_data: """"
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 23:27:04.754412 4994 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-workload in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 23:27:04.777437 5234 tablet_service.cc:1467] Processing CreateTablet for tablet c38e225db7cf4e11bd045f8e8127d79e (DEFAULT_TABLE table=test-workload [id=00fe2013bb2c478089860377060f6f88]), partition=RANGE (key) PARTITION 715827882 <= VALUES < 1431655764
I20240503 23:27:04.778103 5234 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c38e225db7cf4e11bd045f8e8127d79e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 23:27:04.778419 5233 tablet_service.cc:1467] Processing CreateTablet for tablet 3418f665dd5b4131add5ea54d2c2eec5 (DEFAULT_TABLE table=test-workload [id=00fe2013bb2c478089860377060f6f88]), partition=RANGE (key) PARTITION 1431655764 <= VALUES
I20240503 23:27:04.777491 5235 tablet_service.cc:1467] Processing CreateTablet for tablet 50314a53e23a42eeaa12b4babbc74afa (DEFAULT_TABLE table=test-workload [id=00fe2013bb2c478089860377060f6f88]), partition=RANGE (key) PARTITION VALUES < 715827882
I20240503 23:27:04.778856 5233 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3418f665dd5b4131add5ea54d2c2eec5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 23:27:04.779060 5235 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 50314a53e23a42eeaa12b4babbc74afa. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 23:27:04.783696 5086 tablet_service.cc:1467] Processing CreateTablet for tablet 50314a53e23a42eeaa12b4babbc74afa (DEFAULT_TABLE table=test-workload [id=00fe2013bb2c478089860377060f6f88]), partition=RANGE (key) PARTITION VALUES < 715827882
I20240503 23:27:04.784262 5086 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 50314a53e23a42eeaa12b4babbc74afa. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 23:27:04.787989 5085 tablet_service.cc:1467] Processing CreateTablet for tablet c38e225db7cf4e11bd045f8e8127d79e (DEFAULT_TABLE table=test-workload [id=00fe2013bb2c478089860377060f6f88]), partition=RANGE (key) PARTITION 715827882 <= VALUES < 1431655764
I20240503 23:27:04.788623 5085 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c38e225db7cf4e11bd045f8e8127d79e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 23:27:04.792775 5084 tablet_service.cc:1467] Processing CreateTablet for tablet 3418f665dd5b4131add5ea54d2c2eec5 (DEFAULT_TABLE table=test-workload [id=00fe2013bb2c478089860377060f6f88]), partition=RANGE (key) PARTITION 1431655764 <= VALUES
I20240503 23:27:04.793421 5084 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3418f665dd5b4131add5ea54d2c2eec5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 23:27:04.794026 5290 tablet_bootstrap.cc:492] T 3418f665dd5b4131add5ea54d2c2eec5 P 3e0974b73a504d5cbcbab1aa6593966b: Bootstrap starting.
I20240503 23:27:04.796161 5291 tablet_bootstrap.cc:492] T c38e225db7cf4e11bd045f8e8127d79e P 7642a2c0fa914384bfe0d910f91b06c2: Bootstrap starting.
I20240503 23:27:04.796667 5290 tablet_bootstrap.cc:654] T 3418f665dd5b4131add5ea54d2c2eec5 P 3e0974b73a504d5cbcbab1aa6593966b: Neither blocks nor log segments found. Creating new log.
I20240503 23:27:04.799111 5291 tablet_bootstrap.cc:654] T c38e225db7cf4e11bd045f8e8127d79e P 7642a2c0fa914384bfe0d910f91b06c2: Neither blocks nor log segments found. Creating new log.
I20240503 23:27:04.804013 5291 tablet_bootstrap.cc:492] T c38e225db7cf4e11bd045f8e8127d79e P 7642a2c0fa914384bfe0d910f91b06c2: No bootstrap required, opened a new log
I20240503 23:27:04.804013 5290 tablet_bootstrap.cc:492] T 3418f665dd5b4131add5ea54d2c2eec5 P 3e0974b73a504d5cbcbab1aa6593966b: No bootstrap required, opened a new log
I20240503 23:27:04.804221 5291 ts_tablet_manager.cc:1367] T c38e225db7cf4e11bd045f8e8127d79e P 7642a2c0fa914384bfe0d910f91b06c2: Time spent bootstrapping tablet: real 0.008s user 0.003s sys 0.002s
I20240503 23:27:04.804219 5290 ts_tablet_manager.cc:1367] T 3418f665dd5b4131add5ea54d2c2eec5 P 3e0974b73a504d5cbcbab1aa6593966b: Time spent bootstrapping tablet: real 0.010s user 0.001s sys 0.008s
I20240503 23:27:04.804833 5161 tablet_service.cc:1467] Processing CreateTablet for tablet 50314a53e23a42eeaa12b4babbc74afa (DEFAULT_TABLE table=test-workload [id=00fe2013bb2c478089860377060f6f88]), partition=RANGE (key) PARTITION VALUES < 715827882
I20240503 23:27:04.804893 5159 tablet_service.cc:1467] Processing CreateTablet for tablet 3418f665dd5b4131add5ea54d2c2eec5 (DEFAULT_TABLE table=test-workload [id=00fe2013bb2c478089860377060f6f88]), partition=RANGE (key) PARTITION 1431655764 <= VALUES
I20240503 23:27:04.805207 5290 raft_consensus.cc:348] T 3418f665dd5b4131add5ea54d2c2eec5 P 3e0974b73a504d5cbcbab1aa6593966b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7642a2c0fa914384bfe0d910f91b06c2" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 43223 } } peers { permanent_uuid: "eed2fd17595345ef928fff404160a150" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 36927 } } peers { permanent_uuid: "3e0974b73a504d5cbcbab1aa6593966b" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 39269 } }
I20240503 23:27:04.805452 5161 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 50314a53e23a42eeaa12b4babbc74afa. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 23:27:04.805383 5160 tablet_service.cc:1467] Processing CreateTablet for tablet c38e225db7cf4e11bd045f8e8127d79e (DEFAULT_TABLE table=test-workload [id=00fe2013bb2c478089860377060f6f88]), partition=RANGE (key) PARTITION 715827882 <= VALUES < 1431655764
I20240503 23:27:04.805449 5290 raft_consensus.cc:374] T 3418f665dd5b4131add5ea54d2c2eec5 P 3e0974b73a504d5cbcbab1aa6593966b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 23:27:04.805552 5290 raft_consensus.cc:729] T 3418f665dd5b4131add5ea54d2c2eec5 P 3e0974b73a504d5cbcbab1aa6593966b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3e0974b73a504d5cbcbab1aa6593966b, State: Initialized, Role: FOLLOWER
I20240503 23:27:04.805681 5159 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3418f665dd5b4131add5ea54d2c2eec5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 23:27:04.805881 5160 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c38e225db7cf4e11bd045f8e8127d79e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 23:27:04.805791 5290 consensus_queue.cc:260] T 3418f665dd5b4131add5ea54d2c2eec5 P 3e0974b73a504d5cbcbab1aa6593966b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7642a2c0fa914384bfe0d910f91b06c2" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 43223 } } peers { permanent_uuid: "eed2fd17595345ef928fff404160a150" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 36927 } } peers { permanent_uuid: "3e0974b73a504d5cbcbab1aa6593966b" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 39269 } }
I20240503 23:27:04.813117 5291 raft_consensus.cc:348] T c38e225db7cf4e11bd045f8e8127d79e P 7642a2c0fa914384bfe0d910f91b06c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7642a2c0fa914384bfe0d910f91b06c2" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 43223 } } peers { permanent_uuid: "eed2fd17595345ef928fff404160a150" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 36927 } } peers { permanent_uuid: "3e0974b73a504d5cbcbab1aa6593966b" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 39269 } }
I20240503 23:27:04.813419 5291 raft_consensus.cc:374] T c38e225db7cf4e11bd045f8e8127d79e P 7642a2c0fa914384bfe0d910f91b06c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 23:27:04.813501 5291 raft_consensus.cc:729] T c38e225db7cf4e11bd045f8e8127d79e P 7642a2c0fa914384bfe0d910f91b06c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7642a2c0fa914384bfe0d910f91b06c2, State: Initialized, Role: FOLLOWER
I20240503 23:27:04.813817 5291 consensus_queue.cc:260] T c38e225db7cf4e11bd045f8e8127d79e P 7642a2c0fa914384bfe0d910f91b06c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7642a2c0fa914384bfe0d910f91b06c2" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 43223 } } peers { permanent_uuid: "eed2fd17595345ef928fff404160a150" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 36927 } } peers { permanent_uuid: "3e0974b73a504d5cbcbab1aa6593966b" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 39269 } }
I20240503 23:27:04.819141 5295 tablet_bootstrap.cc:492] T 3418f665dd5b4131add5ea54d2c2eec5 P eed2fd17595345ef928fff404160a150: Bootstrap starting.
I20240503 23:27:04.821247 5290 ts_tablet_manager.cc:1398] T 3418f665dd5b4131add5ea54d2c2eec5 P 3e0974b73a504d5cbcbab1aa6593966b: Time spent starting tablet: real 0.017s user 0.002s sys 0.008s
I20240503 23:27:04.821650 5290 tablet_bootstrap.cc:492] T c38e225db7cf4e11bd045f8e8127d79e P 3e0974b73a504d5cbcbab1aa6593966b: Bootstrap starting.
I20240503 23:27:04.821945 5297 raft_consensus.cc:482] T c38e225db7cf4e11bd045f8e8127d79e P 7642a2c0fa914384bfe0d910f91b06c2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 23:27:04.822192 5295 tablet_bootstrap.cc:654] T 3418f665dd5b4131add5ea54d2c2eec5 P eed2fd17595345ef928fff404160a150: Neither blocks nor log segments found. Creating new log.
I20240503 23:27:04.822113 5297 raft_consensus.cc:504] T c38e225db7cf4e11bd045f8e8127d79e P 7642a2c0fa914384bfe0d910f91b06c2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7642a2c0fa914384bfe0d910f91b06c2" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 43223 } } peers { permanent_uuid: "eed2fd17595345ef928fff404160a150" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 36927 } } peers { permanent_uuid: "3e0974b73a504d5cbcbab1aa6593966b" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 39269 } }
I20240503 23:27:04.827490 5297 leader_election.cc:288] T c38e225db7cf4e11bd045f8e8127d79e P 7642a2c0fa914384bfe0d910f91b06c2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers eed2fd17595345ef928fff404160a150 (127.0.100.2:36927), 3e0974b73a504d5cbcbab1aa6593966b (127.0.100.3:39269)
I20240503 23:27:04.828197 5291 ts_tablet_manager.cc:1398] T c38e225db7cf4e11bd045f8e8127d79e P 7642a2c0fa914384bfe0d910f91b06c2: Time spent starting tablet: real 0.024s user 0.003s sys 0.011s
I20240503 23:27:04.828632 5291 tablet_bootstrap.cc:492] T 3418f665dd5b4131add5ea54d2c2eec5 P 7642a2c0fa914384bfe0d910f91b06c2: Bootstrap starting.
I20240503 23:27:04.829624 5290 tablet_bootstrap.cc:654] T c38e225db7cf4e11bd045f8e8127d79e P 3e0974b73a504d5cbcbab1aa6593966b: Neither blocks nor log segments found. Creating new log.
I20240503 23:27:04.830495 5291 tablet_bootstrap.cc:654] T 3418f665dd5b4131add5ea54d2c2eec5 P 7642a2c0fa914384bfe0d910f91b06c2: Neither blocks nor log segments found. Creating new log.
I20240503 23:27:04.832585 5290 tablet_bootstrap.cc:492] T c38e225db7cf4e11bd045f8e8127d79e P 3e0974b73a504d5cbcbab1aa6593966b: No bootstrap required, opened a new log
I20240503 23:27:04.832805 5290 ts_tablet_manager.cc:1367] T c38e225db7cf4e11bd045f8e8127d79e P 3e0974b73a504d5cbcbab1aa6593966b: Time spent bootstrapping tablet: real 0.011s user 0.001s sys 0.004s
I20240503 23:27:04.833892 5290 raft_consensus.cc:348] T c38e225db7cf4e11bd045f8e8127d79e P 3e0974b73a504d5cbcbab1aa6593966b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7642a2c0fa914384bfe0d910f91b06c2" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 43223 } } peers { permanent_uuid: "eed2fd17595345ef928fff404160a150" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 36927 } } peers { permanent_uuid: "3e0974b73a504d5cbcbab1aa6593966b" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 39269 } }
I20240503 23:27:04.834128 5290 raft_consensus.cc:374] T c38e225db7cf4e11bd045f8e8127d79e P 3e0974b73a504d5cbcbab1aa6593966b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 23:27:04.834180 5290 raft_consensus.cc:729] T c38e225db7cf4e11bd045f8e8127d79e P 3e0974b73a504d5cbcbab1aa6593966b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3e0974b73a504d5cbcbab1aa6593966b, State: Initialized, Role: FOLLOWER
I20240503 23:27:04.834506 5290 consensus_queue.cc:260] T c38e225db7cf4e11bd045f8e8127d79e P 3e0974b73a504d5cbcbab1aa6593966b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7642a2c0fa914384bfe0d910f91b06c2" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 43223 } } peers { permanent_uuid: "eed2fd17595345ef928fff404160a150" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 36927 } } peers { permanent_uuid: "3e0974b73a504d5cbcbab1aa6593966b" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 39269 } }
I20240503 23:27:04.837987 5290 ts_tablet_manager.cc:1398] T c38e225db7cf4e11bd045f8e8127d79e P 3e0974b73a504d5cbcbab1aa6593966b: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20240503 23:27:04.838775 5171 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c38e225db7cf4e11bd045f8e8127d79e" candidate_uuid: "7642a2c0fa914384bfe0d910f91b06c2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eed2fd17595345ef928fff404160a150" is_pre_election: true
W20240503 23:27:04.839489 5059 leader_election.cc:341] T c38e225db7cf4e11bd045f8e8127d79e P 7642a2c0fa914384bfe0d910f91b06c2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer eed2fd17595345ef928fff404160a150 (127.0.100.2:36927): Illegal state: must be running to vote when last-logged opid is not known
I20240503 23:27:04.840209 5295 tablet_bootstrap.cc:492] T 3418f665dd5b4131add5ea54d2c2eec5 P eed2fd17595345ef928fff404160a150: No bootstrap required, opened a new log
I20240503 23:27:04.840443 5295 ts_tablet_manager.cc:1367] T 3418f665dd5b4131add5ea54d2c2eec5 P eed2fd17595345ef928fff404160a150: Time spent bootstrapping tablet: real 0.021s user 0.002s sys 0.006s
I20240503 23:27:04.840327 5245 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c38e225db7cf4e11bd045f8e8127d79e" candidate_uuid: "7642a2c0fa914384bfe0d910f91b06c2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3e0974b73a504d5cbcbab1aa6593966b" is_pre_election: true
I20240503 23:27:04.840737 5245 raft_consensus.cc:2454] T c38e225db7cf4e11bd045f8e8127d79e P 3e0974b73a504d5cbcbab1aa6593966b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7642a2c0fa914384bfe0d910f91b06c2 in term 0.
I20240503 23:27:04.841151 5290 tablet_bootstrap.cc:492] T 50314a53e23a42eeaa12b4babbc74afa P 3e0974b73a504d5cbcbab1aa6593966b: Bootstrap starting.
I20240503 23:27:04.841176 5061 leader_election.cc:302] T c38e225db7cf4e11bd045f8e8127d79e P 7642a2c0fa914384bfe0d910f91b06c2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3e0974b73a504d5cbcbab1aa6593966b, 7642a2c0fa914384bfe0d910f91b06c2; no voters: eed2fd17595345ef928fff404160a150
I20240503 23:27:04.841708 5296 raft_consensus.cc:2781] T c38e225db7cf4e11bd045f8e8127d79e P 7642a2c0fa914384bfe0d910f91b06c2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 23:27:04.841811 5296 raft_consensus.cc:482] T c38e225db7cf4e11bd045f8e8127d79e P 7642a2c0fa914384bfe0d910f91b06c2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 23:27:04.841902 5296 raft_consensus.cc:3037] T c38e225db7cf4e11bd045f8e8127d79e P 7642a2c0fa914384bfe0d910f91b06c2 [term 0 FOLLOWER]: Advancing to term 1
I20240503 23:27:04.842219 5295 raft_consensus.cc:348] T 3418f665dd5b4131add5ea54d2c2eec5 P eed2fd17595345ef928fff404160a150 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7642a2c0fa914384bfe0d910f91b06c2" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 43223 } } peers { permanent_uuid: "eed2fd17595345ef928fff404160a150" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 36927 } } peers { permanent_uuid: "3e0974b73a504d5cbcbab1aa6593966b" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 39269 } }
I20240503 23:27:04.842588 5295 raft_consensus.cc:374] T 3418f665dd5b4131add5ea54d2c2eec5 P eed2fd17595345ef928fff404160a150 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 23:27:04.842687 5295 raft_consensus.cc:729] T 3418f665dd5b4131add5ea54d2c2eec5 P eed2fd17595345ef928fff404160a150 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eed2fd17595345ef928fff404160a150, State: Initialized, Role: FOLLOWER
I20240503 23:27:04.843040 5295 consensus_queue.cc:260] T 3418f665dd5b4131add5ea54d2c2eec5 P eed2fd17595345ef928fff404160a150 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7642a2c0fa914384bfe0d910f91b06c2" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 43223 } } peers { permanent_uuid: "eed2fd17595345ef928fff404160a150" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 36927 } } peers { permanent_uuid: "3e0974b73a504d5cbcbab1aa6593966b" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 39269 } }
I20240503 23:27:04.843780 5290 tablet_bootstrap.cc:654] T 50314a53e23a42eeaa12b4babbc74afa P 3e0974b73a504d5cbcbab1aa6593966b: Neither blocks nor log segments found. Creating new log.
I20240503 23:27:04.845253 5294 raft_consensus.cc:482] T 3418f665dd5b4131add5ea54d2c2eec5 P 3e0974b73a504d5cbcbab1aa6593966b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 23:27:04.845451 5294 raft_consensus.cc:504] T 3418f665dd5b4131add5ea54d2c2eec5 P 3e0974b73a504d5cbcbab1aa6593966b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7642a2c0fa914384bfe0d910f91b06c2" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 43223 } } peers { permanent_uuid: "eed2fd17595345ef928fff404160a150" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 36927 } } peers { permanent_uuid: "3e0974b73a504d5cbcbab1aa6593966b" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 39269 } }
I20240503 23:27:04.846729 5291 tablet_bootstrap.cc:492] T 3418f665dd5b4131add5ea54d2c2eec5 P 7642a2c0fa914384bfe0d910f91b06c2: No bootstrap required, opened a new log
I20240503 23:27:04.847030 5291 ts_tablet_manager.cc:1367] T 3418f665dd5b4131add5ea54d2c2eec5 P 7642a2c0fa914384bfe0d910f91b06c2: Time spent bootstrapping tablet: real 0.019s user 0.010s sys 0.007s
I20240503 23:27:04.848199 5291 raft_consensus.cc:348] T 3418f665dd5b4131add5ea54d2c2eec5 P 7642a2c0fa914384bfe0d910f91b06c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7642a2c0fa914384bfe0d910f91b06c2" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 43223 } } peers { permanent_uuid: "eed2fd17595345ef928fff404160a150" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 36927 } } peers { permanent_uuid: "3e0974b73a504d5cbcbab1aa6593966b" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 39269 } }
I20240503 23:27:04.848503 5291 raft_consensus.cc:374] T 3418f665dd5b4131add5ea54d2c2eec5 P 7642a2c0fa914384bfe0d910f91b06c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 23:27:04.848590 5291 raft_consensus.cc:729] T 3418f665dd5b4131add5ea54d2c2eec5 P 7642a2c0fa914384bfe0d910f91b06c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7642a2c0fa914384bfe0d910f91b06c2, State: Initialized, Role: FOLLOWER
I20240503 23:27:04.846755 5294 leader_election.cc:288] T 3418f665dd5b4131add5ea54d2c2eec5 P 3e0974b73a504d5cbcbab1aa6593966b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7642a2c0fa914384bfe0d910f91b06c2 (127.0.100.1:43223), eed2fd17595345ef928fff404160a150 (127.0.100.2:36927)
I20240503 23:27:04.848846 5291 consensus_queue.cc:260] T 3418f665dd5b4131add5ea54d2c2eec5 P 7642a2c0fa914384bfe0d910f91b06c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7642a2c0fa914384bfe0d910f91b06c2" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 43223 } } peers { permanent_uuid: "eed2fd17595345ef928fff404160a150" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 36927 } } peers { permanent_uuid: "3e0974b73a504d5cbcbab1aa6593966b" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 39269 } }
I20240503 23:27:04.849457 5291 ts_tablet_manager.cc:1398] T 3418f665dd5b4131add5ea54d2c2eec5 P 7642a2c0fa914384bfe0d910f91b06c2: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 23:27:04.849916 5296 raft_consensus.cc:504] T c38e225db7cf4e11bd045f8e8127d79e P 7642a2c0fa914384bfe0d910f91b06c2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7642a2c0fa914384bfe0d910f91b06c2" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 43223 } } peers { permanent_uuid: "eed2fd17595345ef928fff404160a150" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 36927 } } peers { permanent_uuid: "3e0974b73a504d5cbcbab1aa6593966b" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 39269 } }
I20240503 23:27:04.851758 5171 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c38e225db7cf4e11bd045f8e8127d79e" candidate_uuid: "7642a2c0fa914384bfe0d910f91b06c2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eed2fd17595345ef928fff404160a150"
W20240503 23:27:04.852665 5059 leader_election.cc:341] T c38e225db7cf4e11bd045f8e8127d79e P 7642a2c0fa914384bfe0d910f91b06c2 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer eed2fd17595345ef928fff404160a150 (127.0.100.2:36927): Illegal state: must be running to vote when last-logged opid is not known
I20240503 23:27:04.853421 5291 tablet_bootstrap.cc:492] T 50314a53e23a42eeaa12b4babbc74afa P 7642a2c0fa914384bfe0d910f91b06c2: Bootstrap starting.
I20240503 23:27:04.854444 5245 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c38e225db7cf4e11bd045f8e8127d79e" candidate_uuid: "7642a2c0fa914384bfe0d910f91b06c2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3e0974b73a504d5cbcbab1aa6593966b"
I20240503 23:27:04.854676 5245 raft_consensus.cc:3037] T c38e225db7cf4e11bd045f8e8127d79e P 3e0974b73a504d5cbcbab1aa6593966b [term 0 FOLLOWER]: Advancing to term 1
I20240503 23:27:04.855438 5291 tablet_bootstrap.cc:654] T 50314a53e23a42eeaa12b4babbc74afa P 7642a2c0fa914384bfe0d910f91b06c2: Neither blocks nor log segments found. Creating new log.
I20240503 23:27:04.848186 5295 ts_tablet_manager.cc:1398] T 3418f665dd5b4131add5ea54d2c2eec5 P eed2fd17595345ef928fff404160a150: Time spent starting tablet: real 0.007s user 0.004s sys 0.001s
I20240503 23:27:04.859897 5295 tablet_bootstrap.cc:492] T c38e225db7cf4e11bd045f8e8127d79e P eed2fd17595345ef928fff404160a150: Bootstrap starting.
I20240503 23:27:04.864133 5245 raft_consensus.cc:2454] T c38e225db7cf4e11bd045f8e8127d79e P 3e0974b73a504d5cbcbab1aa6593966b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7642a2c0fa914384bfe0d910f91b06c2 in term 1.
I20240503 23:27:04.864936 5061 leader_election.cc:302] T c38e225db7cf4e11bd045f8e8127d79e P 7642a2c0fa914384bfe0d910f91b06c2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3e0974b73a504d5cbcbab1aa6593966b, 7642a2c0fa914384bfe0d910f91b06c2; no voters: eed2fd17595345ef928fff404160a150
I20240503 23:27:04.865450 5297 raft_consensus.cc:2781] T c38e225db7cf4e11bd045f8e8127d79e P 7642a2c0fa914384bfe0d910f91b06c2 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 23:27:04.854470 5296 leader_election.cc:288] T c38e225db7cf4e11bd045f8e8127d79e P 7642a2c0fa914384bfe0d910f91b06c2 [CANDIDATE]: Term 1 election: Requested vote from peers eed2fd17595345ef928fff404160a150 (127.0.100.2:36927), 3e0974b73a504d5cbcbab1aa6593966b (127.0.100.3:39269)
I20240503 23:27:04.871438 5296 raft_consensus.cc:482] T 3418f665dd5b4131add5ea54d2c2eec5 P 7642a2c0fa914384bfe0d910f91b06c2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 23:27:04.871672 5296 raft_consensus.cc:504] T 3418f665dd5b4131add5ea54d2c2eec5 P 7642a2c0fa914384bfe0d910f91b06c2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7642a2c0fa914384bfe0d910f91b06c2" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 43223 } } peers { permanent_uuid: "eed2fd17595345ef928fff404160a150" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 36927 } } peers { permanent_uuid: "3e0974b73a504d5cbcbab1aa6593966b" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 39269 } }
I20240503 23:27:04.873392 5171 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3418f665dd5b4131add5ea54d2c2eec5" candidate_uuid: "7642a2c0fa914384bfe0d910f91b06c2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eed2fd17595345ef928fff404160a150" is_pre_election: true
I20240503 23:27:04.873783 5171 raft_consensus.cc:2454] T 3418f665dd5b4131add5ea54d2c2eec5 P eed2fd17595345ef928fff404160a150 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7642a2c0fa914384bfe0d910f91b06c2 in term 0.
I20240503 23:27:04.874394 5059 leader_election.cc:302] T 3418f665dd5b4131add5ea54d2c2eec5 P 7642a2c0fa914384bfe0d910f91b06c2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7642a2c0fa914384bfe0d910f91b06c2, eed2fd17595345ef928fff404160a150; no voters:
I20240503 23:27:04.876230 5296 leader_election.cc:288] T 3418f665dd5b4131add5ea54d2c2eec5 P 7642a2c0fa914384bfe0d910f91b06c2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers eed2fd17595345ef928fff404160a150 (127.0.100.2:36927), 3e0974b73a504d5cbcbab1aa6593966b (127.0.100.3:39269)
I20240503 23:27:04.876439 5296 raft_consensus.cc:2781] T 3418f665dd5b4131add5ea54d2c2eec5 P 7642a2c0fa914384bfe0d910f91b06c2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 23:27:04.876576 5296 raft_consensus.cc:482] T 3418f665dd5b4131add5ea54d2c2eec5 P 7642a2c0fa914384bfe0d910f91b06c2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 23:27:04.876681 5296 raft_consensus.cc:3037] T 3418f665dd5b4131add5ea54d2c2eec5 P 7642a2c0fa914384bfe0d910f91b06c2 [term 0 FOLLOWER]: Advancing to term 1
I20240503 23:27:04.880820 5295 tablet_bootstrap.cc:654] T c38e225db7cf4e11bd045f8e8127d79e P eed2fd17595345ef928fff404160a150: Neither blocks nor log segments found. Creating new log.
I20240503 23:27:04.886567 5245 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3418f665dd5b4131add5ea54d2c2eec5" candidate_uuid: "7642a2c0fa914384bfe0d910f91b06c2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3e0974b73a504d5cbcbab1aa6593966b" is_pre_election: true
I20240503 23:27:04.887037 5245 raft_consensus.cc:2454] T 3418f665dd5b4131add5ea54d2c2eec5 P 3e0974b73a504d5cbcbab1aa6593966b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7642a2c0fa914384bfe0d910f91b06c2 in term 0.
I20240503 23:27:04.890908 5296 raft_consensus.cc:504] T 3418f665dd5b4131add5ea54d2c2eec5 P 7642a2c0fa914384bfe0d910f91b06c2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7642a2c0fa914384bfe0d910f91b06c2" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 43223 } } peers { permanent_uuid: "eed2fd17595345ef928fff404160a150" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 36927 } } peers { permanent_uuid: "3e0974b73a504d5cbcbab1aa6593966b" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 39269 } }
I20240503 23:27:04.892614 5171 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3418f665dd5b4131add5ea54d2c2eec5" candidate_uuid: "7642a2c0fa914384bfe0d910f91b06c2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eed2fd17595345ef928fff404160a150"
I20240503 23:27:04.892983 5171 raft_consensus.cc:3037] T 3418f665dd5b4131add5ea54d2c2eec5 P eed2fd17595345ef928fff404160a150 [term 0 FOLLOWER]: Advancing to term 1
I20240503 23:27:04.893903 5096 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3418f665dd5b4131add5ea54d2c2eec5" candidate_uuid: "3e0974b73a504d5cbcbab1aa6593966b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7642a2c0fa914384bfe0d910f91b06c2" is_pre_election: true
I20240503 23:27:04.894176 5096 raft_consensus.cc:2379] T 3418f665dd5b4131add5ea54d2c2eec5 P 7642a2c0fa914384bfe0d910f91b06c2 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3e0974b73a504d5cbcbab1aa6593966b in current term 1: Already voted for candidate 7642a2c0fa914384bfe0d910f91b06c2 in this term.
W20240503 23:27:04.895871 5197 tablet.cc:2369] T 3418f665dd5b4131add5ea54d2c2eec5 P eed2fd17595345ef928fff404160a150: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240503 23:27:04.897429 5171 raft_consensus.cc:2454] T 3418f665dd5b4131add5ea54d2c2eec5 P eed2fd17595345ef928fff404160a150 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7642a2c0fa914384bfe0d910f91b06c2 in term 1.
I20240503 23:27:04.898365 5059 leader_election.cc:302] T 3418f665dd5b4131add5ea54d2c2eec5 P 7642a2c0fa914384bfe0d910f91b06c2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7642a2c0fa914384bfe0d910f91b06c2, eed2fd17595345ef928fff404160a150; no voters:
I20240503 23:27:04.898901 5309 raft_consensus.cc:2781] T 3418f665dd5b4131add5ea54d2c2eec5 P 7642a2c0fa914384bfe0d910f91b06c2 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 23:27:04.899102 5309 raft_consensus.cc:686] T 3418f665dd5b4131add5ea54d2c2eec5 P 7642a2c0fa914384bfe0d910f91b06c2 [term 1 LEADER]: Becoming Leader. State: Replica: 7642a2c0fa914384bfe0d910f91b06c2, State: Running, Role: LEADER
I20240503 23:27:04.899580 5309 consensus_queue.cc:237] T 3418f665dd5b4131add5ea54d2c2eec5 P 7642a2c0fa914384bfe0d910f91b06c2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7642a2c0fa914384bfe0d910f91b06c2" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 43223 } } peers { permanent_uuid: "eed2fd17595345ef928fff404160a150" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 36927 } } peers { permanent_uuid: "3e0974b73a504d5cbcbab1aa6593966b" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 39269 } }
I20240503 23:27:04.900182 5290 tablet_bootstrap.cc:492] T 50314a53e23a42eeaa12b4babbc74afa P 3e0974b73a504d5cbcbab1aa6593966b: No bootstrap required, opened a new log
I20240503 23:27:04.900435 5290 ts_tablet_manager.cc:1367] T 50314a53e23a42eeaa12b4babbc74afa P 3e0974b73a504d5cbcbab1aa6593966b: Time spent bootstrapping tablet: real 0.059s user 0.029s sys 0.028s
I20240503 23:27:04.901364 5290 raft_consensus.cc:348] T 50314a53e23a42eeaa12b4babbc74afa P 3e0974b73a504d5cbcbab1aa6593966b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e0974b73a504d5cbcbab1aa6593966b" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 39269 } } peers { permanent_uuid: "7642a2c0fa914384bfe0d910f91b06c2" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 43223 } } peers { permanent_uuid: "eed2fd17595345ef928fff404160a150" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 36927 } }
I20240503 23:27:04.901681 5290 raft_consensus.cc:374] T 50314a53e23a42eeaa12b4babbc74afa P 3e0974b73a504d5cbcbab1aa6593966b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 23:27:04.901578 5245 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3418f665dd5b4131add5ea54d2c2eec5" candidate_uuid: "7642a2c0fa914384bfe0d910f91b06c2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3e0974b73a504d5cbcbab1aa6593966b"
I20240503 23:27:04.901806 5290 raft_consensus.cc:729] T 50314a53e23a42eeaa12b4babbc74afa P 3e0974b73a504d5cbcbab1aa6593966b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3e0974b73a504d5cbcbab1aa6593966b, State: Initialized, Role: FOLLOWER
I20240503 23:27:04.901922 5245 raft_consensus.cc:3037] T 3418f665dd5b4131add5ea54d2c2eec5 P 3e0974b73a504d5cbcbab1aa6593966b [term 0 FOLLOWER]: Advancing to term 1
I20240503 23:27:04.902220 5290 consensus_queue.cc:260] T 50314a53e23a42eeaa12b4babbc74afa P 3e0974b73a504d5cbcbab1aa6593966b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e0974b73a504d5cbcbab1aa6593966b" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 39269 } } peers { permanent_uuid: "7642a2c0fa914384bfe0d910f91b06c2" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 43223 } } peers { permanent_uuid: "eed2fd17595345ef928fff404160a150" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 36927 } }
I20240503 23:27:04.902488 5171 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3418f665dd5b4131add5ea54d2c2eec5" candidate_uuid: "3e0974b73a504d5cbcbab1aa6593966b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eed2fd17595345ef928fff404160a150" is_pre_election: true
I20240503 23:27:04.903070 5290 ts_tablet_manager.cc:1398] T 50314a53e23a42eeaa12b4babbc74afa P 3e0974b73a504d5cbcbab1aa6593966b: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 23:27:04.903347 5297 raft_consensus.cc:686] T c38e225db7cf4e11bd045f8e8127d79e P 7642a2c0fa914384bfe0d910f91b06c2 [term 1 LEADER]: Becoming Leader. State: Replica: 7642a2c0fa914384bfe0d910f91b06c2, State: Running, Role: LEADER
I20240503 23:27:04.903705 5297 consensus_queue.cc:237] T c38e225db7cf4e11bd045f8e8127d79e P 7642a2c0fa914384bfe0d910f91b06c2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7642a2c0fa914384bfe0d910f91b06c2" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 43223 } } peers { permanent_uuid: "eed2fd17595345ef928fff404160a150" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 36927 } } peers { permanent_uuid: "3e0974b73a504d5cbcbab1aa6593966b" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 39269 } }
I20240503 23:27:04.906631 5245 raft_consensus.cc:2454] T 3418f665dd5b4131add5ea54d2c2eec5 P 3e0974b73a504d5cbcbab1aa6593966b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7642a2c0fa914384bfe0d910f91b06c2 in term 1.
I20240503 23:27:04.908356 5296 leader_election.cc:288] T 3418f665dd5b4131add5ea54d2c2eec5 P 7642a2c0fa914384bfe0d910f91b06c2 [CANDIDATE]: Term 1 election: Requested vote from peers eed2fd17595345ef928fff404160a150 (127.0.100.2:36927), 3e0974b73a504d5cbcbab1aa6593966b (127.0.100.3:39269)
I20240503 23:27:04.908772 5291 tablet_bootstrap.cc:492] T 50314a53e23a42eeaa12b4babbc74afa P 7642a2c0fa914384bfe0d910f91b06c2: No bootstrap required, opened a new log
I20240503 23:27:04.908910 5171 raft_consensus.cc:2379] T 3418f665dd5b4131add5ea54d2c2eec5 P eed2fd17595345ef928fff404160a150 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3e0974b73a504d5cbcbab1aa6593966b in current term 1: Already voted for candidate 7642a2c0fa914384bfe0d910f91b06c2 in this term.
I20240503 23:27:04.908936 5291 ts_tablet_manager.cc:1367] T 50314a53e23a42eeaa12b4babbc74afa P 7642a2c0fa914384bfe0d910f91b06c2: Time spent bootstrapping tablet: real 0.056s user 0.025s sys 0.020s
I20240503 23:27:04.909585 5208 leader_election.cc:302] T 3418f665dd5b4131add5ea54d2c2eec5 P 3e0974b73a504d5cbcbab1aa6593966b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3e0974b73a504d5cbcbab1aa6593966b; no voters: 7642a2c0fa914384bfe0d910f91b06c2, eed2fd17595345ef928fff404160a150
I20240503 23:27:04.910053 5294 raft_consensus.cc:2726] T 3418f665dd5b4131add5ea54d2c2eec5 P 3e0974b73a504d5cbcbab1aa6593966b [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 23:27:04.910076 5291 raft_consensus.cc:348] T 50314a53e23a42eeaa12b4babbc74afa P 7642a2c0fa914384bfe0d910f91b06c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e0974b73a504d5cbcbab1aa6593966b" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 39269 } } peers { permanent_uuid: "7642a2c0fa914384bfe0d910f91b06c2" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 43223 } } peers { permanent_uuid: "eed2fd17595345ef928fff404160a150" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 36927 } }
I20240503 23:27:04.910439 5291 raft_consensus.cc:374] T 50314a53e23a42eeaa12b4babbc74afa P 7642a2c0fa914384bfe0d910f91b06c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 23:27:04.910513 5291 raft_consensus.cc:729] T 50314a53e23a42eeaa12b4babbc74afa P 7642a2c0fa914384bfe0d910f91b06c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7642a2c0fa914384bfe0d910f91b06c2, State: Initialized, Role: FOLLOWER
I20240503 23:27:04.910742 5291 consensus_queue.cc:260] T 50314a53e23a42eeaa12b4babbc74afa P 7642a2c0fa914384bfe0d910f91b06c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e0974b73a504d5cbcbab1aa6593966b" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 39269 } } peers { permanent_uuid: "7642a2c0fa914384bfe0d910f91b06c2" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 43223 } } peers { permanent_uuid: "eed2fd17595345ef928fff404160a150" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 36927 } }
I20240503 23:27:04.911362 5291 ts_tablet_manager.cc:1398] T 50314a53e23a42eeaa12b4babbc74afa P 7642a2c0fa914384bfe0d910f91b06c2: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 23:27:04.911789 5295 tablet_bootstrap.cc:492] T c38e225db7cf4e11bd045f8e8127d79e P eed2fd17595345ef928fff404160a150: No bootstrap required, opened a new log
I20240503 23:27:04.911954 5295 ts_tablet_manager.cc:1367] T c38e225db7cf4e11bd045f8e8127d79e P eed2fd17595345ef928fff404160a150: Time spent bootstrapping tablet: real 0.052s user 0.003s sys 0.003s
I20240503 23:27:04.912696 5295 raft_consensus.cc:348] T c38e225db7cf4e11bd045f8e8127d79e P eed2fd17595345ef928fff404160a150 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7642a2c0fa914384bfe0d910f91b06c2" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 43223 } } peers { permanent_uuid: "eed2fd17595345ef928fff404160a150" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 36927 } } peers { permanent_uuid: "3e0974b73a504d5cbcbab1aa6593966b" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 39269 } }
I20240503 23:27:04.912909 5295 raft_consensus.cc:374] T c38e225db7cf4e11bd045f8e8127d79e P eed2fd17595345ef928fff404160a150 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 23:27:04.912964 5295 raft_consensus.cc:729] T c38e225db7cf4e11bd045f8e8127d79e P eed2fd17595345ef928fff404160a150 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eed2fd17595345ef928fff404160a150, State: Initialized, Role: FOLLOWER
I20240503 23:27:04.913223 5295 consensus_queue.cc:260] T c38e225db7cf4e11bd045f8e8127d79e P eed2fd17595345ef928fff404160a150 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7642a2c0fa914384bfe0d910f91b06c2" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 43223 } } peers { permanent_uuid: "eed2fd17595345ef928fff404160a150" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 36927 } } peers { permanent_uuid: "3e0974b73a504d5cbcbab1aa6593966b" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 39269 } }
I20240503 23:27:04.913708 5295 ts_tablet_manager.cc:1398] T c38e225db7cf4e11bd045f8e8127d79e P eed2fd17595345ef928fff404160a150: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 23:27:04.913949 5295 tablet_bootstrap.cc:492] T 50314a53e23a42eeaa12b4babbc74afa P eed2fd17595345ef928fff404160a150: Bootstrap starting.
I20240503 23:27:04.913885 4993 catalog_manager.cc:5484] T c38e225db7cf4e11bd045f8e8127d79e P 7642a2c0fa914384bfe0d910f91b06c2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7642a2c0fa914384bfe0d910f91b06c2 (127.0.100.1). New cstate: current_term: 1 leader_uuid: "7642a2c0fa914384bfe0d910f91b06c2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7642a2c0fa914384bfe0d910f91b06c2" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 43223 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "eed2fd17595345ef928fff404160a150" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 36927 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3e0974b73a504d5cbcbab1aa6593966b" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 39269 } health_report { overall_health: UNKNOWN } } }
I20240503 23:27:04.915422 5295 tablet_bootstrap.cc:654] T 50314a53e23a42eeaa12b4babbc74afa P eed2fd17595345ef928fff404160a150: Neither blocks nor log segments found. Creating new log.
I20240503 23:27:04.917944 5295 tablet_bootstrap.cc:492] T 50314a53e23a42eeaa12b4babbc74afa P eed2fd17595345ef928fff404160a150: No bootstrap required, opened a new log
I20240503 23:27:04.918118 5295 ts_tablet_manager.cc:1367] T 50314a53e23a42eeaa12b4babbc74afa P eed2fd17595345ef928fff404160a150: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.003s
I20240503 23:27:04.918872 5295 raft_consensus.cc:348] T 50314a53e23a42eeaa12b4babbc74afa P eed2fd17595345ef928fff404160a150 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e0974b73a504d5cbcbab1aa6593966b" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 39269 } } peers { permanent_uuid: "7642a2c0fa914384bfe0d910f91b06c2" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 43223 } } peers { permanent_uuid: "eed2fd17595345ef928fff404160a150" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 36927 } }
I20240503 23:27:04.919092 5295 raft_consensus.cc:374] T 50314a53e23a42eeaa12b4babbc74afa P eed2fd17595345ef928fff404160a150 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 23:27:04.919160 5295 raft_consensus.cc:729] T 50314a53e23a42eeaa12b4babbc74afa P eed2fd17595345ef928fff404160a150 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eed2fd17595345ef928fff404160a150, State: Initialized, Role: FOLLOWER
I20240503 23:27:04.919397 5295 consensus_queue.cc:260] T 50314a53e23a42eeaa12b4babbc74afa P eed2fd17595345ef928fff404160a150 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e0974b73a504d5cbcbab1aa6593966b" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 39269 } } peers { permanent_uuid: "7642a2c0fa914384bfe0d910f91b06c2" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 43223 } } peers { permanent_uuid: "eed2fd17595345ef928fff404160a150" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 36927 } }
I20240503 23:27:04.919672 4994 catalog_manager.cc:5484] T 3418f665dd5b4131add5ea54d2c2eec5 P 7642a2c0fa914384bfe0d910f91b06c2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7642a2c0fa914384bfe0d910f91b06c2 (127.0.100.1). New cstate: current_term: 1 leader_uuid: "7642a2c0fa914384bfe0d910f91b06c2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7642a2c0fa914384bfe0d910f91b06c2" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 43223 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "eed2fd17595345ef928fff404160a150" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 36927 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3e0974b73a504d5cbcbab1aa6593966b" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 39269 } health_report { overall_health: UNKNOWN } } }
I20240503 23:27:04.920011 5295 ts_tablet_manager.cc:1398] T 50314a53e23a42eeaa12b4babbc74afa P eed2fd17595345ef928fff404160a150: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240503 23:27:04.936720 5303 raft_consensus.cc:482] T c38e225db7cf4e11bd045f8e8127d79e P eed2fd17595345ef928fff404160a150 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 23:27:04.936952 5303 raft_consensus.cc:504] T c38e225db7cf4e11bd045f8e8127d79e P eed2fd17595345ef928fff404160a150 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7642a2c0fa914384bfe0d910f91b06c2" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 43223 } } peers { permanent_uuid: "eed2fd17595345ef928fff404160a150" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 36927 } } peers { permanent_uuid: "3e0974b73a504d5cbcbab1aa6593966b" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 39269 } }
I20240503 23:27:04.937924 5303 leader_election.cc:288] T c38e225db7cf4e11bd045f8e8127d79e P eed2fd17595345ef928fff404160a150 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7642a2c0fa914384bfe0d910f91b06c2 (127.0.100.1:43223), 3e0974b73a504d5cbcbab1aa6593966b (127.0.100.3:39269)
I20240503 23:27:04.944432 5096 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c38e225db7cf4e11bd045f8e8127d79e" candidate_uuid: "eed2fd17595345ef928fff404160a150" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7642a2c0fa914384bfe0d910f91b06c2" is_pre_election: true
I20240503 23:27:04.944761 5245 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c38e225db7cf4e11bd045f8e8127d79e" candidate_uuid: "eed2fd17595345ef928fff404160a150" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3e0974b73a504d5cbcbab1aa6593966b" is_pre_election: true
I20240503 23:27:04.945031 5245 raft_consensus.cc:2379] T c38e225db7cf4e11bd045f8e8127d79e P 3e0974b73a504d5cbcbab1aa6593966b [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate eed2fd17595345ef928fff404160a150 in current term 1: Already voted for candidate 7642a2c0fa914384bfe0d910f91b06c2 in this term.
I20240503 23:27:04.945384 5136 leader_election.cc:302] T c38e225db7cf4e11bd045f8e8127d79e P eed2fd17595345ef928fff404160a150 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: eed2fd17595345ef928fff404160a150; no voters: 3e0974b73a504d5cbcbab1aa6593966b, 7642a2c0fa914384bfe0d910f91b06c2
I20240503 23:27:04.945673 5303 raft_consensus.cc:3037] T c38e225db7cf4e11bd045f8e8127d79e P eed2fd17595345ef928fff404160a150 [term 0 FOLLOWER]: Advancing to term 1
I20240503 23:27:04.947841 5303 raft_consensus.cc:2726] T c38e225db7cf4e11bd045f8e8127d79e P eed2fd17595345ef928fff404160a150 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 23:27:04.960301 5303 raft_consensus.cc:482] T 50314a53e23a42eeaa12b4babbc74afa P eed2fd17595345ef928fff404160a150 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 23:27:04.960579 5303 raft_consensus.cc:504] T 50314a53e23a42eeaa12b4babbc74afa P eed2fd17595345ef928fff404160a150 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e0974b73a504d5cbcbab1aa6593966b" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 39269 } } peers { permanent_uuid: "7642a2c0fa914384bfe0d910f91b06c2" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 43223 } } peers { permanent_uuid: "eed2fd17595345ef928fff404160a150" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 36927 } }
I20240503 23:27:04.961328 5303 leader_election.cc:288] T 50314a53e23a42eeaa12b4babbc74afa P eed2fd17595345ef928fff404160a150 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3e0974b73a504d5cbcbab1aa6593966b (127.0.100.3:39269), 7642a2c0fa914384bfe0d910f91b06c2 (127.0.100.1:43223)
I20240503 23:27:04.961951 5096 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "50314a53e23a42eeaa12b4babbc74afa" candidate_uuid: "eed2fd17595345ef928fff404160a150" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7642a2c0fa914384bfe0d910f91b06c2" is_pre_election: true
I20240503 23:27:04.961956 5245 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "50314a53e23a42eeaa12b4babbc74afa" candidate_uuid: "eed2fd17595345ef928fff404160a150" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3e0974b73a504d5cbcbab1aa6593966b" is_pre_election: true
I20240503 23:27:04.962178 5096 raft_consensus.cc:2454] T 50314a53e23a42eeaa12b4babbc74afa P 7642a2c0fa914384bfe0d910f91b06c2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eed2fd17595345ef928fff404160a150 in term 0.
I20240503 23:27:04.962188 5245 raft_consensus.cc:2454] T 50314a53e23a42eeaa12b4babbc74afa P 3e0974b73a504d5cbcbab1aa6593966b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eed2fd17595345ef928fff404160a150 in term 0.
I20240503 23:27:04.962581 5137 leader_election.cc:302] T 50314a53e23a42eeaa12b4babbc74afa P eed2fd17595345ef928fff404160a150 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7642a2c0fa914384bfe0d910f91b06c2, eed2fd17595345ef928fff404160a150; no voters:
I20240503 23:27:04.962900 5303 raft_consensus.cc:2781] T 50314a53e23a42eeaa12b4babbc74afa P eed2fd17595345ef928fff404160a150 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 23:27:04.962976 5303 raft_consensus.cc:482] T 50314a53e23a42eeaa12b4babbc74afa P eed2fd17595345ef928fff404160a150 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 23:27:04.963027 5303 raft_consensus.cc:3037] T 50314a53e23a42eeaa12b4babbc74afa P eed2fd17595345ef928fff404160a150 [term 0 FOLLOWER]: Advancing to term 1
I20240503 23:27:04.965205 5303 raft_consensus.cc:504] T 50314a53e23a42eeaa12b4babbc74afa P eed2fd17595345ef928fff404160a150 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e0974b73a504d5cbcbab1aa6593966b" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 39269 } } peers { permanent_uuid: "7642a2c0fa914384bfe0d910f91b06c2" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 43223 } } peers { permanent_uuid: "eed2fd17595345ef928fff404160a150" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 36927 } }
I20240503 23:27:04.965924 5303 leader_election.cc:288] T 50314a53e23a42eeaa12b4babbc74afa P eed2fd17595345ef928fff404160a150 [CANDIDATE]: Term 1 election: Requested vote from peers 3e0974b73a504d5cbcbab1aa6593966b (127.0.100.3:39269), 7642a2c0fa914384bfe0d910f91b06c2 (127.0.100.1:43223)
I20240503 23:27:04.966425 5245 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "50314a53e23a42eeaa12b4babbc74afa" candidate_uuid: "eed2fd17595345ef928fff404160a150" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3e0974b73a504d5cbcbab1aa6593966b"
I20240503 23:27:04.966428 5096 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "50314a53e23a42eeaa12b4babbc74afa" candidate_uuid: "eed2fd17595345ef928fff404160a150" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7642a2c0fa914384bfe0d910f91b06c2"
I20240503 23:27:04.966600 5245 raft_consensus.cc:3037] T 50314a53e23a42eeaa12b4babbc74afa P 3e0974b73a504d5cbcbab1aa6593966b [term 0 FOLLOWER]: Advancing to term 1
I20240503 23:27:04.966624 5096 raft_consensus.cc:3037] T 50314a53e23a42eeaa12b4babbc74afa P 7642a2c0fa914384bfe0d910f91b06c2 [term 0 FOLLOWER]: Advancing to term 1
I20240503 23:27:04.968709 5096 raft_consensus.cc:2454] T 50314a53e23a42eeaa12b4babbc74afa P 7642a2c0fa914384bfe0d910f91b06c2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eed2fd17595345ef928fff404160a150 in term 1.
I20240503 23:27:04.968709 5245 raft_consensus.cc:2454] T 50314a53e23a42eeaa12b4babbc74afa P 3e0974b73a504d5cbcbab1aa6593966b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eed2fd17595345ef928fff404160a150 in term 1.
I20240503 23:27:04.969250 5136 leader_election.cc:302] T 50314a53e23a42eeaa12b4babbc74afa P eed2fd17595345ef928fff404160a150 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 3e0974b73a504d5cbcbab1aa6593966b, 7642a2c0fa914384bfe0d910f91b06c2, eed2fd17595345ef928fff404160a150; no voters:
I20240503 23:27:04.969547 5303 raft_consensus.cc:2781] T 50314a53e23a42eeaa12b4babbc74afa P eed2fd17595345ef928fff404160a150 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 23:27:04.970007 5303 raft_consensus.cc:686] T 50314a53e23a42eeaa12b4babbc74afa P eed2fd17595345ef928fff404160a150 [term 1 LEADER]: Becoming Leader. State: Replica: eed2fd17595345ef928fff404160a150, State: Running, Role: LEADER
I20240503 23:27:04.970305 5303 consensus_queue.cc:237] T 50314a53e23a42eeaa12b4babbc74afa P eed2fd17595345ef928fff404160a150 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e0974b73a504d5cbcbab1aa6593966b" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 39269 } } peers { permanent_uuid: "7642a2c0fa914384bfe0d910f91b06c2" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 43223 } } peers { permanent_uuid: "eed2fd17595345ef928fff404160a150" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 36927 } }
I20240503 23:27:04.973215 4993 catalog_manager.cc:5484] T 50314a53e23a42eeaa12b4babbc74afa P eed2fd17595345ef928fff404160a150 reported cstate change: term changed from 0 to 1, leader changed from <none> to eed2fd17595345ef928fff404160a150 (127.0.100.2). New cstate: current_term: 1 leader_uuid: "eed2fd17595345ef928fff404160a150" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e0974b73a504d5cbcbab1aa6593966b" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 39269 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7642a2c0fa914384bfe0d910f91b06c2" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 43223 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "eed2fd17595345ef928fff404160a150" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 36927 } health_report { overall_health: HEALTHY } } }
I20240503 23:27:04.981421 400 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 23:27:04.984800 5318 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 23:27:04.985842 5319 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 23:27:04.986795 5321 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 23:27:04.991935 5297 consensus_queue.cc:1035] T c38e225db7cf4e11bd045f8e8127d79e P 7642a2c0fa914384bfe0d910f91b06c2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3e0974b73a504d5cbcbab1aa6593966b" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 39269 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 23:27:04.999789 5297 consensus_queue.cc:1035] T 3418f665dd5b4131add5ea54d2c2eec5 P 7642a2c0fa914384bfe0d910f91b06c2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eed2fd17595345ef928fff404160a150" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 36927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 23:27:05.000188 5309 consensus_queue.cc:1035] T 3418f665dd5b4131add5ea54d2c2eec5 P 7642a2c0fa914384bfe0d910f91b06c2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3e0974b73a504d5cbcbab1aa6593966b" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 39269 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240503 23:27:05.016675 5296 fault_injection.cc:43] FAULT INJECTION ENABLED!
W20240503 23:27:05.016800 5296 fault_injection.cc:44] THIS SERVER MAY CRASH!
I20240503 23:27:05.019134 5309 consensus_queue.cc:1035] T c38e225db7cf4e11bd045f8e8127d79e P 7642a2c0fa914384bfe0d910f91b06c2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eed2fd17595345ef928fff404160a150" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 36927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 23:27:05.044374 400 server_base.cc:1034] running on GCE node
I20240503 23:27:05.047058 400 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240503 23:27:05.047170 400 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240503 23:27:05.047209 400 hybrid_clock.cc:648] HybridClock initialized: now 1714778825047203 us; error 0 us; skew 500 ppm
I20240503 23:27:05.047384 400 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 23:27:05.049820 400 webserver.cc:460] Webserver started at http://127.0.100.4:33877/ using document root <none> and password file <none>
I20240503 23:27:05.050127 400 fs_manager.cc:359] Metadata directory not provided
I20240503 23:27:05.050208 400 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 23:27:05.050356 400 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 23:27:05.051146 400 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZhIiPz/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingUnstableCluster.1714778688451250-400-0/minicluster-data/ts-3-root/instance:
uuid: "3beb612d5d214657a5a40830f5226b97"
format_stamp: "Formatted at 2024-05-03 23:27:05 on dist-test-slave-jvbp"
I20240503 23:27:05.054764 400 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20240503 23:27:05.054921 5303 consensus_queue.cc:1035] T 50314a53e23a42eeaa12b4babbc74afa P eed2fd17595345ef928fff404160a150 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7642a2c0fa914384bfe0d910f91b06c2" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 43223 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 23:27:05.057159 5335 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 23:27:05.057665 400 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.002s sys 0.000s
I20240503 23:27:05.057821 400 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZhIiPz/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingUnstableCluster.1714778688451250-400-0/minicluster-data/ts-3-root
uuid: "3beb612d5d214657a5a40830f5226b97"
format_stamp: "Formatted at 2024-05-03 23:27:05 on dist-test-slave-jvbp"
I20240503 23:27:05.058101 400 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZhIiPz/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingUnstableCluster.1714778688451250-400-0/minicluster-data/ts-3-root
metadata directory: /tmp/dist-test-taskZhIiPz/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingUnstableCluster.1714778688451250-400-0/minicluster-data/ts-3-root
1 data directories: /tmp/dist-test-taskZhIiPz/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingUnstableCluster.1714778688451250-400-0/minicluster-data/ts-3-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 23:27:05.068284 400 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 23:27:05.069171 400 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 23:27:05.077353 400 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 23:27:05.078933 400 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 23:27:05.079046 400 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 23:27:05.079150 400 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 23:27:05.079195 400 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 23:27:05.083930 5303 consensus_queue.cc:1035] T 50314a53e23a42eeaa12b4babbc74afa P eed2fd17595345ef928fff404160a150 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3e0974b73a504d5cbcbab1aa6593966b" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 39269 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 23:27:05.113207 5400 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.100.4:34613 every 8 connection(s)
I20240503 23:27:05.113209 400 rpc_server.cc:307] RPC server started. Bound to: 127.0.100.4:34613
I20240503 23:27:05.126123 5401 heartbeater.cc:347] Connected to a master server at 127.0.100.62:39883
I20240503 23:27:05.126286 5401 heartbeater.cc:464] Registering TS with master...
I20240503 23:27:05.126690 5401 heartbeater.cc:511] Master 127.0.100.62:39883 requested a full tablet report, sending...
I20240503 23:27:05.127523 4994 ts_manager.cc:194] Registered new tserver with Master: 3beb612d5d214657a5a40830f5226b97 (127.0.100.4:34613)
I20240503 23:27:05.128304 4994 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:41894
I20240503 23:27:05.448063 5047 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240503 23:27:05.463074 5095 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "3418f665dd5b4131add5ea54d2c2eec5"
dest_uuid: "7642a2c0fa914384bfe0d910f91b06c2"
mode: GRACEFUL
new_leader_uuid: "3e0974b73a504d5cbcbab1aa6593966b"
from {username='slave'} at 127.0.0.1:52774
I20240503 23:27:05.463323 5095 raft_consensus.cc:595] T 3418f665dd5b4131add5ea54d2c2eec5 P 7642a2c0fa914384bfe0d910f91b06c2 [term 1 LEADER]: Received request to transfer leadership to TS 3e0974b73a504d5cbcbab1aa6593966b
I20240503 23:27:05.463881 5047 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 1
I20240503 23:27:05.463830 5096 consensus_queue.cc:237] T c38e225db7cf4e11bd045f8e8127d79e P 7642a2c0fa914384bfe0d910f91b06c2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "7642a2c0fa914384bfe0d910f91b06c2" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 43223 } } peers { permanent_uuid: "eed2fd17595345ef928fff404160a150" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 36927 } attrs { replace: true } } peers { permanent_uuid: "3e0974b73a504d5cbcbab1aa6593966b" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 39269 } } peers { permanent_uuid: "3beb612d5d214657a5a40830f5226b97" member_type: NON_VOTER last_known_addr { host: "127.0.100.4" port: 34613 } attrs { promote: true } }
I20240503 23:27:05.464043 5047 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240503 23:27:05.466782 5169 raft_consensus.cc:1261] T c38e225db7cf4e11bd045f8e8127d79e P eed2fd17595345ef928fff404160a150 [term 1 FOLLOWER]: Refusing update from remote peer 7642a2c0fa914384bfe0d910f91b06c2: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 23:27:05.467468 5245 raft_consensus.cc:1261] T c38e225db7cf4e11bd045f8e8127d79e P 3e0974b73a504d5cbcbab1aa6593966b [term 1 FOLLOWER]: Refusing update from remote peer 7642a2c0fa914384bfe0d910f91b06c2: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 23:27:05.467404 5297 consensus_queue.cc:1035] T c38e225db7cf4e11bd045f8e8127d79e P 7642a2c0fa914384bfe0d910f91b06c2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eed2fd17595345ef928fff404160a150" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 36927 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 23:27:05.468274 5297 consensus_queue.cc:1035] T c38e225db7cf4e11bd045f8e8127d79e P 7642a2c0fa914384bfe0d910f91b06c2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3e0974b73a504d5cbcbab1aa6593966b" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 39269 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 23:27:05.471072 5245 raft_consensus.cc:2932] T c38e225db7cf4e11bd045f8e8127d79e P 3e0974b73a504d5cbcbab1aa6593966b [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 3beb612d5d214657a5a40830f5226b97 (127.0.100.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "7642a2c0fa914384bfe0d910f91b06c2" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 43223 } } peers { permanent_uuid: "eed2fd17595345ef928fff404160a150" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 36927 } attrs { replace: true } } peers { permanent_uuid: "3e0974b73a504d5cbcbab1aa6593966b" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 39269 } } peers { permanent_uuid: "3beb612d5d214657a5a40830f5226b97" member_type: NON_VOTER last_known_addr { host: "127.0.100.4" port: 34613 } attrs { promote: true } } }
I20240503 23:27:05.471170 5169 raft_consensus.cc:2932] T c38e225db7cf4e11bd045f8e8127d79e P eed2fd17595345ef928fff404160a150 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 3beb612d5d214657a5a40830f5226b97 (127.0.100.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "7642a2c0fa914384bfe0d910f91b06c2" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 43223 } } peers { permanent_uuid: "eed2fd17595345ef928fff404160a150" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 36927 } attrs { replace: true } } peers { permanent_uuid: "3e0974b73a504d5cbcbab1aa6593966b" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 39269 } } peers { permanent_uuid: "3beb612d5d214657a5a40830f5226b97" member_type: NON_VOTER last_known_addr { host: "127.0.100.4" port: 34613 } attrs { promote: true } } }
W20240503 23:27:05.474200 5059 consensus_peers.cc:485] T c38e225db7cf4e11bd045f8e8127d79e P 7642a2c0fa914384bfe0d910f91b06c2 -> Peer 3beb612d5d214657a5a40830f5226b97 (127.0.100.4:34613): Couldn't send request to peer 3beb612d5d214657a5a40830f5226b97. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: c38e225db7cf4e11bd045f8e8127d79e. This is attempt 1: this message will repeat every 5th retry.
I20240503 23:27:05.477114 4994 catalog_manager.cc:5484] T c38e225db7cf4e11bd045f8e8127d79e P 3e0974b73a504d5cbcbab1aa6593966b reported cstate change: config changed from index -1 to 2, NON_VOTER 3beb612d5d214657a5a40830f5226b97 (127.0.100.4) added. New cstate: current_term: 1 leader_uuid: "7642a2c0fa914384bfe0d910f91b06c2" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "7642a2c0fa914384bfe0d910f91b06c2" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 43223 } } peers { permanent_uuid: "eed2fd17595345ef928fff404160a150" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 36927 } attrs { replace: true } } peers { permanent_uuid: "3e0974b73a504d5cbcbab1aa6593966b" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 39269 } } peers { permanent_uuid: "3beb612d5d214657a5a40830f5226b97" member_type: NON_VOTER last_known_addr { host: "127.0.100.4" port: 34613 } attrs { promote: true } } }
I20240503 23:27:05.513650 5296 raft_consensus.cc:2932] T c38e225db7cf4e11bd045f8e8127d79e P 7642a2c0fa914384bfe0d910f91b06c2 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 3beb612d5d214657a5a40830f5226b97 (127.0.100.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "7642a2c0fa914384bfe0d910f91b06c2" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 43223 } } peers { permanent_uuid: "eed2fd17595345ef928fff404160a150" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 36927 } attrs { replace: true } } peers { permanent_uuid: "3e0974b73a504d5cbcbab1aa6593966b" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 39269 } } peers { permanent_uuid: "3beb612d5d214657a5a40830f5226b97" member_type: NON_VOTER last_known_addr { host: "127.0.100.4" port: 34613 } attrs { promote: true } } }
I20240503 23:27:05.524480 5169 consensus_queue.cc:237] T 50314a53e23a42eeaa12b4babbc74afa P eed2fd17595345ef928fff404160a150 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3e0974b73a504d5cbcbab1aa6593966b" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 39269 } } peers { permanent_uuid: "7642a2c0fa914384bfe0d910f91b06c2" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 43223 } attrs { replace: true } } peers { permanent_uuid: "eed2fd17595345ef928fff404160a150" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 36927 } } peers { permanent_uuid: "3beb612d5d214657a5a40830f5226b97" member_type: NON_VOTER last_known_addr { host: "127.0.100.4" port: 34613 } attrs { promote: true } }
I20240503 23:27:05.526883 5245 raft_consensus.cc:1261] T 50314a53e23a42eeaa12b4babbc74afa P 3e0974b73a504d5cbcbab1aa6593966b [term 1 FOLLOWER]: Refusing update from remote peer eed2fd17595345ef928fff404160a150: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 23:27:05.526883 5096 raft_consensus.cc:1261] T 50314a53e23a42eeaa12b4babbc74afa P 7642a2c0fa914384bfe0d910f91b06c2 [term 1 FOLLOWER]: Refusing update from remote peer eed2fd17595345ef928fff404160a150: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 23:27:05.527647 5303 consensus_queue.cc:1035] T 50314a53e23a42eeaa12b4babbc74afa P eed2fd17595345ef928fff404160a150 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3e0974b73a504d5cbcbab1aa6593966b" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 39269 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 23:27:05.527969 5316 consensus_queue.cc:1035] T 50314a53e23a42eeaa12b4babbc74afa P eed2fd17595345ef928fff404160a150 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7642a2c0fa914384bfe0d910f91b06c2" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 43223 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 23:27:05.531388 5096 raft_consensus.cc:2932] T 50314a53e23a42eeaa12b4babbc74afa P 7642a2c0fa914384bfe0d910f91b06c2 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 3beb612d5d214657a5a40830f5226b97 (127.0.100.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3e0974b73a504d5cbcbab1aa6593966b" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 39269 } } peers { permanent_uuid: "7642a2c0fa914384bfe0d910f91b06c2" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 43223 } attrs { replace: true } } peers { permanent_uuid: "eed2fd17595345ef928fff404160a150" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 36927 } } peers { permanent_uuid: "3beb612d5d214657a5a40830f5226b97" member_type: NON_VOTER last_known_addr { host: "127.0.100.4" port: 34613 } attrs { promote: true } } }
I20240503 23:27:05.531714 5245 raft_consensus.cc:2932] T 50314a53e23a42eeaa12b4babbc74afa P 3e0974b73a504d5cbcbab1aa6593966b [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 3beb612d5d214657a5a40830f5226b97 (127.0.100.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3e0974b73a504d5cbcbab1aa6593966b" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 39269 } } peers { permanent_uuid: "7642a2c0fa914384bfe0d910f91b06c2" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 43223 } attrs { replace: true } } peers { permanent_uuid: "eed2fd17595345ef928fff404160a150" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 36927 } } peers { permanent_uuid: "3beb612d5d214657a5a40830f5226b97" member_type: NON_VOTER last_known_addr { host: "127.0.100.4" port: 34613 } attrs { promote: true } } }
W20240503 23:27:05.534273 5134 consensus_peers.cc:485] T 50314a53e23a42eeaa12b4babbc74afa P eed2fd17595345ef928fff404160a150 -> Peer 3beb612d5d214657a5a40830f5226b97 (127.0.100.4:34613): Couldn't send request to peer 3beb612d5d214657a5a40830f5226b97. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 50314a53e23a42eeaa12b4babbc74afa. This is attempt 1: this message will repeat every 5th retry.
I20240503 23:27:05.538970 4994 catalog_manager.cc:5484] T 50314a53e23a42eeaa12b4babbc74afa P 7642a2c0fa914384bfe0d910f91b06c2 reported cstate change: config changed from index -1 to 2, NON_VOTER 3beb612d5d214657a5a40830f5226b97 (127.0.100.4) added. New cstate: current_term: 1 leader_uuid: "eed2fd17595345ef928fff404160a150" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3e0974b73a504d5cbcbab1aa6593966b" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 39269 } } peers { permanent_uuid: "7642a2c0fa914384bfe0d910f91b06c2" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 43223 } attrs { replace: true } } peers { permanent_uuid: "eed2fd17595345ef928fff404160a150" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 36927 } } peers { permanent_uuid: "3beb612d5d214657a5a40830f5226b97" member_type: NON_VOTER last_known_addr { host: "127.0.100.4" port: 34613 } attrs { promote: true } } }
I20240503 23:27:05.588608 5415 ts_tablet_manager.cc:902] T c38e225db7cf4e11bd045f8e8127d79e P 3beb612d5d214657a5a40830f5226b97: Initiating tablet copy from peer 7642a2c0fa914384bfe0d910f91b06c2 (127.0.100.1:43223)
I20240503 23:27:05.589262 5415 tablet_copy_client.cc:316] T c38e225db7cf4e11bd045f8e8127d79e P 3beb612d5d214657a5a40830f5226b97: tablet copy: Beginning tablet copy session from remote peer at address 127.0.100.1:43223
I20240503 23:27:05.590685 5303 raft_consensus.cc:2932] T 50314a53e23a42eeaa12b4babbc74afa P eed2fd17595345ef928fff404160a150 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 3beb612d5d214657a5a40830f5226b97 (127.0.100.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3e0974b73a504d5cbcbab1aa6593966b" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 39269 } } peers { permanent_uuid: "7642a2c0fa914384bfe0d910f91b06c2" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 43223 } attrs { replace: true } } peers { permanent_uuid: "eed2fd17595345ef928fff404160a150" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 36927 } } peers { permanent_uuid: "3beb612d5d214657a5a40830f5226b97" member_type: NON_VOTER last_known_addr { host: "127.0.100.4" port: 34613 } attrs { promote: true } } }
I20240503 23:27:05.595712 5106 tablet_copy_service.cc:140] P 7642a2c0fa914384bfe0d910f91b06c2: Received BeginTabletCopySession request for tablet c38e225db7cf4e11bd045f8e8127d79e from peer 3beb612d5d214657a5a40830f5226b97 ({username='slave'} at 127.0.0.1:52782)
I20240503 23:27:05.595911 5106 tablet_copy_service.cc:161] P 7642a2c0fa914384bfe0d910f91b06c2: Beginning new tablet copy session on tablet c38e225db7cf4e11bd045f8e8127d79e from peer 3beb612d5d214657a5a40830f5226b97 at {username='slave'} at 127.0.0.1:52782: session id = 3beb612d5d214657a5a40830f5226b97-c38e225db7cf4e11bd045f8e8127d79e
I20240503 23:27:05.598922 5106 tablet_copy_source_session.cc:205] T c38e225db7cf4e11bd045f8e8127d79e P 7642a2c0fa914384bfe0d910f91b06c2: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 23:27:05.600291 5415 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c38e225db7cf4e11bd045f8e8127d79e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 23:27:05.605980 5415 tablet_copy_client.cc:791] T c38e225db7cf4e11bd045f8e8127d79e P 3beb612d5d214657a5a40830f5226b97: tablet copy: Starting download of 0 data blocks...
I20240503 23:27:05.606245 5415 tablet_copy_client.cc:655] T c38e225db7cf4e11bd045f8e8127d79e P 3beb612d5d214657a5a40830f5226b97: tablet copy: Starting download of 1 WAL segments...
I20240503 23:27:05.607889 5415 tablet_copy_client.cc:531] T c38e225db7cf4e11bd045f8e8127d79e P 3beb612d5d214657a5a40830f5226b97: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 23:27:05.611464 5415 tablet_bootstrap.cc:492] T c38e225db7cf4e11bd045f8e8127d79e P 3beb612d5d214657a5a40830f5226b97: Bootstrap starting.
I20240503 23:27:05.618404 5415 tablet_bootstrap.cc:492] T c38e225db7cf4e11bd045f8e8127d79e P 3beb612d5d214657a5a40830f5226b97: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 23:27:05.618891 5415 tablet_bootstrap.cc:492] T c38e225db7cf4e11bd045f8e8127d79e P 3beb612d5d214657a5a40830f5226b97: Bootstrap complete.
I20240503 23:27:05.619184 5415 ts_tablet_manager.cc:1367] T c38e225db7cf4e11bd045f8e8127d79e P 3beb612d5d214657a5a40830f5226b97: Time spent bootstrapping tablet: real 0.008s user 0.007s sys 0.000s
I20240503 23:27:05.619995 5415 raft_consensus.cc:348] T c38e225db7cf4e11bd045f8e8127d79e P 3beb612d5d214657a5a40830f5226b97 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "7642a2c0fa914384bfe0d910f91b06c2" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 43223 } } peers { permanent_uuid: "eed2fd17595345ef928fff404160a150" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 36927 } attrs { replace: true } } peers { permanent_uuid: "3e0974b73a504d5cbcbab1aa6593966b" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 39269 } } peers { permanent_uuid: "3beb612d5d214657a5a40830f5226b97" member_type: NON_VOTER last_known_addr { host: "127.0.100.4" port: 34613 } attrs { promote: true } }
I20240503 23:27:05.620280 5415 raft_consensus.cc:729] T c38e225db7cf4e11bd045f8e8127d79e P 3beb612d5d214657a5a40830f5226b97 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 3beb612d5d214657a5a40830f5226b97, State: Initialized, Role: LEARNER
I20240503 23:27:05.620534 5415 consensus_queue.cc:260] T c38e225db7cf4e11bd045f8e8127d79e P 3beb612d5d214657a5a40830f5226b97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "7642a2c0fa914384bfe0d910f91b06c2" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 43223 } } peers { permanent_uuid: "eed2fd17595345ef928fff404160a150" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 36927 } attrs { replace: true } } peers { permanent_uuid: "3e0974b73a504d5cbcbab1aa6593966b" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 39269 } } peers { permanent_uuid: "3beb612d5d214657a5a40830f5226b97" member_type: NON_VOTER last_known_addr { host: "127.0.100.4" port: 34613 } attrs { promote: true } }
I20240503 23:27:05.621632 5401 heartbeater.cc:503] Master 127.0.100.62:39883 was elected leader, sending a full tablet report...
I20240503 23:27:05.621639 5415 ts_tablet_manager.cc:1398] T c38e225db7cf4e11bd045f8e8127d79e P 3beb612d5d214657a5a40830f5226b97: Time spent starting tablet: real 0.002s user 0.005s sys 0.000s
I20240503 23:27:05.622568 5106 tablet_copy_service.cc:342] P 7642a2c0fa914384bfe0d910f91b06c2: Request end of tablet copy session 3beb612d5d214657a5a40830f5226b97-c38e225db7cf4e11bd045f8e8127d79e received from {username='slave'} at 127.0.0.1:52782
I20240503 23:27:05.622674 5106 tablet_copy_service.cc:433] P 7642a2c0fa914384bfe0d910f91b06c2: ending tablet copy session 3beb612d5d214657a5a40830f5226b97-c38e225db7cf4e11bd045f8e8127d79e on tablet c38e225db7cf4e11bd045f8e8127d79e with peer 3beb612d5d214657a5a40830f5226b97
I20240503 23:27:05.624164 5415 ts_tablet_manager.cc:902] T 50314a53e23a42eeaa12b4babbc74afa P 3beb612d5d214657a5a40830f5226b97: Initiating tablet copy from peer eed2fd17595345ef928fff404160a150 (127.0.100.2:36927)
I20240503 23:27:05.625643 5415 tablet_copy_client.cc:316] T 50314a53e23a42eeaa12b4babbc74afa P 3beb612d5d214657a5a40830f5226b97: tablet copy: Beginning tablet copy session from remote peer at address 127.0.100.2:36927
I20240503 23:27:05.625864 5309 raft_consensus.cc:979] T 3418f665dd5b4131add5ea54d2c2eec5 P 7642a2c0fa914384bfe0d910f91b06c2: : Instructing follower 3e0974b73a504d5cbcbab1aa6593966b to start an election
I20240503 23:27:05.626048 5309 raft_consensus.cc:1067] T 3418f665dd5b4131add5ea54d2c2eec5 P 7642a2c0fa914384bfe0d910f91b06c2 [term 1 LEADER]: Signalling peer 3e0974b73a504d5cbcbab1aa6593966b to start an election
I20240503 23:27:05.626648 5245 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "3418f665dd5b4131add5ea54d2c2eec5"
dest_uuid: "3e0974b73a504d5cbcbab1aa6593966b"
from {username='slave'} at 127.0.0.1:36566
I20240503 23:27:05.626856 5245 raft_consensus.cc:482] T 3418f665dd5b4131add5ea54d2c2eec5 P 3e0974b73a504d5cbcbab1aa6593966b [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 23:27:05.626950 5245 raft_consensus.cc:3037] T 3418f665dd5b4131add5ea54d2c2eec5 P 3e0974b73a504d5cbcbab1aa6593966b [term 1 FOLLOWER]: Advancing to term 2
I20240503 23:27:05.630393 5245 raft_consensus.cc:504] T 3418f665dd5b4131add5ea54d2c2eec5 P 3e0974b73a504d5cbcbab1aa6593966b [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7642a2c0fa914384bfe0d910f91b06c2" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 43223 } } peers { permanent_uuid: "eed2fd17595345ef928fff404160a150" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 36927 } } peers { permanent_uuid: "3e0974b73a504d5cbcbab1aa6593966b" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 39269 } }
I20240503 23:27:05.631335 5245 leader_election.cc:288] T 3418f665dd5b4131add5ea54d2c2eec5 P 3e0974b73a504d5cbcbab1aa6593966b [CANDIDATE]: Term 2 election: Requested vote from peers 7642a2c0fa914384bfe0d910f91b06c2 (127.0.100.1:43223), eed2fd17595345ef928fff404160a150 (127.0.100.2:36927)
I20240503 23:27:05.631906 5169 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3418f665dd5b4131add5ea54d2c2eec5" candidate_uuid: "3e0974b73a504d5cbcbab1aa6593966b" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "eed2fd17595345ef928fff404160a150"
I20240503 23:27:05.631971 5095 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3418f665dd5b4131add5ea54d2c2eec5" candidate_uuid: "3e0974b73a504d5cbcbab1aa6593966b" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "7642a2c0fa914384bfe0d910f91b06c2"
I20240503 23:27:05.632130 5169 raft_consensus.cc:3037] T 3418f665dd5b4131add5ea54d2c2eec5 P eed2fd17595345ef928fff404160a150 [term 1 FOLLOWER]: Advancing to term 2
I20240503 23:27:05.632138 5095 raft_consensus.cc:3032] T 3418f665dd5b4131add5ea54d2c2eec5 P 7642a2c0fa914384bfe0d910f91b06c2 [term 1 LEADER]: Stepping down as leader of term 1
I20240503 23:27:05.632273 5095 raft_consensus.cc:729] T 3418f665dd5b4131add5ea54d2c2eec5 P 7642a2c0fa914384bfe0d910f91b06c2 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 7642a2c0fa914384bfe0d910f91b06c2, State: Running, Role: LEADER
I20240503 23:27:05.632606 5095 consensus_queue.cc:260] T 3418f665dd5b4131add5ea54d2c2eec5 P 7642a2c0fa914384bfe0d910f91b06c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7642a2c0fa914384bfe0d910f91b06c2" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 43223 } } peers { permanent_uuid: "eed2fd17595345ef928fff404160a150" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 36927 } } peers { permanent_uuid: "3e0974b73a504d5cbcbab1aa6593966b" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 39269 } }
I20240503 23:27:05.633003 5095 raft_consensus.cc:3037] T 3418f665dd5b4131add5ea54d2c2eec5 P 7642a2c0fa914384bfe0d910f91b06c2 [term 1 FOLLOWER]: Advancing to term 2
I20240503 23:27:05.633883 5181 tablet_copy_service.cc:140] P eed2fd17595345ef928fff404160a150: Received BeginTabletCopySession request for tablet 50314a53e23a42eeaa12b4babbc74afa from peer 3beb612d5d214657a5a40830f5226b97 ({username='slave'} at 127.0.0.1:33504)
I20240503 23:27:05.634054 5181 tablet_copy_service.cc:161] P eed2fd17595345ef928fff404160a150: Beginning new tablet copy session on tablet 50314a53e23a42eeaa12b4babbc74afa from peer 3beb612d5d214657a5a40830f5226b97 at {username='slave'} at 127.0.0.1:33504: session id = 3beb612d5d214657a5a40830f5226b97-50314a53e23a42eeaa12b4babbc74afa
I20240503 23:27:05.636451 5169 raft_consensus.cc:2454] T 3418f665dd5b4131add5ea54d2c2eec5 P eed2fd17595345ef928fff404160a150 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3e0974b73a504d5cbcbab1aa6593966b in term 2.
I20240503 23:27:05.636719 5095 raft_consensus.cc:2454] T 3418f665dd5b4131add5ea54d2c2eec5 P 7642a2c0fa914384bfe0d910f91b06c2 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3e0974b73a504d5cbcbab1aa6593966b in term 2.
I20240503 23:27:05.637195 5208 leader_election.cc:302] T 3418f665dd5b4131add5ea54d2c2eec5 P 3e0974b73a504d5cbcbab1aa6593966b [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3e0974b73a504d5cbcbab1aa6593966b, eed2fd17595345ef928fff404160a150; no voters:
I20240503 23:27:05.637533 5323 raft_consensus.cc:2781] T 3418f665dd5b4131add5ea54d2c2eec5 P 3e0974b73a504d5cbcbab1aa6593966b [term 2 FOLLOWER]: Leader election won for term 2
I20240503 23:27:05.638016 5323 raft_consensus.cc:686] T 3418f665dd5b4131add5ea54d2c2eec5 P 3e0974b73a504d5cbcbab1aa6593966b [term 2 LEADER]: Becoming Leader. State: Replica: 3e0974b73a504d5cbcbab1aa6593966b, State: Running, Role: LEADER
I20240503 23:27:05.638360 5181 tablet_copy_source_session.cc:205] T 50314a53e23a42eeaa12b4babbc74afa P eed2fd17595345ef928fff404160a150: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 23:27:05.638325 5323 consensus_queue.cc:237] T 3418f665dd5b4131add5ea54d2c2eec5 P 3e0974b73a504d5cbcbab1aa6593966b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7642a2c0fa914384bfe0d910f91b06c2" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 43223 } } peers { permanent_uuid: "eed2fd17595345ef928fff404160a150" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 36927 } } peers { permanent_uuid: "3e0974b73a504d5cbcbab1aa6593966b" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 39269 } }
I20240503 23:27:05.639945 5415 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 50314a53e23a42eeaa12b4babbc74afa. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 23:27:05.641255 4992 catalog_manager.cc:5484] T 3418f665dd5b4131add5ea54d2c2eec5 P 3e0974b73a504d5cbcbab1aa6593966b reported cstate change: term changed from 1 to 2, leader changed from 7642a2c0fa914384bfe0d910f91b06c2 (127.0.100.1) to 3e0974b73a504d5cbcbab1aa6593966b (127.0.100.3). New cstate: current_term: 2 leader_uuid: "3e0974b73a504d5cbcbab1aa6593966b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7642a2c0fa914384bfe0d910f91b06c2" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 43223 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "eed2fd17595345ef928fff404160a150" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 36927 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3e0974b73a504d5cbcbab1aa6593966b" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 39269 } health_report { overall_health: HEALTHY } } }
I20240503 23:27:05.646598 5415 tablet_copy_client.cc:791] T 50314a53e23a42eeaa12b4babbc74afa P 3beb612d5d214657a5a40830f5226b97: tablet copy: Starting download of 0 data blocks...
I20240503 23:27:05.646879 5415 tablet_copy_client.cc:655] T 50314a53e23a42eeaa12b4babbc74afa P 3beb612d5d214657a5a40830f5226b97: tablet copy: Starting download of 1 WAL segments...
I20240503 23:27:05.648782 5415 tablet_copy_client.cc:531] T 50314a53e23a42eeaa12b4babbc74afa P 3beb612d5d214657a5a40830f5226b97: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 23:27:05.652200 5415 tablet_bootstrap.cc:492] T 50314a53e23a42eeaa12b4babbc74afa P 3beb612d5d214657a5a40830f5226b97: Bootstrap starting.
I20240503 23:27:05.659003 5415 tablet_bootstrap.cc:492] T 50314a53e23a42eeaa12b4babbc74afa P 3beb612d5d214657a5a40830f5226b97: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 23:27:05.659442 5415 tablet_bootstrap.cc:492] T 50314a53e23a42eeaa12b4babbc74afa P 3beb612d5d214657a5a40830f5226b97: Bootstrap complete.
I20240503 23:27:05.659708 5415 ts_tablet_manager.cc:1367] T 50314a53e23a42eeaa12b4babbc74afa P 3beb612d5d214657a5a40830f5226b97: Time spent bootstrapping tablet: real 0.008s user 0.007s sys 0.000s
I20240503 23:27:05.660621 5415 raft_consensus.cc:348] T 50314a53e23a42eeaa12b4babbc74afa P 3beb612d5d214657a5a40830f5226b97 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3e0974b73a504d5cbcbab1aa6593966b" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 39269 } } peers { permanent_uuid: "7642a2c0fa914384bfe0d910f91b06c2" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 43223 } attrs { replace: true } } peers { permanent_uuid: "eed2fd17595345ef928fff404160a150" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 36927 } } peers { permanent_uuid: "3beb612d5d214657a5a40830f5226b97" member_type: NON_VOTER last_known_addr { host: "127.0.100.4" port: 34613 } attrs { promote: true } }
I20240503 23:27:05.660843 5415 raft_consensus.cc:729] T 50314a53e23a42eeaa12b4babbc74afa P 3beb612d5d214657a5a40830f5226b97 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 3beb612d5d214657a5a40830f5226b97, State: Initialized, Role: LEARNER
I20240503 23:27:05.661039 5415 consensus_queue.cc:260] T 50314a53e23a42eeaa12b4babbc74afa P 3beb612d5d214657a5a40830f5226b97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3e0974b73a504d5cbcbab1aa6593966b" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 39269 } } peers { permanent_uuid: "7642a2c0fa914384bfe0d910f91b06c2" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 43223 } attrs { replace: true } } peers { permanent_uuid: "eed2fd17595345ef928fff404160a150" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 36927 } } peers { permanent_uuid: "3beb612d5d214657a5a40830f5226b97" member_type: NON_VOTER last_known_addr { host: "127.0.100.4" port: 34613 } attrs { promote: true } }
I20240503 23:27:05.661643 5415 ts_tablet_manager.cc:1398] T 50314a53e23a42eeaa12b4babbc74afa P 3beb612d5d214657a5a40830f5226b97: Time spent starting tablet: real 0.002s user 0.004s sys 0.000s
I20240503 23:27:05.662292 5181 tablet_copy_service.cc:342] P eed2fd17595345ef928fff404160a150: Request end of tablet copy session 3beb612d5d214657a5a40830f5226b97-50314a53e23a42eeaa12b4babbc74afa received from {username='slave'} at 127.0.0.1:33504
I20240503 23:27:05.662436 5181 tablet_copy_service.cc:433] P eed2fd17595345ef928fff404160a150: ending tablet copy session 3beb612d5d214657a5a40830f5226b97-50314a53e23a42eeaa12b4babbc74afa on tablet 50314a53e23a42eeaa12b4babbc74afa with peer 3beb612d5d214657a5a40830f5226b97
I20240503 23:27:05.685344 5376 raft_consensus.cc:1203] T c38e225db7cf4e11bd045f8e8127d79e P 3beb612d5d214657a5a40830f5226b97 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240503 23:27:05.717662 5095 raft_consensus.cc:1261] T 3418f665dd5b4131add5ea54d2c2eec5 P 7642a2c0fa914384bfe0d910f91b06c2 [term 2 FOLLOWER]: Refusing update from remote peer 3e0974b73a504d5cbcbab1aa6593966b: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240503 23:27:05.718271 5323 consensus_queue.cc:1035] T 3418f665dd5b4131add5ea54d2c2eec5 P 3e0974b73a504d5cbcbab1aa6593966b [LEADER]: Connected to new peer: Peer: permanent_uuid: "7642a2c0fa914384bfe0d910f91b06c2" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 43223 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 23:27:05.729754 5376 raft_consensus.cc:1203] T 50314a53e23a42eeaa12b4babbc74afa P 3beb612d5d214657a5a40830f5226b97 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240503 23:27:05.743443 5171 raft_consensus.cc:1261] T 3418f665dd5b4131add5ea54d2c2eec5 P eed2fd17595345ef928fff404160a150 [term 2 FOLLOWER]: Refusing update from remote peer 3e0974b73a504d5cbcbab1aa6593966b: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240503 23:27:05.744052 5323 consensus_queue.cc:1035] T 3418f665dd5b4131add5ea54d2c2eec5 P 3e0974b73a504d5cbcbab1aa6593966b [LEADER]: Connected to new peer: Peer: permanent_uuid: "eed2fd17595345ef928fff404160a150" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 36927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 23:27:05.771471 5309 raft_consensus.cc:1050] T c38e225db7cf4e11bd045f8e8127d79e P 7642a2c0fa914384bfe0d910f91b06c2: attempting to promote NON_VOTER 3beb612d5d214657a5a40830f5226b97 to VOTER
I20240503 23:27:05.772461 5309 consensus_queue.cc:237] T c38e225db7cf4e11bd045f8e8127d79e P 7642a2c0fa914384bfe0d910f91b06c2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "7642a2c0fa914384bfe0d910f91b06c2" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 43223 } } peers { permanent_uuid: "eed2fd17595345ef928fff404160a150" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 36927 } attrs { replace: true } } peers { permanent_uuid: "3e0974b73a504d5cbcbab1aa6593966b" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 39269 } } peers { permanent_uuid: "3beb612d5d214657a5a40830f5226b97" member_type: VOTER last_known_addr { host: "127.0.100.4" port: 34613 } attrs { promote: false } }
I20240503 23:27:05.774852 5376 raft_consensus.cc:1261] T c38e225db7cf4e11bd045f8e8127d79e P 3beb612d5d214657a5a40830f5226b97 [term 1 LEARNER]: Refusing update from remote peer 7642a2c0fa914384bfe0d910f91b06c2: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 23:27:05.774852 5245 raft_consensus.cc:1261] T c38e225db7cf4e11bd045f8e8127d79e P 3e0974b73a504d5cbcbab1aa6593966b [term 1 FOLLOWER]: Refusing update from remote peer 7642a2c0fa914384bfe0d910f91b06c2: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 23:27:05.774904 5169 raft_consensus.cc:1261] T c38e225db7cf4e11bd045f8e8127d79e P eed2fd17595345ef928fff404160a150 [term 1 FOLLOWER]: Refusing update from remote peer 7642a2c0fa914384bfe0d910f91b06c2: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 23:27:05.775449 5309 consensus_queue.cc:1035] T c38e225db7cf4e11bd045f8e8127d79e P 7642a2c0fa914384bfe0d910f91b06c2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3e0974b73a504d5cbcbab1aa6593966b" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 39269 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 23:27:05.775874 5296 consensus_queue.cc:1035] T c38e225db7cf4e11bd045f8e8127d79e P 7642a2c0fa914384bfe0d910f91b06c2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3beb612d5d214657a5a40830f5226b97" member_type: VOTER last_known_addr { host: "127.0.100.4" port: 34613 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 23:27:05.776651 5296 consensus_queue.cc:1035] T c38e225db7cf4e11bd045f8e8127d79e P 7642a2c0fa914384bfe0d910f91b06c2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eed2fd17595345ef928fff404160a150" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 36927 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 23:27:05.780031 5171 raft_consensus.cc:2932] T c38e225db7cf4e11bd045f8e8127d79e P eed2fd17595345ef928fff404160a150 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 3beb612d5d214657a5a40830f5226b97 (127.0.100.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "7642a2c0fa914384bfe0d910f91b06c2" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 43223 } } peers { permanent_uuid: "eed2fd17595345ef928fff404160a150" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 36927 } attrs { replace: true } } peers { permanent_uuid: "3e0974b73a504d5cbcbab1aa6593966b" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 39269 } } peers { permanent_uuid: "3beb612d5d214657a5a40830f5226b97" member_type: VOTER last_known_addr { host: "127.0.100.4" port: 34613 } attrs { promote: false } } }
I20240503 23:27:05.784087 5376 raft_consensus.cc:2932] T c38e225db7cf4e11bd045f8e8127d79e P 3beb612d5d214657a5a40830f5226b97 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 3beb612d5d214657a5a40830f5226b97 (127.0.100.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "7642a2c0fa914384bfe0d910f91b06c2" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 43223 } } peers { permanent_uuid: "eed2fd17595345ef928fff404160a150" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 36927 } attrs { replace: true } } peers { permanent_uuid: "3e0974b73a504d5cbcbab1aa6593966b" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 39269 } } peers { permanent_uuid: "3beb612d5d214657a5a40830f5226b97" member_type: VOTER last_known_addr { host: "127.0.100.4" port: 34613 } attrs { promote: false } } }
I20240503 23:27:05.785116 4992 catalog_manager.cc:5484] T c38e225db7cf4e11bd045f8e8127d79e P eed2fd17595345ef928fff404160a150 reported cstate change: config changed from index 2 to 3, 3beb612d5d214657a5a40830f5226b97 (127.0.100.4) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "7642a2c0fa914384bfe0d910f91b06c2" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "7642a2c0fa914384bfe0d910f91b06c2" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 43223 } } peers { permanent_uuid: "eed2fd17595345ef928fff404160a150" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 36927 } attrs { replace: true } } peers { permanent_uuid: "3e0974b73a504d5cbcbab1aa6593966b" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 39269 } } peers { permanent_uuid: "3beb612d5d214657a5a40830f5226b97" member_type: VOTER last_known_addr { host: "127.0.100.4" port: 34613 } attrs { promote: false } } }
I20240503 23:27:05.786250 5309 raft_consensus.cc:2932] T c38e225db7cf4e11bd045f8e8127d79e P 7642a2c0fa914384bfe0d910f91b06c2 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 3beb612d5d214657a5a40830f5226b97 (127.0.100.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "7642a2c0fa914384bfe0d910f91b06c2" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 43223 } } peers { permanent_uuid: "eed2fd17595345ef928fff404160a150" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 36927 } attrs { replace: true } } peers { permanent_uuid: "3e0974b73a504d5cbcbab1aa6593966b" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 39269 } } peers { permanent_uuid: "3beb612d5d214657a5a40830f5226b97" member_type: VOTER last_known_addr { host: "127.0.100.4" port: 34613 } attrs { promote: false } } }
I20240503 23:27:05.791754 5245 raft_consensus.cc:2932] T c38e225db7cf4e11bd045f8e8127d79e P 3e0974b73a504d5cbcbab1aa6593966b [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 3beb612d5d214657a5a40830f5226b97 (127.0.100.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "7642a2c0fa914384bfe0d910f91b06c2" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 43223 } } peers { permanent_uuid: "eed2fd17595345ef928fff404160a150" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 36927 } attrs { replace: true } } peers { permanent_uuid: "3e0974b73a504d5cbcbab1aa6593966b" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 39269 } } peers { permanent_uuid: "3beb612d5d214657a5a40830f5226b97" member_type: VOTER last_known_addr { host: "127.0.100.4" port: 34613 } attrs { promote: false } } }
I20240503 23:27:05.876233 5095 consensus_queue.cc:237] T c38e225db7cf4e11bd045f8e8127d79e P 7642a2c0fa914384bfe0d910f91b06c2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7642a2c0fa914384bfe0d910f91b06c2" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 43223 } } peers { permanent_uuid: "3e0974b73a504d5cbcbab1aa6593966b" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 39269 } } peers { permanent_uuid: "3beb612d5d214657a5a40830f5226b97" member_type: VOTER last_known_addr { host: "127.0.100.4" port: 34613 } attrs { promote: false } }
I20240503 23:27:05.878341 5376 raft_consensus.cc:1261] T c38e225db7cf4e11bd045f8e8127d79e P 3beb612d5d214657a5a40830f5226b97 [term 1 FOLLOWER]: Refusing update from remote peer 7642a2c0fa914384bfe0d910f91b06c2: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 23:27:05.878926 5296 consensus_queue.cc:1035] T c38e225db7cf4e11bd045f8e8127d79e P 7642a2c0fa914384bfe0d910f91b06c2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3beb612d5d214657a5a40830f5226b97" member_type: VOTER last_known_addr { host: "127.0.100.4" port: 34613 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 23:27:05.879041 5245 raft_consensus.cc:1261] T c38e225db7cf4e11bd045f8e8127d79e P 3e0974b73a504d5cbcbab1aa6593966b [term 1 FOLLOWER]: Refusing update from remote peer 7642a2c0fa914384bfe0d910f91b06c2: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 23:27:05.879562 5296 consensus_queue.cc:1035] T c38e225db7cf4e11bd045f8e8127d79e P 7642a2c0fa914384bfe0d910f91b06c2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3e0974b73a504d5cbcbab1aa6593966b" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 39269 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 23:27:05.882270 5376 raft_consensus.cc:2932] T c38e225db7cf4e11bd045f8e8127d79e P 3beb612d5d214657a5a40830f5226b97 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER eed2fd17595345ef928fff404160a150 (127.0.100.2) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7642a2c0fa914384bfe0d910f91b06c2" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 43223 } } peers { permanent_uuid: "3e0974b73a504d5cbcbab1aa6593966b" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 39269 } } peers { permanent_uuid: "3beb612d5d214657a5a40830f5226b97" member_type: VOTER last_known_addr { host: "127.0.100.4" port: 34613 } attrs { promote: false } } }
I20240503 23:27:05.882759 5245 raft_consensus.cc:2932] T c38e225db7cf4e11bd045f8e8127d79e P 3e0974b73a504d5cbcbab1aa6593966b [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER eed2fd17595345ef928fff404160a150 (127.0.100.2) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7642a2c0fa914384bfe0d910f91b06c2" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 43223 } } peers { permanent_uuid: "3e0974b73a504d5cbcbab1aa6593966b" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 39269 } } peers { permanent_uuid: "3beb612d5d214657a5a40830f5226b97" member_type: VOTER last_known_addr { host: "127.0.100.4" port: 34613 } attrs { promote: false } } }
I20240503 23:27:05.887647 4994 catalog_manager.cc:5484] T c38e225db7cf4e11bd045f8e8127d79e P 3beb612d5d214657a5a40830f5226b97 reported cstate change: config changed from index 3 to 4, VOTER eed2fd17595345ef928fff404160a150 (127.0.100.2) evicted. New cstate: current_term: 1 leader_uuid: "7642a2c0fa914384bfe0d910f91b06c2" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7642a2c0fa914384bfe0d910f91b06c2" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 43223 } } peers { permanent_uuid: "3e0974b73a504d5cbcbab1aa6593966b" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 39269 } } peers { permanent_uuid: "3beb612d5d214657a5a40830f5226b97" member_type: VOTER last_known_addr { host: "127.0.100.4" port: 34613 } attrs { promote: false } } }
I20240503 23:27:05.908887 5159 tablet_service.cc:1514] Processing DeleteTablet for tablet c38e225db7cf4e11bd045f8e8127d79e with delete_type TABLET_DATA_TOMBSTONED (TS eed2fd17595345ef928fff404160a150 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:33486
I20240503 23:27:05.909703 5428 tablet_replica.cc:330] T c38e225db7cf4e11bd045f8e8127d79e P eed2fd17595345ef928fff404160a150: stopping tablet replica
I20240503 23:27:05.909910 5428 raft_consensus.cc:2229] T c38e225db7cf4e11bd045f8e8127d79e P eed2fd17595345ef928fff404160a150 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 23:27:05.910244 5428 raft_consensus.cc:2258] T c38e225db7cf4e11bd045f8e8127d79e P eed2fd17595345ef928fff404160a150 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 23:27:05.911281 5428 ts_tablet_manager.cc:1875] T c38e225db7cf4e11bd045f8e8127d79e P eed2fd17595345ef928fff404160a150: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 23:27:05.913762 5316 raft_consensus.cc:1050] T 50314a53e23a42eeaa12b4babbc74afa P eed2fd17595345ef928fff404160a150: attempting to promote NON_VOTER 3beb612d5d214657a5a40830f5226b97 to VOTER
I20240503 23:27:05.914803 5316 consensus_queue.cc:237] T 50314a53e23a42eeaa12b4babbc74afa P eed2fd17595345ef928fff404160a150 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3e0974b73a504d5cbcbab1aa6593966b" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 39269 } } peers { permanent_uuid: "7642a2c0fa914384bfe0d910f91b06c2" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 43223 } attrs { replace: true } } peers { permanent_uuid: "eed2fd17595345ef928fff404160a150" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 36927 } } peers { permanent_uuid: "3beb612d5d214657a5a40830f5226b97" member_type: VOTER last_known_addr { host: "127.0.100.4" port: 34613 } attrs { promote: false } }
I20240503 23:27:05.917659 5376 raft_consensus.cc:1261] T 50314a53e23a42eeaa12b4babbc74afa P 3beb612d5d214657a5a40830f5226b97 [term 1 LEARNER]: Refusing update from remote peer eed2fd17595345ef928fff404160a150: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 23:27:05.918375 5344 consensus_queue.cc:1035] T 50314a53e23a42eeaa12b4babbc74afa P eed2fd17595345ef928fff404160a150 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3beb612d5d214657a5a40830f5226b97" member_type: VOTER last_known_addr { host: "127.0.100.4" port: 34613 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 23:27:05.919373 5428 ts_tablet_manager.cc:1888] T c38e225db7cf4e11bd045f8e8127d79e P eed2fd17595345ef928fff404160a150: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240503 23:27:05.919557 5428 log.cc:1196] T c38e225db7cf4e11bd045f8e8127d79e P eed2fd17595345ef928fff404160a150: Deleting WAL directory at /tmp/dist-test-taskZhIiPz/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingUnstableCluster.1714778688451250-400-0/minicluster-data/ts-1-root/wals/c38e225db7cf4e11bd045f8e8127d79e
I20240503 23:27:05.919379 5245 raft_consensus.cc:1261] T 50314a53e23a42eeaa12b4babbc74afa P 3e0974b73a504d5cbcbab1aa6593966b [term 1 FOLLOWER]: Refusing update from remote peer eed2fd17595345ef928fff404160a150: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 23:27:05.920229 5095 raft_consensus.cc:1261] T 50314a53e23a42eeaa12b4babbc74afa P 7642a2c0fa914384bfe0d910f91b06c2 [term 1 FOLLOWER]: Refusing update from remote peer eed2fd17595345ef928fff404160a150: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 23:27:05.920804 5344 consensus_queue.cc:1035] T 50314a53e23a42eeaa12b4babbc74afa P eed2fd17595345ef928fff404160a150 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7642a2c0fa914384bfe0d910f91b06c2" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 43223 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 23:27:05.922026 4978 catalog_manager.cc:4830] TS eed2fd17595345ef928fff404160a150 (127.0.100.2:36927): tablet c38e225db7cf4e11bd045f8e8127d79e (table test-workload [id=00fe2013bb2c478089860377060f6f88]) successfully deleted
I20240503 23:27:05.922300 5344 consensus_queue.cc:1035] T 50314a53e23a42eeaa12b4babbc74afa P eed2fd17595345ef928fff404160a150 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3e0974b73a504d5cbcbab1aa6593966b" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 39269 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 23:27:05.925457 5376 raft_consensus.cc:2932] T 50314a53e23a42eeaa12b4babbc74afa P 3beb612d5d214657a5a40830f5226b97 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 3beb612d5d214657a5a40830f5226b97 (127.0.100.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3e0974b73a504d5cbcbab1aa6593966b" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 39269 } } peers { permanent_uuid: "7642a2c0fa914384bfe0d910f91b06c2" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 43223 } attrs { replace: true } } peers { permanent_uuid: "eed2fd17595345ef928fff404160a150" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 36927 } } peers { permanent_uuid: "3beb612d5d214657a5a40830f5226b97" member_type: VOTER last_known_addr { host: "127.0.100.4" port: 34613 } attrs { promote: false } } }
I20240503 23:27:05.929284 5245 raft_consensus.cc:2932] T 50314a53e23a42eeaa12b4babbc74afa P 3e0974b73a504d5cbcbab1aa6593966b [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 3beb612d5d214657a5a40830f5226b97 (127.0.100.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3e0974b73a504d5cbcbab1aa6593966b" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 39269 } } peers { permanent_uuid: "7642a2c0fa914384bfe0d910f91b06c2" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 43223 } attrs { replace: true } } peers { permanent_uuid: "eed2fd17595345ef928fff404160a150" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 36927 } } peers { permanent_uuid: "3beb612d5d214657a5a40830f5226b97" member_type: VOTER last_known_addr { host: "127.0.100.4" port: 34613 } attrs { promote: false } } }
I20240503 23:27:05.930286 4991 catalog_manager.cc:5484] T 50314a53e23a42eeaa12b4babbc74afa P 3beb612d5d214657a5a40830f5226b97 reported cstate change: config changed from index 2 to 3, 3beb612d5d214657a5a40830f5226b97 (127.0.100.4) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "eed2fd17595345ef928fff404160a150" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3e0974b73a504d5cbcbab1aa6593966b" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 39269 } } peers { permanent_uuid: "7642a2c0fa914384bfe0d910f91b06c2" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 43223 } attrs { replace: true } } peers { permanent_uuid: "eed2fd17595345ef928fff404160a150" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 36927 } } peers { permanent_uuid: "3beb612d5d214657a5a40830f5226b97" member_type: VOTER last_known_addr { host: "127.0.100.4" port: 34613 } attrs { promote: false } } }
I20240503 23:27:05.986395 5308 raft_consensus.cc:2932] T c38e225db7cf4e11bd045f8e8127d79e P 7642a2c0fa914384bfe0d910f91b06c2 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER eed2fd17595345ef928fff404160a150 (127.0.100.2) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7642a2c0fa914384bfe0d910f91b06c2" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 43223 } } peers { permanent_uuid: "3e0974b73a504d5cbcbab1aa6593966b" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 39269 } } peers { permanent_uuid: "3beb612d5d214657a5a40830f5226b97" member_type: VOTER last_known_addr { host: "127.0.100.4" port: 34613 } attrs { promote: false } } }
I20240503 23:27:05.990294 4981 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet c38e225db7cf4e11bd045f8e8127d79e with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 23:27:05.995028 5303 raft_consensus.cc:2932] T 50314a53e23a42eeaa12b4babbc74afa P eed2fd17595345ef928fff404160a150 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 3beb612d5d214657a5a40830f5226b97 (127.0.100.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3e0974b73a504d5cbcbab1aa6593966b" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 39269 } } peers { permanent_uuid: "7642a2c0fa914384bfe0d910f91b06c2" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 43223 } attrs { replace: true } } peers { permanent_uuid: "eed2fd17595345ef928fff404160a150" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 36927 } } peers { permanent_uuid: "3beb612d5d214657a5a40830f5226b97" member_type: VOTER last_known_addr { host: "127.0.100.4" port: 34613 } attrs { promote: false } } }
I20240503 23:27:06.020015 5096 raft_consensus.cc:2932] T 50314a53e23a42eeaa12b4babbc74afa P 7642a2c0fa914384bfe0d910f91b06c2 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 3beb612d5d214657a5a40830f5226b97 (127.0.100.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3e0974b73a504d5cbcbab1aa6593966b" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 39269 } } peers { permanent_uuid: "7642a2c0fa914384bfe0d910f91b06c2" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 43223 } attrs { replace: true } } peers { permanent_uuid: "eed2fd17595345ef928fff404160a150" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 36927 } } peers { permanent_uuid: "3beb612d5d214657a5a40830f5226b97" member_type: VOTER last_known_addr { host: "127.0.100.4" port: 34613 } attrs { promote: false } } }
I20240503 23:27:06.030659 5171 consensus_queue.cc:237] T 50314a53e23a42eeaa12b4babbc74afa P eed2fd17595345ef928fff404160a150 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3e0974b73a504d5cbcbab1aa6593966b" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 39269 } } peers { permanent_uuid: "eed2fd17595345ef928fff404160a150" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 36927 } } peers { permanent_uuid: "3beb612d5d214657a5a40830f5226b97" member_type: VOTER last_known_addr { host: "127.0.100.4" port: 34613 } attrs { promote: false } }
I20240503 23:27:06.032670 5376 raft_consensus.cc:1261] T 50314a53e23a42eeaa12b4babbc74afa P 3beb612d5d214657a5a40830f5226b97 [term 1 FOLLOWER]: Refusing update from remote peer eed2fd17595345ef928fff404160a150: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 23:27:06.032805 5245 raft_consensus.cc:1261] T 50314a53e23a42eeaa12b4babbc74afa P 3e0974b73a504d5cbcbab1aa6593966b [term 1 FOLLOWER]: Refusing update from remote peer eed2fd17595345ef928fff404160a150: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 23:27:06.033299 5316 consensus_queue.cc:1035] T 50314a53e23a42eeaa12b4babbc74afa P eed2fd17595345ef928fff404160a150 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3beb612d5d214657a5a40830f5226b97" member_type: VOTER last_known_addr { host: "127.0.100.4" port: 34613 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 23:27:06.033835 5344 consensus_queue.cc:1035] T 50314a53e23a42eeaa12b4babbc74afa P eed2fd17595345ef928fff404160a150 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3e0974b73a504d5cbcbab1aa6593966b" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 39269 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 23:27:06.036674 5245 raft_consensus.cc:2932] T 50314a53e23a42eeaa12b4babbc74afa P 3e0974b73a504d5cbcbab1aa6593966b [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 7642a2c0fa914384bfe0d910f91b06c2 (127.0.100.1) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3e0974b73a504d5cbcbab1aa6593966b" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 39269 } } peers { permanent_uuid: "eed2fd17595345ef928fff404160a150" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 36927 } } peers { permanent_uuid: "3beb612d5d214657a5a40830f5226b97" member_type: VOTER last_known_addr { host: "127.0.100.4" port: 34613 } attrs { promote: false } } }
I20240503 23:27:06.036572 5376 raft_consensus.cc:2932] T 50314a53e23a42eeaa12b4babbc74afa P 3beb612d5d214657a5a40830f5226b97 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 7642a2c0fa914384bfe0d910f91b06c2 (127.0.100.1) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3e0974b73a504d5cbcbab1aa6593966b" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 39269 } } peers { permanent_uuid: "eed2fd17595345ef928fff404160a150" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 36927 } } peers { permanent_uuid: "3beb612d5d214657a5a40830f5226b97" member_type: VOTER last_known_addr { host: "127.0.100.4" port: 34613 } attrs { promote: false } } }
I20240503 23:27:06.041375 4993 catalog_manager.cc:5484] T 50314a53e23a42eeaa12b4babbc74afa P 3beb612d5d214657a5a40830f5226b97 reported cstate change: config changed from index 3 to 4, VOTER 7642a2c0fa914384bfe0d910f91b06c2 (127.0.100.1) evicted. New cstate: current_term: 1 leader_uuid: "eed2fd17595345ef928fff404160a150" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3e0974b73a504d5cbcbab1aa6593966b" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 39269 } } peers { permanent_uuid: "eed2fd17595345ef928fff404160a150" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 36927 } } peers { permanent_uuid: "3beb612d5d214657a5a40830f5226b97" member_type: VOTER last_known_addr { host: "127.0.100.4" port: 34613 } attrs { promote: false } } }
I20240503 23:27:06.098851 5084 tablet_service.cc:1514] Processing DeleteTablet for tablet 50314a53e23a42eeaa12b4babbc74afa with delete_type TABLET_DATA_TOMBSTONED (TS 7642a2c0fa914384bfe0d910f91b06c2 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:52732
I20240503 23:27:06.099685 5431 tablet_replica.cc:330] T 50314a53e23a42eeaa12b4babbc74afa P 7642a2c0fa914384bfe0d910f91b06c2: stopping tablet replica
I20240503 23:27:06.099931 5431 raft_consensus.cc:2229] T 50314a53e23a42eeaa12b4babbc74afa P 7642a2c0fa914384bfe0d910f91b06c2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 23:27:06.100216 5431 raft_consensus.cc:2258] T 50314a53e23a42eeaa12b4babbc74afa P 7642a2c0fa914384bfe0d910f91b06c2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 23:27:06.101244 5431 ts_tablet_manager.cc:1875] T 50314a53e23a42eeaa12b4babbc74afa P 7642a2c0fa914384bfe0d910f91b06c2: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 23:27:06.106544 5431 ts_tablet_manager.cc:1888] T 50314a53e23a42eeaa12b4babbc74afa P 7642a2c0fa914384bfe0d910f91b06c2: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240503 23:27:06.106667 5431 log.cc:1196] T 50314a53e23a42eeaa12b4babbc74afa P 7642a2c0fa914384bfe0d910f91b06c2: Deleting WAL directory at /tmp/dist-test-taskZhIiPz/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingUnstableCluster.1714778688451250-400-0/minicluster-data/ts-0-root/wals/50314a53e23a42eeaa12b4babbc74afa
I20240503 23:27:06.107458 4981 catalog_manager.cc:4830] TS 7642a2c0fa914384bfe0d910f91b06c2 (127.0.100.1:43223): tablet 50314a53e23a42eeaa12b4babbc74afa (table test-workload [id=00fe2013bb2c478089860377060f6f88]) successfully deleted
I20240503 23:27:06.176072 5316 raft_consensus.cc:2932] T 50314a53e23a42eeaa12b4babbc74afa P eed2fd17595345ef928fff404160a150 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 7642a2c0fa914384bfe0d910f91b06c2 (127.0.100.1) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3e0974b73a504d5cbcbab1aa6593966b" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 39269 } } peers { permanent_uuid: "eed2fd17595345ef928fff404160a150" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 36927 } } peers { permanent_uuid: "3beb612d5d214657a5a40830f5226b97" member_type: VOTER last_known_addr { host: "127.0.100.4" port: 34613 } attrs { promote: false } } }
I20240503 23:27:06.179172 4978 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 50314a53e23a42eeaa12b4babbc74afa with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 23:27:07.464535 5047 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240503 23:27:07.465461 5047 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240503 23:27:07.465608 5047 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240503 23:27:09.466109 5047 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240503 23:27:09.466969 5047 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240503 23:27:09.467109 5047 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240503 23:27:11.467664 5047 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240503 23:27:11.468878 5047 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240503 23:27:11.469034 5047 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240503 23:27:13.469523 5047 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240503 23:27:13.470381 5047 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240503 23:27:13.470520 5047 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240503 23:27:13.660750 400 tablet_server.cc:178] TabletServer@127.0.100.1:0 shutting down...
I20240503 23:27:13.675990 400 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240503 23:27:13.676374 400 tablet_replica.cc:330] T c38e225db7cf4e11bd045f8e8127d79e P 7642a2c0fa914384bfe0d910f91b06c2: stopping tablet replica
I20240503 23:27:13.676606 400 raft_consensus.cc:2229] T c38e225db7cf4e11bd045f8e8127d79e P 7642a2c0fa914384bfe0d910f91b06c2 [term 1 LEADER]: Raft consensus shutting down.
I20240503 23:27:13.677000 400 raft_consensus.cc:2258] T c38e225db7cf4e11bd045f8e8127d79e P 7642a2c0fa914384bfe0d910f91b06c2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 23:27:13.677718 400 tablet_replica.cc:330] T 3418f665dd5b4131add5ea54d2c2eec5 P 7642a2c0fa914384bfe0d910f91b06c2: stopping tablet replica
I20240503 23:27:13.677853 400 raft_consensus.cc:2229] T 3418f665dd5b4131add5ea54d2c2eec5 P 7642a2c0fa914384bfe0d910f91b06c2 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 23:27:13.677990 400 raft_consensus.cc:2258] T 3418f665dd5b4131add5ea54d2c2eec5 P 7642a2c0fa914384bfe0d910f91b06c2 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 23:27:13.693687 400 tablet_server.cc:194] TabletServer@127.0.100.1:0 shutdown complete.
I20240503 23:27:13.697918 400 tablet_server.cc:178] TabletServer@127.0.100.2:0 shutting down...
I20240503 23:27:13.727236 400 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240503 23:27:13.727587 400 tablet_replica.cc:330] T 50314a53e23a42eeaa12b4babbc74afa P eed2fd17595345ef928fff404160a150: stopping tablet replica
I20240503 23:27:13.727813 400 raft_consensus.cc:2229] T 50314a53e23a42eeaa12b4babbc74afa P eed2fd17595345ef928fff404160a150 [term 1 LEADER]: Raft consensus shutting down.
I20240503 23:27:13.728180 400 raft_consensus.cc:2258] T 50314a53e23a42eeaa12b4babbc74afa P eed2fd17595345ef928fff404160a150 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 23:27:13.728920 400 tablet_replica.cc:330] T 3418f665dd5b4131add5ea54d2c2eec5 P eed2fd17595345ef928fff404160a150: stopping tablet replica
I20240503 23:27:13.729048 400 raft_consensus.cc:2229] T 3418f665dd5b4131add5ea54d2c2eec5 P eed2fd17595345ef928fff404160a150 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 23:27:13.729167 400 raft_consensus.cc:2258] T 3418f665dd5b4131add5ea54d2c2eec5 P eed2fd17595345ef928fff404160a150 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 23:27:13.744323 400 tablet_server.cc:194] TabletServer@127.0.100.2:0 shutdown complete.
W20240503 23:27:13.745426 5208 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.100.2:36927: connect: Connection refused (error 111) [suppressed 2 similar messages]
W20240503 23:27:13.746798 5208 consensus_peers.cc:485] T 3418f665dd5b4131add5ea54d2c2eec5 P 3e0974b73a504d5cbcbab1aa6593966b -> Peer eed2fd17595345ef928fff404160a150 (127.0.100.2:36927): Couldn't send request to peer eed2fd17595345ef928fff404160a150. Status: Network error: Client connection negotiation failed: client connection to 127.0.100.2:36927: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 23:27:13.748961 400 tablet_server.cc:178] TabletServer@127.0.100.3:0 shutting down...
I20240503 23:27:13.759622 400 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240503 23:27:13.759928 400 tablet_replica.cc:330] T 50314a53e23a42eeaa12b4babbc74afa P 3e0974b73a504d5cbcbab1aa6593966b: stopping tablet replica
W20240503 23:27:13.759963 5211 consensus_peers.cc:485] T 3418f665dd5b4131add5ea54d2c2eec5 P 3e0974b73a504d5cbcbab1aa6593966b -> Peer 7642a2c0fa914384bfe0d910f91b06c2 (127.0.100.1:43223): Couldn't send request to peer 7642a2c0fa914384bfe0d910f91b06c2. Status: Network error: Client connection negotiation failed: client connection to 127.0.100.1:43223: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 23:27:13.760185 400 raft_consensus.cc:2229] T 50314a53e23a42eeaa12b4babbc74afa P 3e0974b73a504d5cbcbab1aa6593966b [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 23:27:13.760468 400 raft_consensus.cc:2258] T 50314a53e23a42eeaa12b4babbc74afa P 3e0974b73a504d5cbcbab1aa6593966b [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 23:27:13.761207 400 tablet_replica.cc:330] T 3418f665dd5b4131add5ea54d2c2eec5 P 3e0974b73a504d5cbcbab1aa6593966b: stopping tablet replica
I20240503 23:27:13.761387 400 raft_consensus.cc:2229] T 3418f665dd5b4131add5ea54d2c2eec5 P 3e0974b73a504d5cbcbab1aa6593966b [term 2 LEADER]: Raft consensus shutting down.
I20240503 23:27:13.761704 400 raft_consensus.cc:2258] T 3418f665dd5b4131add5ea54d2c2eec5 P 3e0974b73a504d5cbcbab1aa6593966b [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 23:27:13.762280 400 tablet_replica.cc:330] T c38e225db7cf4e11bd045f8e8127d79e P 3e0974b73a504d5cbcbab1aa6593966b: stopping tablet replica
I20240503 23:27:13.762398 400 raft_consensus.cc:2229] T c38e225db7cf4e11bd045f8e8127d79e P 3e0974b73a504d5cbcbab1aa6593966b [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 23:27:13.762653 400 raft_consensus.cc:2258] T c38e225db7cf4e11bd045f8e8127d79e P 3e0974b73a504d5cbcbab1aa6593966b [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 23:27:13.777375 400 tablet_server.cc:194] TabletServer@127.0.100.3:0 shutdown complete.
I20240503 23:27:13.781700 400 tablet_server.cc:178] TabletServer@127.0.100.4:0 shutting down...
I20240503 23:27:13.790916 400 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240503 23:27:13.791260 400 tablet_replica.cc:330] T 50314a53e23a42eeaa12b4babbc74afa P 3beb612d5d214657a5a40830f5226b97: stopping tablet replica
I20240503 23:27:13.791491 400 raft_consensus.cc:2229] T 50314a53e23a42eeaa12b4babbc74afa P 3beb612d5d214657a5a40830f5226b97 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 23:27:13.791772 400 raft_consensus.cc:2258] T 50314a53e23a42eeaa12b4babbc74afa P 3beb612d5d214657a5a40830f5226b97 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 23:27:13.792503 400 tablet_replica.cc:330] T c38e225db7cf4e11bd045f8e8127d79e P 3beb612d5d214657a5a40830f5226b97: stopping tablet replica
I20240503 23:27:13.792665 400 raft_consensus.cc:2229] T c38e225db7cf4e11bd045f8e8127d79e P 3beb612d5d214657a5a40830f5226b97 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 23:27:13.792927 400 raft_consensus.cc:2258] T c38e225db7cf4e11bd045f8e8127d79e P 3beb612d5d214657a5a40830f5226b97 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 23:27:13.807392 400 tablet_server.cc:194] TabletServer@127.0.100.4:0 shutdown complete.
I20240503 23:27:13.811363 400 master.cc:537] Master@127.0.100.62:39883 shutting down...
I20240503 23:27:13.820829 400 raft_consensus.cc:2229] T 00000000000000000000000000000000 P d2c8d94c8b904af0b5807cc170c1ee96 [term 1 LEADER]: Raft consensus shutting down.
I20240503 23:27:13.821161 400 raft_consensus.cc:2258] T 00000000000000000000000000000000 P d2c8d94c8b904af0b5807cc170c1ee96 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 23:27:13.821269 400 tablet_replica.cc:330] T 00000000000000000000000000000000 P d2c8d94c8b904af0b5807cc170c1ee96: stopping tablet replica
I20240503 23:27:13.835393 400 master.cc:552] Master@127.0.100.62:39883 shutdown complete.
[ OK ] AutoRebalancerTest.AutoRebalancingUnstableCluster (10475 ms)
[ RUN ] AutoRebalancerTest.NoReplicaMovesIfCannotMeetReplicationFactor
I20240503 23:27:13.852146 400 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.0.100.62:37225
I20240503 23:27:13.853017 400 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 23:27:13.856807 5439 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 23:27:13.857683 5440 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 23:27:13.858570 400 server_base.cc:1034] running on GCE node
W20240503 23:27:13.859457 5442 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 23:27:13.859848 400 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240503 23:27:13.859925 400 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240503 23:27:13.859983 400 hybrid_clock.cc:648] HybridClock initialized: now 1714778833859977 us; error 0 us; skew 500 ppm
I20240503 23:27:13.860177 400 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 23:27:13.866449 400 webserver.cc:460] Webserver started at http://127.0.100.62:45743/ using document root <none> and password file <none>
I20240503 23:27:13.866763 400 fs_manager.cc:359] Metadata directory not provided
I20240503 23:27:13.866844 400 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 23:27:13.867004 400 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 23:27:13.867825 400 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZhIiPz/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotMeetReplicationFactor.1714778688451250-400-0/minicluster-data/master-0-root/instance:
uuid: "39da52ddfea94a2c9fa8c5f90c2b0bbc"
format_stamp: "Formatted at 2024-05-03 23:27:13 on dist-test-slave-jvbp"
I20240503 23:27:13.870843 400 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.004s
I20240503 23:27:13.872874 5447 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 23:27:13.873306 400 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.001s
I20240503 23:27:13.873414 400 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZhIiPz/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotMeetReplicationFactor.1714778688451250-400-0/minicluster-data/master-0-root
uuid: "39da52ddfea94a2c9fa8c5f90c2b0bbc"
format_stamp: "Formatted at 2024-05-03 23:27:13 on dist-test-slave-jvbp"
I20240503 23:27:13.873546 400 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZhIiPz/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotMeetReplicationFactor.1714778688451250-400-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskZhIiPz/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotMeetReplicationFactor.1714778688451250-400-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskZhIiPz/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotMeetReplicationFactor.1714778688451250-400-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 23:27:13.902900 400 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 23:27:13.903582 400 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 23:27:13.917112 400 rpc_server.cc:307] RPC server started. Bound to: 127.0.100.62:37225
I20240503 23:27:13.917117 5498 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.100.62:37225 every 8 connection(s)
I20240503 23:27:13.918933 5499 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 23:27:13.923789 5499 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 39da52ddfea94a2c9fa8c5f90c2b0bbc: Bootstrap starting.
I20240503 23:27:13.925407 5499 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 39da52ddfea94a2c9fa8c5f90c2b0bbc: Neither blocks nor log segments found. Creating new log.
I20240503 23:27:13.927325 5499 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 39da52ddfea94a2c9fa8c5f90c2b0bbc: No bootstrap required, opened a new log
I20240503 23:27:13.928056 5499 raft_consensus.cc:348] T 00000000000000000000000000000000 P 39da52ddfea94a2c9fa8c5f90c2b0bbc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "39da52ddfea94a2c9fa8c5f90c2b0bbc" member_type: VOTER }
I20240503 23:27:13.928221 5499 raft_consensus.cc:374] T 00000000000000000000000000000000 P 39da52ddfea94a2c9fa8c5f90c2b0bbc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 23:27:13.928277 5499 raft_consensus.cc:729] T 00000000000000000000000000000000 P 39da52ddfea94a2c9fa8c5f90c2b0bbc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 39da52ddfea94a2c9fa8c5f90c2b0bbc, State: Initialized, Role: FOLLOWER
I20240503 23:27:13.928504 5499 consensus_queue.cc:260] T 00000000000000000000000000000000 P 39da52ddfea94a2c9fa8c5f90c2b0bbc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "39da52ddfea94a2c9fa8c5f90c2b0bbc" member_type: VOTER }
I20240503 23:27:13.928661 5499 raft_consensus.cc:388] T 00000000000000000000000000000000 P 39da52ddfea94a2c9fa8c5f90c2b0bbc [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 23:27:13.928731 5499 raft_consensus.cc:482] T 00000000000000000000000000000000 P 39da52ddfea94a2c9fa8c5f90c2b0bbc [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 23:27:13.928817 5499 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 39da52ddfea94a2c9fa8c5f90c2b0bbc [term 0 FOLLOWER]: Advancing to term 1
I20240503 23:27:13.930824 5499 raft_consensus.cc:504] T 00000000000000000000000000000000 P 39da52ddfea94a2c9fa8c5f90c2b0bbc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "39da52ddfea94a2c9fa8c5f90c2b0bbc" member_type: VOTER }
I20240503 23:27:13.931064 5499 leader_election.cc:302] T 00000000000000000000000000000000 P 39da52ddfea94a2c9fa8c5f90c2b0bbc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 39da52ddfea94a2c9fa8c5f90c2b0bbc; no voters:
I20240503 23:27:13.931548 5499 leader_election.cc:288] T 00000000000000000000000000000000 P 39da52ddfea94a2c9fa8c5f90c2b0bbc [CANDIDATE]: Term 1 election: Requested vote from peers
I20240503 23:27:13.931704 5502 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 39da52ddfea94a2c9fa8c5f90c2b0bbc [term 1 FOLLOWER]: Leader election won for term 1
I20240503 23:27:13.932240 5502 raft_consensus.cc:686] T 00000000000000000000000000000000 P 39da52ddfea94a2c9fa8c5f90c2b0bbc [term 1 LEADER]: Becoming Leader. State: Replica: 39da52ddfea94a2c9fa8c5f90c2b0bbc, State: Running, Role: LEADER
I20240503 23:27:13.932494 5502 consensus_queue.cc:237] T 00000000000000000000000000000000 P 39da52ddfea94a2c9fa8c5f90c2b0bbc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "39da52ddfea94a2c9fa8c5f90c2b0bbc" member_type: VOTER }
I20240503 23:27:13.932623 5499 sys_catalog.cc:564] T 00000000000000000000000000000000 P 39da52ddfea94a2c9fa8c5f90c2b0bbc [sys.catalog]: configured and running, proceeding with master startup.
I20240503 23:27:13.933805 5504 sys_catalog.cc:455] T 00000000000000000000000000000000 P 39da52ddfea94a2c9fa8c5f90c2b0bbc [sys.catalog]: SysCatalogTable state changed. Reason: New leader 39da52ddfea94a2c9fa8c5f90c2b0bbc. Latest consensus state: current_term: 1 leader_uuid: "39da52ddfea94a2c9fa8c5f90c2b0bbc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "39da52ddfea94a2c9fa8c5f90c2b0bbc" member_type: VOTER } }
I20240503 23:27:13.933768 5503 sys_catalog.cc:455] T 00000000000000000000000000000000 P 39da52ddfea94a2c9fa8c5f90c2b0bbc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "39da52ddfea94a2c9fa8c5f90c2b0bbc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "39da52ddfea94a2c9fa8c5f90c2b0bbc" member_type: VOTER } }
I20240503 23:27:13.934029 5504 sys_catalog.cc:458] T 00000000000000000000000000000000 P 39da52ddfea94a2c9fa8c5f90c2b0bbc [sys.catalog]: This master's current role is: LEADER
I20240503 23:27:13.934058 5503 sys_catalog.cc:458] T 00000000000000000000000000000000 P 39da52ddfea94a2c9fa8c5f90c2b0bbc [sys.catalog]: This master's current role is: LEADER
I20240503 23:27:13.935329 5507 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240503 23:27:13.937546 5507 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240503 23:27:13.938534 400 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20240503 23:27:13.941324 5507 catalog_manager.cc:1347] Generated new cluster ID: eced9a8744af4f228566e5c7056d7626
I20240503 23:27:13.941453 5507 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240503 23:27:13.962165 5507 catalog_manager.cc:1370] Generated new certificate authority record
I20240503 23:27:13.963078 5507 catalog_manager.cc:1504] Loading token signing keys...
I20240503 23:27:13.972899 5507 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 39da52ddfea94a2c9fa8c5f90c2b0bbc: Generated new TSK 0
I20240503 23:27:13.973212 5507 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240503 23:27:14.004127 400 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 23:27:14.007639 5520 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 23:27:14.008648 5521 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 23:27:14.009418 5523 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 23:27:14.010205 400 server_base.cc:1034] running on GCE node
I20240503 23:27:14.010471 400 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240503 23:27:14.010525 400 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240503 23:27:14.010556 400 hybrid_clock.cc:648] HybridClock initialized: now 1714778834010551 us; error 0 us; skew 500 ppm
I20240503 23:27:14.010725 400 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 23:27:14.012411 400 webserver.cc:460] Webserver started at http://127.0.100.1:44747/ using document root <none> and password file <none>
I20240503 23:27:14.012709 400 fs_manager.cc:359] Metadata directory not provided
I20240503 23:27:14.012794 400 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 23:27:14.012909 400 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 23:27:14.013644 400 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZhIiPz/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotMeetReplicationFactor.1714778688451250-400-0/minicluster-data/ts-0-root/instance:
uuid: "df682363faab414db67b9be5ef7d7af7"
format_stamp: "Formatted at 2024-05-03 23:27:14 on dist-test-slave-jvbp"
I20240503 23:27:14.016717 400 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20240503 23:27:14.018744 5528 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 23:27:14.019167 400 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.002s sys 0.000s
I20240503 23:27:14.019290 400 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZhIiPz/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotMeetReplicationFactor.1714778688451250-400-0/minicluster-data/ts-0-root
uuid: "df682363faab414db67b9be5ef7d7af7"
format_stamp: "Formatted at 2024-05-03 23:27:14 on dist-test-slave-jvbp"
I20240503 23:27:14.019434 400 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZhIiPz/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotMeetReplicationFactor.1714778688451250-400-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskZhIiPz/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotMeetReplicationFactor.1714778688451250-400-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskZhIiPz/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotMeetReplicationFactor.1714778688451250-400-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 23:27:14.044355 400 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 23:27:14.045099 400 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 23:27:14.045783 400 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 23:27:14.046849 400 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 23:27:14.046914 400 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 23:27:14.047000 400 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 23:27:14.047039 400 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 23:27:14.062460 400 rpc_server.cc:307] RPC server started. Bound to: 127.0.100.1:36189
I20240503 23:27:14.062472 5590 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.100.1:36189 every 8 connection(s)
I20240503 23:27:14.064438 400 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 23:27:14.068815 5595 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 23:27:14.069537 5596 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 23:27:14.070220 5598 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 23:27:14.071036 400 server_base.cc:1034] running on GCE node
I20240503 23:27:14.071372 400 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240503 23:27:14.071460 400 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240503 23:27:14.071501 400 hybrid_clock.cc:648] HybridClock initialized: now 1714778834071494 us; error 0 us; skew 500 ppm
I20240503 23:27:14.071655 400 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 23:27:14.072068 5591 heartbeater.cc:347] Connected to a master server at 127.0.100.62:37225
I20240503 23:27:14.072222 5591 heartbeater.cc:464] Registering TS with master...
I20240503 23:27:14.072737 5591 heartbeater.cc:511] Master 127.0.100.62:37225 requested a full tablet report, sending...
I20240503 23:27:14.073146 400 webserver.cc:460] Webserver started at http://127.0.100.2:44795/ using document root <none> and password file <none>
I20240503 23:27:14.073397 400 fs_manager.cc:359] Metadata directory not provided
I20240503 23:27:14.073477 400 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 23:27:14.073629 400 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 23:27:14.073896 5464 ts_manager.cc:194] Registered new tserver with Master: df682363faab414db67b9be5ef7d7af7 (127.0.100.1:36189)
I20240503 23:27:14.074358 400 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZhIiPz/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotMeetReplicationFactor.1714778688451250-400-0/minicluster-data/ts-1-root/instance:
uuid: "3e509ab3d9e5463c9f048f5623bcf7d0"
format_stamp: "Formatted at 2024-05-03 23:27:14 on dist-test-slave-jvbp"
I20240503 23:27:14.075132 5464 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:51812
I20240503 23:27:14.077234 400 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20240503 23:27:14.079012 5603 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 23:27:14.079432 400 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.002s sys 0.000s
I20240503 23:27:14.079531 400 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZhIiPz/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotMeetReplicationFactor.1714778688451250-400-0/minicluster-data/ts-1-root
uuid: "3e509ab3d9e5463c9f048f5623bcf7d0"
format_stamp: "Formatted at 2024-05-03 23:27:14 on dist-test-slave-jvbp"
I20240503 23:27:14.079638 400 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZhIiPz/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotMeetReplicationFactor.1714778688451250-400-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskZhIiPz/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotMeetReplicationFactor.1714778688451250-400-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskZhIiPz/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotMeetReplicationFactor.1714778688451250-400-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 23:27:14.105057 400 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 23:27:14.105973 400 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 23:27:14.106813 400 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 23:27:14.108328 400 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 23:27:14.108419 400 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 23:27:14.108510 400 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 23:27:14.108587 400 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.001s
I20240503 23:27:14.128070 400 rpc_server.cc:307] RPC server started. Bound to: 127.0.100.2:43463
I20240503 23:27:14.128072 5665 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.100.2:43463 every 8 connection(s)
I20240503 23:27:14.130957 400 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20240503 23:27:14.137362 5666 heartbeater.cc:347] Connected to a master server at 127.0.100.62:37225
I20240503 23:27:14.137554 5666 heartbeater.cc:464] Registering TS with master...
I20240503 23:27:14.138005 5666 heartbeater.cc:511] Master 127.0.100.62:37225 requested a full tablet report, sending...
I20240503 23:27:14.138950 5464 ts_manager.cc:194] Registered new tserver with Master: 3e509ab3d9e5463c9f048f5623bcf7d0 (127.0.100.2:43463)
I20240503 23:27:14.139776 5464 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:51824
W20240503 23:27:14.140959 5669 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 23:27:14.147049 5670 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 23:27:14.147894 5672 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 23:27:14.148723 400 server_base.cc:1034] running on GCE node
I20240503 23:27:14.149106 400 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240503 23:27:14.149174 400 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240503 23:27:14.149226 400 hybrid_clock.cc:648] HybridClock initialized: now 1714778834149221 us; error 0 us; skew 500 ppm
I20240503 23:27:14.149405 400 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 23:27:14.151043 400 webserver.cc:460] Webserver started at http://127.0.100.3:42869/ using document root <none> and password file <none>
I20240503 23:27:14.151293 400 fs_manager.cc:359] Metadata directory not provided
I20240503 23:27:14.151382 400 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 23:27:14.151510 400 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 23:27:14.152277 400 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZhIiPz/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotMeetReplicationFactor.1714778688451250-400-0/minicluster-data/ts-2-root/instance:
uuid: "0868ac3905ce4a67901d5c40aa0d60c5"
format_stamp: "Formatted at 2024-05-03 23:27:14 on dist-test-slave-jvbp"
I20240503 23:27:14.155416 400 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.002s
I20240503 23:27:14.158185 5677 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 23:27:14.158674 400 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20240503 23:27:14.158800 400 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZhIiPz/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotMeetReplicationFactor.1714778688451250-400-0/minicluster-data/ts-2-root
uuid: "0868ac3905ce4a67901d5c40aa0d60c5"
format_stamp: "Formatted at 2024-05-03 23:27:14 on dist-test-slave-jvbp"
I20240503 23:27:14.158953 400 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZhIiPz/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotMeetReplicationFactor.1714778688451250-400-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-taskZhIiPz/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotMeetReplicationFactor.1714778688451250-400-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-taskZhIiPz/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotMeetReplicationFactor.1714778688451250-400-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 23:27:14.171417 400 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 23:27:14.172135 400 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 23:27:14.172891 400 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 23:27:14.173995 400 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 23:27:14.174088 400 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 23:27:14.174208 400 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 23:27:14.174278 400 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 23:27:14.195246 400 rpc_server.cc:307] RPC server started. Bound to: 127.0.100.3:38111
I20240503 23:27:14.195307 5739 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.100.3:38111 every 8 connection(s)
I20240503 23:27:14.205081 5740 heartbeater.cc:347] Connected to a master server at 127.0.100.62:37225
I20240503 23:27:14.205323 5740 heartbeater.cc:464] Registering TS with master...
I20240503 23:27:14.205844 5740 heartbeater.cc:511] Master 127.0.100.62:37225 requested a full tablet report, sending...
I20240503 23:27:14.206983 5464 ts_manager.cc:194] Registered new tserver with Master: 0868ac3905ce4a67901d5c40aa0d60c5 (127.0.100.3:38111)
I20240503 23:27:14.207742 400 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.011399596s
I20240503 23:27:14.208011 5464 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:51828
I20240503 23:27:15.076594 5591 heartbeater.cc:503] Master 127.0.100.62:37225 was elected leader, sending a full tablet report...
I20240503 23:27:15.141160 5666 heartbeater.cc:503] Master 127.0.100.62:37225 was elected leader, sending a full tablet report...
I20240503 23:27:15.209952 5740 heartbeater.cc:503] Master 127.0.100.62:37225 was elected leader, sending a full tablet report...
I20240503 23:27:15.237093 400 test_util.cc:268] Using random seed: 257407194
I20240503 23:27:15.253779 5464 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51840:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 23:27:15.254593 5464 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-workload in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 23:27:15.280923 5705 tablet_service.cc:1467] Processing CreateTablet for tablet a03e446728754cfabd499ffbc3e36e76 (DEFAULT_TABLE table=test-workload [id=bea261d47d3144df8ae3033cac2429bd]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 23:27:15.281512 5705 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a03e446728754cfabd499ffbc3e36e76. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 23:27:15.283891 5631 tablet_service.cc:1467] Processing CreateTablet for tablet a03e446728754cfabd499ffbc3e36e76 (DEFAULT_TABLE table=test-workload [id=bea261d47d3144df8ae3033cac2429bd]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 23:27:15.284111 5556 tablet_service.cc:1467] Processing CreateTablet for tablet a03e446728754cfabd499ffbc3e36e76 (DEFAULT_TABLE table=test-workload [id=bea261d47d3144df8ae3033cac2429bd]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 23:27:15.284437 5631 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a03e446728754cfabd499ffbc3e36e76. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 23:27:15.284576 5556 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a03e446728754cfabd499ffbc3e36e76. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 23:27:15.289119 5760 tablet_bootstrap.cc:492] T a03e446728754cfabd499ffbc3e36e76 P 0868ac3905ce4a67901d5c40aa0d60c5: Bootstrap starting.
I20240503 23:27:15.291628 5762 tablet_bootstrap.cc:492] T a03e446728754cfabd499ffbc3e36e76 P df682363faab414db67b9be5ef7d7af7: Bootstrap starting.
I20240503 23:27:15.291883 5760 tablet_bootstrap.cc:654] T a03e446728754cfabd499ffbc3e36e76 P 0868ac3905ce4a67901d5c40aa0d60c5: Neither blocks nor log segments found. Creating new log.
I20240503 23:27:15.292285 5761 tablet_bootstrap.cc:492] T a03e446728754cfabd499ffbc3e36e76 P 3e509ab3d9e5463c9f048f5623bcf7d0: Bootstrap starting.
I20240503 23:27:15.293864 5762 tablet_bootstrap.cc:654] T a03e446728754cfabd499ffbc3e36e76 P df682363faab414db67b9be5ef7d7af7: Neither blocks nor log segments found. Creating new log.
I20240503 23:27:15.294178 5761 tablet_bootstrap.cc:654] T a03e446728754cfabd499ffbc3e36e76 P 3e509ab3d9e5463c9f048f5623bcf7d0: Neither blocks nor log segments found. Creating new log.
I20240503 23:27:15.294282 5760 tablet_bootstrap.cc:492] T a03e446728754cfabd499ffbc3e36e76 P 0868ac3905ce4a67901d5c40aa0d60c5: No bootstrap required, opened a new log
I20240503 23:27:15.294471 5760 ts_tablet_manager.cc:1367] T a03e446728754cfabd499ffbc3e36e76 P 0868ac3905ce4a67901d5c40aa0d60c5: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20240503 23:27:15.295342 5760 raft_consensus.cc:348] T a03e446728754cfabd499ffbc3e36e76 P 0868ac3905ce4a67901d5c40aa0d60c5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0868ac3905ce4a67901d5c40aa0d60c5" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 38111 } } peers { permanent_uuid: "3e509ab3d9e5463c9f048f5623bcf7d0" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 43463 } } peers { permanent_uuid: "df682363faab414db67b9be5ef7d7af7" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 36189 } }
I20240503 23:27:15.295758 5760 raft_consensus.cc:374] T a03e446728754cfabd499ffbc3e36e76 P 0868ac3905ce4a67901d5c40aa0d60c5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 23:27:15.295914 5760 raft_consensus.cc:729] T a03e446728754cfabd499ffbc3e36e76 P 0868ac3905ce4a67901d5c40aa0d60c5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0868ac3905ce4a67901d5c40aa0d60c5, State: Initialized, Role: FOLLOWER
I20240503 23:27:15.296274 5760 consensus_queue.cc:260] T a03e446728754cfabd499ffbc3e36e76 P 0868ac3905ce4a67901d5c40aa0d60c5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0868ac3905ce4a67901d5c40aa0d60c5" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 38111 } } peers { permanent_uuid: "3e509ab3d9e5463c9f048f5623bcf7d0" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 43463 } } peers { permanent_uuid: "df682363faab414db67b9be5ef7d7af7" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 36189 } }
I20240503 23:27:15.297570 5762 tablet_bootstrap.cc:492] T a03e446728754cfabd499ffbc3e36e76 P df682363faab414db67b9be5ef7d7af7: No bootstrap required, opened a new log
I20240503 23:27:15.297860 5760 ts_tablet_manager.cc:1398] T a03e446728754cfabd499ffbc3e36e76 P 0868ac3905ce4a67901d5c40aa0d60c5: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20240503 23:27:15.298219 5762 ts_tablet_manager.cc:1367] T a03e446728754cfabd499ffbc3e36e76 P df682363faab414db67b9be5ef7d7af7: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20240503 23:27:15.299261 5761 tablet_bootstrap.cc:492] T a03e446728754cfabd499ffbc3e36e76 P 3e509ab3d9e5463c9f048f5623bcf7d0: No bootstrap required, opened a new log
I20240503 23:27:15.299132 5762 raft_consensus.cc:348] T a03e446728754cfabd499ffbc3e36e76 P df682363faab414db67b9be5ef7d7af7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0868ac3905ce4a67901d5c40aa0d60c5" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 38111 } } peers { permanent_uuid: "3e509ab3d9e5463c9f048f5623bcf7d0" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 43463 } } peers { permanent_uuid: "df682363faab414db67b9be5ef7d7af7" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 36189 } }
I20240503 23:27:15.299432 5761 ts_tablet_manager.cc:1367] T a03e446728754cfabd499ffbc3e36e76 P 3e509ab3d9e5463c9f048f5623bcf7d0: Time spent bootstrapping tablet: real 0.007s user 0.003s sys 0.003s
I20240503 23:27:15.299576 5762 raft_consensus.cc:374] T a03e446728754cfabd499ffbc3e36e76 P df682363faab414db67b9be5ef7d7af7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 23:27:15.299705 5762 raft_consensus.cc:729] T a03e446728754cfabd499ffbc3e36e76 P df682363faab414db67b9be5ef7d7af7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df682363faab414db67b9be5ef7d7af7, State: Initialized, Role: FOLLOWER
I20240503 23:27:15.300083 5762 consensus_queue.cc:260] T a03e446728754cfabd499ffbc3e36e76 P df682363faab414db67b9be5ef7d7af7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0868ac3905ce4a67901d5c40aa0d60c5" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 38111 } } peers { permanent_uuid: "3e509ab3d9e5463c9f048f5623bcf7d0" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 43463 } } peers { permanent_uuid: "df682363faab414db67b9be5ef7d7af7" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 36189 } }
I20240503 23:27:15.300422 5761 raft_consensus.cc:348] T a03e446728754cfabd499ffbc3e36e76 P 3e509ab3d9e5463c9f048f5623bcf7d0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0868ac3905ce4a67901d5c40aa0d60c5" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 38111 } } peers { permanent_uuid: "3e509ab3d9e5463c9f048f5623bcf7d0" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 43463 } } peers { permanent_uuid: "df682363faab414db67b9be5ef7d7af7" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 36189 } }
I20240503 23:27:15.300789 5761 raft_consensus.cc:374] T a03e446728754cfabd499ffbc3e36e76 P 3e509ab3d9e5463c9f048f5623bcf7d0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 23:27:15.300922 5761 raft_consensus.cc:729] T a03e446728754cfabd499ffbc3e36e76 P 3e509ab3d9e5463c9f048f5623bcf7d0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3e509ab3d9e5463c9f048f5623bcf7d0, State: Initialized, Role: FOLLOWER
I20240503 23:27:15.301131 5762 ts_tablet_manager.cc:1398] T a03e446728754cfabd499ffbc3e36e76 P df682363faab414db67b9be5ef7d7af7: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240503 23:27:15.301311 5761 consensus_queue.cc:260] T a03e446728754cfabd499ffbc3e36e76 P 3e509ab3d9e5463c9f048f5623bcf7d0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0868ac3905ce4a67901d5c40aa0d60c5" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 38111 } } peers { permanent_uuid: "3e509ab3d9e5463c9f048f5623bcf7d0" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 43463 } } peers { permanent_uuid: "df682363faab414db67b9be5ef7d7af7" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 36189 } }
I20240503 23:27:15.302173 5761 ts_tablet_manager.cc:1398] T a03e446728754cfabd499ffbc3e36e76 P 3e509ab3d9e5463c9f048f5623bcf7d0: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20240503 23:27:15.626617 5766 raft_consensus.cc:482] T a03e446728754cfabd499ffbc3e36e76 P 0868ac3905ce4a67901d5c40aa0d60c5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 23:27:15.626891 5766 raft_consensus.cc:504] T a03e446728754cfabd499ffbc3e36e76 P 0868ac3905ce4a67901d5c40aa0d60c5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0868ac3905ce4a67901d5c40aa0d60c5" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 38111 } } peers { permanent_uuid: "3e509ab3d9e5463c9f048f5623bcf7d0" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 43463 } } peers { permanent_uuid: "df682363faab414db67b9be5ef7d7af7" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 36189 } }
I20240503 23:27:15.627815 5766 leader_election.cc:288] T a03e446728754cfabd499ffbc3e36e76 P 0868ac3905ce4a67901d5c40aa0d60c5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3e509ab3d9e5463c9f048f5623bcf7d0 (127.0.100.2:43463), df682363faab414db67b9be5ef7d7af7 (127.0.100.1:36189)
I20240503 23:27:15.636150 5566 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a03e446728754cfabd499ffbc3e36e76" candidate_uuid: "0868ac3905ce4a67901d5c40aa0d60c5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df682363faab414db67b9be5ef7d7af7" is_pre_election: true
I20240503 23:27:15.636137 5641 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a03e446728754cfabd499ffbc3e36e76" candidate_uuid: "0868ac3905ce4a67901d5c40aa0d60c5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3e509ab3d9e5463c9f048f5623bcf7d0" is_pre_election: true
I20240503 23:27:15.636595 5641 raft_consensus.cc:2454] T a03e446728754cfabd499ffbc3e36e76 P 3e509ab3d9e5463c9f048f5623bcf7d0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0868ac3905ce4a67901d5c40aa0d60c5 in term 0.
I20240503 23:27:15.636598 5566 raft_consensus.cc:2454] T a03e446728754cfabd499ffbc3e36e76 P df682363faab414db67b9be5ef7d7af7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0868ac3905ce4a67901d5c40aa0d60c5 in term 0.
I20240503 23:27:15.637207 5679 leader_election.cc:302] T a03e446728754cfabd499ffbc3e36e76 P 0868ac3905ce4a67901d5c40aa0d60c5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0868ac3905ce4a67901d5c40aa0d60c5, df682363faab414db67b9be5ef7d7af7; no voters:
I20240503 23:27:15.637512 5766 raft_consensus.cc:2781] T a03e446728754cfabd499ffbc3e36e76 P 0868ac3905ce4a67901d5c40aa0d60c5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 23:27:15.637629 5766 raft_consensus.cc:482] T a03e446728754cfabd499ffbc3e36e76 P 0868ac3905ce4a67901d5c40aa0d60c5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 23:27:15.637725 5766 raft_consensus.cc:3037] T a03e446728754cfabd499ffbc3e36e76 P 0868ac3905ce4a67901d5c40aa0d60c5 [term 0 FOLLOWER]: Advancing to term 1
I20240503 23:27:15.639937 5766 raft_consensus.cc:504] T a03e446728754cfabd499ffbc3e36e76 P 0868ac3905ce4a67901d5c40aa0d60c5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0868ac3905ce4a67901d5c40aa0d60c5" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 38111 } } peers { permanent_uuid: "3e509ab3d9e5463c9f048f5623bcf7d0" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 43463 } } peers { permanent_uuid: "df682363faab414db67b9be5ef7d7af7" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 36189 } }
I20240503 23:27:15.640724 5766 leader_election.cc:288] T a03e446728754cfabd499ffbc3e36e76 P 0868ac3905ce4a67901d5c40aa0d60c5 [CANDIDATE]: Term 1 election: Requested vote from peers 3e509ab3d9e5463c9f048f5623bcf7d0 (127.0.100.2:43463), df682363faab414db67b9be5ef7d7af7 (127.0.100.1:36189)
I20240503 23:27:15.641280 5641 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a03e446728754cfabd499ffbc3e36e76" candidate_uuid: "0868ac3905ce4a67901d5c40aa0d60c5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3e509ab3d9e5463c9f048f5623bcf7d0"
I20240503 23:27:15.641270 5566 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a03e446728754cfabd499ffbc3e36e76" candidate_uuid: "0868ac3905ce4a67901d5c40aa0d60c5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df682363faab414db67b9be5ef7d7af7"
I20240503 23:27:15.641527 5641 raft_consensus.cc:3037] T a03e446728754cfabd499ffbc3e36e76 P 3e509ab3d9e5463c9f048f5623bcf7d0 [term 0 FOLLOWER]: Advancing to term 1
I20240503 23:27:15.641527 5566 raft_consensus.cc:3037] T a03e446728754cfabd499ffbc3e36e76 P df682363faab414db67b9be5ef7d7af7 [term 0 FOLLOWER]: Advancing to term 1
I20240503 23:27:15.644594 5641 raft_consensus.cc:2454] T a03e446728754cfabd499ffbc3e36e76 P 3e509ab3d9e5463c9f048f5623bcf7d0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0868ac3905ce4a67901d5c40aa0d60c5 in term 1.
I20240503 23:27:15.644594 5566 raft_consensus.cc:2454] T a03e446728754cfabd499ffbc3e36e76 P df682363faab414db67b9be5ef7d7af7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0868ac3905ce4a67901d5c40aa0d60c5 in term 1.
I20240503 23:27:15.645318 5681 leader_election.cc:302] T a03e446728754cfabd499ffbc3e36e76 P 0868ac3905ce4a67901d5c40aa0d60c5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 0868ac3905ce4a67901d5c40aa0d60c5, 3e509ab3d9e5463c9f048f5623bcf7d0, df682363faab414db67b9be5ef7d7af7; no voters:
I20240503 23:27:15.645702 5766 raft_consensus.cc:2781] T a03e446728754cfabd499ffbc3e36e76 P 0868ac3905ce4a67901d5c40aa0d60c5 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 23:27:15.646232 5766 raft_consensus.cc:686] T a03e446728754cfabd499ffbc3e36e76 P 0868ac3905ce4a67901d5c40aa0d60c5 [term 1 LEADER]: Becoming Leader. State: Replica: 0868ac3905ce4a67901d5c40aa0d60c5, State: Running, Role: LEADER
I20240503 23:27:15.646544 5766 consensus_queue.cc:237] T a03e446728754cfabd499ffbc3e36e76 P 0868ac3905ce4a67901d5c40aa0d60c5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0868ac3905ce4a67901d5c40aa0d60c5" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 38111 } } peers { permanent_uuid: "3e509ab3d9e5463c9f048f5623bcf7d0" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 43463 } } peers { permanent_uuid: "df682363faab414db67b9be5ef7d7af7" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 36189 } }
I20240503 23:27:15.649613 5464 catalog_manager.cc:5484] T a03e446728754cfabd499ffbc3e36e76 P 0868ac3905ce4a67901d5c40aa0d60c5 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0868ac3905ce4a67901d5c40aa0d60c5 (127.0.100.3). New cstate: current_term: 1 leader_uuid: "0868ac3905ce4a67901d5c40aa0d60c5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0868ac3905ce4a67901d5c40aa0d60c5" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 38111 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3e509ab3d9e5463c9f048f5623bcf7d0" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 43463 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "df682363faab414db67b9be5ef7d7af7" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 36189 } health_report { overall_health: UNKNOWN } } }
I20240503 23:27:15.731132 400 tablet_server.cc:178] TabletServer@127.0.100.1:0 shutting down...
I20240503 23:27:15.764858 400 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240503 23:27:15.765501 400 tablet_replica.cc:330] T a03e446728754cfabd499ffbc3e36e76 P df682363faab414db67b9be5ef7d7af7: stopping tablet replica
I20240503 23:27:15.765766 400 raft_consensus.cc:2229] T a03e446728754cfabd499ffbc3e36e76 P df682363faab414db67b9be5ef7d7af7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 23:27:15.765976 400 raft_consensus.cc:2258] T a03e446728754cfabd499ffbc3e36e76 P df682363faab414db67b9be5ef7d7af7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 23:27:15.780748 400 tablet_server.cc:194] TabletServer@127.0.100.1:0 shutdown complete.
I20240503 23:27:15.938481 5517 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240503 23:27:15.939049 5517 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240503 23:27:15.939164 5517 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
W20240503 23:27:16.071864 5679 consensus_peers.cc:485] T a03e446728754cfabd499ffbc3e36e76 P 0868ac3905ce4a67901d5c40aa0d60c5 -> Peer df682363faab414db67b9be5ef7d7af7 (127.0.100.1:36189): Couldn't send request to peer df682363faab414db67b9be5ef7d7af7. Status: Network error: Client connection negotiation failed: client connection to 127.0.100.1:36189: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 23:27:16.120805 5766 consensus_queue.cc:1035] T a03e446728754cfabd499ffbc3e36e76 P 0868ac3905ce4a67901d5c40aa0d60c5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3e509ab3d9e5463c9f048f5623bcf7d0" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 43463 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 23:27:17.939726 5517 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240503 23:27:17.940325 5517 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240503 23:27:17.940448 5517 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
W20240503 23:27:18.698415 5679 consensus_peers.cc:485] T a03e446728754cfabd499ffbc3e36e76 P 0868ac3905ce4a67901d5c40aa0d60c5 -> Peer df682363faab414db67b9be5ef7d7af7 (127.0.100.1:36189): Couldn't send request to peer df682363faab414db67b9be5ef7d7af7. Status: Network error: Client connection negotiation failed: client connection to 127.0.100.1:36189: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 23:27:19.216615 5679 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.100.1:36189: connect: Connection refused (error 111) [suppressed 7 similar messages]
I20240503 23:27:19.940991 5517 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240503 23:27:19.941594 5517 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240503 23:27:19.941735 5517 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
W20240503 23:27:21.154927 5679 consensus_peers.cc:485] T a03e446728754cfabd499ffbc3e36e76 P 0868ac3905ce4a67901d5c40aa0d60c5 -> Peer df682363faab414db67b9be5ef7d7af7 (127.0.100.1:36189): Couldn't send request to peer df682363faab414db67b9be5ef7d7af7. Status: Network error: Client connection negotiation failed: client connection to 127.0.100.1:36189: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240503 23:27:21.942251 5517 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240503 23:27:21.942824 5517 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240503 23:27:21.942975 5517 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
W20240503 23:27:23.705719 5679 consensus_peers.cc:485] T a03e446728754cfabd499ffbc3e36e76 P 0868ac3905ce4a67901d5c40aa0d60c5 -> Peer df682363faab414db67b9be5ef7d7af7 (127.0.100.1:36189): Couldn't send request to peer df682363faab414db67b9be5ef7d7af7. Status: Network error: Client connection negotiation failed: client connection to 127.0.100.1:36189: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240503 23:27:23.943528 5517 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240503 23:27:23.944152 5517 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240503 23:27:23.944285 5517 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
W20240503 23:27:24.602002 5679 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.100.1:36189: connect: Connection refused (error 111) [suppressed 10 similar messages]
I20240503 23:27:25.944937 5517 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240503 23:27:25.945639 5517 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240503 23:27:25.945796 5517 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
W20240503 23:27:26.078608 5679 consensus_peers.cc:485] T a03e446728754cfabd499ffbc3e36e76 P 0868ac3905ce4a67901d5c40aa0d60c5 -> Peer df682363faab414db67b9be5ef7d7af7 (127.0.100.1:36189): Couldn't send request to peer df682363faab414db67b9be5ef7d7af7. Status: Network error: Client connection negotiation failed: client connection to 127.0.100.1:36189: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240503 23:27:26.840011 400 tablet_server.cc:178] TabletServer@127.0.100.2:0 shutting down...
I20240503 23:27:26.869850 400 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240503 23:27:26.870218 400 tablet_replica.cc:330] T a03e446728754cfabd499ffbc3e36e76 P 3e509ab3d9e5463c9f048f5623bcf7d0: stopping tablet replica
I20240503 23:27:26.870455 400 raft_consensus.cc:2229] T a03e446728754cfabd499ffbc3e36e76 P 3e509ab3d9e5463c9f048f5623bcf7d0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 23:27:26.870714 400 raft_consensus.cc:2258] T a03e446728754cfabd499ffbc3e36e76 P 3e509ab3d9e5463c9f048f5623bcf7d0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 23:27:26.886739 400 tablet_server.cc:194] TabletServer@127.0.100.2:0 shutdown complete.
I20240503 23:27:26.890796 400 tablet_server.cc:178] TabletServer@127.0.100.3:0 shutting down...
I20240503 23:27:26.901065 400 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240503 23:27:26.901419 400 tablet_replica.cc:330] T a03e446728754cfabd499ffbc3e36e76 P 0868ac3905ce4a67901d5c40aa0d60c5: stopping tablet replica
I20240503 23:27:26.901620 400 raft_consensus.cc:2229] T a03e446728754cfabd499ffbc3e36e76 P 0868ac3905ce4a67901d5c40aa0d60c5 [term 1 LEADER]: Raft consensus shutting down.
I20240503 23:27:26.902040 400 raft_consensus.cc:2258] T a03e446728754cfabd499ffbc3e36e76 P 0868ac3905ce4a67901d5c40aa0d60c5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 23:27:26.938524 400 tablet_server.cc:194] TabletServer@127.0.100.3:0 shutdown complete.
I20240503 23:27:26.942433 400 master.cc:537] Master@127.0.100.62:37225 shutting down...
I20240503 23:27:26.952057 400 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 39da52ddfea94a2c9fa8c5f90c2b0bbc [term 1 LEADER]: Raft consensus shutting down.
I20240503 23:27:26.952390 400 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 39da52ddfea94a2c9fa8c5f90c2b0bbc [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 23:27:26.952498 400 tablet_replica.cc:330] T 00000000000000000000000000000000 P 39da52ddfea94a2c9fa8c5f90c2b0bbc: stopping tablet replica
I20240503 23:27:26.968448 400 master.cc:552] Master@127.0.100.62:37225 shutdown complete.
[ OK ] AutoRebalancerTest.NoReplicaMovesIfCannotMeetReplicationFactor (13131 ms)
[ RUN ] AutoRebalancerTest.NoRebalancingIfReplicasRecovering
I20240503 23:27:26.983098 400 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.0.100.62:35103
I20240503 23:27:26.983742 400 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 23:27:26.988402 5801 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 23:27:26.989200 5798 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 23:27:26.989204 5799 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 23:27:26.989959 400 server_base.cc:1034] running on GCE node
I20240503 23:27:26.990268 400 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240503 23:27:26.990319 400 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240503 23:27:26.990345 400 hybrid_clock.cc:648] HybridClock initialized: now 1714778846990341 us; error 0 us; skew 500 ppm
I20240503 23:27:26.990487 400 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 23:27:26.992075 400 webserver.cc:460] Webserver started at http://127.0.100.62:43933/ using document root <none> and password file <none>
I20240503 23:27:26.992317 400 fs_manager.cc:359] Metadata directory not provided
I20240503 23:27:26.992377 400 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 23:27:26.992496 400 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 23:27:26.993294 400 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZhIiPz/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoRebalancingIfReplicasRecovering.1714778688451250-400-0/minicluster-data/master-0-root/instance:
uuid: "2849b178c1b2448e9ca7c2130795d41d"
format_stamp: "Formatted at 2024-05-03 23:27:26 on dist-test-slave-jvbp"
I20240503 23:27:26.996325 400 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20240503 23:27:26.998591 5806 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 23:27:26.999053 400 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.002s
I20240503 23:27:26.999164 400 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZhIiPz/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoRebalancingIfReplicasRecovering.1714778688451250-400-0/minicluster-data/master-0-root
uuid: "2849b178c1b2448e9ca7c2130795d41d"
format_stamp: "Formatted at 2024-05-03 23:27:26 on dist-test-slave-jvbp"
I20240503 23:27:26.999302 400 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZhIiPz/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoRebalancingIfReplicasRecovering.1714778688451250-400-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskZhIiPz/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoRebalancingIfReplicasRecovering.1714778688451250-400-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskZhIiPz/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoRebalancingIfReplicasRecovering.1714778688451250-400-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 23:27:27.023150 400 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 23:27:27.023873 400 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 23:27:27.039258 400 rpc_server.cc:307] RPC server started. Bound to: 127.0.100.62:35103
I20240503 23:27:27.039273 5857 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.100.62:35103 every 8 connection(s)
I20240503 23:27:27.040891 5858 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 23:27:27.046133 5858 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2849b178c1b2448e9ca7c2130795d41d: Bootstrap starting.
I20240503 23:27:27.047694 5858 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2849b178c1b2448e9ca7c2130795d41d: Neither blocks nor log segments found. Creating new log.
I20240503 23:27:27.049834 5858 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2849b178c1b2448e9ca7c2130795d41d: No bootstrap required, opened a new log
I20240503 23:27:27.050597 5858 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2849b178c1b2448e9ca7c2130795d41d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2849b178c1b2448e9ca7c2130795d41d" member_type: VOTER }
I20240503 23:27:27.050761 5858 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2849b178c1b2448e9ca7c2130795d41d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 23:27:27.050804 5858 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2849b178c1b2448e9ca7c2130795d41d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2849b178c1b2448e9ca7c2130795d41d, State: Initialized, Role: FOLLOWER
I20240503 23:27:27.050998 5858 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2849b178c1b2448e9ca7c2130795d41d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2849b178c1b2448e9ca7c2130795d41d" member_type: VOTER }
I20240503 23:27:27.051106 5858 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2849b178c1b2448e9ca7c2130795d41d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 23:27:27.051151 5858 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2849b178c1b2448e9ca7c2130795d41d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 23:27:27.051208 5858 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2849b178c1b2448e9ca7c2130795d41d [term 0 FOLLOWER]: Advancing to term 1
I20240503 23:27:27.053349 5858 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2849b178c1b2448e9ca7c2130795d41d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2849b178c1b2448e9ca7c2130795d41d" member_type: VOTER }
I20240503 23:27:27.053572 5858 leader_election.cc:302] T 00000000000000000000000000000000 P 2849b178c1b2448e9ca7c2130795d41d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2849b178c1b2448e9ca7c2130795d41d; no voters:
I20240503 23:27:27.054077 5858 leader_election.cc:288] T 00000000000000000000000000000000 P 2849b178c1b2448e9ca7c2130795d41d [CANDIDATE]: Term 1 election: Requested vote from peers
I20240503 23:27:27.054301 5861 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2849b178c1b2448e9ca7c2130795d41d [term 1 FOLLOWER]: Leader election won for term 1
I20240503 23:27:27.054771 5861 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2849b178c1b2448e9ca7c2130795d41d [term 1 LEADER]: Becoming Leader. State: Replica: 2849b178c1b2448e9ca7c2130795d41d, State: Running, Role: LEADER
I20240503 23:27:27.055042 5861 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2849b178c1b2448e9ca7c2130795d41d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2849b178c1b2448e9ca7c2130795d41d" member_type: VOTER }
I20240503 23:27:27.055155 5858 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2849b178c1b2448e9ca7c2130795d41d [sys.catalog]: configured and running, proceeding with master startup.
I20240503 23:27:27.056227 5862 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2849b178c1b2448e9ca7c2130795d41d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2849b178c1b2448e9ca7c2130795d41d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2849b178c1b2448e9ca7c2130795d41d" member_type: VOTER } }
I20240503 23:27:27.056280 5863 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2849b178c1b2448e9ca7c2130795d41d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2849b178c1b2448e9ca7c2130795d41d. Latest consensus state: current_term: 1 leader_uuid: "2849b178c1b2448e9ca7c2130795d41d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2849b178c1b2448e9ca7c2130795d41d" member_type: VOTER } }
I20240503 23:27:27.056622 5863 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2849b178c1b2448e9ca7c2130795d41d [sys.catalog]: This master's current role is: LEADER
I20240503 23:27:27.059558 5862 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2849b178c1b2448e9ca7c2130795d41d [sys.catalog]: This master's current role is: LEADER
I20240503 23:27:27.062685 5867 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240503 23:27:27.062767 400 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20240503 23:27:27.064345 5867 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240503 23:27:27.067873 5867 catalog_manager.cc:1347] Generated new cluster ID: bccc786defd4453f8e4797288c5d4dd3
I20240503 23:27:27.068012 5867 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240503 23:27:27.081321 5867 catalog_manager.cc:1370] Generated new certificate authority record
I20240503 23:27:27.082269 5867 catalog_manager.cc:1504] Loading token signing keys...
I20240503 23:27:27.093616 5867 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 2849b178c1b2448e9ca7c2130795d41d: Generated new TSK 0
I20240503 23:27:27.093958 5867 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240503 23:27:27.096019 400 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 23:27:27.099627 5879 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 23:27:27.100838 5880 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 23:27:27.101637 5882 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 23:27:27.102710 400 server_base.cc:1034] running on GCE node
I20240503 23:27:27.103186 400 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240503 23:27:27.103276 400 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240503 23:27:27.103315 400 hybrid_clock.cc:648] HybridClock initialized: now 1714778847103311 us; error 0 us; skew 500 ppm
I20240503 23:27:27.103530 400 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 23:27:27.105965 400 webserver.cc:460] Webserver started at http://127.0.100.1:40345/ using document root <none> and password file <none>
I20240503 23:27:27.106534 400 fs_manager.cc:359] Metadata directory not provided
I20240503 23:27:27.106662 400 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 23:27:27.106838 400 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 23:27:27.107920 400 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZhIiPz/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoRebalancingIfReplicasRecovering.1714778688451250-400-0/minicluster-data/ts-0-root/instance:
uuid: "bb85d6712d2242f9b99303387b4bbf87"
format_stamp: "Formatted at 2024-05-03 23:27:27 on dist-test-slave-jvbp"
I20240503 23:27:27.111661 400 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.004s
I20240503 23:27:27.113813 5887 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 23:27:27.114275 400 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20240503 23:27:27.114413 400 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZhIiPz/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoRebalancingIfReplicasRecovering.1714778688451250-400-0/minicluster-data/ts-0-root
uuid: "bb85d6712d2242f9b99303387b4bbf87"
format_stamp: "Formatted at 2024-05-03 23:27:27 on dist-test-slave-jvbp"
I20240503 23:27:27.114558 400 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZhIiPz/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoRebalancingIfReplicasRecovering.1714778688451250-400-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskZhIiPz/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoRebalancingIfReplicasRecovering.1714778688451250-400-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskZhIiPz/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoRebalancingIfReplicasRecovering.1714778688451250-400-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 23:27:27.125759 400 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 23:27:27.126507 400 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 23:27:27.127166 400 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 23:27:27.128222 400 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 23:27:27.128285 400 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 23:27:27.128373 400 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 23:27:27.128417 400 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 23:27:27.144003 400 rpc_server.cc:307] RPC server started. Bound to: 127.0.100.1:41717
I20240503 23:27:27.144019 5949 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.100.1:41717 every 8 connection(s)
I20240503 23:27:27.145931 400 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 23:27:27.149834 5954 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 23:27:27.150890 5955 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 23:27:27.151151 5957 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 23:27:27.154075 400 server_base.cc:1034] running on GCE node
I20240503 23:27:27.154453 400 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240503 23:27:27.154513 400 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240503 23:27:27.154565 400 hybrid_clock.cc:648] HybridClock initialized: now 1714778847154558 us; error 0 us; skew 500 ppm
I20240503 23:27:27.154606 5950 heartbeater.cc:347] Connected to a master server at 127.0.100.62:35103
I20240503 23:27:27.154732 5950 heartbeater.cc:464] Registering TS with master...
I20240503 23:27:27.154739 400 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 23:27:27.155437 5950 heartbeater.cc:511] Master 127.0.100.62:35103 requested a full tablet report, sending...
I20240503 23:27:27.156502 400 webserver.cc:460] Webserver started at http://127.0.100.2:41381/ using document root <none> and password file <none>
I20240503 23:27:27.156775 400 fs_manager.cc:359] Metadata directory not provided
I20240503 23:27:27.156750 5823 ts_manager.cc:194] Registered new tserver with Master: bb85d6712d2242f9b99303387b4bbf87 (127.0.100.1:41717)
I20240503 23:27:27.156884 400 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 23:27:27.157016 400 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 23:27:27.157814 400 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZhIiPz/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoRebalancingIfReplicasRecovering.1714778688451250-400-0/minicluster-data/ts-1-root/instance:
uuid: "5bed7b22c6d240668345b7d0dc336ea9"
format_stamp: "Formatted at 2024-05-03 23:27:27 on dist-test-slave-jvbp"
I20240503 23:27:27.158092 5823 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:42714
I20240503 23:27:27.160924 400 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20240503 23:27:27.162986 5962 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 23:27:27.163482 400 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.001s
I20240503 23:27:27.163678 400 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZhIiPz/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoRebalancingIfReplicasRecovering.1714778688451250-400-0/minicluster-data/ts-1-root
uuid: "5bed7b22c6d240668345b7d0dc336ea9"
format_stamp: "Formatted at 2024-05-03 23:27:27 on dist-test-slave-jvbp"
I20240503 23:27:27.163817 400 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZhIiPz/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoRebalancingIfReplicasRecovering.1714778688451250-400-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskZhIiPz/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoRebalancingIfReplicasRecovering.1714778688451250-400-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskZhIiPz/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoRebalancingIfReplicasRecovering.1714778688451250-400-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 23:27:27.181500 400 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 23:27:27.182183 400 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 23:27:27.182849 400 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 23:27:27.183823 400 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 23:27:27.183887 400 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 23:27:27.183966 400 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 23:27:27.184002 400 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 23:27:27.198791 400 rpc_server.cc:307] RPC server started. Bound to: 127.0.100.2:45643
I20240503 23:27:27.198828 6024 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.100.2:45643 every 8 connection(s)
I20240503 23:27:27.200848 400 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 23:27:27.204824 6028 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 23:27:27.205808 6031 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 23:27:27.206704 6029 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 23:27:27.207196 400 server_base.cc:1034] running on GCE node
I20240503 23:27:27.207538 400 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240503 23:27:27.207612 400 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240503 23:27:27.207655 400 hybrid_clock.cc:648] HybridClock initialized: now 1714778847207651 us; error 0 us; skew 500 ppm
I20240503 23:27:27.207836 400 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 23:27:27.207911 6025 heartbeater.cc:347] Connected to a master server at 127.0.100.62:35103
I20240503 23:27:27.208076 6025 heartbeater.cc:464] Registering TS with master...
I20240503 23:27:27.208642 6025 heartbeater.cc:511] Master 127.0.100.62:35103 requested a full tablet report, sending...
I20240503 23:27:27.209509 400 webserver.cc:460] Webserver started at http://127.0.100.3:44897/ using document root <none> and password file <none>
I20240503 23:27:27.209800 400 fs_manager.cc:359] Metadata directory not provided
I20240503 23:27:27.209774 5823 ts_manager.cc:194] Registered new tserver with Master: 5bed7b22c6d240668345b7d0dc336ea9 (127.0.100.2:45643)
I20240503 23:27:27.209895 400 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 23:27:27.210055 400 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 23:27:27.210908 5823 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:42722
I20240503 23:27:27.210973 400 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZhIiPz/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoRebalancingIfReplicasRecovering.1714778688451250-400-0/minicluster-data/ts-2-root/instance:
uuid: "d8998ef4c11346fb89defef6f414514b"
format_stamp: "Formatted at 2024-05-03 23:27:27 on dist-test-slave-jvbp"
I20240503 23:27:27.213930 400 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20240503 23:27:27.216063 6036 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 23:27:27.216578 400 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.002s sys 0.000s
I20240503 23:27:27.216701 400 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZhIiPz/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoRebalancingIfReplicasRecovering.1714778688451250-400-0/minicluster-data/ts-2-root
uuid: "d8998ef4c11346fb89defef6f414514b"
format_stamp: "Formatted at 2024-05-03 23:27:27 on dist-test-slave-jvbp"
I20240503 23:27:27.216828 400 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZhIiPz/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoRebalancingIfReplicasRecovering.1714778688451250-400-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-taskZhIiPz/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoRebalancingIfReplicasRecovering.1714778688451250-400-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-taskZhIiPz/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoRebalancingIfReplicasRecovering.1714778688451250-400-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 23:27:27.227349 400 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 23:27:27.228130 400 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 23:27:27.228837 400 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 23:27:27.229928 400 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 23:27:27.229991 400 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 23:27:27.230057 400 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 23:27:27.230103 400 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 23:27:27.245957 400 rpc_server.cc:307] RPC server started. Bound to: 127.0.100.3:42325
I20240503 23:27:27.245975 6098 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.100.3:42325 every 8 connection(s)
I20240503 23:27:27.254901 6099 heartbeater.cc:347] Connected to a master server at 127.0.100.62:35103
I20240503 23:27:27.255132 6099 heartbeater.cc:464] Registering TS with master...
I20240503 23:27:27.255621 6099 heartbeater.cc:511] Master 127.0.100.62:35103 requested a full tablet report, sending...
I20240503 23:27:27.257030 5823 ts_manager.cc:194] Registered new tserver with Master: d8998ef4c11346fb89defef6f414514b (127.0.100.3:42325)
I20240503 23:27:27.257442 400 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.010372427s
I20240503 23:27:27.258244 5823 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:42736
I20240503 23:27:28.159812 5950 heartbeater.cc:503] Master 127.0.100.62:35103 was elected leader, sending a full tablet report...
I20240503 23:27:28.212409 6025 heartbeater.cc:503] Master 127.0.100.62:35103 was elected leader, sending a full tablet report...
I20240503 23:27:28.260082 6099 heartbeater.cc:503] Master 127.0.100.62:35103 was elected leader, sending a full tablet report...
I20240503 23:27:28.287146 400 test_util.cc:268] Using random seed: 270457247
I20240503 23:27:28.303014 5823 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:42752:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "\004\001\000\377\377\377\037\004\001\000\376\377\377?\004\001\000\375\377\377_""\004\001\000\377\377\377\037\004\001\000\376\377\377?\004\001\000\375\377\377_"
indirect_data: """"
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 23:27:28.303969 5823 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-workload in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 23:27:28.325676 5915 tablet_service.cc:1467] Processing CreateTablet for tablet 3b169d0763a9492c83a8a406a16069f6 (DEFAULT_TABLE table=test-workload [id=96c264844e10403eaaed3223f26ed6de]), partition=RANGE (key) PARTITION VALUES < 536870911
I20240503 23:27:28.326349 5913 tablet_service.cc:1467] Processing CreateTablet for tablet f657a9c7f5244a13ad0a03c28cbf3f1f (DEFAULT_TABLE table=test-workload [id=96c264844e10403eaaed3223f26ed6de]), partition=RANGE (key) PARTITION 1073741822 <= VALUES < 1610612733
I20240503 23:27:28.326462 5915 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3b169d0763a9492c83a8a406a16069f6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 23:27:28.325676 5912 tablet_service.cc:1467] Processing CreateTablet for tablet 5a588f46c5ae427bbde5cf6a988cd824 (DEFAULT_TABLE table=test-workload [id=96c264844e10403eaaed3223f26ed6de]), partition=RANGE (key) PARTITION 1610612733 <= VALUES
I20240503 23:27:28.325790 5914 tablet_service.cc:1467] Processing CreateTablet for tablet f5cee53ca97a47b9a77b8a2a912f1349 (DEFAULT_TABLE table=test-workload [id=96c264844e10403eaaed3223f26ed6de]), partition=RANGE (key) PARTITION 536870911 <= VALUES < 1073741822
I20240503 23:27:28.327445 5914 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f5cee53ca97a47b9a77b8a2a912f1349. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 23:27:28.329656 6061 tablet_service.cc:1467] Processing CreateTablet for tablet 5a588f46c5ae427bbde5cf6a988cd824 (DEFAULT_TABLE table=test-workload [id=96c264844e10403eaaed3223f26ed6de]), partition=RANGE (key) PARTITION 1610612733 <= VALUES
I20240503 23:27:28.329643 6063 tablet_service.cc:1467] Processing CreateTablet for tablet f5cee53ca97a47b9a77b8a2a912f1349 (DEFAULT_TABLE table=test-workload [id=96c264844e10403eaaed3223f26ed6de]), partition=RANGE (key) PARTITION 536870911 <= VALUES < 1073741822
I20240503 23:27:28.329643 6062 tablet_service.cc:1467] Processing CreateTablet for tablet f657a9c7f5244a13ad0a03c28cbf3f1f (DEFAULT_TABLE table=test-workload [id=96c264844e10403eaaed3223f26ed6de]), partition=RANGE (key) PARTITION 1073741822 <= VALUES < 1610612733
I20240503 23:27:28.330195 5913 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f657a9c7f5244a13ad0a03c28cbf3f1f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 23:27:28.332796 5912 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5a588f46c5ae427bbde5cf6a988cd824. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 23:27:28.330202 6061 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5a588f46c5ae427bbde5cf6a988cd824. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 23:27:28.333087 6064 tablet_service.cc:1467] Processing CreateTablet for tablet 3b169d0763a9492c83a8a406a16069f6 (DEFAULT_TABLE table=test-workload [id=96c264844e10403eaaed3223f26ed6de]), partition=RANGE (key) PARTITION VALUES < 536870911
I20240503 23:27:28.333559 6064 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3b169d0763a9492c83a8a406a16069f6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 23:27:28.339917 6062 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f657a9c7f5244a13ad0a03c28cbf3f1f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 23:27:28.343286 6063 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f5cee53ca97a47b9a77b8a2a912f1349. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 23:27:28.345575 5990 tablet_service.cc:1467] Processing CreateTablet for tablet 3b169d0763a9492c83a8a406a16069f6 (DEFAULT_TABLE table=test-workload [id=96c264844e10403eaaed3223f26ed6de]), partition=RANGE (key) PARTITION VALUES < 536870911
I20240503 23:27:28.346187 5990 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3b169d0763a9492c83a8a406a16069f6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 23:27:28.346292 6120 tablet_bootstrap.cc:492] T 5a588f46c5ae427bbde5cf6a988cd824 P d8998ef4c11346fb89defef6f414514b: Bootstrap starting.
I20240503 23:27:28.348294 6120 tablet_bootstrap.cc:654] T 5a588f46c5ae427bbde5cf6a988cd824 P d8998ef4c11346fb89defef6f414514b: Neither blocks nor log segments found. Creating new log.
I20240503 23:27:28.349077 5989 tablet_service.cc:1467] Processing CreateTablet for tablet f5cee53ca97a47b9a77b8a2a912f1349 (DEFAULT_TABLE table=test-workload [id=96c264844e10403eaaed3223f26ed6de]), partition=RANGE (key) PARTITION 536870911 <= VALUES < 1073741822
I20240503 23:27:28.349607 5989 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f5cee53ca97a47b9a77b8a2a912f1349. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 23:27:28.351310 6119 tablet_bootstrap.cc:492] T f657a9c7f5244a13ad0a03c28cbf3f1f P bb85d6712d2242f9b99303387b4bbf87: Bootstrap starting.
I20240503 23:27:28.353103 5987 tablet_service.cc:1467] Processing CreateTablet for tablet 5a588f46c5ae427bbde5cf6a988cd824 (DEFAULT_TABLE table=test-workload [id=96c264844e10403eaaed3223f26ed6de]), partition=RANGE (key) PARTITION 1610612733 <= VALUES
I20240503 23:27:28.353461 6119 tablet_bootstrap.cc:654] T f657a9c7f5244a13ad0a03c28cbf3f1f P bb85d6712d2242f9b99303387b4bbf87: Neither blocks nor log segments found. Creating new log.
I20240503 23:27:28.353669 5987 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5a588f46c5ae427bbde5cf6a988cd824. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 23:27:28.356093 6120 tablet_bootstrap.cc:492] T 5a588f46c5ae427bbde5cf6a988cd824 P d8998ef4c11346fb89defef6f414514b: No bootstrap required, opened a new log
I20240503 23:27:28.356272 6120 ts_tablet_manager.cc:1367] T 5a588f46c5ae427bbde5cf6a988cd824 P d8998ef4c11346fb89defef6f414514b: Time spent bootstrapping tablet: real 0.010s user 0.000s sys 0.004s
I20240503 23:27:28.357229 6119 tablet_bootstrap.cc:492] T f657a9c7f5244a13ad0a03c28cbf3f1f P bb85d6712d2242f9b99303387b4bbf87: No bootstrap required, opened a new log
I20240503 23:27:28.357091 5988 tablet_service.cc:1467] Processing CreateTablet for tablet f657a9c7f5244a13ad0a03c28cbf3f1f (DEFAULT_TABLE table=test-workload [id=96c264844e10403eaaed3223f26ed6de]), partition=RANGE (key) PARTITION 1073741822 <= VALUES < 1610612733
I20240503 23:27:28.357403 6119 ts_tablet_manager.cc:1367] T f657a9c7f5244a13ad0a03c28cbf3f1f P bb85d6712d2242f9b99303387b4bbf87: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20240503 23:27:28.357299 6120 raft_consensus.cc:348] T 5a588f46c5ae427bbde5cf6a988cd824 P d8998ef4c11346fb89defef6f414514b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb85d6712d2242f9b99303387b4bbf87" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 41717 } } peers { permanent_uuid: "5bed7b22c6d240668345b7d0dc336ea9" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 45643 } } peers { permanent_uuid: "d8998ef4c11346fb89defef6f414514b" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 42325 } }
I20240503 23:27:28.357609 6120 raft_consensus.cc:374] T 5a588f46c5ae427bbde5cf6a988cd824 P d8998ef4c11346fb89defef6f414514b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 23:27:28.357726 6120 raft_consensus.cc:729] T 5a588f46c5ae427bbde5cf6a988cd824 P d8998ef4c11346fb89defef6f414514b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d8998ef4c11346fb89defef6f414514b, State: Initialized, Role: FOLLOWER
I20240503 23:27:28.357846 5988 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f657a9c7f5244a13ad0a03c28cbf3f1f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 23:27:28.358076 6120 consensus_queue.cc:260] T 5a588f46c5ae427bbde5cf6a988cd824 P d8998ef4c11346fb89defef6f414514b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb85d6712d2242f9b99303387b4bbf87" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 41717 } } peers { permanent_uuid: "5bed7b22c6d240668345b7d0dc336ea9" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 45643 } } peers { permanent_uuid: "d8998ef4c11346fb89defef6f414514b" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 42325 } }
I20240503 23:27:28.358355 6119 raft_consensus.cc:348] T f657a9c7f5244a13ad0a03c28cbf3f1f P bb85d6712d2242f9b99303387b4bbf87 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb85d6712d2242f9b99303387b4bbf87" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 41717 } } peers { permanent_uuid: "5bed7b22c6d240668345b7d0dc336ea9" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 45643 } } peers { permanent_uuid: "d8998ef4c11346fb89defef6f414514b" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 42325 } }
I20240503 23:27:28.358587 6119 raft_consensus.cc:374] T f657a9c7f5244a13ad0a03c28cbf3f1f P bb85d6712d2242f9b99303387b4bbf87 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 23:27:28.358687 6119 raft_consensus.cc:729] T f657a9c7f5244a13ad0a03c28cbf3f1f P bb85d6712d2242f9b99303387b4bbf87 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bb85d6712d2242f9b99303387b4bbf87, State: Initialized, Role: FOLLOWER
I20240503 23:27:28.359006 6119 consensus_queue.cc:260] T f657a9c7f5244a13ad0a03c28cbf3f1f P bb85d6712d2242f9b99303387b4bbf87 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb85d6712d2242f9b99303387b4bbf87" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 41717 } } peers { permanent_uuid: "5bed7b22c6d240668345b7d0dc336ea9" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 45643 } } peers { permanent_uuid: "d8998ef4c11346fb89defef6f414514b" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 42325 } }
I20240503 23:27:28.363884 6123 tablet_bootstrap.cc:492] T f5cee53ca97a47b9a77b8a2a912f1349 P 5bed7b22c6d240668345b7d0dc336ea9: Bootstrap starting.
I20240503 23:27:28.367201 6123 tablet_bootstrap.cc:654] T f5cee53ca97a47b9a77b8a2a912f1349 P 5bed7b22c6d240668345b7d0dc336ea9: Neither blocks nor log segments found. Creating new log.
I20240503 23:27:28.371080 6120 ts_tablet_manager.cc:1398] T 5a588f46c5ae427bbde5cf6a988cd824 P d8998ef4c11346fb89defef6f414514b: Time spent starting tablet: real 0.015s user 0.007s sys 0.008s
I20240503 23:27:28.371168 6119 ts_tablet_manager.cc:1398] T f657a9c7f5244a13ad0a03c28cbf3f1f P bb85d6712d2242f9b99303387b4bbf87: Time spent starting tablet: real 0.014s user 0.004s sys 0.008s
I20240503 23:27:28.371645 6119 tablet_bootstrap.cc:492] T f5cee53ca97a47b9a77b8a2a912f1349 P bb85d6712d2242f9b99303387b4bbf87: Bootstrap starting.
I20240503 23:27:28.372911 6120 tablet_bootstrap.cc:492] T 3b169d0763a9492c83a8a406a16069f6 P d8998ef4c11346fb89defef6f414514b: Bootstrap starting.
I20240503 23:27:28.373862 6119 tablet_bootstrap.cc:654] T f5cee53ca97a47b9a77b8a2a912f1349 P bb85d6712d2242f9b99303387b4bbf87: Neither blocks nor log segments found. Creating new log.
I20240503 23:27:28.374797 6120 tablet_bootstrap.cc:654] T 3b169d0763a9492c83a8a406a16069f6 P d8998ef4c11346fb89defef6f414514b: Neither blocks nor log segments found. Creating new log.
I20240503 23:27:28.382977 6123 tablet_bootstrap.cc:492] T f5cee53ca97a47b9a77b8a2a912f1349 P 5bed7b22c6d240668345b7d0dc336ea9: No bootstrap required, opened a new log
I20240503 23:27:28.383316 6123 ts_tablet_manager.cc:1367] T f5cee53ca97a47b9a77b8a2a912f1349 P 5bed7b22c6d240668345b7d0dc336ea9: Time spent bootstrapping tablet: real 0.020s user 0.003s sys 0.006s
I20240503 23:27:28.384704 6123 raft_consensus.cc:348] T f5cee53ca97a47b9a77b8a2a912f1349 P 5bed7b22c6d240668345b7d0dc336ea9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb85d6712d2242f9b99303387b4bbf87" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 41717 } } peers { permanent_uuid: "5bed7b22c6d240668345b7d0dc336ea9" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 45643 } } peers { permanent_uuid: "d8998ef4c11346fb89defef6f414514b" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 42325 } }
I20240503 23:27:28.385108 6123 raft_consensus.cc:374] T f5cee53ca97a47b9a77b8a2a912f1349 P 5bed7b22c6d240668345b7d0dc336ea9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 23:27:28.385244 6123 raft_consensus.cc:729] T f5cee53ca97a47b9a77b8a2a912f1349 P 5bed7b22c6d240668345b7d0dc336ea9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5bed7b22c6d240668345b7d0dc336ea9, State: Initialized, Role: FOLLOWER
I20240503 23:27:28.385555 6123 consensus_queue.cc:260] T f5cee53ca97a47b9a77b8a2a912f1349 P 5bed7b22c6d240668345b7d0dc336ea9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb85d6712d2242f9b99303387b4bbf87" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 41717 } } peers { permanent_uuid: "5bed7b22c6d240668345b7d0dc336ea9" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 45643 } } peers { permanent_uuid: "d8998ef4c11346fb89defef6f414514b" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 42325 } }
I20240503 23:27:28.385774 6120 tablet_bootstrap.cc:492] T 3b169d0763a9492c83a8a406a16069f6 P d8998ef4c11346fb89defef6f414514b: No bootstrap required, opened a new log
I20240503 23:27:28.385799 6119 tablet_bootstrap.cc:492] T f5cee53ca97a47b9a77b8a2a912f1349 P bb85d6712d2242f9b99303387b4bbf87: No bootstrap required, opened a new log
I20240503 23:27:28.386024 6119 ts_tablet_manager.cc:1367] T f5cee53ca97a47b9a77b8a2a912f1349 P bb85d6712d2242f9b99303387b4bbf87: Time spent bootstrapping tablet: real 0.014s user 0.000s sys 0.014s
I20240503 23:27:28.386021 6120 ts_tablet_manager.cc:1367] T 3b169d0763a9492c83a8a406a16069f6 P d8998ef4c11346fb89defef6f414514b: Time spent bootstrapping tablet: real 0.013s user 0.002s sys 0.007s
I20240503 23:27:28.386592 6123 ts_tablet_manager.cc:1398] T f5cee53ca97a47b9a77b8a2a912f1349 P 5bed7b22c6d240668345b7d0dc336ea9: Time spent starting tablet: real 0.003s user 0.001s sys 0.003s
I20240503 23:27:28.387037 6123 tablet_bootstrap.cc:492] T 3b169d0763a9492c83a8a406a16069f6 P 5bed7b22c6d240668345b7d0dc336ea9: Bootstrap starting.
I20240503 23:27:28.387192 6119 raft_consensus.cc:348] T f5cee53ca97a47b9a77b8a2a912f1349 P bb85d6712d2242f9b99303387b4bbf87 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb85d6712d2242f9b99303387b4bbf87" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 41717 } } peers { permanent_uuid: "5bed7b22c6d240668345b7d0dc336ea9" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 45643 } } peers { permanent_uuid: "d8998ef4c11346fb89defef6f414514b" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 42325 } }
I20240503 23:27:28.387264 6120 raft_consensus.cc:348] T 3b169d0763a9492c83a8a406a16069f6 P d8998ef4c11346fb89defef6f414514b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb85d6712d2242f9b99303387b4bbf87" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 41717 } } peers { permanent_uuid: "5bed7b22c6d240668345b7d0dc336ea9" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 45643 } } peers { permanent_uuid: "d8998ef4c11346fb89defef6f414514b" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 42325 } }
I20240503 23:27:28.387460 6119 raft_consensus.cc:374] T f5cee53ca97a47b9a77b8a2a912f1349 P bb85d6712d2242f9b99303387b4bbf87 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 23:27:28.387607 6120 raft_consensus.cc:374] T 3b169d0763a9492c83a8a406a16069f6 P d8998ef4c11346fb89defef6f414514b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 23:27:28.387683 6119 raft_consensus.cc:729] T f5cee53ca97a47b9a77b8a2a912f1349 P bb85d6712d2242f9b99303387b4bbf87 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bb85d6712d2242f9b99303387b4bbf87, State: Initialized, Role: FOLLOWER
I20240503 23:27:28.387780 6120 raft_consensus.cc:729] T 3b169d0763a9492c83a8a406a16069f6 P d8998ef4c11346fb89defef6f414514b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d8998ef4c11346fb89defef6f414514b, State: Initialized, Role: FOLLOWER
I20240503 23:27:28.388135 6120 consensus_queue.cc:260] T 3b169d0763a9492c83a8a406a16069f6 P d8998ef4c11346fb89defef6f414514b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb85d6712d2242f9b99303387b4bbf87" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 41717 } } peers { permanent_uuid: "5bed7b22c6d240668345b7d0dc336ea9" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 45643 } } peers { permanent_uuid: "d8998ef4c11346fb89defef6f414514b" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 42325 } }
I20240503 23:27:28.388837 6120 ts_tablet_manager.cc:1398] T 3b169d0763a9492c83a8a406a16069f6 P d8998ef4c11346fb89defef6f414514b: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240503 23:27:28.388092 6119 consensus_queue.cc:260] T f5cee53ca97a47b9a77b8a2a912f1349 P bb85d6712d2242f9b99303387b4bbf87 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb85d6712d2242f9b99303387b4bbf87" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 41717 } } peers { permanent_uuid: "5bed7b22c6d240668345b7d0dc336ea9" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 45643 } } peers { permanent_uuid: "d8998ef4c11346fb89defef6f414514b" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 42325 } }
I20240503 23:27:28.389333 6123 tablet_bootstrap.cc:654] T 3b169d0763a9492c83a8a406a16069f6 P 5bed7b22c6d240668345b7d0dc336ea9: Neither blocks nor log segments found. Creating new log.
I20240503 23:27:28.389381 6119 ts_tablet_manager.cc:1398] T f5cee53ca97a47b9a77b8a2a912f1349 P bb85d6712d2242f9b99303387b4bbf87: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20240503 23:27:28.389691 6120 tablet_bootstrap.cc:492] T f657a9c7f5244a13ad0a03c28cbf3f1f P d8998ef4c11346fb89defef6f414514b: Bootstrap starting.
I20240503 23:27:28.389729 6119 tablet_bootstrap.cc:492] T 3b169d0763a9492c83a8a406a16069f6 P bb85d6712d2242f9b99303387b4bbf87: Bootstrap starting.
I20240503 23:27:28.391963 6119 tablet_bootstrap.cc:654] T 3b169d0763a9492c83a8a406a16069f6 P bb85d6712d2242f9b99303387b4bbf87: Neither blocks nor log segments found. Creating new log.
I20240503 23:27:28.391997 6120 tablet_bootstrap.cc:654] T f657a9c7f5244a13ad0a03c28cbf3f1f P d8998ef4c11346fb89defef6f414514b: Neither blocks nor log segments found. Creating new log.
I20240503 23:27:28.401484 6119 tablet_bootstrap.cc:492] T 3b169d0763a9492c83a8a406a16069f6 P bb85d6712d2242f9b99303387b4bbf87: No bootstrap required, opened a new log
I20240503 23:27:28.401860 6119 ts_tablet_manager.cc:1367] T 3b169d0763a9492c83a8a406a16069f6 P bb85d6712d2242f9b99303387b4bbf87: Time spent bootstrapping tablet: real 0.012s user 0.000s sys 0.010s
I20240503 23:27:28.402909 6119 raft_consensus.cc:348] T 3b169d0763a9492c83a8a406a16069f6 P bb85d6712d2242f9b99303387b4bbf87 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb85d6712d2242f9b99303387b4bbf87" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 41717 } } peers { permanent_uuid: "5bed7b22c6d240668345b7d0dc336ea9" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 45643 } } peers { permanent_uuid: "d8998ef4c11346fb89defef6f414514b" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 42325 } }
I20240503 23:27:28.403199 6119 raft_consensus.cc:374] T 3b169d0763a9492c83a8a406a16069f6 P bb85d6712d2242f9b99303387b4bbf87 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 23:27:28.403334 6119 raft_consensus.cc:729] T 3b169d0763a9492c83a8a406a16069f6 P bb85d6712d2242f9b99303387b4bbf87 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bb85d6712d2242f9b99303387b4bbf87, State: Initialized, Role: FOLLOWER
I20240503 23:27:28.403617 6119 consensus_queue.cc:260] T 3b169d0763a9492c83a8a406a16069f6 P bb85d6712d2242f9b99303387b4bbf87 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb85d6712d2242f9b99303387b4bbf87" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 41717 } } peers { permanent_uuid: "5bed7b22c6d240668345b7d0dc336ea9" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 45643 } } peers { permanent_uuid: "d8998ef4c11346fb89defef6f414514b" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 42325 } }
I20240503 23:27:28.404279 6119 ts_tablet_manager.cc:1398] T 3b169d0763a9492c83a8a406a16069f6 P bb85d6712d2242f9b99303387b4bbf87: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240503 23:27:28.404717 6119 tablet_bootstrap.cc:492] T 5a588f46c5ae427bbde5cf6a988cd824 P bb85d6712d2242f9b99303387b4bbf87: Bootstrap starting.
I20240503 23:27:28.406934 6119 tablet_bootstrap.cc:654] T 5a588f46c5ae427bbde5cf6a988cd824 P bb85d6712d2242f9b99303387b4bbf87: Neither blocks nor log segments found. Creating new log.
I20240503 23:27:28.407182 6123 tablet_bootstrap.cc:492] T 3b169d0763a9492c83a8a406a16069f6 P 5bed7b22c6d240668345b7d0dc336ea9: No bootstrap required, opened a new log
I20240503 23:27:28.407370 6123 ts_tablet_manager.cc:1367] T 3b169d0763a9492c83a8a406a16069f6 P 5bed7b22c6d240668345b7d0dc336ea9: Time spent bootstrapping tablet: real 0.020s user 0.008s sys 0.011s
I20240503 23:27:28.401489 6120 tablet_bootstrap.cc:492] T f657a9c7f5244a13ad0a03c28cbf3f1f P d8998ef4c11346fb89defef6f414514b: No bootstrap required, opened a new log
I20240503 23:27:28.408321 6120 ts_tablet_manager.cc:1367] T f657a9c7f5244a13ad0a03c28cbf3f1f P d8998ef4c11346fb89defef6f414514b: Time spent bootstrapping tablet: real 0.019s user 0.009s sys 0.003s
I20240503 23:27:28.408352 6123 raft_consensus.cc:348] T 3b169d0763a9492c83a8a406a16069f6 P 5bed7b22c6d240668345b7d0dc336ea9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb85d6712d2242f9b99303387b4bbf87" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 41717 } } peers { permanent_uuid: "5bed7b22c6d240668345b7d0dc336ea9" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 45643 } } peers { permanent_uuid: "d8998ef4c11346fb89defef6f414514b" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 42325 } }
I20240503 23:27:28.408746 6123 raft_consensus.cc:374] T 3b169d0763a9492c83a8a406a16069f6 P 5bed7b22c6d240668345b7d0dc336ea9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 23:27:28.408895 6123 raft_consensus.cc:729] T 3b169d0763a9492c83a8a406a16069f6 P 5bed7b22c6d240668345b7d0dc336ea9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5bed7b22c6d240668345b7d0dc336ea9, State: Initialized, Role: FOLLOWER
I20240503 23:27:28.409240 6123 consensus_queue.cc:260] T 3b169d0763a9492c83a8a406a16069f6 P 5bed7b22c6d240668345b7d0dc336ea9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb85d6712d2242f9b99303387b4bbf87" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 41717 } } peers { permanent_uuid: "5bed7b22c6d240668345b7d0dc336ea9" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 45643 } } peers { permanent_uuid: "d8998ef4c11346fb89defef6f414514b" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 42325 } }
I20240503 23:27:28.409371 6120 raft_consensus.cc:348] T f657a9c7f5244a13ad0a03c28cbf3f1f P d8998ef4c11346fb89defef6f414514b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb85d6712d2242f9b99303387b4bbf87" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 41717 } } peers { permanent_uuid: "5bed7b22c6d240668345b7d0dc336ea9" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 45643 } } peers { permanent_uuid: "d8998ef4c11346fb89defef6f414514b" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 42325 } }
I20240503 23:27:28.410005 6123 ts_tablet_manager.cc:1398] T 3b169d0763a9492c83a8a406a16069f6 P 5bed7b22c6d240668345b7d0dc336ea9: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20240503 23:27:28.410156 6120 raft_consensus.cc:374] T f657a9c7f5244a13ad0a03c28cbf3f1f P d8998ef4c11346fb89defef6f414514b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 23:27:28.410324 6120 raft_consensus.cc:729] T f657a9c7f5244a13ad0a03c28cbf3f1f P d8998ef4c11346fb89defef6f414514b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d8998ef4c11346fb89defef6f414514b, State: Initialized, Role: FOLLOWER
I20240503 23:27:28.410501 6123 tablet_bootstrap.cc:492] T 5a588f46c5ae427bbde5cf6a988cd824 P 5bed7b22c6d240668345b7d0dc336ea9: Bootstrap starting.
I20240503 23:27:28.410677 6120 consensus_queue.cc:260] T f657a9c7f5244a13ad0a03c28cbf3f1f P d8998ef4c11346fb89defef6f414514b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb85d6712d2242f9b99303387b4bbf87" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 41717 } } peers { permanent_uuid: "5bed7b22c6d240668345b7d0dc336ea9" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 45643 } } peers { permanent_uuid: "d8998ef4c11346fb89defef6f414514b" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 42325 } }
I20240503 23:27:28.410965 6119 tablet_bootstrap.cc:492] T 5a588f46c5ae427bbde5cf6a988cd824 P bb85d6712d2242f9b99303387b4bbf87: No bootstrap required, opened a new log
I20240503 23:27:28.411299 6119 ts_tablet_manager.cc:1367] T 5a588f46c5ae427bbde5cf6a988cd824 P bb85d6712d2242f9b99303387b4bbf87: Time spent bootstrapping tablet: real 0.007s user 0.006s sys 0.000s
I20240503 23:27:28.411502 6120 ts_tablet_manager.cc:1398] T f657a9c7f5244a13ad0a03c28cbf3f1f P d8998ef4c11346fb89defef6f414514b: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240503 23:27:28.411964 6120 tablet_bootstrap.cc:492] T f5cee53ca97a47b9a77b8a2a912f1349 P d8998ef4c11346fb89defef6f414514b: Bootstrap starting.
I20240503 23:27:28.412353 6119 raft_consensus.cc:348] T 5a588f46c5ae427bbde5cf6a988cd824 P bb85d6712d2242f9b99303387b4bbf87 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb85d6712d2242f9b99303387b4bbf87" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 41717 } } peers { permanent_uuid: "5bed7b22c6d240668345b7d0dc336ea9" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 45643 } } peers { permanent_uuid: "d8998ef4c11346fb89defef6f414514b" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 42325 } }
I20240503 23:27:28.412701 6119 raft_consensus.cc:374] T 5a588f46c5ae427bbde5cf6a988cd824 P bb85d6712d2242f9b99303387b4bbf87 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 23:27:28.412768 6123 tablet_bootstrap.cc:654] T 5a588f46c5ae427bbde5cf6a988cd824 P 5bed7b22c6d240668345b7d0dc336ea9: Neither blocks nor log segments found. Creating new log.
I20240503 23:27:28.412889 6119 raft_consensus.cc:729] T 5a588f46c5ae427bbde5cf6a988cd824 P bb85d6712d2242f9b99303387b4bbf87 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bb85d6712d2242f9b99303387b4bbf87, State: Initialized, Role: FOLLOWER
I20240503 23:27:28.413311 6119 consensus_queue.cc:260] T 5a588f46c5ae427bbde5cf6a988cd824 P bb85d6712d2242f9b99303387b4bbf87 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb85d6712d2242f9b99303387b4bbf87" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 41717 } } peers { permanent_uuid: "5bed7b22c6d240668345b7d0dc336ea9" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 45643 } } peers { permanent_uuid: "d8998ef4c11346fb89defef6f414514b" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 42325 } }
I20240503 23:27:28.414024 6120 tablet_bootstrap.cc:654] T f5cee53ca97a47b9a77b8a2a912f1349 P d8998ef4c11346fb89defef6f414514b: Neither blocks nor log segments found. Creating new log.
I20240503 23:27:28.414130 6119 ts_tablet_manager.cc:1398] T 5a588f46c5ae427bbde5cf6a988cd824 P bb85d6712d2242f9b99303387b4bbf87: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240503 23:27:28.419107 6123 tablet_bootstrap.cc:492] T 5a588f46c5ae427bbde5cf6a988cd824 P 5bed7b22c6d240668345b7d0dc336ea9: No bootstrap required, opened a new log
I20240503 23:27:28.419142 6120 tablet_bootstrap.cc:492] T f5cee53ca97a47b9a77b8a2a912f1349 P d8998ef4c11346fb89defef6f414514b: No bootstrap required, opened a new log
I20240503 23:27:28.419342 6123 ts_tablet_manager.cc:1367] T 5a588f46c5ae427bbde5cf6a988cd824 P 5bed7b22c6d240668345b7d0dc336ea9: Time spent bootstrapping tablet: real 0.009s user 0.000s sys 0.007s
I20240503 23:27:28.419346 6120 ts_tablet_manager.cc:1367] T f5cee53ca97a47b9a77b8a2a912f1349 P d8998ef4c11346fb89defef6f414514b: Time spent bootstrapping tablet: real 0.007s user 0.006s sys 0.001s
I20240503 23:27:28.420152 6120 raft_consensus.cc:348] T f5cee53ca97a47b9a77b8a2a912f1349 P d8998ef4c11346fb89defef6f414514b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb85d6712d2242f9b99303387b4bbf87" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 41717 } } peers { permanent_uuid: "5bed7b22c6d240668345b7d0dc336ea9" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 45643 } } peers { permanent_uuid: "d8998ef4c11346fb89defef6f414514b" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 42325 } }
I20240503 23:27:28.420189 6123 raft_consensus.cc:348] T 5a588f46c5ae427bbde5cf6a988cd824 P 5bed7b22c6d240668345b7d0dc336ea9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb85d6712d2242f9b99303387b4bbf87" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 41717 } } peers { permanent_uuid: "5bed7b22c6d240668345b7d0dc336ea9" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 45643 } } peers { permanent_uuid: "d8998ef4c11346fb89defef6f414514b" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 42325 } }
I20240503 23:27:28.420374 6120 raft_consensus.cc:374] T f5cee53ca97a47b9a77b8a2a912f1349 P d8998ef4c11346fb89defef6f414514b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 23:27:28.420411 6123 raft_consensus.cc:374] T 5a588f46c5ae427bbde5cf6a988cd824 P 5bed7b22c6d240668345b7d0dc336ea9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 23:27:28.420462 6120 raft_consensus.cc:729] T f5cee53ca97a47b9a77b8a2a912f1349 P d8998ef4c11346fb89defef6f414514b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d8998ef4c11346fb89defef6f414514b, State: Initialized, Role: FOLLOWER
I20240503 23:27:28.420511 6123 raft_consensus.cc:729] T 5a588f46c5ae427bbde5cf6a988cd824 P 5bed7b22c6d240668345b7d0dc336ea9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5bed7b22c6d240668345b7d0dc336ea9, State: Initialized, Role: FOLLOWER
I20240503 23:27:28.420794 6123 consensus_queue.cc:260] T 5a588f46c5ae427bbde5cf6a988cd824 P 5bed7b22c6d240668345b7d0dc336ea9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb85d6712d2242f9b99303387b4bbf87" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 41717 } } peers { permanent_uuid: "5bed7b22c6d240668345b7d0dc336ea9" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 45643 } } peers { permanent_uuid: "d8998ef4c11346fb89defef6f414514b" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 42325 } }
I20240503 23:27:28.420801 6120 consensus_queue.cc:260] T f5cee53ca97a47b9a77b8a2a912f1349 P d8998ef4c11346fb89defef6f414514b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb85d6712d2242f9b99303387b4bbf87" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 41717 } } peers { permanent_uuid: "5bed7b22c6d240668345b7d0dc336ea9" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 45643 } } peers { permanent_uuid: "d8998ef4c11346fb89defef6f414514b" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 42325 } }
I20240503 23:27:28.421360 6120 ts_tablet_manager.cc:1398] T f5cee53ca97a47b9a77b8a2a912f1349 P d8998ef4c11346fb89defef6f414514b: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 23:27:28.421396 6123 ts_tablet_manager.cc:1398] T 5a588f46c5ae427bbde5cf6a988cd824 P 5bed7b22c6d240668345b7d0dc336ea9: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 23:27:28.421725 6123 tablet_bootstrap.cc:492] T f657a9c7f5244a13ad0a03c28cbf3f1f P 5bed7b22c6d240668345b7d0dc336ea9: Bootstrap starting.
I20240503 23:27:28.423034 6125 raft_consensus.cc:482] T 5a588f46c5ae427bbde5cf6a988cd824 P bb85d6712d2242f9b99303387b4bbf87 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 23:27:28.423250 6125 raft_consensus.cc:504] T 5a588f46c5ae427bbde5cf6a988cd824 P bb85d6712d2242f9b99303387b4bbf87 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb85d6712d2242f9b99303387b4bbf87" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 41717 } } peers { permanent_uuid: "5bed7b22c6d240668345b7d0dc336ea9" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 45643 } } peers { permanent_uuid: "d8998ef4c11346fb89defef6f414514b" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 42325 } }
I20240503 23:27:28.424343 6125 leader_election.cc:288] T 5a588f46c5ae427bbde5cf6a988cd824 P bb85d6712d2242f9b99303387b4bbf87 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5bed7b22c6d240668345b7d0dc336ea9 (127.0.100.2:45643), d8998ef4c11346fb89defef6f414514b (127.0.100.3:42325)
I20240503 23:27:28.425702 6123 tablet_bootstrap.cc:654] T f657a9c7f5244a13ad0a03c28cbf3f1f P 5bed7b22c6d240668345b7d0dc336ea9: Neither blocks nor log segments found. Creating new log.
I20240503 23:27:28.428957 6123 tablet_bootstrap.cc:492] T f657a9c7f5244a13ad0a03c28cbf3f1f P 5bed7b22c6d240668345b7d0dc336ea9: No bootstrap required, opened a new log
I20240503 23:27:28.429193 6123 ts_tablet_manager.cc:1367] T f657a9c7f5244a13ad0a03c28cbf3f1f P 5bed7b22c6d240668345b7d0dc336ea9: Time spent bootstrapping tablet: real 0.008s user 0.004s sys 0.001s
I20240503 23:27:28.430222 6123 raft_consensus.cc:348] T f657a9c7f5244a13ad0a03c28cbf3f1f P 5bed7b22c6d240668345b7d0dc336ea9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb85d6712d2242f9b99303387b4bbf87" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 41717 } } peers { permanent_uuid: "5bed7b22c6d240668345b7d0dc336ea9" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 45643 } } peers { permanent_uuid: "d8998ef4c11346fb89defef6f414514b" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 42325 } }
I20240503 23:27:28.430518 6123 raft_consensus.cc:374] T f657a9c7f5244a13ad0a03c28cbf3f1f P 5bed7b22c6d240668345b7d0dc336ea9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 23:27:28.430640 6123 raft_consensus.cc:729] T f657a9c7f5244a13ad0a03c28cbf3f1f P 5bed7b22c6d240668345b7d0dc336ea9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5bed7b22c6d240668345b7d0dc336ea9, State: Initialized, Role: FOLLOWER
I20240503 23:27:28.430968 6123 consensus_queue.cc:260] T f657a9c7f5244a13ad0a03c28cbf3f1f P 5bed7b22c6d240668345b7d0dc336ea9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb85d6712d2242f9b99303387b4bbf87" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 41717 } } peers { permanent_uuid: "5bed7b22c6d240668345b7d0dc336ea9" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 45643 } } peers { permanent_uuid: "d8998ef4c11346fb89defef6f414514b" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 42325 } }
I20240503 23:27:28.431635 6123 ts_tablet_manager.cc:1398] T f657a9c7f5244a13ad0a03c28cbf3f1f P 5bed7b22c6d240668345b7d0dc336ea9: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240503 23:27:28.432217 6129 raft_consensus.cc:482] T f657a9c7f5244a13ad0a03c28cbf3f1f P 5bed7b22c6d240668345b7d0dc336ea9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 23:27:28.432389 6129 raft_consensus.cc:504] T f657a9c7f5244a13ad0a03c28cbf3f1f P 5bed7b22c6d240668345b7d0dc336ea9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb85d6712d2242f9b99303387b4bbf87" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 41717 } } peers { permanent_uuid: "5bed7b22c6d240668345b7d0dc336ea9" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 45643 } } peers { permanent_uuid: "d8998ef4c11346fb89defef6f414514b" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 42325 } }
I20240503 23:27:28.433574 6129 leader_election.cc:288] T f657a9c7f5244a13ad0a03c28cbf3f1f P 5bed7b22c6d240668345b7d0dc336ea9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bb85d6712d2242f9b99303387b4bbf87 (127.0.100.1:41717), d8998ef4c11346fb89defef6f414514b (127.0.100.3:42325)
I20240503 23:27:28.434487 6074 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5a588f46c5ae427bbde5cf6a988cd824" candidate_uuid: "bb85d6712d2242f9b99303387b4bbf87" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d8998ef4c11346fb89defef6f414514b" is_pre_election: true
I20240503 23:27:28.434794 6074 raft_consensus.cc:2454] T 5a588f46c5ae427bbde5cf6a988cd824 P d8998ef4c11346fb89defef6f414514b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bb85d6712d2242f9b99303387b4bbf87 in term 0.
I20240503 23:27:28.435338 5891 leader_election.cc:302] T 5a588f46c5ae427bbde5cf6a988cd824 P bb85d6712d2242f9b99303387b4bbf87 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bb85d6712d2242f9b99303387b4bbf87, d8998ef4c11346fb89defef6f414514b; no voters:
I20240503 23:27:28.435719 6125 raft_consensus.cc:2781] T 5a588f46c5ae427bbde5cf6a988cd824 P bb85d6712d2242f9b99303387b4bbf87 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 23:27:28.435848 6125 raft_consensus.cc:482] T 5a588f46c5ae427bbde5cf6a988cd824 P bb85d6712d2242f9b99303387b4bbf87 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 23:27:28.435963 6125 raft_consensus.cc:3037] T 5a588f46c5ae427bbde5cf6a988cd824 P bb85d6712d2242f9b99303387b4bbf87 [term 0 FOLLOWER]: Advancing to term 1
I20240503 23:27:28.439585 6125 raft_consensus.cc:504] T 5a588f46c5ae427bbde5cf6a988cd824 P bb85d6712d2242f9b99303387b4bbf87 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb85d6712d2242f9b99303387b4bbf87" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 41717 } } peers { permanent_uuid: "5bed7b22c6d240668345b7d0dc336ea9" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 45643 } } peers { permanent_uuid: "d8998ef4c11346fb89defef6f414514b" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 42325 } }
I20240503 23:27:28.440500 6125 leader_election.cc:288] T 5a588f46c5ae427bbde5cf6a988cd824 P bb85d6712d2242f9b99303387b4bbf87 [CANDIDATE]: Term 1 election: Requested vote from peers 5bed7b22c6d240668345b7d0dc336ea9 (127.0.100.2:45643), d8998ef4c11346fb89defef6f414514b (127.0.100.3:42325)
I20240503 23:27:28.441298 6074 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5a588f46c5ae427bbde5cf6a988cd824" candidate_uuid: "bb85d6712d2242f9b99303387b4bbf87" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d8998ef4c11346fb89defef6f414514b"
I20240503 23:27:28.441565 6074 raft_consensus.cc:3037] T 5a588f46c5ae427bbde5cf6a988cd824 P d8998ef4c11346fb89defef6f414514b [term 0 FOLLOWER]: Advancing to term 1
I20240503 23:27:28.442363 6000 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5a588f46c5ae427bbde5cf6a988cd824" candidate_uuid: "bb85d6712d2242f9b99303387b4bbf87" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5bed7b22c6d240668345b7d0dc336ea9" is_pre_election: true
I20240503 23:27:28.442584 6000 raft_consensus.cc:2454] T 5a588f46c5ae427bbde5cf6a988cd824 P 5bed7b22c6d240668345b7d0dc336ea9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bb85d6712d2242f9b99303387b4bbf87 in term 0.
I20240503 23:27:28.442951 5999 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5a588f46c5ae427bbde5cf6a988cd824" candidate_uuid: "bb85d6712d2242f9b99303387b4bbf87" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5bed7b22c6d240668345b7d0dc336ea9"
I20240503 23:27:28.443187 5999 raft_consensus.cc:3037] T 5a588f46c5ae427bbde5cf6a988cd824 P 5bed7b22c6d240668345b7d0dc336ea9 [term 0 FOLLOWER]: Advancing to term 1
I20240503 23:27:28.444815 6074 raft_consensus.cc:2454] T 5a588f46c5ae427bbde5cf6a988cd824 P d8998ef4c11346fb89defef6f414514b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bb85d6712d2242f9b99303387b4bbf87 in term 1.
I20240503 23:27:28.445334 5891 leader_election.cc:302] T 5a588f46c5ae427bbde5cf6a988cd824 P bb85d6712d2242f9b99303387b4bbf87 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bb85d6712d2242f9b99303387b4bbf87, d8998ef4c11346fb89defef6f414514b; no voters:
I20240503 23:27:28.445565 6074 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f657a9c7f5244a13ad0a03c28cbf3f1f" candidate_uuid: "5bed7b22c6d240668345b7d0dc336ea9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d8998ef4c11346fb89defef6f414514b" is_pre_election: true
I20240503 23:27:28.445750 6125 raft_consensus.cc:2781] T 5a588f46c5ae427bbde5cf6a988cd824 P bb85d6712d2242f9b99303387b4bbf87 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 23:27:28.445765 6074 raft_consensus.cc:2454] T f657a9c7f5244a13ad0a03c28cbf3f1f P d8998ef4c11346fb89defef6f414514b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5bed7b22c6d240668345b7d0dc336ea9 in term 0.
I20240503 23:27:28.446250 6124 raft_consensus.cc:482] T 3b169d0763a9492c83a8a406a16069f6 P d8998ef4c11346fb89defef6f414514b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 23:27:28.446337 5966 leader_election.cc:302] T f657a9c7f5244a13ad0a03c28cbf3f1f P 5bed7b22c6d240668345b7d0dc336ea9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5bed7b22c6d240668345b7d0dc336ea9, d8998ef4c11346fb89defef6f414514b; no voters:
I20240503 23:27:28.446363 5999 raft_consensus.cc:2454] T 5a588f46c5ae427bbde5cf6a988cd824 P 5bed7b22c6d240668345b7d0dc336ea9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bb85d6712d2242f9b99303387b4bbf87 in term 1.
I20240503 23:27:28.446444 6124 raft_consensus.cc:504] T 3b169d0763a9492c83a8a406a16069f6 P d8998ef4c11346fb89defef6f414514b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb85d6712d2242f9b99303387b4bbf87" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 41717 } } peers { permanent_uuid: "5bed7b22c6d240668345b7d0dc336ea9" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 45643 } } peers { permanent_uuid: "d8998ef4c11346fb89defef6f414514b" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 42325 } }
I20240503 23:27:28.447399 6124 leader_election.cc:288] T 3b169d0763a9492c83a8a406a16069f6 P d8998ef4c11346fb89defef6f414514b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bb85d6712d2242f9b99303387b4bbf87 (127.0.100.1:41717), 5bed7b22c6d240668345b7d0dc336ea9 (127.0.100.2:45643)
I20240503 23:27:28.447643 6139 raft_consensus.cc:2781] T f657a9c7f5244a13ad0a03c28cbf3f1f P 5bed7b22c6d240668345b7d0dc336ea9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 23:27:28.447919 6139 raft_consensus.cc:482] T f657a9c7f5244a13ad0a03c28cbf3f1f P 5bed7b22c6d240668345b7d0dc336ea9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 23:27:28.448117 6139 raft_consensus.cc:3037] T f657a9c7f5244a13ad0a03c28cbf3f1f P 5bed7b22c6d240668345b7d0dc336ea9 [term 0 FOLLOWER]: Advancing to term 1
I20240503 23:27:28.450887 5925 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f657a9c7f5244a13ad0a03c28cbf3f1f" candidate_uuid: "5bed7b22c6d240668345b7d0dc336ea9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bb85d6712d2242f9b99303387b4bbf87" is_pre_election: true
I20240503 23:27:28.451275 5925 raft_consensus.cc:2454] T f657a9c7f5244a13ad0a03c28cbf3f1f P bb85d6712d2242f9b99303387b4bbf87 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5bed7b22c6d240668345b7d0dc336ea9 in term 0.
I20240503 23:27:28.451867 6139 raft_consensus.cc:504] T f657a9c7f5244a13ad0a03c28cbf3f1f P 5bed7b22c6d240668345b7d0dc336ea9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb85d6712d2242f9b99303387b4bbf87" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 41717 } } peers { permanent_uuid: "5bed7b22c6d240668345b7d0dc336ea9" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 45643 } } peers { permanent_uuid: "d8998ef4c11346fb89defef6f414514b" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 42325 } }
I20240503 23:27:28.452844 6139 leader_election.cc:288] T f657a9c7f5244a13ad0a03c28cbf3f1f P 5bed7b22c6d240668345b7d0dc336ea9 [CANDIDATE]: Term 1 election: Requested vote from peers bb85d6712d2242f9b99303387b4bbf87 (127.0.100.1:41717), d8998ef4c11346fb89defef6f414514b (127.0.100.3:42325)
I20240503 23:27:28.453045 5925 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f657a9c7f5244a13ad0a03c28cbf3f1f" candidate_uuid: "5bed7b22c6d240668345b7d0dc336ea9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bb85d6712d2242f9b99303387b4bbf87"
I20240503 23:27:28.453212 5925 raft_consensus.cc:3037] T f657a9c7f5244a13ad0a03c28cbf3f1f P bb85d6712d2242f9b99303387b4bbf87 [term 0 FOLLOWER]: Advancing to term 1
I20240503 23:27:28.454325 6074 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f657a9c7f5244a13ad0a03c28cbf3f1f" candidate_uuid: "5bed7b22c6d240668345b7d0dc336ea9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d8998ef4c11346fb89defef6f414514b"
I20240503 23:27:28.454630 6074 raft_consensus.cc:3037] T f657a9c7f5244a13ad0a03c28cbf3f1f P d8998ef4c11346fb89defef6f414514b [term 0 FOLLOWER]: Advancing to term 1
I20240503 23:27:28.456804 5925 raft_consensus.cc:2454] T f657a9c7f5244a13ad0a03c28cbf3f1f P bb85d6712d2242f9b99303387b4bbf87 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5bed7b22c6d240668345b7d0dc336ea9 in term 1.
I20240503 23:27:28.457399 5966 leader_election.cc:302] T f657a9c7f5244a13ad0a03c28cbf3f1f P 5bed7b22c6d240668345b7d0dc336ea9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5bed7b22c6d240668345b7d0dc336ea9, bb85d6712d2242f9b99303387b4bbf87; no voters:
I20240503 23:27:28.457823 6139 raft_consensus.cc:2781] T f657a9c7f5244a13ad0a03c28cbf3f1f P 5bed7b22c6d240668345b7d0dc336ea9 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 23:27:28.457990 6139 raft_consensus.cc:686] T f657a9c7f5244a13ad0a03c28cbf3f1f P 5bed7b22c6d240668345b7d0dc336ea9 [term 1 LEADER]: Becoming Leader. State: Replica: 5bed7b22c6d240668345b7d0dc336ea9, State: Running, Role: LEADER
I20240503 23:27:28.458366 6139 consensus_queue.cc:237] T f657a9c7f5244a13ad0a03c28cbf3f1f P 5bed7b22c6d240668345b7d0dc336ea9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb85d6712d2242f9b99303387b4bbf87" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 41717 } } peers { permanent_uuid: "5bed7b22c6d240668345b7d0dc336ea9" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 45643 } } peers { permanent_uuid: "d8998ef4c11346fb89defef6f414514b" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 42325 } }
I20240503 23:27:28.458942 6074 raft_consensus.cc:2454] T f657a9c7f5244a13ad0a03c28cbf3f1f P d8998ef4c11346fb89defef6f414514b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5bed7b22c6d240668345b7d0dc336ea9 in term 1.
I20240503 23:27:28.462436 5999 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b169d0763a9492c83a8a406a16069f6" candidate_uuid: "d8998ef4c11346fb89defef6f414514b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5bed7b22c6d240668345b7d0dc336ea9" is_pre_election: true
I20240503 23:27:28.462822 5999 raft_consensus.cc:2454] T 3b169d0763a9492c83a8a406a16069f6 P 5bed7b22c6d240668345b7d0dc336ea9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d8998ef4c11346fb89defef6f414514b in term 0.
I20240503 23:27:28.463397 6040 leader_election.cc:302] T 3b169d0763a9492c83a8a406a16069f6 P d8998ef4c11346fb89defef6f414514b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5bed7b22c6d240668345b7d0dc336ea9, d8998ef4c11346fb89defef6f414514b; no voters:
I20240503 23:27:28.463789 6124 raft_consensus.cc:2781] T 3b169d0763a9492c83a8a406a16069f6 P d8998ef4c11346fb89defef6f414514b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 23:27:28.463898 6124 raft_consensus.cc:482] T 3b169d0763a9492c83a8a406a16069f6 P d8998ef4c11346fb89defef6f414514b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 23:27:28.464015 6124 raft_consensus.cc:3037] T 3b169d0763a9492c83a8a406a16069f6 P d8998ef4c11346fb89defef6f414514b [term 0 FOLLOWER]: Advancing to term 1
I20240503 23:27:28.467784 6124 raft_consensus.cc:504] T 3b169d0763a9492c83a8a406a16069f6 P d8998ef4c11346fb89defef6f414514b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb85d6712d2242f9b99303387b4bbf87" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 41717 } } peers { permanent_uuid: "5bed7b22c6d240668345b7d0dc336ea9" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 45643 } } peers { permanent_uuid: "d8998ef4c11346fb89defef6f414514b" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 42325 } }
I20240503 23:27:28.468197 6125 raft_consensus.cc:686] T 5a588f46c5ae427bbde5cf6a988cd824 P bb85d6712d2242f9b99303387b4bbf87 [term 1 LEADER]: Becoming Leader. State: Replica: bb85d6712d2242f9b99303387b4bbf87, State: Running, Role: LEADER
I20240503 23:27:28.468611 6125 consensus_queue.cc:237] T 5a588f46c5ae427bbde5cf6a988cd824 P bb85d6712d2242f9b99303387b4bbf87 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb85d6712d2242f9b99303387b4bbf87" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 41717 } } peers { permanent_uuid: "5bed7b22c6d240668345b7d0dc336ea9" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 45643 } } peers { permanent_uuid: "d8998ef4c11346fb89defef6f414514b" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 42325 } }
I20240503 23:27:28.469228 6124 leader_election.cc:288] T 3b169d0763a9492c83a8a406a16069f6 P d8998ef4c11346fb89defef6f414514b [CANDIDATE]: Term 1 election: Requested vote from peers bb85d6712d2242f9b99303387b4bbf87 (127.0.100.1:41717), 5bed7b22c6d240668345b7d0dc336ea9 (127.0.100.2:45643)
I20240503 23:27:28.469898 5999 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b169d0763a9492c83a8a406a16069f6" candidate_uuid: "d8998ef4c11346fb89defef6f414514b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5bed7b22c6d240668345b7d0dc336ea9"
I20240503 23:27:28.470142 5999 raft_consensus.cc:3037] T 3b169d0763a9492c83a8a406a16069f6 P 5bed7b22c6d240668345b7d0dc336ea9 [term 0 FOLLOWER]: Advancing to term 1
I20240503 23:27:28.470075 5823 catalog_manager.cc:5484] T f657a9c7f5244a13ad0a03c28cbf3f1f P 5bed7b22c6d240668345b7d0dc336ea9 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5bed7b22c6d240668345b7d0dc336ea9 (127.0.100.2). New cstate: current_term: 1 leader_uuid: "5bed7b22c6d240668345b7d0dc336ea9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb85d6712d2242f9b99303387b4bbf87" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 41717 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5bed7b22c6d240668345b7d0dc336ea9" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 45643 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d8998ef4c11346fb89defef6f414514b" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 42325 } health_report { overall_health: UNKNOWN } } }
I20240503 23:27:28.473189 5999 raft_consensus.cc:2454] T 3b169d0763a9492c83a8a406a16069f6 P 5bed7b22c6d240668345b7d0dc336ea9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d8998ef4c11346fb89defef6f414514b in term 1.
I20240503 23:27:28.473716 6040 leader_election.cc:302] T 3b169d0763a9492c83a8a406a16069f6 P d8998ef4c11346fb89defef6f414514b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5bed7b22c6d240668345b7d0dc336ea9, d8998ef4c11346fb89defef6f414514b; no voters:
I20240503 23:27:28.475332 6124 raft_consensus.cc:2781] T 3b169d0763a9492c83a8a406a16069f6 P d8998ef4c11346fb89defef6f414514b [term 1 FOLLOWER]: Leader election won for term 1
I20240503 23:27:28.475147 5821 catalog_manager.cc:5484] T 5a588f46c5ae427bbde5cf6a988cd824 P bb85d6712d2242f9b99303387b4bbf87 reported cstate change: term changed from 0 to 1, leader changed from <none> to bb85d6712d2242f9b99303387b4bbf87 (127.0.100.1). New cstate: current_term: 1 leader_uuid: "bb85d6712d2242f9b99303387b4bbf87" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb85d6712d2242f9b99303387b4bbf87" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 41717 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5bed7b22c6d240668345b7d0dc336ea9" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 45643 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d8998ef4c11346fb89defef6f414514b" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 42325 } health_report { overall_health: UNKNOWN } } }
I20240503 23:27:28.475663 5925 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b169d0763a9492c83a8a406a16069f6" candidate_uuid: "d8998ef4c11346fb89defef6f414514b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bb85d6712d2242f9b99303387b4bbf87" is_pre_election: true
I20240503 23:27:28.475879 5925 raft_consensus.cc:2454] T 3b169d0763a9492c83a8a406a16069f6 P bb85d6712d2242f9b99303387b4bbf87 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d8998ef4c11346fb89defef6f414514b in term 0.
I20240503 23:27:28.476202 5924 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b169d0763a9492c83a8a406a16069f6" candidate_uuid: "d8998ef4c11346fb89defef6f414514b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bb85d6712d2242f9b99303387b4bbf87"
I20240503 23:27:28.476435 5924 raft_consensus.cc:3037] T 3b169d0763a9492c83a8a406a16069f6 P bb85d6712d2242f9b99303387b4bbf87 [term 0 FOLLOWER]: Advancing to term 1
I20240503 23:27:28.479269 5924 raft_consensus.cc:2454] T 3b169d0763a9492c83a8a406a16069f6 P bb85d6712d2242f9b99303387b4bbf87 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d8998ef4c11346fb89defef6f414514b in term 1.
I20240503 23:27:28.480840 6124 raft_consensus.cc:686] T 3b169d0763a9492c83a8a406a16069f6 P d8998ef4c11346fb89defef6f414514b [term 1 LEADER]: Becoming Leader. State: Replica: d8998ef4c11346fb89defef6f414514b, State: Running, Role: LEADER
I20240503 23:27:28.481200 6124 consensus_queue.cc:237] T 3b169d0763a9492c83a8a406a16069f6 P d8998ef4c11346fb89defef6f414514b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb85d6712d2242f9b99303387b4bbf87" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 41717 } } peers { permanent_uuid: "5bed7b22c6d240668345b7d0dc336ea9" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 45643 } } peers { permanent_uuid: "d8998ef4c11346fb89defef6f414514b" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 42325 } }
I20240503 23:27:28.483937 5821 catalog_manager.cc:5484] T 3b169d0763a9492c83a8a406a16069f6 P d8998ef4c11346fb89defef6f414514b reported cstate change: term changed from 0 to 1, leader changed from <none> to d8998ef4c11346fb89defef6f414514b (127.0.100.3). New cstate: current_term: 1 leader_uuid: "d8998ef4c11346fb89defef6f414514b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb85d6712d2242f9b99303387b4bbf87" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 41717 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5bed7b22c6d240668345b7d0dc336ea9" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 45643 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d8998ef4c11346fb89defef6f414514b" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 42325 } health_report { overall_health: HEALTHY } } }
I20240503 23:27:28.488265 6125 raft_consensus.cc:482] T f5cee53ca97a47b9a77b8a2a912f1349 P bb85d6712d2242f9b99303387b4bbf87 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 23:27:28.488512 6125 raft_consensus.cc:504] T f5cee53ca97a47b9a77b8a2a912f1349 P bb85d6712d2242f9b99303387b4bbf87 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb85d6712d2242f9b99303387b4bbf87" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 41717 } } peers { permanent_uuid: "5bed7b22c6d240668345b7d0dc336ea9" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 45643 } } peers { permanent_uuid: "d8998ef4c11346fb89defef6f414514b" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 42325 } }
I20240503 23:27:28.489239 6125 leader_election.cc:288] T f5cee53ca97a47b9a77b8a2a912f1349 P bb85d6712d2242f9b99303387b4bbf87 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5bed7b22c6d240668345b7d0dc336ea9 (127.0.100.2:45643), d8998ef4c11346fb89defef6f414514b (127.0.100.3:42325)
I20240503 23:27:28.489773 5999 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f5cee53ca97a47b9a77b8a2a912f1349" candidate_uuid: "bb85d6712d2242f9b99303387b4bbf87" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5bed7b22c6d240668345b7d0dc336ea9" is_pre_election: true
I20240503 23:27:28.489820 6074 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f5cee53ca97a47b9a77b8a2a912f1349" candidate_uuid: "bb85d6712d2242f9b99303387b4bbf87" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d8998ef4c11346fb89defef6f414514b" is_pre_election: true
I20240503 23:27:28.490051 6074 raft_consensus.cc:2454] T f5cee53ca97a47b9a77b8a2a912f1349 P d8998ef4c11346fb89defef6f414514b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bb85d6712d2242f9b99303387b4bbf87 in term 0.
I20240503 23:27:28.490059 5999 raft_consensus.cc:2454] T f5cee53ca97a47b9a77b8a2a912f1349 P 5bed7b22c6d240668345b7d0dc336ea9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bb85d6712d2242f9b99303387b4bbf87 in term 0.
I20240503 23:27:28.490482 5891 leader_election.cc:302] T f5cee53ca97a47b9a77b8a2a912f1349 P bb85d6712d2242f9b99303387b4bbf87 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5bed7b22c6d240668345b7d0dc336ea9, bb85d6712d2242f9b99303387b4bbf87; no voters:
I20240503 23:27:28.490805 6125 raft_consensus.cc:2781] T f5cee53ca97a47b9a77b8a2a912f1349 P bb85d6712d2242f9b99303387b4bbf87 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 23:27:28.490936 6125 raft_consensus.cc:482] T f5cee53ca97a47b9a77b8a2a912f1349 P bb85d6712d2242f9b99303387b4bbf87 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 23:27:28.491042 6125 raft_consensus.cc:3037] T f5cee53ca97a47b9a77b8a2a912f1349 P bb85d6712d2242f9b99303387b4bbf87 [term 0 FOLLOWER]: Advancing to term 1
I20240503 23:27:28.493345 6125 raft_consensus.cc:504] T f5cee53ca97a47b9a77b8a2a912f1349 P bb85d6712d2242f9b99303387b4bbf87 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb85d6712d2242f9b99303387b4bbf87" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 41717 } } peers { permanent_uuid: "5bed7b22c6d240668345b7d0dc336ea9" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 45643 } } peers { permanent_uuid: "d8998ef4c11346fb89defef6f414514b" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 42325 } }
I20240503 23:27:28.494073 6125 leader_election.cc:288] T f5cee53ca97a47b9a77b8a2a912f1349 P bb85d6712d2242f9b99303387b4bbf87 [CANDIDATE]: Term 1 election: Requested vote from peers 5bed7b22c6d240668345b7d0dc336ea9 (127.0.100.2:45643), d8998ef4c11346fb89defef6f414514b (127.0.100.3:42325)
I20240503 23:27:28.494606 5999 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f5cee53ca97a47b9a77b8a2a912f1349" candidate_uuid: "bb85d6712d2242f9b99303387b4bbf87" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5bed7b22c6d240668345b7d0dc336ea9"
I20240503 23:27:28.494607 6074 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f5cee53ca97a47b9a77b8a2a912f1349" candidate_uuid: "bb85d6712d2242f9b99303387b4bbf87" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d8998ef4c11346fb89defef6f414514b"
I20240503 23:27:28.494822 5999 raft_consensus.cc:3037] T f5cee53ca97a47b9a77b8a2a912f1349 P 5bed7b22c6d240668345b7d0dc336ea9 [term 0 FOLLOWER]: Advancing to term 1
I20240503 23:27:28.494869 6074 raft_consensus.cc:3037] T f5cee53ca97a47b9a77b8a2a912f1349 P d8998ef4c11346fb89defef6f414514b [term 0 FOLLOWER]: Advancing to term 1
I20240503 23:27:28.497020 6074 raft_consensus.cc:2454] T f5cee53ca97a47b9a77b8a2a912f1349 P d8998ef4c11346fb89defef6f414514b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bb85d6712d2242f9b99303387b4bbf87 in term 1.
I20240503 23:27:28.497020 5999 raft_consensus.cc:2454] T f5cee53ca97a47b9a77b8a2a912f1349 P 5bed7b22c6d240668345b7d0dc336ea9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bb85d6712d2242f9b99303387b4bbf87 in term 1.
I20240503 23:27:28.497540 5891 leader_election.cc:302] T f5cee53ca97a47b9a77b8a2a912f1349 P bb85d6712d2242f9b99303387b4bbf87 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bb85d6712d2242f9b99303387b4bbf87, d8998ef4c11346fb89defef6f414514b; no voters:
I20240503 23:27:28.497862 6125 raft_consensus.cc:2781] T f5cee53ca97a47b9a77b8a2a912f1349 P bb85d6712d2242f9b99303387b4bbf87 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 23:27:28.498029 6125 raft_consensus.cc:686] T f5cee53ca97a47b9a77b8a2a912f1349 P bb85d6712d2242f9b99303387b4bbf87 [term 1 LEADER]: Becoming Leader. State: Replica: bb85d6712d2242f9b99303387b4bbf87, State: Running, Role: LEADER
I20240503 23:27:28.498353 6125 consensus_queue.cc:237] T f5cee53ca97a47b9a77b8a2a912f1349 P bb85d6712d2242f9b99303387b4bbf87 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb85d6712d2242f9b99303387b4bbf87" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 41717 } } peers { permanent_uuid: "5bed7b22c6d240668345b7d0dc336ea9" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 45643 } } peers { permanent_uuid: "d8998ef4c11346fb89defef6f414514b" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 42325 } }
I20240503 23:27:28.501261 5821 catalog_manager.cc:5484] T f5cee53ca97a47b9a77b8a2a912f1349 P bb85d6712d2242f9b99303387b4bbf87 reported cstate change: term changed from 0 to 1, leader changed from <none> to bb85d6712d2242f9b99303387b4bbf87 (127.0.100.1). New cstate: current_term: 1 leader_uuid: "bb85d6712d2242f9b99303387b4bbf87" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb85d6712d2242f9b99303387b4bbf87" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 41717 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5bed7b22c6d240668345b7d0dc336ea9" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 45643 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d8998ef4c11346fb89defef6f414514b" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 42325 } health_report { overall_health: UNKNOWN } } }
I20240503 23:27:28.528898 400 tablet_server.cc:178] TabletServer@127.0.100.1:0 shutting down...
I20240503 23:27:28.553566 400 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240503 23:27:28.554018 400 tablet_replica.cc:330] T 5a588f46c5ae427bbde5cf6a988cd824 P bb85d6712d2242f9b99303387b4bbf87: stopping tablet replica
I20240503 23:27:28.554271 400 raft_consensus.cc:2229] T 5a588f46c5ae427bbde5cf6a988cd824 P bb85d6712d2242f9b99303387b4bbf87 [term 1 LEADER]: Raft consensus shutting down.
I20240503 23:27:28.554615 400 pending_rounds.cc:62] T 5a588f46c5ae427bbde5cf6a988cd824 P bb85d6712d2242f9b99303387b4bbf87: Trying to abort 1 pending ops.
I20240503 23:27:28.554662 400 pending_rounds.cc:68] T 5a588f46c5ae427bbde5cf6a988cd824 P bb85d6712d2242f9b99303387b4bbf87: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7023734163330441216 op_type: NO_OP noop_request { }
I20240503 23:27:28.554868 400 raft_consensus.cc:2866] T 5a588f46c5ae427bbde5cf6a988cd824 P bb85d6712d2242f9b99303387b4bbf87 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20240503 23:27:28.554953 400 raft_consensus.cc:2258] T 5a588f46c5ae427bbde5cf6a988cd824 P bb85d6712d2242f9b99303387b4bbf87 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 23:27:28.555984 400 tablet_replica.cc:330] T f657a9c7f5244a13ad0a03c28cbf3f1f P bb85d6712d2242f9b99303387b4bbf87: stopping tablet replica
I20240503 23:27:28.556133 400 raft_consensus.cc:2229] T f657a9c7f5244a13ad0a03c28cbf3f1f P bb85d6712d2242f9b99303387b4bbf87 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 23:27:28.556264 400 raft_consensus.cc:2258] T f657a9c7f5244a13ad0a03c28cbf3f1f P bb85d6712d2242f9b99303387b4bbf87 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 23:27:28.556895 400 tablet_replica.cc:330] T 3b169d0763a9492c83a8a406a16069f6 P bb85d6712d2242f9b99303387b4bbf87: stopping tablet replica
I20240503 23:27:28.557013 400 raft_consensus.cc:2229] T 3b169d0763a9492c83a8a406a16069f6 P bb85d6712d2242f9b99303387b4bbf87 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 23:27:28.557117 400 raft_consensus.cc:2258] T 3b169d0763a9492c83a8a406a16069f6 P bb85d6712d2242f9b99303387b4bbf87 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 23:27:28.557715 400 tablet_replica.cc:330] T f5cee53ca97a47b9a77b8a2a912f1349 P bb85d6712d2242f9b99303387b4bbf87: stopping tablet replica
I20240503 23:27:28.557826 400 raft_consensus.cc:2229] T f5cee53ca97a47b9a77b8a2a912f1349 P bb85d6712d2242f9b99303387b4bbf87 [term 1 LEADER]: Raft consensus shutting down.
I20240503 23:27:28.558034 400 pending_rounds.cc:62] T f5cee53ca97a47b9a77b8a2a912f1349 P bb85d6712d2242f9b99303387b4bbf87: Trying to abort 1 pending ops.
I20240503 23:27:28.558081 400 pending_rounds.cc:68] T f5cee53ca97a47b9a77b8a2a912f1349 P bb85d6712d2242f9b99303387b4bbf87: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7023734163452071936 op_type: NO_OP noop_request { }
I20240503 23:27:28.558204 400 raft_consensus.cc:2866] T f5cee53ca97a47b9a77b8a2a912f1349 P bb85d6712d2242f9b99303387b4bbf87 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20240503 23:27:28.558267 400 raft_consensus.cc:2258] T f5cee53ca97a47b9a77b8a2a912f1349 P bb85d6712d2242f9b99303387b4bbf87 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 23:27:28.574133 400 tablet_server.cc:194] TabletServer@127.0.100.1:0 shutdown complete.
I20240503 23:27:28.971868 6124 consensus_queue.cc:1035] T 3b169d0763a9492c83a8a406a16069f6 P d8998ef4c11346fb89defef6f414514b [LEADER]: Connected to new peer: Peer: permanent_uuid: "5bed7b22c6d240668345b7d0dc336ea9" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 45643 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240503 23:27:28.979983 6040 consensus_peers.cc:485] T 3b169d0763a9492c83a8a406a16069f6 P d8998ef4c11346fb89defef6f414514b -> Peer bb85d6712d2242f9b99303387b4bbf87 (127.0.100.1:41717): Couldn't send request to peer bb85d6712d2242f9b99303387b4bbf87. Status: Network error: Client connection negotiation failed: client connection to 127.0.100.1:41717: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 23:27:29.021905 5966 consensus_peers.cc:485] T f657a9c7f5244a13ad0a03c28cbf3f1f P 5bed7b22c6d240668345b7d0dc336ea9 -> Peer bb85d6712d2242f9b99303387b4bbf87 (127.0.100.1:41717): Couldn't send request to peer bb85d6712d2242f9b99303387b4bbf87. Status: Network error: Client connection negotiation failed: client connection to 127.0.100.1:41717: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 23:27:29.080397 6151 consensus_queue.cc:1035] T f657a9c7f5244a13ad0a03c28cbf3f1f P 5bed7b22c6d240668345b7d0dc336ea9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d8998ef4c11346fb89defef6f414514b" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 42325 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 23:27:29.514292 6154 consensus_queue.cc:579] T 3b169d0763a9492c83a8a406a16069f6 P d8998ef4c11346fb89defef6f414514b [LEADER]: Leader has been unable to successfully communicate with peer bb85d6712d2242f9b99303387b4bbf87 for more than 1 seconds (1.033s)
I20240503 23:27:29.516069 6161 consensus_queue.cc:579] T f657a9c7f5244a13ad0a03c28cbf3f1f P 5bed7b22c6d240668345b7d0dc336ea9 [LEADER]: Leader has been unable to successfully communicate with peer bb85d6712d2242f9b99303387b4bbf87 for more than 1 seconds (1.057s)
W20240503 23:27:29.518515 5821 catalog_manager.cc:5162] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 3b169d0763a9492c83a8a406a16069f6 with cas_config_opid_index -1: no extra replica candidate found for tablet 3b169d0763a9492c83a8a406a16069f6 (table test-workload [id=96c264844e10403eaaed3223f26ed6de]): Not found: could not select location for extra replica: not enough tablet servers to satisfy replica placement policy: the total number of registered tablet servers (3) does not allow for adding an extra replica; consider bringing up more to have at least 4 tablet servers up and running
I20240503 23:27:29.580108 400 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 23:27:29.584177 6167 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 23:27:29.585132 6168 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 23:27:29.585891 6170 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 23:27:29.586797 400 server_base.cc:1034] running on GCE node
I20240503 23:27:29.587224 400 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240503 23:27:29.587289 400 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240503 23:27:29.587332 400 hybrid_clock.cc:648] HybridClock initialized: now 1714778849587326 us; error 0 us; skew 500 ppm
I20240503 23:27:29.587502 400 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 23:27:29.589531 400 webserver.cc:460] Webserver started at http://127.0.100.4:44991/ using document root <none> and password file <none>
I20240503 23:27:29.589839 400 fs_manager.cc:359] Metadata directory not provided
I20240503 23:27:29.589929 400 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 23:27:29.590073 400 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 23:27:29.590938 400 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZhIiPz/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoRebalancingIfReplicasRecovering.1714778688451250-400-0/minicluster-data/ts-3-root/instance:
uuid: "c4adb558d3eb47b0ad3c5019472c2b28"
format_stamp: "Formatted at 2024-05-03 23:27:29 on dist-test-slave-jvbp"
I20240503 23:27:29.594202 400 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20240503 23:27:29.596267 6175 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 23:27:29.596710 400 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.002s sys 0.000s
I20240503 23:27:29.596848 400 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZhIiPz/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoRebalancingIfReplicasRecovering.1714778688451250-400-0/minicluster-data/ts-3-root
uuid: "c4adb558d3eb47b0ad3c5019472c2b28"
format_stamp: "Formatted at 2024-05-03 23:27:29 on dist-test-slave-jvbp"
I20240503 23:27:29.596990 400 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZhIiPz/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoRebalancingIfReplicasRecovering.1714778688451250-400-0/minicluster-data/ts-3-root
metadata directory: /tmp/dist-test-taskZhIiPz/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoRebalancingIfReplicasRecovering.1714778688451250-400-0/minicluster-data/ts-3-root
1 data directories: /tmp/dist-test-taskZhIiPz/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoRebalancingIfReplicasRecovering.1714778688451250-400-0/minicluster-data/ts-3-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 23:27:29.611233 400 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 23:27:29.611991 400 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 23:27:29.612779 400 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 23:27:29.613931 400 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 23:27:29.614001 400 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 23:27:29.614097 400 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 23:27:29.614136 400 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 23:27:29.629655 400 rpc_server.cc:307] RPC server started. Bound to: 127.0.100.4:37269
I20240503 23:27:29.629668 6237 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.100.4:37269 every 8 connection(s)
I20240503 23:27:29.638298 6238 heartbeater.cc:347] Connected to a master server at 127.0.100.62:35103
I20240503 23:27:29.638563 6238 heartbeater.cc:464] Registering TS with master...
I20240503 23:27:29.639110 6238 heartbeater.cc:511] Master 127.0.100.62:35103 requested a full tablet report, sending...
I20240503 23:27:29.640369 5823 ts_manager.cc:194] Registered new tserver with Master: c4adb558d3eb47b0ad3c5019472c2b28 (127.0.100.4:37269)
I20240503 23:27:29.641366 5823 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:35476
I20240503 23:27:29.646979 6074 consensus_queue.cc:237] T 3b169d0763a9492c83a8a406a16069f6 P d8998ef4c11346fb89defef6f414514b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "bb85d6712d2242f9b99303387b4bbf87" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 41717 } } peers { permanent_uuid: "5bed7b22c6d240668345b7d0dc336ea9" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 45643 } } peers { permanent_uuid: "d8998ef4c11346fb89defef6f414514b" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 42325 } } peers { permanent_uuid: "c4adb558d3eb47b0ad3c5019472c2b28" member_type: NON_VOTER last_known_addr { host: "127.0.100.4" port: 37269 } attrs { promote: true } }
W20240503 23:27:29.649845 6040 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.100.1:41717: connect: Connection refused (error 111) [suppressed 8 similar messages]
I20240503 23:27:29.650197 5999 raft_consensus.cc:1261] T 3b169d0763a9492c83a8a406a16069f6 P 5bed7b22c6d240668345b7d0dc336ea9 [term 1 FOLLOWER]: Refusing update from remote peer d8998ef4c11346fb89defef6f414514b: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 23:27:29.650810 6163 consensus_queue.cc:1035] T 3b169d0763a9492c83a8a406a16069f6 P d8998ef4c11346fb89defef6f414514b [LEADER]: Connected to new peer: Peer: permanent_uuid: "5bed7b22c6d240668345b7d0dc336ea9" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 45643 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20240503 23:27:29.651106 6040 consensus_peers.cc:485] T 3b169d0763a9492c83a8a406a16069f6 P d8998ef4c11346fb89defef6f414514b -> Peer bb85d6712d2242f9b99303387b4bbf87 (127.0.100.1:41717): Couldn't send request to peer bb85d6712d2242f9b99303387b4bbf87. Status: Network error: Client connection negotiation failed: client connection to 127.0.100.1:41717: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 23:27:29.653438 6154 raft_consensus.cc:2932] T 3b169d0763a9492c83a8a406a16069f6 P d8998ef4c11346fb89defef6f414514b [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER c4adb558d3eb47b0ad3c5019472c2b28 (127.0.100.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "bb85d6712d2242f9b99303387b4bbf87" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 41717 } } peers { permanent_uuid: "5bed7b22c6d240668345b7d0dc336ea9" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 45643 } } peers { permanent_uuid: "d8998ef4c11346fb89defef6f414514b" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 42325 } } peers { permanent_uuid: "c4adb558d3eb47b0ad3c5019472c2b28" member_type: NON_VOTER last_known_addr { host: "127.0.100.4" port: 37269 } attrs { promote: true } } }
I20240503 23:27:29.654217 5999 raft_consensus.cc:2932] T 3b169d0763a9492c83a8a406a16069f6 P 5bed7b22c6d240668345b7d0dc336ea9 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER c4adb558d3eb47b0ad3c5019472c2b28 (127.0.100.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "bb85d6712d2242f9b99303387b4bbf87" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 41717 } } peers { permanent_uuid: "5bed7b22c6d240668345b7d0dc336ea9" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 45643 } } peers { permanent_uuid: "d8998ef4c11346fb89defef6f414514b" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 42325 } } peers { permanent_uuid: "c4adb558d3eb47b0ad3c5019472c2b28" member_type: NON_VOTER last_known_addr { host: "127.0.100.4" port: 37269 } attrs { promote: true } } }
I20240503 23:27:29.657989 5810 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 3b169d0763a9492c83a8a406a16069f6 with cas_config_opid_index -1: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 3)
W20240503 23:27:29.658188 6037 consensus_peers.cc:485] T 3b169d0763a9492c83a8a406a16069f6 P d8998ef4c11346fb89defef6f414514b -> Peer c4adb558d3eb47b0ad3c5019472c2b28 (127.0.100.4:37269): Couldn't send request to peer c4adb558d3eb47b0ad3c5019472c2b28. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 3b169d0763a9492c83a8a406a16069f6. This is attempt 1: this message will repeat every 5th retry.
I20240503 23:27:29.659355 5823 catalog_manager.cc:5484] T 3b169d0763a9492c83a8a406a16069f6 P d8998ef4c11346fb89defef6f414514b reported cstate change: config changed from index -1 to 2, NON_VOTER c4adb558d3eb47b0ad3c5019472c2b28 (127.0.100.4) added. New cstate: current_term: 1 leader_uuid: "d8998ef4c11346fb89defef6f414514b" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "bb85d6712d2242f9b99303387b4bbf87" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 41717 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5bed7b22c6d240668345b7d0dc336ea9" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 45643 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d8998ef4c11346fb89defef6f414514b" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 42325 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c4adb558d3eb47b0ad3c5019472c2b28" member_type: NON_VOTER last_known_addr { host: "127.0.100.4" port: 37269 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 23:27:29.686653 5999 consensus_queue.cc:237] T f657a9c7f5244a13ad0a03c28cbf3f1f P 5bed7b22c6d240668345b7d0dc336ea9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "bb85d6712d2242f9b99303387b4bbf87" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 41717 } } peers { permanent_uuid: "5bed7b22c6d240668345b7d0dc336ea9" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 45643 } } peers { permanent_uuid: "d8998ef4c11346fb89defef6f414514b" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 42325 } } peers { permanent_uuid: "c4adb558d3eb47b0ad3c5019472c2b28" member_type: NON_VOTER last_known_addr { host: "127.0.100.4" port: 37269 } attrs { promote: true } }
I20240503 23:27:29.690344 6074 raft_consensus.cc:1261] T f657a9c7f5244a13ad0a03c28cbf3f1f P d8998ef4c11346fb89defef6f414514b [term 1 FOLLOWER]: Refusing update from remote peer 5bed7b22c6d240668345b7d0dc336ea9: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240503 23:27:29.690877 5966 consensus_peers.cc:485] T f657a9c7f5244a13ad0a03c28cbf3f1f P 5bed7b22c6d240668345b7d0dc336ea9 -> Peer bb85d6712d2242f9b99303387b4bbf87 (127.0.100.1:41717): Couldn't send request to peer bb85d6712d2242f9b99303387b4bbf87. Status: Network error: Client connection negotiation failed: client connection to 127.0.100.1:41717: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 23:27:29.691246 6151 consensus_queue.cc:1035] T f657a9c7f5244a13ad0a03c28cbf3f1f P 5bed7b22c6d240668345b7d0dc336ea9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d8998ef4c11346fb89defef6f414514b" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 42325 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 23:27:29.693773 6161 raft_consensus.cc:2932] T f657a9c7f5244a13ad0a03c28cbf3f1f P 5bed7b22c6d240668345b7d0dc336ea9 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER c4adb558d3eb47b0ad3c5019472c2b28 (127.0.100.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "bb85d6712d2242f9b99303387b4bbf87" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 41717 } } peers { permanent_uuid: "5bed7b22c6d240668345b7d0dc336ea9" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 45643 } } peers { permanent_uuid: "d8998ef4c11346fb89defef6f414514b" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 42325 } } peers { permanent_uuid: "c4adb558d3eb47b0ad3c5019472c2b28" member_type: NON_VOTER last_known_addr { host: "127.0.100.4" port: 37269 } attrs { promote: true } } }
I20240503 23:27:29.694499 6074 raft_consensus.cc:2932] T f657a9c7f5244a13ad0a03c28cbf3f1f P d8998ef4c11346fb89defef6f414514b [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER c4adb558d3eb47b0ad3c5019472c2b28 (127.0.100.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "bb85d6712d2242f9b99303387b4bbf87" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 41717 } } peers { permanent_uuid: "5bed7b22c6d240668345b7d0dc336ea9" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 45643 } } peers { permanent_uuid: "d8998ef4c11346fb89defef6f414514b" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 42325 } } peers { permanent_uuid: "c4adb558d3eb47b0ad3c5019472c2b28" member_type: NON_VOTER last_known_addr { host: "127.0.100.4" port: 37269 } attrs { promote: true } } }
W20240503 23:27:29.695511 5963 consensus_peers.cc:485] T f657a9c7f5244a13ad0a03c28cbf3f1f P 5bed7b22c6d240668345b7d0dc336ea9 -> Peer c4adb558d3eb47b0ad3c5019472c2b28 (127.0.100.4:37269): Couldn't send request to peer c4adb558d3eb47b0ad3c5019472c2b28. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f657a9c7f5244a13ad0a03c28cbf3f1f. This is attempt 1: this message will repeat every 5th retry.
I20240503 23:27:29.698387 5810 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet f657a9c7f5244a13ad0a03c28cbf3f1f with cas_config_opid_index -1: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 4)
I20240503 23:27:29.699344 5821 catalog_manager.cc:5484] T f657a9c7f5244a13ad0a03c28cbf3f1f P d8998ef4c11346fb89defef6f414514b reported cstate change: config changed from index -1 to 2, NON_VOTER c4adb558d3eb47b0ad3c5019472c2b28 (127.0.100.4) added. New cstate: current_term: 1 leader_uuid: "5bed7b22c6d240668345b7d0dc336ea9" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "bb85d6712d2242f9b99303387b4bbf87" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 41717 } } peers { permanent_uuid: "5bed7b22c6d240668345b7d0dc336ea9" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 45643 } } peers { permanent_uuid: "d8998ef4c11346fb89defef6f414514b" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 42325 } } peers { permanent_uuid: "c4adb558d3eb47b0ad3c5019472c2b28" member_type: NON_VOTER last_known_addr { host: "127.0.100.4" port: 37269 } attrs { promote: true } } }
I20240503 23:27:30.039086 6250 ts_tablet_manager.cc:902] T 3b169d0763a9492c83a8a406a16069f6 P c4adb558d3eb47b0ad3c5019472c2b28: Initiating tablet copy from peer d8998ef4c11346fb89defef6f414514b (127.0.100.3:42325)
I20240503 23:27:30.039876 6250 tablet_copy_client.cc:316] T 3b169d0763a9492c83a8a406a16069f6 P c4adb558d3eb47b0ad3c5019472c2b28: tablet copy: Beginning tablet copy session from remote peer at address 127.0.100.3:42325
I20240503 23:27:30.046129 6084 tablet_copy_service.cc:140] P d8998ef4c11346fb89defef6f414514b: Received BeginTabletCopySession request for tablet 3b169d0763a9492c83a8a406a16069f6 from peer c4adb558d3eb47b0ad3c5019472c2b28 ({username='slave'} at 127.0.0.1:58342)
I20240503 23:27:30.046362 6084 tablet_copy_service.cc:161] P d8998ef4c11346fb89defef6f414514b: Beginning new tablet copy session on tablet 3b169d0763a9492c83a8a406a16069f6 from peer c4adb558d3eb47b0ad3c5019472c2b28 at {username='slave'} at 127.0.0.1:58342: session id = c4adb558d3eb47b0ad3c5019472c2b28-3b169d0763a9492c83a8a406a16069f6
I20240503 23:27:30.048682 6084 tablet_copy_source_session.cc:205] T 3b169d0763a9492c83a8a406a16069f6 P d8998ef4c11346fb89defef6f414514b: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 23:27:30.050313 6250 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3b169d0763a9492c83a8a406a16069f6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 23:27:30.056365 6250 tablet_copy_client.cc:791] T 3b169d0763a9492c83a8a406a16069f6 P c4adb558d3eb47b0ad3c5019472c2b28: tablet copy: Starting download of 0 data blocks...
I20240503 23:27:30.056671 6250 tablet_copy_client.cc:655] T 3b169d0763a9492c83a8a406a16069f6 P c4adb558d3eb47b0ad3c5019472c2b28: tablet copy: Starting download of 1 WAL segments...
I20240503 23:27:30.058418 6250 tablet_copy_client.cc:531] T 3b169d0763a9492c83a8a406a16069f6 P c4adb558d3eb47b0ad3c5019472c2b28: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 23:27:30.061388 6250 tablet_bootstrap.cc:492] T 3b169d0763a9492c83a8a406a16069f6 P c4adb558d3eb47b0ad3c5019472c2b28: Bootstrap starting.
W20240503 23:27:30.061448 5871 auto_rebalancer.cc:227] Could not retrieve cluster info: Not found: tserver bb85d6712d2242f9b99303387b4bbf87 not available for placement
I20240503 23:27:30.064779 6151 raft_consensus.cc:482] T f5cee53ca97a47b9a77b8a2a912f1349 P 5bed7b22c6d240668345b7d0dc336ea9 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 23:27:30.065013 6151 raft_consensus.cc:504] T f5cee53ca97a47b9a77b8a2a912f1349 P 5bed7b22c6d240668345b7d0dc336ea9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb85d6712d2242f9b99303387b4bbf87" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 41717 } } peers { permanent_uuid: "5bed7b22c6d240668345b7d0dc336ea9" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 45643 } } peers { permanent_uuid: "d8998ef4c11346fb89defef6f414514b" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 42325 } }
I20240503 23:27:30.066079 6151 leader_election.cc:288] T f5cee53ca97a47b9a77b8a2a912f1349 P 5bed7b22c6d240668345b7d0dc336ea9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers bb85d6712d2242f9b99303387b4bbf87 (127.0.100.1:41717), d8998ef4c11346fb89defef6f414514b (127.0.100.3:42325)
I20240503 23:27:30.067025 6074 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f5cee53ca97a47b9a77b8a2a912f1349" candidate_uuid: "5bed7b22c6d240668345b7d0dc336ea9" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d8998ef4c11346fb89defef6f414514b" is_pre_election: true
I20240503 23:27:30.067298 6074 raft_consensus.cc:2454] T f5cee53ca97a47b9a77b8a2a912f1349 P d8998ef4c11346fb89defef6f414514b [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5bed7b22c6d240668345b7d0dc336ea9 in term 1.
W20240503 23:27:30.067448 5966 leader_election.cc:334] T f5cee53ca97a47b9a77b8a2a912f1349 P 5bed7b22c6d240668345b7d0dc336ea9 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer bb85d6712d2242f9b99303387b4bbf87 (127.0.100.1:41717): Network error: Client connection negotiation failed: client connection to 127.0.100.1:41717: connect: Connection refused (error 111)
I20240503 23:27:30.067775 5966 leader_election.cc:302] T f5cee53ca97a47b9a77b8a2a912f1349 P 5bed7b22c6d240668345b7d0dc336ea9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5bed7b22c6d240668345b7d0dc336ea9, d8998ef4c11346fb89defef6f414514b; no voters: bb85d6712d2242f9b99303387b4bbf87
I20240503 23:27:30.068058 6151 raft_consensus.cc:2781] T f5cee53ca97a47b9a77b8a2a912f1349 P 5bed7b22c6d240668345b7d0dc336ea9 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240503 23:27:30.068151 6151 raft_consensus.cc:482] T f5cee53ca97a47b9a77b8a2a912f1349 P 5bed7b22c6d240668345b7d0dc336ea9 [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 23:27:30.068233 6151 raft_consensus.cc:3037] T f5cee53ca97a47b9a77b8a2a912f1349 P 5bed7b22c6d240668345b7d0dc336ea9 [term 1 FOLLOWER]: Advancing to term 2
I20240503 23:27:30.068434 6250 tablet_bootstrap.cc:492] T 3b169d0763a9492c83a8a406a16069f6 P c4adb558d3eb47b0ad3c5019472c2b28: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 23:27:30.068866 6250 tablet_bootstrap.cc:492] T 3b169d0763a9492c83a8a406a16069f6 P c4adb558d3eb47b0ad3c5019472c2b28: Bootstrap complete.
I20240503 23:27:30.069150 6250 ts_tablet_manager.cc:1367] T 3b169d0763a9492c83a8a406a16069f6 P c4adb558d3eb47b0ad3c5019472c2b28: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.002s
I20240503 23:27:30.070005 6250 raft_consensus.cc:348] T 3b169d0763a9492c83a8a406a16069f6 P c4adb558d3eb47b0ad3c5019472c2b28 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "bb85d6712d2242f9b99303387b4bbf87" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 41717 } } peers { permanent_uuid: "5bed7b22c6d240668345b7d0dc336ea9" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 45643 } } peers { permanent_uuid: "d8998ef4c11346fb89defef6f414514b" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 42325 } } peers { permanent_uuid: "c4adb558d3eb47b0ad3c5019472c2b28" member_type: NON_VOTER last_known_addr { host: "127.0.100.4" port: 37269 } attrs { promote: true } }
I20240503 23:27:30.070250 6250 raft_consensus.cc:729] T 3b169d0763a9492c83a8a406a16069f6 P c4adb558d3eb47b0ad3c5019472c2b28 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: c4adb558d3eb47b0ad3c5019472c2b28, State: Initialized, Role: LEARNER
I20240503 23:27:30.070461 6250 consensus_queue.cc:260] T 3b169d0763a9492c83a8a406a16069f6 P c4adb558d3eb47b0ad3c5019472c2b28 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "bb85d6712d2242f9b99303387b4bbf87" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 41717 } } peers { permanent_uuid: "5bed7b22c6d240668345b7d0dc336ea9" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 45643 } } peers { permanent_uuid: "d8998ef4c11346fb89defef6f414514b" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 42325 } } peers { permanent_uuid: "c4adb558d3eb47b0ad3c5019472c2b28" member_type: NON_VOTER last_known_addr { host: "127.0.100.4" port: 37269 } attrs { promote: true } }
I20240503 23:27:30.071403 6250 ts_tablet_manager.cc:1398] T 3b169d0763a9492c83a8a406a16069f6 P c4adb558d3eb47b0ad3c5019472c2b28: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 23:27:30.071368 6151 raft_consensus.cc:504] T f5cee53ca97a47b9a77b8a2a912f1349 P 5bed7b22c6d240668345b7d0dc336ea9 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb85d6712d2242f9b99303387b4bbf87" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 41717 } } peers { permanent_uuid: "5bed7b22c6d240668345b7d0dc336ea9" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 45643 } } peers { permanent_uuid: "d8998ef4c11346fb89defef6f414514b" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 42325 } }
I20240503 23:27:30.071453 6238 heartbeater.cc:503] Master 127.0.100.62:35103 was elected leader, sending a full tablet report...
I20240503 23:27:30.071960 6151 leader_election.cc:288] T f5cee53ca97a47b9a77b8a2a912f1349 P 5bed7b22c6d240668345b7d0dc336ea9 [CANDIDATE]: Term 2 election: Requested vote from peers bb85d6712d2242f9b99303387b4bbf87 (127.0.100.1:41717), d8998ef4c11346fb89defef6f414514b (127.0.100.3:42325)
I20240503 23:27:30.072775 6074 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f5cee53ca97a47b9a77b8a2a912f1349" candidate_uuid: "5bed7b22c6d240668345b7d0dc336ea9" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d8998ef4c11346fb89defef6f414514b"
I20240503 23:27:30.072957 6074 raft_consensus.cc:3037] T f5cee53ca97a47b9a77b8a2a912f1349 P d8998ef4c11346fb89defef6f414514b [term 1 FOLLOWER]: Advancing to term 2
W20240503 23:27:30.073549 5966 leader_election.cc:334] T f5cee53ca97a47b9a77b8a2a912f1349 P 5bed7b22c6d240668345b7d0dc336ea9 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer bb85d6712d2242f9b99303387b4bbf87 (127.0.100.1:41717): Network error: Client connection negotiation failed: client connection to 127.0.100.1:41717: connect: Connection refused (error 111)
I20240503 23:27:30.075256 6074 raft_consensus.cc:2454] T f5cee53ca97a47b9a77b8a2a912f1349 P d8998ef4c11346fb89defef6f414514b [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5bed7b22c6d240668345b7d0dc336ea9 in term 2.
I20240503 23:27:30.075671 6084 tablet_copy_service.cc:342] P d8998ef4c11346fb89defef6f414514b: Request end of tablet copy session c4adb558d3eb47b0ad3c5019472c2b28-3b169d0763a9492c83a8a406a16069f6 received from {username='slave'} at 127.0.0.1:58342
I20240503 23:27:30.075770 6084 tablet_copy_service.cc:433] P d8998ef4c11346fb89defef6f414514b: ending tablet copy session c4adb558d3eb47b0ad3c5019472c2b28-3b169d0763a9492c83a8a406a16069f6 on tablet 3b169d0763a9492c83a8a406a16069f6 with peer c4adb558d3eb47b0ad3c5019472c2b28
I20240503 23:27:30.075812 5966 leader_election.cc:302] T f5cee53ca97a47b9a77b8a2a912f1349 P 5bed7b22c6d240668345b7d0dc336ea9 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5bed7b22c6d240668345b7d0dc336ea9, d8998ef4c11346fb89defef6f414514b; no voters: bb85d6712d2242f9b99303387b4bbf87
I20240503 23:27:30.076133 6161 raft_consensus.cc:2781] T f5cee53ca97a47b9a77b8a2a912f1349 P 5bed7b22c6d240668345b7d0dc336ea9 [term 2 FOLLOWER]: Leader election won for term 2
I20240503 23:27:30.076278 6161 raft_consensus.cc:686] T f5cee53ca97a47b9a77b8a2a912f1349 P 5bed7b22c6d240668345b7d0dc336ea9 [term 2 LEADER]: Becoming Leader. State: Replica: 5bed7b22c6d240668345b7d0dc336ea9, State: Running, Role: LEADER
I20240503 23:27:30.076696 6161 consensus_queue.cc:237] T f5cee53ca97a47b9a77b8a2a912f1349 P 5bed7b22c6d240668345b7d0dc336ea9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb85d6712d2242f9b99303387b4bbf87" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 41717 } } peers { permanent_uuid: "5bed7b22c6d240668345b7d0dc336ea9" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 45643 } } peers { permanent_uuid: "d8998ef4c11346fb89defef6f414514b" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 42325 } }
I20240503 23:27:30.080071 5823 catalog_manager.cc:5484] T f5cee53ca97a47b9a77b8a2a912f1349 P 5bed7b22c6d240668345b7d0dc336ea9 reported cstate change: term changed from 1 to 2, leader changed from bb85d6712d2242f9b99303387b4bbf87 (127.0.100.1) to 5bed7b22c6d240668345b7d0dc336ea9 (127.0.100.2). New cstate: current_term: 2 leader_uuid: "5bed7b22c6d240668345b7d0dc336ea9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb85d6712d2242f9b99303387b4bbf87" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 41717 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5bed7b22c6d240668345b7d0dc336ea9" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 45643 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d8998ef4c11346fb89defef6f414514b" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 42325 } health_report { overall_health: UNKNOWN } } }
I20240503 23:27:30.104249 6250 ts_tablet_manager.cc:902] T f657a9c7f5244a13ad0a03c28cbf3f1f P c4adb558d3eb47b0ad3c5019472c2b28: Initiating tablet copy from peer 5bed7b22c6d240668345b7d0dc336ea9 (127.0.100.2:45643)
I20240503 23:27:30.105077 6250 tablet_copy_client.cc:316] T f657a9c7f5244a13ad0a03c28cbf3f1f P c4adb558d3eb47b0ad3c5019472c2b28: tablet copy: Beginning tablet copy session from remote peer at address 127.0.100.2:45643
I20240503 23:27:30.111033 6010 tablet_copy_service.cc:140] P 5bed7b22c6d240668345b7d0dc336ea9: Received BeginTabletCopySession request for tablet f657a9c7f5244a13ad0a03c28cbf3f1f from peer c4adb558d3eb47b0ad3c5019472c2b28 ({username='slave'} at 127.0.0.1:52634)
I20240503 23:27:30.111224 6010 tablet_copy_service.cc:161] P 5bed7b22c6d240668345b7d0dc336ea9: Beginning new tablet copy session on tablet f657a9c7f5244a13ad0a03c28cbf3f1f from peer c4adb558d3eb47b0ad3c5019472c2b28 at {username='slave'} at 127.0.0.1:52634: session id = c4adb558d3eb47b0ad3c5019472c2b28-f657a9c7f5244a13ad0a03c28cbf3f1f
I20240503 23:27:30.113510 6010 tablet_copy_source_session.cc:205] T f657a9c7f5244a13ad0a03c28cbf3f1f P 5bed7b22c6d240668345b7d0dc336ea9: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 23:27:30.114845 6250 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f657a9c7f5244a13ad0a03c28cbf3f1f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 23:27:30.119904 6250 tablet_copy_client.cc:791] T f657a9c7f5244a13ad0a03c28cbf3f1f P c4adb558d3eb47b0ad3c5019472c2b28: tablet copy: Starting download of 0 data blocks...
I20240503 23:27:30.120182 6250 tablet_copy_client.cc:655] T f657a9c7f5244a13ad0a03c28cbf3f1f P c4adb558d3eb47b0ad3c5019472c2b28: tablet copy: Starting download of 1 WAL segments...
I20240503 23:27:30.121882 6250 tablet_copy_client.cc:531] T f657a9c7f5244a13ad0a03c28cbf3f1f P c4adb558d3eb47b0ad3c5019472c2b28: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 23:27:30.124596 6250 tablet_bootstrap.cc:492] T f657a9c7f5244a13ad0a03c28cbf3f1f P c4adb558d3eb47b0ad3c5019472c2b28: Bootstrap starting.
I20240503 23:27:30.130216 6250 tablet_bootstrap.cc:492] T f657a9c7f5244a13ad0a03c28cbf3f1f P c4adb558d3eb47b0ad3c5019472c2b28: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 23:27:30.130570 6250 tablet_bootstrap.cc:492] T f657a9c7f5244a13ad0a03c28cbf3f1f P c4adb558d3eb47b0ad3c5019472c2b28: Bootstrap complete.
I20240503 23:27:30.130800 6250 ts_tablet_manager.cc:1367] T f657a9c7f5244a13ad0a03c28cbf3f1f P c4adb558d3eb47b0ad3c5019472c2b28: Time spent bootstrapping tablet: real 0.006s user 0.006s sys 0.001s
I20240503 23:27:30.131513 6250 raft_consensus.cc:348] T f657a9c7f5244a13ad0a03c28cbf3f1f P c4adb558d3eb47b0ad3c5019472c2b28 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "bb85d6712d2242f9b99303387b4bbf87" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 41717 } } peers { permanent_uuid: "5bed7b22c6d240668345b7d0dc336ea9" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 45643 } } peers { permanent_uuid: "d8998ef4c11346fb89defef6f414514b" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 42325 } } peers { permanent_uuid: "c4adb558d3eb47b0ad3c5019472c2b28" member_type: NON_VOTER last_known_addr { host: "127.0.100.4" port: 37269 } attrs { promote: true } }
I20240503 23:27:30.131722 6250 raft_consensus.cc:729] T f657a9c7f5244a13ad0a03c28cbf3f1f P c4adb558d3eb47b0ad3c5019472c2b28 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: c4adb558d3eb47b0ad3c5019472c2b28, State: Initialized, Role: LEARNER
I20240503 23:27:30.131880 6250 consensus_queue.cc:260] T f657a9c7f5244a13ad0a03c28cbf3f1f P c4adb558d3eb47b0ad3c5019472c2b28 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "bb85d6712d2242f9b99303387b4bbf87" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 41717 } } peers { permanent_uuid: "5bed7b22c6d240668345b7d0dc336ea9" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 45643 } } peers { permanent_uuid: "d8998ef4c11346fb89defef6f414514b" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 42325 } } peers { permanent_uuid: "c4adb558d3eb47b0ad3c5019472c2b28" member_type: NON_VOTER last_known_addr { host: "127.0.100.4" port: 37269 } attrs { promote: true } }
I20240503 23:27:30.132351 6250 ts_tablet_manager.cc:1398] T f657a9c7f5244a13ad0a03c28cbf3f1f P c4adb558d3eb47b0ad3c5019472c2b28: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 23:27:30.133270 6010 tablet_copy_service.cc:342] P 5bed7b22c6d240668345b7d0dc336ea9: Request end of tablet copy session c4adb558d3eb47b0ad3c5019472c2b28-f657a9c7f5244a13ad0a03c28cbf3f1f received from {username='slave'} at 127.0.0.1:52634
I20240503 23:27:30.133395 6010 tablet_copy_service.cc:433] P 5bed7b22c6d240668345b7d0dc336ea9: ending tablet copy session c4adb558d3eb47b0ad3c5019472c2b28-f657a9c7f5244a13ad0a03c28cbf3f1f on tablet f657a9c7f5244a13ad0a03c28cbf3f1f with peer c4adb558d3eb47b0ad3c5019472c2b28
I20240503 23:27:30.172513 6151 raft_consensus.cc:482] T 5a588f46c5ae427bbde5cf6a988cd824 P 5bed7b22c6d240668345b7d0dc336ea9 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 23:27:30.172822 6151 raft_consensus.cc:504] T 5a588f46c5ae427bbde5cf6a988cd824 P 5bed7b22c6d240668345b7d0dc336ea9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb85d6712d2242f9b99303387b4bbf87" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 41717 } } peers { permanent_uuid: "5bed7b22c6d240668345b7d0dc336ea9" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 45643 } } peers { permanent_uuid: "d8998ef4c11346fb89defef6f414514b" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 42325 } }
I20240503 23:27:30.173561 6151 leader_election.cc:288] T 5a588f46c5ae427bbde5cf6a988cd824 P 5bed7b22c6d240668345b7d0dc336ea9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers bb85d6712d2242f9b99303387b4bbf87 (127.0.100.1:41717), d8998ef4c11346fb89defef6f414514b (127.0.100.3:42325)
I20240503 23:27:30.174549 6074 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5a588f46c5ae427bbde5cf6a988cd824" candidate_uuid: "5bed7b22c6d240668345b7d0dc336ea9" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d8998ef4c11346fb89defef6f414514b" is_pre_election: true
I20240503 23:27:30.174825 6074 raft_consensus.cc:2454] T 5a588f46c5ae427bbde5cf6a988cd824 P d8998ef4c11346fb89defef6f414514b [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5bed7b22c6d240668345b7d0dc336ea9 in term 1.
W20240503 23:27:30.175216 5966 leader_election.cc:334] T 5a588f46c5ae427bbde5cf6a988cd824 P 5bed7b22c6d240668345b7d0dc336ea9 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer bb85d6712d2242f9b99303387b4bbf87 (127.0.100.1:41717): Network error: Client connection negotiation failed: client connection to 127.0.100.1:41717: connect: Connection refused (error 111)
I20240503 23:27:30.175446 5966 leader_election.cc:302] T 5a588f46c5ae427bbde5cf6a988cd824 P 5bed7b22c6d240668345b7d0dc336ea9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5bed7b22c6d240668345b7d0dc336ea9, d8998ef4c11346fb89defef6f414514b; no voters: bb85d6712d2242f9b99303387b4bbf87
I20240503 23:27:30.175657 6151 raft_consensus.cc:2781] T 5a588f46c5ae427bbde5cf6a988cd824 P 5bed7b22c6d240668345b7d0dc336ea9 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240503 23:27:30.175735 6151 raft_consensus.cc:482] T 5a588f46c5ae427bbde5cf6a988cd824 P 5bed7b22c6d240668345b7d0dc336ea9 [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 23:27:30.175781 6151 raft_consensus.cc:3037] T 5a588f46c5ae427bbde5cf6a988cd824 P 5bed7b22c6d240668345b7d0dc336ea9 [term 1 FOLLOWER]: Advancing to term 2
I20240503 23:27:30.177876 6151 raft_consensus.cc:504] T 5a588f46c5ae427bbde5cf6a988cd824 P 5bed7b22c6d240668345b7d0dc336ea9 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb85d6712d2242f9b99303387b4bbf87" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 41717 } } peers { permanent_uuid: "5bed7b22c6d240668345b7d0dc336ea9" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 45643 } } peers { permanent_uuid: "d8998ef4c11346fb89defef6f414514b" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 42325 } }
I20240503 23:27:30.178453 6151 leader_election.cc:288] T 5a588f46c5ae427bbde5cf6a988cd824 P 5bed7b22c6d240668345b7d0dc336ea9 [CANDIDATE]: Term 2 election: Requested vote from peers bb85d6712d2242f9b99303387b4bbf87 (127.0.100.1:41717), d8998ef4c11346fb89defef6f414514b (127.0.100.3:42325)
I20240503 23:27:30.179266 6074 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5a588f46c5ae427bbde5cf6a988cd824" candidate_uuid: "5bed7b22c6d240668345b7d0dc336ea9" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d8998ef4c11346fb89defef6f414514b"
I20240503 23:27:30.179527 6074 raft_consensus.cc:3037] T 5a588f46c5ae427bbde5cf6a988cd824 P d8998ef4c11346fb89defef6f414514b [term 1 FOLLOWER]: Advancing to term 2
W20240503 23:27:30.179821 5966 leader_election.cc:334] T 5a588f46c5ae427bbde5cf6a988cd824 P 5bed7b22c6d240668345b7d0dc336ea9 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer bb85d6712d2242f9b99303387b4bbf87 (127.0.100.1:41717): Network error: Client connection negotiation failed: client connection to 127.0.100.1:41717: connect: Connection refused (error 111)
I20240503 23:27:30.181820 6074 raft_consensus.cc:2454] T 5a588f46c5ae427bbde5cf6a988cd824 P d8998ef4c11346fb89defef6f414514b [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5bed7b22c6d240668345b7d0dc336ea9 in term 2.
I20240503 23:27:30.182277 5966 leader_election.cc:302] T 5a588f46c5ae427bbde5cf6a988cd824 P 5bed7b22c6d240668345b7d0dc336ea9 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5bed7b22c6d240668345b7d0dc336ea9, d8998ef4c11346fb89defef6f414514b; no voters: bb85d6712d2242f9b99303387b4bbf87
I20240503 23:27:30.182551 6151 raft_consensus.cc:2781] T 5a588f46c5ae427bbde5cf6a988cd824 P 5bed7b22c6d240668345b7d0dc336ea9 [term 2 FOLLOWER]: Leader election won for term 2
I20240503 23:27:30.182701 6151 raft_consensus.cc:686] T 5a588f46c5ae427bbde5cf6a988cd824 P 5bed7b22c6d240668345b7d0dc336ea9 [term 2 LEADER]: Becoming Leader. State: Replica: 5bed7b22c6d240668345b7d0dc336ea9, State: Running, Role: LEADER
I20240503 23:27:30.182971 6151 consensus_queue.cc:237] T 5a588f46c5ae427bbde5cf6a988cd824 P 5bed7b22c6d240668345b7d0dc336ea9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb85d6712d2242f9b99303387b4bbf87" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 41717 } } peers { permanent_uuid: "5bed7b22c6d240668345b7d0dc336ea9" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 45643 } } peers { permanent_uuid: "d8998ef4c11346fb89defef6f414514b" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 42325 } }
I20240503 23:27:30.186067 5823 catalog_manager.cc:5484] T 5a588f46c5ae427bbde5cf6a988cd824 P 5bed7b22c6d240668345b7d0dc336ea9 reported cstate change: term changed from 1 to 2, leader changed from bb85d6712d2242f9b99303387b4bbf87 (127.0.100.1) to 5bed7b22c6d240668345b7d0dc336ea9 (127.0.100.2). New cstate: current_term: 2 leader_uuid: "5bed7b22c6d240668345b7d0dc336ea9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb85d6712d2242f9b99303387b4bbf87" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 41717 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5bed7b22c6d240668345b7d0dc336ea9" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 45643 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d8998ef4c11346fb89defef6f414514b" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 42325 } health_report { overall_health: UNKNOWN } } }
I20240503 23:27:30.508222 6213 raft_consensus.cc:1203] T f657a9c7f5244a13ad0a03c28cbf3f1f P c4adb558d3eb47b0ad3c5019472c2b28 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
W20240503 23:27:30.563829 5966 consensus_peers.cc:485] T f5cee53ca97a47b9a77b8a2a912f1349 P 5bed7b22c6d240668345b7d0dc336ea9 -> Peer bb85d6712d2242f9b99303387b4bbf87 (127.0.100.1:41717): Couldn't send request to peer bb85d6712d2242f9b99303387b4bbf87. Status: Network error: Client connection negotiation failed: client connection to 127.0.100.1:41717: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 23:27:30.572319 6213 raft_consensus.cc:1203] T 3b169d0763a9492c83a8a406a16069f6 P c4adb558d3eb47b0ad3c5019472c2b28 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240503 23:27:30.595990 6074 raft_consensus.cc:1261] T 5a588f46c5ae427bbde5cf6a988cd824 P d8998ef4c11346fb89defef6f414514b [term 2 FOLLOWER]: Refusing update from remote peer 5bed7b22c6d240668345b7d0dc336ea9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 2 index: 1. (index mismatch)
I20240503 23:27:30.596029 6073 raft_consensus.cc:1261] T f5cee53ca97a47b9a77b8a2a912f1349 P d8998ef4c11346fb89defef6f414514b [term 2 FOLLOWER]: Refusing update from remote peer 5bed7b22c6d240668345b7d0dc336ea9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 2 index: 1. (index mismatch)
I20240503 23:27:30.596684 6151 consensus_queue.cc:1035] T 5a588f46c5ae427bbde5cf6a988cd824 P 5bed7b22c6d240668345b7d0dc336ea9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d8998ef4c11346fb89defef6f414514b" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 42325 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 23:27:30.596755 6260 consensus_queue.cc:1035] T f5cee53ca97a47b9a77b8a2a912f1349 P 5bed7b22c6d240668345b7d0dc336ea9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d8998ef4c11346fb89defef6f414514b" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 42325 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240503 23:27:30.602988 5966 consensus_peers.cc:485] T 5a588f46c5ae427bbde5cf6a988cd824 P 5bed7b22c6d240668345b7d0dc336ea9 -> Peer bb85d6712d2242f9b99303387b4bbf87 (127.0.100.1:41717): Couldn't send request to peer bb85d6712d2242f9b99303387b4bbf87. Status: Network error: Client connection negotiation failed: client connection to 127.0.100.1:41717: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 23:27:30.863040 6260 consensus_queue.cc:579] T f657a9c7f5244a13ad0a03c28cbf3f1f P 5bed7b22c6d240668345b7d0dc336ea9 [LEADER]: Leader has been unable to successfully communicate with peer bb85d6712d2242f9b99303387b4bbf87 for more than 1 seconds (1.176s)
I20240503 23:27:30.892100 6161 raft_consensus.cc:1050] T f657a9c7f5244a13ad0a03c28cbf3f1f P 5bed7b22c6d240668345b7d0dc336ea9: attempting to promote NON_VOTER c4adb558d3eb47b0ad3c5019472c2b28 to VOTER
I20240503 23:27:30.893016 6161 consensus_queue.cc:237] T f657a9c7f5244a13ad0a03c28cbf3f1f P 5bed7b22c6d240668345b7d0dc336ea9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "bb85d6712d2242f9b99303387b4bbf87" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 41717 } } peers { permanent_uuid: "5bed7b22c6d240668345b7d0dc336ea9" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 45643 } } peers { permanent_uuid: "d8998ef4c11346fb89defef6f414514b" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 42325 } } peers { permanent_uuid: "c4adb558d3eb47b0ad3c5019472c2b28" member_type: VOTER last_known_addr { host: "127.0.100.4" port: 37269 } attrs { promote: false } }
I20240503 23:27:30.895758 6213 raft_consensus.cc:1261] T f657a9c7f5244a13ad0a03c28cbf3f1f P c4adb558d3eb47b0ad3c5019472c2b28 [term 1 LEARNER]: Refusing update from remote peer 5bed7b22c6d240668345b7d0dc336ea9: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 23:27:30.895767 6074 raft_consensus.cc:1261] T f657a9c7f5244a13ad0a03c28cbf3f1f P d8998ef4c11346fb89defef6f414514b [term 1 FOLLOWER]: Refusing update from remote peer 5bed7b22c6d240668345b7d0dc336ea9: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 23:27:30.896421 6161 consensus_queue.cc:1035] T f657a9c7f5244a13ad0a03c28cbf3f1f P 5bed7b22c6d240668345b7d0dc336ea9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c4adb558d3eb47b0ad3c5019472c2b28" member_type: VOTER last_known_addr { host: "127.0.100.4" port: 37269 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 23:27:30.896786 6273 consensus_queue.cc:1035] T f657a9c7f5244a13ad0a03c28cbf3f1f P 5bed7b22c6d240668345b7d0dc336ea9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d8998ef4c11346fb89defef6f414514b" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 42325 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
W20240503 23:27:30.897111 5966 consensus_peers.cc:485] T f657a9c7f5244a13ad0a03c28cbf3f1f P 5bed7b22c6d240668345b7d0dc336ea9 -> Peer bb85d6712d2242f9b99303387b4bbf87 (127.0.100.1:41717): Couldn't send request to peer bb85d6712d2242f9b99303387b4bbf87. Status: Network error: Client connection negotiation failed: client connection to 127.0.100.1:41717: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 23:27:30.899854 6161 raft_consensus.cc:2932] T f657a9c7f5244a13ad0a03c28cbf3f1f P 5bed7b22c6d240668345b7d0dc336ea9 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, c4adb558d3eb47b0ad3c5019472c2b28 (127.0.100.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "bb85d6712d2242f9b99303387b4bbf87" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 41717 } } peers { permanent_uuid: "5bed7b22c6d240668345b7d0dc336ea9" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 45643 } } peers { permanent_uuid: "d8998ef4c11346fb89defef6f414514b" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 42325 } } peers { permanent_uuid: "c4adb558d3eb47b0ad3c5019472c2b28" member_type: VOTER last_known_addr { host: "127.0.100.4" port: 37269 } attrs { promote: false } } }
I20240503 23:27:30.900620 6213 raft_consensus.cc:2932] T f657a9c7f5244a13ad0a03c28cbf3f1f P c4adb558d3eb47b0ad3c5019472c2b28 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, c4adb558d3eb47b0ad3c5019472c2b28 (127.0.100.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "bb85d6712d2242f9b99303387b4bbf87" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 41717 } } peers { permanent_uuid: "5bed7b22c6d240668345b7d0dc336ea9" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 45643 } } peers { permanent_uuid: "d8998ef4c11346fb89defef6f414514b" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 42325 } } peers { permanent_uuid: "c4adb558d3eb47b0ad3c5019472c2b28" member_type: VOTER last_known_addr { host: "127.0.100.4" port: 37269 } attrs { promote: false } } }
I20240503 23:27:30.904385 6074 raft_consensus.cc:2932] T f657a9c7f5244a13ad0a03c28cbf3f1f P d8998ef4c11346fb89defef6f414514b [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, c4adb558d3eb47b0ad3c5019472c2b28 (127.0.100.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "bb85d6712d2242f9b99303387b4bbf87" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 41717 } } peers { permanent_uuid: "5bed7b22c6d240668345b7d0dc336ea9" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 45643 } } peers { permanent_uuid: "d8998ef4c11346fb89defef6f414514b" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 42325 } } peers { permanent_uuid: "c4adb558d3eb47b0ad3c5019472c2b28" member_type: VOTER last_known_addr { host: "127.0.100.4" port: 37269 } attrs { promote: false } } }
I20240503 23:27:30.904896 5821 catalog_manager.cc:5484] T f657a9c7f5244a13ad0a03c28cbf3f1f P c4adb558d3eb47b0ad3c5019472c2b28 reported cstate change: config changed from index 2 to 3, c4adb558d3eb47b0ad3c5019472c2b28 (127.0.100.4) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "5bed7b22c6d240668345b7d0dc336ea9" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "bb85d6712d2242f9b99303387b4bbf87" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 41717 } } peers { permanent_uuid: "5bed7b22c6d240668345b7d0dc336ea9" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 45643 } } peers { permanent_uuid: "d8998ef4c11346fb89defef6f414514b" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 42325 } } peers { permanent_uuid: "c4adb558d3eb47b0ad3c5019472c2b28" member_type: VOTER last_known_addr { host: "127.0.100.4" port: 37269 } attrs { promote: false } } }
I20240503 23:27:30.912185 5999 consensus_queue.cc:237] T f657a9c7f5244a13ad0a03c28cbf3f1f P 5bed7b22c6d240668345b7d0dc336ea9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5bed7b22c6d240668345b7d0dc336ea9" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 45643 } } peers { permanent_uuid: "d8998ef4c11346fb89defef6f414514b" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 42325 } } peers { permanent_uuid: "c4adb558d3eb47b0ad3c5019472c2b28" member_type: VOTER last_known_addr { host: "127.0.100.4" port: 37269 } attrs { promote: false } }
I20240503 23:27:30.914391 6213 raft_consensus.cc:1261] T f657a9c7f5244a13ad0a03c28cbf3f1f P c4adb558d3eb47b0ad3c5019472c2b28 [term 1 FOLLOWER]: Refusing update from remote peer 5bed7b22c6d240668345b7d0dc336ea9: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 23:27:30.914886 6074 raft_consensus.cc:1261] T f657a9c7f5244a13ad0a03c28cbf3f1f P d8998ef4c11346fb89defef6f414514b [term 1 FOLLOWER]: Refusing update from remote peer 5bed7b22c6d240668345b7d0dc336ea9: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 23:27:30.915019 6260 consensus_queue.cc:1035] T f657a9c7f5244a13ad0a03c28cbf3f1f P 5bed7b22c6d240668345b7d0dc336ea9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c4adb558d3eb47b0ad3c5019472c2b28" member_type: VOTER last_known_addr { host: "127.0.100.4" port: 37269 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 23:27:30.915413 6161 consensus_queue.cc:1035] T f657a9c7f5244a13ad0a03c28cbf3f1f P 5bed7b22c6d240668345b7d0dc336ea9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d8998ef4c11346fb89defef6f414514b" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 42325 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 23:27:30.917614 6273 raft_consensus.cc:2932] T f657a9c7f5244a13ad0a03c28cbf3f1f P 5bed7b22c6d240668345b7d0dc336ea9 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER bb85d6712d2242f9b99303387b4bbf87 (127.0.100.1) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5bed7b22c6d240668345b7d0dc336ea9" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 45643 } } peers { permanent_uuid: "d8998ef4c11346fb89defef6f414514b" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 42325 } } peers { permanent_uuid: "c4adb558d3eb47b0ad3c5019472c2b28" member_type: VOTER last_known_addr { host: "127.0.100.4" port: 37269 } attrs { promote: false } } }
I20240503 23:27:30.918263 6074 raft_consensus.cc:2932] T f657a9c7f5244a13ad0a03c28cbf3f1f P d8998ef4c11346fb89defef6f414514b [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER bb85d6712d2242f9b99303387b4bbf87 (127.0.100.1) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5bed7b22c6d240668345b7d0dc336ea9" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 45643 } } peers { permanent_uuid: "d8998ef4c11346fb89defef6f414514b" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 42325 } } peers { permanent_uuid: "c4adb558d3eb47b0ad3c5019472c2b28" member_type: VOTER last_known_addr { host: "127.0.100.4" port: 37269 } attrs { promote: false } } }
I20240503 23:27:30.918319 6213 raft_consensus.cc:2932] T f657a9c7f5244a13ad0a03c28cbf3f1f P c4adb558d3eb47b0ad3c5019472c2b28 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER bb85d6712d2242f9b99303387b4bbf87 (127.0.100.1) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5bed7b22c6d240668345b7d0dc336ea9" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 45643 } } peers { permanent_uuid: "d8998ef4c11346fb89defef6f414514b" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 42325 } } peers { permanent_uuid: "c4adb558d3eb47b0ad3c5019472c2b28" member_type: VOTER last_known_addr { host: "127.0.100.4" port: 37269 } attrs { promote: false } } }
I20240503 23:27:30.921183 5810 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet f657a9c7f5244a13ad0a03c28cbf3f1f with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 23:27:30.922420 5823 catalog_manager.cc:5484] T f657a9c7f5244a13ad0a03c28cbf3f1f P 5bed7b22c6d240668345b7d0dc336ea9 reported cstate change: config changed from index 3 to 4, VOTER bb85d6712d2242f9b99303387b4bbf87 (127.0.100.1) evicted. New cstate: current_term: 1 leader_uuid: "5bed7b22c6d240668345b7d0dc336ea9" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5bed7b22c6d240668345b7d0dc336ea9" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 45643 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d8998ef4c11346fb89defef6f414514b" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 42325 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c4adb558d3eb47b0ad3c5019472c2b28" member_type: VOTER last_known_addr { host: "127.0.100.4" port: 37269 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
W20240503 23:27:30.928269 5810 catalog_manager.cc:4573] TS bb85d6712d2242f9b99303387b4bbf87 (127.0.100.1:41717): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet f657a9c7f5244a13ad0a03c28cbf3f1f: Network error: Client connection negotiation failed: client connection to 127.0.100.1:41717: connect: Connection refused (error 111)
W20240503 23:27:31.062433 5871 auto_rebalancer.cc:227] Could not retrieve cluster info: Not found: tserver bb85d6712d2242f9b99303387b4bbf87 not available for placement
I20240503 23:27:31.103752 6263 consensus_queue.cc:579] T 3b169d0763a9492c83a8a406a16069f6 P d8998ef4c11346fb89defef6f414514b [LEADER]: Leader has been unable to successfully communicate with peer bb85d6712d2242f9b99303387b4bbf87 for more than 1 seconds (1.456s)
I20240503 23:27:31.191987 6281 raft_consensus.cc:1050] T 3b169d0763a9492c83a8a406a16069f6 P d8998ef4c11346fb89defef6f414514b: attempting to promote NON_VOTER c4adb558d3eb47b0ad3c5019472c2b28 to VOTER
I20240503 23:27:31.192847 6281 consensus_queue.cc:237] T 3b169d0763a9492c83a8a406a16069f6 P d8998ef4c11346fb89defef6f414514b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "bb85d6712d2242f9b99303387b4bbf87" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 41717 } } peers { permanent_uuid: "5bed7b22c6d240668345b7d0dc336ea9" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 45643 } } peers { permanent_uuid: "d8998ef4c11346fb89defef6f414514b" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 42325 } } peers { permanent_uuid: "c4adb558d3eb47b0ad3c5019472c2b28" member_type: VOTER last_known_addr { host: "127.0.100.4" port: 37269 } attrs { promote: false } }
I20240503 23:27:31.195636 5999 raft_consensus.cc:1261] T 3b169d0763a9492c83a8a406a16069f6 P 5bed7b22c6d240668345b7d0dc336ea9 [term 1 FOLLOWER]: Refusing update from remote peer d8998ef4c11346fb89defef6f414514b: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 23:27:31.195636 6213 raft_consensus.cc:1261] T 3b169d0763a9492c83a8a406a16069f6 P c4adb558d3eb47b0ad3c5019472c2b28 [term 1 LEARNER]: Refusing update from remote peer d8998ef4c11346fb89defef6f414514b: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 23:27:31.196326 6284 consensus_queue.cc:1035] T 3b169d0763a9492c83a8a406a16069f6 P d8998ef4c11346fb89defef6f414514b [LEADER]: Connected to new peer: Peer: permanent_uuid: "c4adb558d3eb47b0ad3c5019472c2b28" member_type: VOTER last_known_addr { host: "127.0.100.4" port: 37269 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
W20240503 23:27:31.196632 6040 consensus_peers.cc:485] T 3b169d0763a9492c83a8a406a16069f6 P d8998ef4c11346fb89defef6f414514b -> Peer bb85d6712d2242f9b99303387b4bbf87 (127.0.100.1:41717): Couldn't send request to peer bb85d6712d2242f9b99303387b4bbf87. Status: Network error: Client connection negotiation failed: client connection to 127.0.100.1:41717: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 23:27:31.196652 6282 consensus_queue.cc:1035] T 3b169d0763a9492c83a8a406a16069f6 P d8998ef4c11346fb89defef6f414514b [LEADER]: Connected to new peer: Peer: permanent_uuid: "5bed7b22c6d240668345b7d0dc336ea9" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 45643 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 23:27:31.199800 6282 raft_consensus.cc:2932] T 3b169d0763a9492c83a8a406a16069f6 P d8998ef4c11346fb89defef6f414514b [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, c4adb558d3eb47b0ad3c5019472c2b28 (127.0.100.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "bb85d6712d2242f9b99303387b4bbf87" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 41717 } } peers { permanent_uuid: "5bed7b22c6d240668345b7d0dc336ea9" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 45643 } } peers { permanent_uuid: "d8998ef4c11346fb89defef6f414514b" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 42325 } } peers { permanent_uuid: "c4adb558d3eb47b0ad3c5019472c2b28" member_type: VOTER last_known_addr { host: "127.0.100.4" port: 37269 } attrs { promote: false } } }
I20240503 23:27:31.200783 6213 raft_consensus.cc:2932] T 3b169d0763a9492c83a8a406a16069f6 P c4adb558d3eb47b0ad3c5019472c2b28 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, c4adb558d3eb47b0ad3c5019472c2b28 (127.0.100.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "bb85d6712d2242f9b99303387b4bbf87" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 41717 } } peers { permanent_uuid: "5bed7b22c6d240668345b7d0dc336ea9" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 45643 } } peers { permanent_uuid: "d8998ef4c11346fb89defef6f414514b" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 42325 } } peers { permanent_uuid: "c4adb558d3eb47b0ad3c5019472c2b28" member_type: VOTER last_known_addr { host: "127.0.100.4" port: 37269 } attrs { promote: false } } }
I20240503 23:27:31.205164 5999 raft_consensus.cc:2932] T 3b169d0763a9492c83a8a406a16069f6 P 5bed7b22c6d240668345b7d0dc336ea9 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, c4adb558d3eb47b0ad3c5019472c2b28 (127.0.100.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "bb85d6712d2242f9b99303387b4bbf87" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 41717 } } peers { permanent_uuid: "5bed7b22c6d240668345b7d0dc336ea9" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 45643 } } peers { permanent_uuid: "d8998ef4c11346fb89defef6f414514b" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 42325 } } peers { permanent_uuid: "c4adb558d3eb47b0ad3c5019472c2b28" member_type: VOTER last_known_addr { host: "127.0.100.4" port: 37269 } attrs { promote: false } } }
I20240503 23:27:31.206084 5822 catalog_manager.cc:5484] T 3b169d0763a9492c83a8a406a16069f6 P c4adb558d3eb47b0ad3c5019472c2b28 reported cstate change: config changed from index 2 to 3, c4adb558d3eb47b0ad3c5019472c2b28 (127.0.100.4) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "d8998ef4c11346fb89defef6f414514b" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "bb85d6712d2242f9b99303387b4bbf87" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 41717 } } peers { permanent_uuid: "5bed7b22c6d240668345b7d0dc336ea9" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 45643 } } peers { permanent_uuid: "d8998ef4c11346fb89defef6f414514b" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 42325 } } peers { permanent_uuid: "c4adb558d3eb47b0ad3c5019472c2b28" member_type: VOTER last_known_addr { host: "127.0.100.4" port: 37269 } attrs { promote: false } } }
I20240503 23:27:31.211429 6074 consensus_queue.cc:237] T 3b169d0763a9492c83a8a406a16069f6 P d8998ef4c11346fb89defef6f414514b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5bed7b22c6d240668345b7d0dc336ea9" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 45643 } } peers { permanent_uuid: "d8998ef4c11346fb89defef6f414514b" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 42325 } } peers { permanent_uuid: "c4adb558d3eb47b0ad3c5019472c2b28" member_type: VOTER last_known_addr { host: "127.0.100.4" port: 37269 } attrs { promote: false } }
I20240503 23:27:31.213605 5999 raft_consensus.cc:1261] T 3b169d0763a9492c83a8a406a16069f6 P 5bed7b22c6d240668345b7d0dc336ea9 [term 1 FOLLOWER]: Refusing update from remote peer d8998ef4c11346fb89defef6f414514b: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 23:27:31.213716 6213 raft_consensus.cc:1261] T 3b169d0763a9492c83a8a406a16069f6 P c4adb558d3eb47b0ad3c5019472c2b28 [term 1 FOLLOWER]: Refusing update from remote peer d8998ef4c11346fb89defef6f414514b: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 23:27:31.214121 6281 consensus_queue.cc:1035] T 3b169d0763a9492c83a8a406a16069f6 P d8998ef4c11346fb89defef6f414514b [LEADER]: Connected to new peer: Peer: permanent_uuid: "5bed7b22c6d240668345b7d0dc336ea9" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 45643 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 23:27:31.214375 6284 consensus_queue.cc:1035] T 3b169d0763a9492c83a8a406a16069f6 P d8998ef4c11346fb89defef6f414514b [LEADER]: Connected to new peer: Peer: permanent_uuid: "c4adb558d3eb47b0ad3c5019472c2b28" member_type: VOTER last_known_addr { host: "127.0.100.4" port: 37269 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 23:27:31.216825 6282 raft_consensus.cc:2932] T 3b169d0763a9492c83a8a406a16069f6 P d8998ef4c11346fb89defef6f414514b [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER bb85d6712d2242f9b99303387b4bbf87 (127.0.100.1) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5bed7b22c6d240668345b7d0dc336ea9" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 45643 } } peers { permanent_uuid: "d8998ef4c11346fb89defef6f414514b" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 42325 } } peers { permanent_uuid: "c4adb558d3eb47b0ad3c5019472c2b28" member_type: VOTER last_known_addr { host: "127.0.100.4" port: 37269 } attrs { promote: false } } }
I20240503 23:27:31.217425 6213 raft_consensus.cc:2932] T 3b169d0763a9492c83a8a406a16069f6 P c4adb558d3eb47b0ad3c5019472c2b28 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER bb85d6712d2242f9b99303387b4bbf87 (127.0.100.1) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5bed7b22c6d240668345b7d0dc336ea9" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 45643 } } peers { permanent_uuid: "d8998ef4c11346fb89defef6f414514b" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 42325 } } peers { permanent_uuid: "c4adb558d3eb47b0ad3c5019472c2b28" member_type: VOTER last_known_addr { host: "127.0.100.4" port: 37269 } attrs { promote: false } } }
I20240503 23:27:31.218703 6161 consensus_queue.cc:579] T 5a588f46c5ae427bbde5cf6a988cd824 P 5bed7b22c6d240668345b7d0dc336ea9 [LEADER]: Leader has been unable to successfully communicate with peer bb85d6712d2242f9b99303387b4bbf87 for more than 1 seconds (1.035s)
I20240503 23:27:31.221053 5999 raft_consensus.cc:2932] T 3b169d0763a9492c83a8a406a16069f6 P 5bed7b22c6d240668345b7d0dc336ea9 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER bb85d6712d2242f9b99303387b4bbf87 (127.0.100.1) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5bed7b22c6d240668345b7d0dc336ea9" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 45643 } } peers { permanent_uuid: "d8998ef4c11346fb89defef6f414514b" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 42325 } } peers { permanent_uuid: "c4adb558d3eb47b0ad3c5019472c2b28" member_type: VOTER last_known_addr { host: "127.0.100.4" port: 37269 } attrs { promote: false } } }
I20240503 23:27:31.223284 6000 consensus_queue.cc:237] T 5a588f46c5ae427bbde5cf6a988cd824 P 5bed7b22c6d240668345b7d0dc336ea9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.1, Last appended by leader: 0, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "bb85d6712d2242f9b99303387b4bbf87" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 41717 } } peers { permanent_uuid: "5bed7b22c6d240668345b7d0dc336ea9" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 45643 } } peers { permanent_uuid: "d8998ef4c11346fb89defef6f414514b" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 42325 } } peers { permanent_uuid: "c4adb558d3eb47b0ad3c5019472c2b28" member_type: NON_VOTER last_known_addr { host: "127.0.100.4" port: 37269 } attrs { promote: true } }
I20240503 23:27:31.225912 5821 catalog_manager.cc:5484] T 3b169d0763a9492c83a8a406a16069f6 P c4adb558d3eb47b0ad3c5019472c2b28 reported cstate change: config changed from index 3 to 4, VOTER bb85d6712d2242f9b99303387b4bbf87 (127.0.100.1) evicted. New cstate: current_term: 1 leader_uuid: "d8998ef4c11346fb89defef6f414514b" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5bed7b22c6d240668345b7d0dc336ea9" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 45643 } } peers { permanent_uuid: "d8998ef4c11346fb89defef6f414514b" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 42325 } } peers { permanent_uuid: "c4adb558d3eb47b0ad3c5019472c2b28" member_type: VOTER last_known_addr { host: "127.0.100.4" port: 37269 } attrs { promote: false } } }
I20240503 23:27:31.226728 6074 raft_consensus.cc:1261] T 5a588f46c5ae427bbde5cf6a988cd824 P d8998ef4c11346fb89defef6f414514b [term 2 FOLLOWER]: Refusing update from remote peer 5bed7b22c6d240668345b7d0dc336ea9: Log matching property violated. Preceding OpId in replica: term: 2 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
W20240503 23:27:31.226926 5966 consensus_peers.cc:485] T 5a588f46c5ae427bbde5cf6a988cd824 P 5bed7b22c6d240668345b7d0dc336ea9 -> Peer bb85d6712d2242f9b99303387b4bbf87 (127.0.100.1:41717): Couldn't send request to peer bb85d6712d2242f9b99303387b4bbf87. Status: Network error: Client connection negotiation failed: client connection to 127.0.100.1:41717: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 23:27:31.227326 5963 consensus_peers.cc:485] T 5a588f46c5ae427bbde5cf6a988cd824 P 5bed7b22c6d240668345b7d0dc336ea9 -> Peer c4adb558d3eb47b0ad3c5019472c2b28 (127.0.100.4:37269): Couldn't send request to peer c4adb558d3eb47b0ad3c5019472c2b28. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 5a588f46c5ae427bbde5cf6a988cd824. This is attempt 1: this message will repeat every 5th retry.
I20240503 23:27:31.227427 6161 consensus_queue.cc:1035] T 5a588f46c5ae427bbde5cf6a988cd824 P 5bed7b22c6d240668345b7d0dc336ea9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d8998ef4c11346fb89defef6f414514b" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 42325 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 23:27:31.229638 6260 raft_consensus.cc:2932] T 5a588f46c5ae427bbde5cf6a988cd824 P 5bed7b22c6d240668345b7d0dc336ea9 [term 2 LEADER]: Committing config change with OpId 2.2: config changed from index -1 to 2, NON_VOTER c4adb558d3eb47b0ad3c5019472c2b28 (127.0.100.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "bb85d6712d2242f9b99303387b4bbf87" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 41717 } } peers { permanent_uuid: "5bed7b22c6d240668345b7d0dc336ea9" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 45643 } } peers { permanent_uuid: "d8998ef4c11346fb89defef6f414514b" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 42325 } } peers { permanent_uuid: "c4adb558d3eb47b0ad3c5019472c2b28" member_type: NON_VOTER last_known_addr { host: "127.0.100.4" port: 37269 } attrs { promote: true } } }
I20240503 23:27:31.230239 6074 raft_consensus.cc:2932] T 5a588f46c5ae427bbde5cf6a988cd824 P d8998ef4c11346fb89defef6f414514b [term 2 FOLLOWER]: Committing config change with OpId 2.2: config changed from index -1 to 2, NON_VOTER c4adb558d3eb47b0ad3c5019472c2b28 (127.0.100.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "bb85d6712d2242f9b99303387b4bbf87" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 41717 } } peers { permanent_uuid: "5bed7b22c6d240668345b7d0dc336ea9" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 45643 } } peers { permanent_uuid: "d8998ef4c11346fb89defef6f414514b" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 42325 } } peers { permanent_uuid: "c4adb558d3eb47b0ad3c5019472c2b28" member_type: NON_VOTER last_known_addr { host: "127.0.100.4" port: 37269 } attrs { promote: true } } }
I20240503 23:27:31.233071 5810 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 3b169d0763a9492c83a8a406a16069f6 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 23:27:31.234241 5810 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 5a588f46c5ae427bbde5cf6a988cd824 with cas_config_opid_index -1: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20240503 23:27:31.235608 5821 catalog_manager.cc:5484] T 5a588f46c5ae427bbde5cf6a988cd824 P 5bed7b22c6d240668345b7d0dc336ea9 reported cstate change: config changed from index -1 to 2, NON_VOTER c4adb558d3eb47b0ad3c5019472c2b28 (127.0.100.4) added. New cstate: current_term: 2 leader_uuid: "5bed7b22c6d240668345b7d0dc336ea9" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "bb85d6712d2242f9b99303387b4bbf87" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 41717 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5bed7b22c6d240668345b7d0dc336ea9" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 45643 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d8998ef4c11346fb89defef6f414514b" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 42325 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c4adb558d3eb47b0ad3c5019472c2b28" member_type: NON_VOTER last_known_addr { host: "127.0.100.4" port: 37269 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 23:27:31.393240 6161 consensus_queue.cc:579] T f5cee53ca97a47b9a77b8a2a912f1349 P 5bed7b22c6d240668345b7d0dc336ea9 [LEADER]: Leader has been unable to successfully communicate with peer bb85d6712d2242f9b99303387b4bbf87 for more than 1 seconds (1.316s)
I20240503 23:27:31.397634 6000 consensus_queue.cc:237] T f5cee53ca97a47b9a77b8a2a912f1349 P 5bed7b22c6d240668345b7d0dc336ea9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.1, Last appended by leader: 0, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "bb85d6712d2242f9b99303387b4bbf87" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 41717 } } peers { permanent_uuid: "5bed7b22c6d240668345b7d0dc336ea9" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 45643 } } peers { permanent_uuid: "d8998ef4c11346fb89defef6f414514b" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 42325 } } peers { permanent_uuid: "c4adb558d3eb47b0ad3c5019472c2b28" member_type: NON_VOTER last_known_addr { host: "127.0.100.4" port: 37269 } attrs { promote: true } }
I20240503 23:27:31.400130 6074 raft_consensus.cc:1261] T f5cee53ca97a47b9a77b8a2a912f1349 P d8998ef4c11346fb89defef6f414514b [term 2 FOLLOWER]: Refusing update from remote peer 5bed7b22c6d240668345b7d0dc336ea9: Log matching property violated. Preceding OpId in replica: term: 2 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
W20240503 23:27:31.400336 5963 consensus_peers.cc:485] T f5cee53ca97a47b9a77b8a2a912f1349 P 5bed7b22c6d240668345b7d0dc336ea9 -> Peer c4adb558d3eb47b0ad3c5019472c2b28 (127.0.100.4:37269): Couldn't send request to peer c4adb558d3eb47b0ad3c5019472c2b28. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f5cee53ca97a47b9a77b8a2a912f1349. This is attempt 1: this message will repeat every 5th retry.
W20240503 23:27:31.401005 5966 consensus_peers.cc:485] T f5cee53ca97a47b9a77b8a2a912f1349 P 5bed7b22c6d240668345b7d0dc336ea9 -> Peer bb85d6712d2242f9b99303387b4bbf87 (127.0.100.1:41717): Couldn't send request to peer bb85d6712d2242f9b99303387b4bbf87. Status: Network error: Client connection negotiation failed: client connection to 127.0.100.1:41717: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 23:27:31.401111 6260 consensus_queue.cc:1035] T f5cee53ca97a47b9a77b8a2a912f1349 P 5bed7b22c6d240668345b7d0dc336ea9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d8998ef4c11346fb89defef6f414514b" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 42325 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 23:27:31.403745 6161 raft_consensus.cc:2932] T f5cee53ca97a47b9a77b8a2a912f1349 P 5bed7b22c6d240668345b7d0dc336ea9 [term 2 LEADER]: Committing config change with OpId 2.2: config changed from index -1 to 2, NON_VOTER c4adb558d3eb47b0ad3c5019472c2b28 (127.0.100.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "bb85d6712d2242f9b99303387b4bbf87" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 41717 } } peers { permanent_uuid: "5bed7b22c6d240668345b7d0dc336ea9" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 45643 } } peers { permanent_uuid: "d8998ef4c11346fb89defef6f414514b" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 42325 } } peers { permanent_uuid: "c4adb558d3eb47b0ad3c5019472c2b28" member_type: NON_VOTER last_known_addr { host: "127.0.100.4" port: 37269 } attrs { promote: true } } }
I20240503 23:27:31.404443 6074 raft_consensus.cc:2932] T f5cee53ca97a47b9a77b8a2a912f1349 P d8998ef4c11346fb89defef6f414514b [term 2 FOLLOWER]: Committing config change with OpId 2.2: config changed from index -1 to 2, NON_VOTER c4adb558d3eb47b0ad3c5019472c2b28 (127.0.100.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "bb85d6712d2242f9b99303387b4bbf87" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 41717 } } peers { permanent_uuid: "5bed7b22c6d240668345b7d0dc336ea9" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 45643 } } peers { permanent_uuid: "d8998ef4c11346fb89defef6f414514b" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 42325 } } peers { permanent_uuid: "c4adb558d3eb47b0ad3c5019472c2b28" member_type: NON_VOTER last_known_addr { host: "127.0.100.4" port: 37269 } attrs { promote: true } } }
I20240503 23:27:31.407035 5810 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet f5cee53ca97a47b9a77b8a2a912f1349 with cas_config_opid_index -1: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20240503 23:27:31.408644 5823 catalog_manager.cc:5484] T f5cee53ca97a47b9a77b8a2a912f1349 P 5bed7b22c6d240668345b7d0dc336ea9 reported cstate change: config changed from index -1 to 2, NON_VOTER c4adb558d3eb47b0ad3c5019472c2b28 (127.0.100.4) added. New cstate: current_term: 2 leader_uuid: "5bed7b22c6d240668345b7d0dc336ea9" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "bb85d6712d2242f9b99303387b4bbf87" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 41717 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5bed7b22c6d240668345b7d0dc336ea9" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 45643 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d8998ef4c11346fb89defef6f414514b" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 42325 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c4adb558d3eb47b0ad3c5019472c2b28" member_type: NON_VOTER last_known_addr { host: "127.0.100.4" port: 37269 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 23:27:31.790464 6287 ts_tablet_manager.cc:902] T 5a588f46c5ae427bbde5cf6a988cd824 P c4adb558d3eb47b0ad3c5019472c2b28: Initiating tablet copy from peer 5bed7b22c6d240668345b7d0dc336ea9 (127.0.100.2:45643)
I20240503 23:27:31.791144 6287 tablet_copy_client.cc:316] T 5a588f46c5ae427bbde5cf6a988cd824 P c4adb558d3eb47b0ad3c5019472c2b28: tablet copy: Beginning tablet copy session from remote peer at address 127.0.100.2:45643
I20240503 23:27:31.791887 6010 tablet_copy_service.cc:140] P 5bed7b22c6d240668345b7d0dc336ea9: Received BeginTabletCopySession request for tablet 5a588f46c5ae427bbde5cf6a988cd824 from peer c4adb558d3eb47b0ad3c5019472c2b28 ({username='slave'} at 127.0.0.1:52634)
I20240503 23:27:31.792047 6010 tablet_copy_service.cc:161] P 5bed7b22c6d240668345b7d0dc336ea9: Beginning new tablet copy session on tablet 5a588f46c5ae427bbde5cf6a988cd824 from peer c4adb558d3eb47b0ad3c5019472c2b28 at {username='slave'} at 127.0.0.1:52634: session id = c4adb558d3eb47b0ad3c5019472c2b28-5a588f46c5ae427bbde5cf6a988cd824
I20240503 23:27:31.794224 6010 tablet_copy_source_session.cc:205] T 5a588f46c5ae427bbde5cf6a988cd824 P 5bed7b22c6d240668345b7d0dc336ea9: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 23:27:31.795621 6287 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5a588f46c5ae427bbde5cf6a988cd824. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 23:27:31.801118 6287 tablet_copy_client.cc:791] T 5a588f46c5ae427bbde5cf6a988cd824 P c4adb558d3eb47b0ad3c5019472c2b28: tablet copy: Starting download of 0 data blocks...
I20240503 23:27:31.801443 6287 tablet_copy_client.cc:655] T 5a588f46c5ae427bbde5cf6a988cd824 P c4adb558d3eb47b0ad3c5019472c2b28: tablet copy: Starting download of 1 WAL segments...
I20240503 23:27:31.802748 6289 ts_tablet_manager.cc:902] T f5cee53ca97a47b9a77b8a2a912f1349 P c4adb558d3eb47b0ad3c5019472c2b28: Initiating tablet copy from peer 5bed7b22c6d240668345b7d0dc336ea9 (127.0.100.2:45643)
I20240503 23:27:31.803321 6287 tablet_copy_client.cc:531] T 5a588f46c5ae427bbde5cf6a988cd824 P c4adb558d3eb47b0ad3c5019472c2b28: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 23:27:31.803520 6289 tablet_copy_client.cc:316] T f5cee53ca97a47b9a77b8a2a912f1349 P c4adb558d3eb47b0ad3c5019472c2b28: tablet copy: Beginning tablet copy session from remote peer at address 127.0.100.2:45643
I20240503 23:27:31.804147 6010 tablet_copy_service.cc:140] P 5bed7b22c6d240668345b7d0dc336ea9: Received BeginTabletCopySession request for tablet f5cee53ca97a47b9a77b8a2a912f1349 from peer c4adb558d3eb47b0ad3c5019472c2b28 ({username='slave'} at 127.0.0.1:52634)
I20240503 23:27:31.804311 6010 tablet_copy_service.cc:161] P 5bed7b22c6d240668345b7d0dc336ea9: Beginning new tablet copy session on tablet f5cee53ca97a47b9a77b8a2a912f1349 from peer c4adb558d3eb47b0ad3c5019472c2b28 at {username='slave'} at 127.0.0.1:52634: session id = c4adb558d3eb47b0ad3c5019472c2b28-f5cee53ca97a47b9a77b8a2a912f1349
I20240503 23:27:31.806375 6010 tablet_copy_source_session.cc:205] T f5cee53ca97a47b9a77b8a2a912f1349 P 5bed7b22c6d240668345b7d0dc336ea9: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 23:27:31.806515 6287 tablet_bootstrap.cc:492] T 5a588f46c5ae427bbde5cf6a988cd824 P c4adb558d3eb47b0ad3c5019472c2b28: Bootstrap starting.
I20240503 23:27:31.807659 6289 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f5cee53ca97a47b9a77b8a2a912f1349. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 23:27:31.813310 6287 tablet_bootstrap.cc:492] T 5a588f46c5ae427bbde5cf6a988cd824 P c4adb558d3eb47b0ad3c5019472c2b28: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 23:27:31.813762 6287 tablet_bootstrap.cc:492] T 5a588f46c5ae427bbde5cf6a988cd824 P c4adb558d3eb47b0ad3c5019472c2b28: Bootstrap complete.
I20240503 23:27:31.814054 6287 ts_tablet_manager.cc:1367] T 5a588f46c5ae427bbde5cf6a988cd824 P c4adb558d3eb47b0ad3c5019472c2b28: Time spent bootstrapping tablet: real 0.008s user 0.008s sys 0.000s
I20240503 23:27:31.814707 6289 tablet_copy_client.cc:791] T f5cee53ca97a47b9a77b8a2a912f1349 P c4adb558d3eb47b0ad3c5019472c2b28: tablet copy: Starting download of 0 data blocks...
I20240503 23:27:31.814988 6289 tablet_copy_client.cc:655] T f5cee53ca97a47b9a77b8a2a912f1349 P c4adb558d3eb47b0ad3c5019472c2b28: tablet copy: Starting download of 1 WAL segments...
I20240503 23:27:31.814989 6287 raft_consensus.cc:348] T 5a588f46c5ae427bbde5cf6a988cd824 P c4adb558d3eb47b0ad3c5019472c2b28 [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "bb85d6712d2242f9b99303387b4bbf87" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 41717 } } peers { permanent_uuid: "5bed7b22c6d240668345b7d0dc336ea9" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 45643 } } peers { permanent_uuid: "d8998ef4c11346fb89defef6f414514b" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 42325 } } peers { permanent_uuid: "c4adb558d3eb47b0ad3c5019472c2b28" member_type: NON_VOTER last_known_addr { host: "127.0.100.4" port: 37269 } attrs { promote: true } }
I20240503 23:27:31.815279 6287 raft_consensus.cc:729] T 5a588f46c5ae427bbde5cf6a988cd824 P c4adb558d3eb47b0ad3c5019472c2b28 [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: c4adb558d3eb47b0ad3c5019472c2b28, State: Initialized, Role: LEARNER
I20240503 23:27:31.815515 6287 consensus_queue.cc:260] T 5a588f46c5ae427bbde5cf6a988cd824 P c4adb558d3eb47b0ad3c5019472c2b28 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "bb85d6712d2242f9b99303387b4bbf87" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 41717 } } peers { permanent_uuid: "5bed7b22c6d240668345b7d0dc336ea9" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 45643 } } peers { permanent_uuid: "d8998ef4c11346fb89defef6f414514b" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 42325 } } peers { permanent_uuid: "c4adb558d3eb47b0ad3c5019472c2b28" member_type: NON_VOTER last_known_addr { host: "127.0.100.4" port: 37269 } attrs { promote: true } }
I20240503 23:27:31.816445 6287 ts_tablet_manager.cc:1398] T 5a588f46c5ae427bbde5cf6a988cd824 P c4adb558d3eb47b0ad3c5019472c2b28: Time spent starting tablet: real 0.002s user 0.004s sys 0.000s
I20240503 23:27:31.816952 6289 tablet_copy_client.cc:531] T f5cee53ca97a47b9a77b8a2a912f1349 P c4adb558d3eb47b0ad3c5019472c2b28: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 23:27:31.817183 6010 tablet_copy_service.cc:342] P 5bed7b22c6d240668345b7d0dc336ea9: Request end of tablet copy session c4adb558d3eb47b0ad3c5019472c2b28-5a588f46c5ae427bbde5cf6a988cd824 received from {username='slave'} at 127.0.0.1:52634
I20240503 23:27:31.817324 6010 tablet_copy_service.cc:433] P 5bed7b22c6d240668345b7d0dc336ea9: ending tablet copy session c4adb558d3eb47b0ad3c5019472c2b28-5a588f46c5ae427bbde5cf6a988cd824 on tablet 5a588f46c5ae427bbde5cf6a988cd824 with peer c4adb558d3eb47b0ad3c5019472c2b28
I20240503 23:27:31.820528 6289 tablet_bootstrap.cc:492] T f5cee53ca97a47b9a77b8a2a912f1349 P c4adb558d3eb47b0ad3c5019472c2b28: Bootstrap starting.
I20240503 23:27:31.826256 6289 tablet_bootstrap.cc:492] T f5cee53ca97a47b9a77b8a2a912f1349 P c4adb558d3eb47b0ad3c5019472c2b28: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 23:27:31.826689 6289 tablet_bootstrap.cc:492] T f5cee53ca97a47b9a77b8a2a912f1349 P c4adb558d3eb47b0ad3c5019472c2b28: Bootstrap complete.
I20240503 23:27:31.826954 6289 ts_tablet_manager.cc:1367] T f5cee53ca97a47b9a77b8a2a912f1349 P c4adb558d3eb47b0ad3c5019472c2b28: Time spent bootstrapping tablet: real 0.007s user 0.007s sys 0.000s
I20240503 23:27:31.827700 6289 raft_consensus.cc:348] T f5cee53ca97a47b9a77b8a2a912f1349 P c4adb558d3eb47b0ad3c5019472c2b28 [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "bb85d6712d2242f9b99303387b4bbf87" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 41717 } } peers { permanent_uuid: "5bed7b22c6d240668345b7d0dc336ea9" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 45643 } } peers { permanent_uuid: "d8998ef4c11346fb89defef6f414514b" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 42325 } } peers { permanent_uuid: "c4adb558d3eb47b0ad3c5019472c2b28" member_type: NON_VOTER last_known_addr { host: "127.0.100.4" port: 37269 } attrs { promote: true } }
I20240503 23:27:31.827914 6289 raft_consensus.cc:729] T f5cee53ca97a47b9a77b8a2a912f1349 P c4adb558d3eb47b0ad3c5019472c2b28 [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: c4adb558d3eb47b0ad3c5019472c2b28, State: Initialized, Role: LEARNER
I20240503 23:27:31.828128 6289 consensus_queue.cc:260] T f5cee53ca97a47b9a77b8a2a912f1349 P c4adb558d3eb47b0ad3c5019472c2b28 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "bb85d6712d2242f9b99303387b4bbf87" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 41717 } } peers { permanent_uuid: "5bed7b22c6d240668345b7d0dc336ea9" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 45643 } } peers { permanent_uuid: "d8998ef4c11346fb89defef6f414514b" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 42325 } } peers { permanent_uuid: "c4adb558d3eb47b0ad3c5019472c2b28" member_type: NON_VOTER last_known_addr { host: "127.0.100.4" port: 37269 } attrs { promote: true } }
I20240503 23:27:31.828665 6289 ts_tablet_manager.cc:1398] T f5cee53ca97a47b9a77b8a2a912f1349 P c4adb558d3eb47b0ad3c5019472c2b28: Time spent starting tablet: real 0.002s user 0.004s sys 0.000s
I20240503 23:27:31.829551 6010 tablet_copy_service.cc:342] P 5bed7b22c6d240668345b7d0dc336ea9: Request end of tablet copy session c4adb558d3eb47b0ad3c5019472c2b28-f5cee53ca97a47b9a77b8a2a912f1349 received from {username='slave'} at 127.0.0.1:52634
I20240503 23:27:31.829671 6010 tablet_copy_service.cc:433] P 5bed7b22c6d240668345b7d0dc336ea9: ending tablet copy session c4adb558d3eb47b0ad3c5019472c2b28-f5cee53ca97a47b9a77b8a2a912f1349 on tablet f5cee53ca97a47b9a77b8a2a912f1349 with peer c4adb558d3eb47b0ad3c5019472c2b28
W20240503 23:27:32.063750 5871 auto_rebalancer.cc:227] Could not retrieve cluster info: Not found: tserver bb85d6712d2242f9b99303387b4bbf87 not available for placement
W20240503 23:27:32.066380 5810 catalog_manager.cc:4573] TS bb85d6712d2242f9b99303387b4bbf87 (127.0.100.1:41717): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet f657a9c7f5244a13ad0a03c28cbf3f1f: Network error: Client connection negotiation failed: client connection to 127.0.100.1:41717: connect: Connection refused (error 111)
I20240503 23:27:32.311054 6213 raft_consensus.cc:1203] T 5a588f46c5ae427bbde5cf6a988cd824 P c4adb558d3eb47b0ad3c5019472c2b28 [term 2 LEARNER]: Deduplicated request from leader. Original: 2.1->[2.2-2.2] Dedup: 2.2->[]
I20240503 23:27:32.314790 6213 raft_consensus.cc:1203] T f5cee53ca97a47b9a77b8a2a912f1349 P c4adb558d3eb47b0ad3c5019472c2b28 [term 2 LEARNER]: Deduplicated request from leader. Original: 2.1->[2.2-2.2] Dedup: 2.2->[]
I20240503 23:27:32.624151 6161 consensus_queue.cc:579] T 5a588f46c5ae427bbde5cf6a988cd824 P 5bed7b22c6d240668345b7d0dc336ea9 [LEADER]: Leader has been unable to successfully communicate with peer bb85d6712d2242f9b99303387b4bbf87 for more than 1 seconds (1.400s)
I20240503 23:27:32.712716 6161 consensus_queue.cc:579] T f5cee53ca97a47b9a77b8a2a912f1349 P 5bed7b22c6d240668345b7d0dc336ea9 [LEADER]: Leader has been unable to successfully communicate with peer bb85d6712d2242f9b99303387b4bbf87 for more than 1 seconds (1.315s)
I20240503 23:27:32.754484 6295 raft_consensus.cc:1050] T 5a588f46c5ae427bbde5cf6a988cd824 P 5bed7b22c6d240668345b7d0dc336ea9: attempting to promote NON_VOTER c4adb558d3eb47b0ad3c5019472c2b28 to VOTER
I20240503 23:27:32.755434 6295 consensus_queue.cc:237] T 5a588f46c5ae427bbde5cf6a988cd824 P 5bed7b22c6d240668345b7d0dc336ea9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 0, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "bb85d6712d2242f9b99303387b4bbf87" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 41717 } } peers { permanent_uuid: "5bed7b22c6d240668345b7d0dc336ea9" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 45643 } } peers { permanent_uuid: "d8998ef4c11346fb89defef6f414514b" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 42325 } } peers { permanent_uuid: "c4adb558d3eb47b0ad3c5019472c2b28" member_type: VOTER last_known_addr { host: "127.0.100.4" port: 37269 } attrs { promote: false } }
I20240503 23:27:32.757941 6074 raft_consensus.cc:1261] T 5a588f46c5ae427bbde5cf6a988cd824 P d8998ef4c11346fb89defef6f414514b [term 2 FOLLOWER]: Refusing update from remote peer 5bed7b22c6d240668345b7d0dc336ea9: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20240503 23:27:32.757936 6213 raft_consensus.cc:1261] T 5a588f46c5ae427bbde5cf6a988cd824 P c4adb558d3eb47b0ad3c5019472c2b28 [term 2 LEARNER]: Refusing update from remote peer 5bed7b22c6d240668345b7d0dc336ea9: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20240503 23:27:32.758594 6295 consensus_queue.cc:1035] T 5a588f46c5ae427bbde5cf6a988cd824 P 5bed7b22c6d240668345b7d0dc336ea9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d8998ef4c11346fb89defef6f414514b" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 42325 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 23:27:32.758852 6294 consensus_queue.cc:1035] T 5a588f46c5ae427bbde5cf6a988cd824 P 5bed7b22c6d240668345b7d0dc336ea9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c4adb558d3eb47b0ad3c5019472c2b28" member_type: VOTER last_known_addr { host: "127.0.100.4" port: 37269 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
W20240503 23:27:32.759539 5966 consensus_peers.cc:485] T 5a588f46c5ae427bbde5cf6a988cd824 P 5bed7b22c6d240668345b7d0dc336ea9 -> Peer bb85d6712d2242f9b99303387b4bbf87 (127.0.100.1:41717): Couldn't send request to peer bb85d6712d2242f9b99303387b4bbf87. Status: Network error: Client connection negotiation failed: client connection to 127.0.100.1:41717: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 23:27:32.762176 6295 raft_consensus.cc:2932] T 5a588f46c5ae427bbde5cf6a988cd824 P 5bed7b22c6d240668345b7d0dc336ea9 [term 2 LEADER]: Committing config change with OpId 2.3: config changed from index 2 to 3, c4adb558d3eb47b0ad3c5019472c2b28 (127.0.100.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "bb85d6712d2242f9b99303387b4bbf87" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 41717 } } peers { permanent_uuid: "5bed7b22c6d240668345b7d0dc336ea9" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 45643 } } peers { permanent_uuid: "d8998ef4c11346fb89defef6f414514b" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 42325 } } peers { permanent_uuid: "c4adb558d3eb47b0ad3c5019472c2b28" member_type: VOTER last_known_addr { host: "127.0.100.4" port: 37269 } attrs { promote: false } } }
I20240503 23:27:32.762945 6074 raft_consensus.cc:2932] T 5a588f46c5ae427bbde5cf6a988cd824 P d8998ef4c11346fb89defef6f414514b [term 2 FOLLOWER]: Committing config change with OpId 2.3: config changed from index 2 to 3, c4adb558d3eb47b0ad3c5019472c2b28 (127.0.100.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "bb85d6712d2242f9b99303387b4bbf87" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 41717 } } peers { permanent_uuid: "5bed7b22c6d240668345b7d0dc336ea9" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 45643 } } peers { permanent_uuid: "d8998ef4c11346fb89defef6f414514b" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 42325 } } peers { permanent_uuid: "c4adb558d3eb47b0ad3c5019472c2b28" member_type: VOTER last_known_addr { host: "127.0.100.4" port: 37269 } attrs { promote: false } } }
I20240503 23:27:32.767076 6213 raft_consensus.cc:2932] T 5a588f46c5ae427bbde5cf6a988cd824 P c4adb558d3eb47b0ad3c5019472c2b28 [term 2 FOLLOWER]: Committing config change with OpId 2.3: config changed from index 2 to 3, c4adb558d3eb47b0ad3c5019472c2b28 (127.0.100.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "bb85d6712d2242f9b99303387b4bbf87" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 41717 } } peers { permanent_uuid: "5bed7b22c6d240668345b7d0dc336ea9" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 45643 } } peers { permanent_uuid: "d8998ef4c11346fb89defef6f414514b" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 42325 } } peers { permanent_uuid: "c4adb558d3eb47b0ad3c5019472c2b28" member_type: VOTER last_known_addr { host: "127.0.100.4" port: 37269 } attrs { promote: false } } }
I20240503 23:27:32.767372 5823 catalog_manager.cc:5484] T 5a588f46c5ae427bbde5cf6a988cd824 P d8998ef4c11346fb89defef6f414514b reported cstate change: config changed from index 2 to 3, c4adb558d3eb47b0ad3c5019472c2b28 (127.0.100.4) changed from NON_VOTER to VOTER. New cstate: current_term: 2 leader_uuid: "5bed7b22c6d240668345b7d0dc336ea9" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "bb85d6712d2242f9b99303387b4bbf87" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 41717 } } peers { permanent_uuid: "5bed7b22c6d240668345b7d0dc336ea9" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 45643 } } peers { permanent_uuid: "d8998ef4c11346fb89defef6f414514b" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 42325 } } peers { permanent_uuid: "c4adb558d3eb47b0ad3c5019472c2b28" member_type: VOTER last_known_addr { host: "127.0.100.4" port: 37269 } attrs { promote: false } } }
I20240503 23:27:32.775146 6000 consensus_queue.cc:237] T 5a588f46c5ae427bbde5cf6a988cd824 P 5bed7b22c6d240668345b7d0dc336ea9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.3, Last appended by leader: 0, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5bed7b22c6d240668345b7d0dc336ea9" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 45643 } } peers { permanent_uuid: "d8998ef4c11346fb89defef6f414514b" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 42325 } } peers { permanent_uuid: "c4adb558d3eb47b0ad3c5019472c2b28" member_type: VOTER last_known_addr { host: "127.0.100.4" port: 37269 } attrs { promote: false } }
I20240503 23:27:32.777308 6213 raft_consensus.cc:1261] T 5a588f46c5ae427bbde5cf6a988cd824 P c4adb558d3eb47b0ad3c5019472c2b28 [term 2 FOLLOWER]: Refusing update from remote peer 5bed7b22c6d240668345b7d0dc336ea9: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240503 23:27:32.777309 6074 raft_consensus.cc:1261] T 5a588f46c5ae427bbde5cf6a988cd824 P d8998ef4c11346fb89defef6f414514b [term 2 FOLLOWER]: Refusing update from remote peer 5bed7b22c6d240668345b7d0dc336ea9: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240503 23:27:32.777936 6294 consensus_queue.cc:1035] T 5a588f46c5ae427bbde5cf6a988cd824 P 5bed7b22c6d240668345b7d0dc336ea9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d8998ef4c11346fb89defef6f414514b" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 42325 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 23:27:32.778623 6294 consensus_queue.cc:1035] T 5a588f46c5ae427bbde5cf6a988cd824 P 5bed7b22c6d240668345b7d0dc336ea9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c4adb558d3eb47b0ad3c5019472c2b28" member_type: VOTER last_known_addr { host: "127.0.100.4" port: 37269 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 23:27:32.780480 6161 raft_consensus.cc:2932] T 5a588f46c5ae427bbde5cf6a988cd824 P 5bed7b22c6d240668345b7d0dc336ea9 [term 2 LEADER]: Committing config change with OpId 2.4: config changed from index 3 to 4, VOTER bb85d6712d2242f9b99303387b4bbf87 (127.0.100.1) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5bed7b22c6d240668345b7d0dc336ea9" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 45643 } } peers { permanent_uuid: "d8998ef4c11346fb89defef6f414514b" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 42325 } } peers { permanent_uuid: "c4adb558d3eb47b0ad3c5019472c2b28" member_type: VOTER last_known_addr { host: "127.0.100.4" port: 37269 } attrs { promote: false } } }
I20240503 23:27:32.781106 6074 raft_consensus.cc:2932] T 5a588f46c5ae427bbde5cf6a988cd824 P d8998ef4c11346fb89defef6f414514b [term 2 FOLLOWER]: Committing config change with OpId 2.4: config changed from index 3 to 4, VOTER bb85d6712d2242f9b99303387b4bbf87 (127.0.100.1) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5bed7b22c6d240668345b7d0dc336ea9" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 45643 } } peers { permanent_uuid: "d8998ef4c11346fb89defef6f414514b" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 42325 } } peers { permanent_uuid: "c4adb558d3eb47b0ad3c5019472c2b28" member_type: VOTER last_known_addr { host: "127.0.100.4" port: 37269 } attrs { promote: false } } }
I20240503 23:27:32.783773 5810 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 5a588f46c5ae427bbde5cf6a988cd824 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 23:27:32.784447 6213 raft_consensus.cc:2932] T 5a588f46c5ae427bbde5cf6a988cd824 P c4adb558d3eb47b0ad3c5019472c2b28 [term 2 FOLLOWER]: Committing config change with OpId 2.4: config changed from index 3 to 4, VOTER bb85d6712d2242f9b99303387b4bbf87 (127.0.100.1) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5bed7b22c6d240668345b7d0dc336ea9" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 45643 } } peers { permanent_uuid: "d8998ef4c11346fb89defef6f414514b" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 42325 } } peers { permanent_uuid: "c4adb558d3eb47b0ad3c5019472c2b28" member_type: VOTER last_known_addr { host: "127.0.100.4" port: 37269 } attrs { promote: false } } }
I20240503 23:27:32.785468 5822 catalog_manager.cc:5484] T 5a588f46c5ae427bbde5cf6a988cd824 P d8998ef4c11346fb89defef6f414514b reported cstate change: config changed from index 3 to 4, VOTER bb85d6712d2242f9b99303387b4bbf87 (127.0.100.1) evicted. New cstate: current_term: 2 leader_uuid: "5bed7b22c6d240668345b7d0dc336ea9" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5bed7b22c6d240668345b7d0dc336ea9" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 45643 } } peers { permanent_uuid: "d8998ef4c11346fb89defef6f414514b" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 42325 } } peers { permanent_uuid: "c4adb558d3eb47b0ad3c5019472c2b28" member_type: VOTER last_known_addr { host: "127.0.100.4" port: 37269 } attrs { promote: false } } }
I20240503 23:27:32.847998 6295 raft_consensus.cc:1050] T f5cee53ca97a47b9a77b8a2a912f1349 P 5bed7b22c6d240668345b7d0dc336ea9: attempting to promote NON_VOTER c4adb558d3eb47b0ad3c5019472c2b28 to VOTER
I20240503 23:27:32.848809 6295 consensus_queue.cc:237] T f5cee53ca97a47b9a77b8a2a912f1349 P 5bed7b22c6d240668345b7d0dc336ea9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 0, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "bb85d6712d2242f9b99303387b4bbf87" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 41717 } } peers { permanent_uuid: "5bed7b22c6d240668345b7d0dc336ea9" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 45643 } } peers { permanent_uuid: "d8998ef4c11346fb89defef6f414514b" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 42325 } } peers { permanent_uuid: "c4adb558d3eb47b0ad3c5019472c2b28" member_type: VOTER last_known_addr { host: "127.0.100.4" port: 37269 } attrs { promote: false } }
I20240503 23:27:32.851119 6213 raft_consensus.cc:1261] T f5cee53ca97a47b9a77b8a2a912f1349 P c4adb558d3eb47b0ad3c5019472c2b28 [term 2 LEARNER]: Refusing update from remote peer 5bed7b22c6d240668345b7d0dc336ea9: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20240503 23:27:32.851153 6074 raft_consensus.cc:1261] T f5cee53ca97a47b9a77b8a2a912f1349 P d8998ef4c11346fb89defef6f414514b [term 2 FOLLOWER]: Refusing update from remote peer 5bed7b22c6d240668345b7d0dc336ea9: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20240503 23:27:32.851699 6295 consensus_queue.cc:1035] T f5cee53ca97a47b9a77b8a2a912f1349 P 5bed7b22c6d240668345b7d0dc336ea9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c4adb558d3eb47b0ad3c5019472c2b28" member_type: VOTER last_known_addr { host: "127.0.100.4" port: 37269 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 23:27:32.852072 6297 consensus_queue.cc:1035] T f5cee53ca97a47b9a77b8a2a912f1349 P 5bed7b22c6d240668345b7d0dc336ea9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d8998ef4c11346fb89defef6f414514b" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 42325 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
W20240503 23:27:32.852272 5966 consensus_peers.cc:485] T f5cee53ca97a47b9a77b8a2a912f1349 P 5bed7b22c6d240668345b7d0dc336ea9 -> Peer bb85d6712d2242f9b99303387b4bbf87 (127.0.100.1:41717): Couldn't send request to peer bb85d6712d2242f9b99303387b4bbf87. Status: Network error: Client connection negotiation failed: client connection to 127.0.100.1:41717: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 23:27:32.855389 6295 raft_consensus.cc:2932] T f5cee53ca97a47b9a77b8a2a912f1349 P 5bed7b22c6d240668345b7d0dc336ea9 [term 2 LEADER]: Committing config change with OpId 2.3: config changed from index 2 to 3, c4adb558d3eb47b0ad3c5019472c2b28 (127.0.100.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "bb85d6712d2242f9b99303387b4bbf87" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 41717 } } peers { permanent_uuid: "5bed7b22c6d240668345b7d0dc336ea9" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 45643 } } peers { permanent_uuid: "d8998ef4c11346fb89defef6f414514b" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 42325 } } peers { permanent_uuid: "c4adb558d3eb47b0ad3c5019472c2b28" member_type: VOTER last_known_addr { host: "127.0.100.4" port: 37269 } attrs { promote: false } } }
I20240503 23:27:32.856062 6074 raft_consensus.cc:2932] T f5cee53ca97a47b9a77b8a2a912f1349 P d8998ef4c11346fb89defef6f414514b [term 2 FOLLOWER]: Committing config change with OpId 2.3: config changed from index 2 to 3, c4adb558d3eb47b0ad3c5019472c2b28 (127.0.100.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "bb85d6712d2242f9b99303387b4bbf87" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 41717 } } peers { permanent_uuid: "5bed7b22c6d240668345b7d0dc336ea9" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 45643 } } peers { permanent_uuid: "d8998ef4c11346fb89defef6f414514b" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 42325 } } peers { permanent_uuid: "c4adb558d3eb47b0ad3c5019472c2b28" member_type: VOTER last_known_addr { host: "127.0.100.4" port: 37269 } attrs { promote: false } } }
I20240503 23:27:32.861325 5823 catalog_manager.cc:5484] T f5cee53ca97a47b9a77b8a2a912f1349 P d8998ef4c11346fb89defef6f414514b reported cstate change: config changed from index 2 to 3, c4adb558d3eb47b0ad3c5019472c2b28 (127.0.100.4) changed from NON_VOTER to VOTER. New cstate: current_term: 2 leader_uuid: "5bed7b22c6d240668345b7d0dc336ea9" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "bb85d6712d2242f9b99303387b4bbf87" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 41717 } } peers { permanent_uuid: "5bed7b22c6d240668345b7d0dc336ea9" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 45643 } } peers { permanent_uuid: "d8998ef4c11346fb89defef6f414514b" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 42325 } } peers { permanent_uuid: "c4adb558d3eb47b0ad3c5019472c2b28" member_type: VOTER last_known_addr { host: "127.0.100.4" port: 37269 } attrs { promote: false } } }
I20240503 23:27:32.861599 6213 raft_consensus.cc:2932] T f5cee53ca97a47b9a77b8a2a912f1349 P c4adb558d3eb47b0ad3c5019472c2b28 [term 2 FOLLOWER]: Committing config change with OpId 2.3: config changed from index 2 to 3, c4adb558d3eb47b0ad3c5019472c2b28 (127.0.100.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "bb85d6712d2242f9b99303387b4bbf87" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 41717 } } peers { permanent_uuid: "5bed7b22c6d240668345b7d0dc336ea9" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 45643 } } peers { permanent_uuid: "d8998ef4c11346fb89defef6f414514b" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 42325 } } peers { permanent_uuid: "c4adb558d3eb47b0ad3c5019472c2b28" member_type: VOTER last_known_addr { host: "127.0.100.4" port: 37269 } attrs { promote: false } } }
I20240503 23:27:32.866678 6000 consensus_queue.cc:237] T f5cee53ca97a47b9a77b8a2a912f1349 P 5bed7b22c6d240668345b7d0dc336ea9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.3, Last appended by leader: 0, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5bed7b22c6d240668345b7d0dc336ea9" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 45643 } } peers { permanent_uuid: "d8998ef4c11346fb89defef6f414514b" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 42325 } } peers { permanent_uuid: "c4adb558d3eb47b0ad3c5019472c2b28" member_type: VOTER last_known_addr { host: "127.0.100.4" port: 37269 } attrs { promote: false } }
I20240503 23:27:32.868602 6213 raft_consensus.cc:1261] T f5cee53ca97a47b9a77b8a2a912f1349 P c4adb558d3eb47b0ad3c5019472c2b28 [term 2 FOLLOWER]: Refusing update from remote peer 5bed7b22c6d240668345b7d0dc336ea9: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240503 23:27:32.868602 6074 raft_consensus.cc:1261] T f5cee53ca97a47b9a77b8a2a912f1349 P d8998ef4c11346fb89defef6f414514b [term 2 FOLLOWER]: Refusing update from remote peer 5bed7b22c6d240668345b7d0dc336ea9: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240503 23:27:32.869242 6161 consensus_queue.cc:1035] T f5cee53ca97a47b9a77b8a2a912f1349 P 5bed7b22c6d240668345b7d0dc336ea9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d8998ef4c11346fb89defef6f414514b" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 42325 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 23:27:32.869484 6295 consensus_queue.cc:1035] T f5cee53ca97a47b9a77b8a2a912f1349 P 5bed7b22c6d240668345b7d0dc336ea9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c4adb558d3eb47b0ad3c5019472c2b28" member_type: VOTER last_known_addr { host: "127.0.100.4" port: 37269 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 23:27:32.871862 6294 raft_consensus.cc:2932] T f5cee53ca97a47b9a77b8a2a912f1349 P 5bed7b22c6d240668345b7d0dc336ea9 [term 2 LEADER]: Committing config change with OpId 2.4: config changed from index 3 to 4, VOTER bb85d6712d2242f9b99303387b4bbf87 (127.0.100.1) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5bed7b22c6d240668345b7d0dc336ea9" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 45643 } } peers { permanent_uuid: "d8998ef4c11346fb89defef6f414514b" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 42325 } } peers { permanent_uuid: "c4adb558d3eb47b0ad3c5019472c2b28" member_type: VOTER last_known_addr { host: "127.0.100.4" port: 37269 } attrs { promote: false } } }
I20240503 23:27:32.872489 6074 raft_consensus.cc:2932] T f5cee53ca97a47b9a77b8a2a912f1349 P d8998ef4c11346fb89defef6f414514b [term 2 FOLLOWER]: Committing config change with OpId 2.4: config changed from index 3 to 4, VOTER bb85d6712d2242f9b99303387b4bbf87 (127.0.100.1) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5bed7b22c6d240668345b7d0dc336ea9" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 45643 } } peers { permanent_uuid: "d8998ef4c11346fb89defef6f414514b" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 42325 } } peers { permanent_uuid: "c4adb558d3eb47b0ad3c5019472c2b28" member_type: VOTER last_known_addr { host: "127.0.100.4" port: 37269 } attrs { promote: false } } }
I20240503 23:27:32.875984 5810 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet f5cee53ca97a47b9a77b8a2a912f1349 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 23:27:32.876991 6213 raft_consensus.cc:2932] T f5cee53ca97a47b9a77b8a2a912f1349 P c4adb558d3eb47b0ad3c5019472c2b28 [term 2 FOLLOWER]: Committing config change with OpId 2.4: config changed from index 3 to 4, VOTER bb85d6712d2242f9b99303387b4bbf87 (127.0.100.1) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5bed7b22c6d240668345b7d0dc336ea9" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 45643 } } peers { permanent_uuid: "d8998ef4c11346fb89defef6f414514b" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 42325 } } peers { permanent_uuid: "c4adb558d3eb47b0ad3c5019472c2b28" member_type: VOTER last_known_addr { host: "127.0.100.4" port: 37269 } attrs { promote: false } } }
I20240503 23:27:32.877324 5823 catalog_manager.cc:5484] T f5cee53ca97a47b9a77b8a2a912f1349 P d8998ef4c11346fb89defef6f414514b reported cstate change: config changed from index 3 to 4, VOTER bb85d6712d2242f9b99303387b4bbf87 (127.0.100.1) evicted. New cstate: current_term: 2 leader_uuid: "5bed7b22c6d240668345b7d0dc336ea9" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5bed7b22c6d240668345b7d0dc336ea9" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 45643 } } peers { permanent_uuid: "d8998ef4c11346fb89defef6f414514b" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 42325 } } peers { permanent_uuid: "c4adb558d3eb47b0ad3c5019472c2b28" member_type: VOTER last_known_addr { host: "127.0.100.4" port: 37269 } attrs { promote: false } } }
W20240503 23:27:33.083546 5810 catalog_manager.cc:4573] TS bb85d6712d2242f9b99303387b4bbf87 (127.0.100.1:41717): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet f5cee53ca97a47b9a77b8a2a912f1349: Network error: Client connection negotiation failed: client connection to 127.0.100.1:41717: connect: Connection refused (error 111)
W20240503 23:27:35.023716 5810 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.100.1:41717: connect: Connection refused (error 111) [suppressed 57 similar messages]
W20240503 23:27:35.025167 5810 catalog_manager.cc:4573] TS bb85d6712d2242f9b99303387b4bbf87 (127.0.100.1:41717): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 5a588f46c5ae427bbde5cf6a988cd824: Network error: Client connection negotiation failed: client connection to 127.0.100.1:41717: connect: Connection refused (error 111)
I20240503 23:27:36.658946 400 tablet_server.cc:178] TabletServer@127.0.100.2:0 shutting down...
I20240503 23:27:36.668430 400 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240503 23:27:36.668823 400 tablet_replica.cc:330] T f657a9c7f5244a13ad0a03c28cbf3f1f P 5bed7b22c6d240668345b7d0dc336ea9: stopping tablet replica
I20240503 23:27:36.669009 400 raft_consensus.cc:2229] T f657a9c7f5244a13ad0a03c28cbf3f1f P 5bed7b22c6d240668345b7d0dc336ea9 [term 1 LEADER]: Raft consensus shutting down.
I20240503 23:27:36.669384 400 raft_consensus.cc:2258] T f657a9c7f5244a13ad0a03c28cbf3f1f P 5bed7b22c6d240668345b7d0dc336ea9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 23:27:36.670091 400 tablet_replica.cc:330] T 5a588f46c5ae427bbde5cf6a988cd824 P 5bed7b22c6d240668345b7d0dc336ea9: stopping tablet replica
I20240503 23:27:36.670223 400 raft_consensus.cc:2229] T 5a588f46c5ae427bbde5cf6a988cd824 P 5bed7b22c6d240668345b7d0dc336ea9 [term 2 LEADER]: Raft consensus shutting down.
I20240503 23:27:36.670469 400 raft_consensus.cc:2258] T 5a588f46c5ae427bbde5cf6a988cd824 P 5bed7b22c6d240668345b7d0dc336ea9 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 23:27:36.670940 400 tablet_replica.cc:330] T f5cee53ca97a47b9a77b8a2a912f1349 P 5bed7b22c6d240668345b7d0dc336ea9: stopping tablet replica
I20240503 23:27:36.671056 400 raft_consensus.cc:2229] T f5cee53ca97a47b9a77b8a2a912f1349 P 5bed7b22c6d240668345b7d0dc336ea9 [term 2 LEADER]: Raft consensus shutting down.
I20240503 23:27:36.671296 400 raft_consensus.cc:2258] T f5cee53ca97a47b9a77b8a2a912f1349 P 5bed7b22c6d240668345b7d0dc336ea9 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 23:27:36.671706 400 tablet_replica.cc:330] T 3b169d0763a9492c83a8a406a16069f6 P 5bed7b22c6d240668345b7d0dc336ea9: stopping tablet replica
I20240503 23:27:36.671811 400 raft_consensus.cc:2229] T 3b169d0763a9492c83a8a406a16069f6 P 5bed7b22c6d240668345b7d0dc336ea9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 23:27:36.672005 400 raft_consensus.cc:2258] T 3b169d0763a9492c83a8a406a16069f6 P 5bed7b22c6d240668345b7d0dc336ea9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 23:27:36.687709 400 tablet_server.cc:194] TabletServer@127.0.100.2:0 shutdown complete.
I20240503 23:27:36.692407 400 tablet_server.cc:178] TabletServer@127.0.100.3:0 shutting down...
I20240503 23:27:36.702106 400 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240503 23:27:36.702435 400 tablet_replica.cc:330] T f657a9c7f5244a13ad0a03c28cbf3f1f P d8998ef4c11346fb89defef6f414514b: stopping tablet replica
I20240503 23:27:36.702678 400 raft_consensus.cc:2229] T f657a9c7f5244a13ad0a03c28cbf3f1f P d8998ef4c11346fb89defef6f414514b [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 23:27:36.702987 400 raft_consensus.cc:2258] T f657a9c7f5244a13ad0a03c28cbf3f1f P d8998ef4c11346fb89defef6f414514b [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 23:27:36.703650 400 tablet_replica.cc:330] T 5a588f46c5ae427bbde5cf6a988cd824 P d8998ef4c11346fb89defef6f414514b: stopping tablet replica
I20240503 23:27:36.703821 400 raft_consensus.cc:2229] T 5a588f46c5ae427bbde5cf6a988cd824 P d8998ef4c11346fb89defef6f414514b [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 23:27:36.704077 400 raft_consensus.cc:2258] T 5a588f46c5ae427bbde5cf6a988cd824 P d8998ef4c11346fb89defef6f414514b [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 23:27:36.704638 400 tablet_replica.cc:330] T f5cee53ca97a47b9a77b8a2a912f1349 P d8998ef4c11346fb89defef6f414514b: stopping tablet replica
I20240503 23:27:36.704779 400 raft_consensus.cc:2229] T f5cee53ca97a47b9a77b8a2a912f1349 P d8998ef4c11346fb89defef6f414514b [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 23:27:36.705008 400 raft_consensus.cc:2258] T f5cee53ca97a47b9a77b8a2a912f1349 P d8998ef4c11346fb89defef6f414514b [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 23:27:36.705453 400 tablet_replica.cc:330] T 3b169d0763a9492c83a8a406a16069f6 P d8998ef4c11346fb89defef6f414514b: stopping tablet replica
I20240503 23:27:36.705586 400 raft_consensus.cc:2229] T 3b169d0763a9492c83a8a406a16069f6 P d8998ef4c11346fb89defef6f414514b [term 1 LEADER]: Raft consensus shutting down.
I20240503 23:27:36.705871 400 raft_consensus.cc:2258] T 3b169d0763a9492c83a8a406a16069f6 P d8998ef4c11346fb89defef6f414514b [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 23:27:36.721222 400 tablet_server.cc:194] TabletServer@127.0.100.3:0 shutdown complete.
I20240503 23:27:36.726012 400 tablet_server.cc:178] TabletServer@127.0.100.4:0 shutting down...
I20240503 23:27:36.736918 400 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240503 23:27:36.737301 400 tablet_replica.cc:330] T f5cee53ca97a47b9a77b8a2a912f1349 P c4adb558d3eb47b0ad3c5019472c2b28: stopping tablet replica
I20240503 23:27:36.737527 400 raft_consensus.cc:2229] T f5cee53ca97a47b9a77b8a2a912f1349 P c4adb558d3eb47b0ad3c5019472c2b28 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 23:27:36.737808 400 raft_consensus.cc:2258] T f5cee53ca97a47b9a77b8a2a912f1349 P c4adb558d3eb47b0ad3c5019472c2b28 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 23:27:36.738502 400 tablet_replica.cc:330] T 3b169d0763a9492c83a8a406a16069f6 P c4adb558d3eb47b0ad3c5019472c2b28: stopping tablet replica
I20240503 23:27:36.738683 400 raft_consensus.cc:2229] T 3b169d0763a9492c83a8a406a16069f6 P c4adb558d3eb47b0ad3c5019472c2b28 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 23:27:36.738896 400 raft_consensus.cc:2258] T 3b169d0763a9492c83a8a406a16069f6 P c4adb558d3eb47b0ad3c5019472c2b28 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 23:27:36.739392 400 tablet_replica.cc:330] T 5a588f46c5ae427bbde5cf6a988cd824 P c4adb558d3eb47b0ad3c5019472c2b28: stopping tablet replica
I20240503 23:27:36.739534 400 raft_consensus.cc:2229] T 5a588f46c5ae427bbde5cf6a988cd824 P c4adb558d3eb47b0ad3c5019472c2b28 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 23:27:36.739728 400 raft_consensus.cc:2258] T 5a588f46c5ae427bbde5cf6a988cd824 P c4adb558d3eb47b0ad3c5019472c2b28 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 23:27:36.740154 400 tablet_replica.cc:330] T f657a9c7f5244a13ad0a03c28cbf3f1f P c4adb558d3eb47b0ad3c5019472c2b28: stopping tablet replica
I20240503 23:27:36.740272 400 raft_consensus.cc:2229] T f657a9c7f5244a13ad0a03c28cbf3f1f P c4adb558d3eb47b0ad3c5019472c2b28 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 23:27:36.740459 400 raft_consensus.cc:2258] T f657a9c7f5244a13ad0a03c28cbf3f1f P c4adb558d3eb47b0ad3c5019472c2b28 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 23:27:36.767220 400 tablet_server.cc:194] TabletServer@127.0.100.4:0 shutdown complete.
I20240503 23:27:36.771682 400 master.cc:537] Master@127.0.100.62:35103 shutting down...
W20240503 23:27:37.092375 5810 catalog_manager.cc:4573] TS bb85d6712d2242f9b99303387b4bbf87 (127.0.100.1:41717): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 5a588f46c5ae427bbde5cf6a988cd824: Network error: Client connection negotiation failed: client connection to 127.0.100.1:41717: connect: Connection refused (error 111)
W20240503 23:27:39.364228 5810 catalog_manager.cc:4573] TS bb85d6712d2242f9b99303387b4bbf87 (127.0.100.1:41717): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet f657a9c7f5244a13ad0a03c28cbf3f1f: Network error: Client connection negotiation failed: client connection to 127.0.100.1:41717: connect: Connection refused (error 111)
I20240503 23:27:40.372437 400 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 2849b178c1b2448e9ca7c2130795d41d [term 1 LEADER]: Raft consensus shutting down.
I20240503 23:27:40.372843 400 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 2849b178c1b2448e9ca7c2130795d41d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 23:27:40.372968 400 tablet_replica.cc:330] T 00000000000000000000000000000000 P 2849b178c1b2448e9ca7c2130795d41d: stopping tablet replica
I20240503 23:27:40.403185 400 master.cc:552] Master@127.0.100.62:35103 shutdown complete.
[ OK ] AutoRebalancerTest.NoRebalancingIfReplicasRecovering (13437 ms)
[ RUN ] AutoRebalancerTest.TestHandlingFailedTservers
I20240503 23:27:40.420487 400 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.0.100.62:44317
I20240503 23:27:40.421185 400 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 23:27:40.424490 6315 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 23:27:40.425519 6316 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 23:27:40.426306 6318 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 23:27:40.427181 400 server_base.cc:1034] running on GCE node
I20240503 23:27:40.427464 400 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240503 23:27:40.427536 400 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240503 23:27:40.427608 400 hybrid_clock.cc:648] HybridClock initialized: now 1714778860427584 us; error 0 us; skew 500 ppm
I20240503 23:27:40.427806 400 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 23:27:40.435104 400 webserver.cc:460] Webserver started at http://127.0.100.62:45353/ using document root <none> and password file <none>
I20240503 23:27:40.435454 400 fs_manager.cc:359] Metadata directory not provided
I20240503 23:27:40.435562 400 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 23:27:40.435726 400 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 23:27:40.436655 400 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZhIiPz/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestHandlingFailedTservers.1714778688451250-400-0/minicluster-data/master-0-root/instance:
uuid: "94ce7459b82d4eebac81c62d8179eec0"
format_stamp: "Formatted at 2024-05-03 23:27:40 on dist-test-slave-jvbp"
I20240503 23:27:40.439924 400 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.003s
I20240503 23:27:40.442430 6323 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 23:27:40.442962 400 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240503 23:27:40.443127 400 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZhIiPz/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestHandlingFailedTservers.1714778688451250-400-0/minicluster-data/master-0-root
uuid: "94ce7459b82d4eebac81c62d8179eec0"
format_stamp: "Formatted at 2024-05-03 23:27:40 on dist-test-slave-jvbp"
I20240503 23:27:40.443290 400 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZhIiPz/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestHandlingFailedTservers.1714778688451250-400-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskZhIiPz/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestHandlingFailedTservers.1714778688451250-400-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskZhIiPz/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestHandlingFailedTservers.1714778688451250-400-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 23:27:40.449944 400 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 23:27:40.450613 400 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 23:27:40.464892 400 rpc_server.cc:307] RPC server started. Bound to: 127.0.100.62:44317
I20240503 23:27:40.464915 6374 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.100.62:44317 every 8 connection(s)
I20240503 23:27:40.466567 6375 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 23:27:40.471609 6375 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 94ce7459b82d4eebac81c62d8179eec0: Bootstrap starting.
I20240503 23:27:40.473186 6375 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 94ce7459b82d4eebac81c62d8179eec0: Neither blocks nor log segments found. Creating new log.
I20240503 23:27:40.475114 6375 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 94ce7459b82d4eebac81c62d8179eec0: No bootstrap required, opened a new log
I20240503 23:27:40.475847 6375 raft_consensus.cc:348] T 00000000000000000000000000000000 P 94ce7459b82d4eebac81c62d8179eec0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "94ce7459b82d4eebac81c62d8179eec0" member_type: VOTER }
I20240503 23:27:40.476052 6375 raft_consensus.cc:374] T 00000000000000000000000000000000 P 94ce7459b82d4eebac81c62d8179eec0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 23:27:40.476140 6375 raft_consensus.cc:729] T 00000000000000000000000000000000 P 94ce7459b82d4eebac81c62d8179eec0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 94ce7459b82d4eebac81c62d8179eec0, State: Initialized, Role: FOLLOWER
I20240503 23:27:40.476392 6375 consensus_queue.cc:260] T 00000000000000000000000000000000 P 94ce7459b82d4eebac81c62d8179eec0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "94ce7459b82d4eebac81c62d8179eec0" member_type: VOTER }
I20240503 23:27:40.476579 6375 raft_consensus.cc:388] T 00000000000000000000000000000000 P 94ce7459b82d4eebac81c62d8179eec0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 23:27:40.476661 6375 raft_consensus.cc:482] T 00000000000000000000000000000000 P 94ce7459b82d4eebac81c62d8179eec0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 23:27:40.476763 6375 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 94ce7459b82d4eebac81c62d8179eec0 [term 0 FOLLOWER]: Advancing to term 1
I20240503 23:27:40.478957 6375 raft_consensus.cc:504] T 00000000000000000000000000000000 P 94ce7459b82d4eebac81c62d8179eec0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "94ce7459b82d4eebac81c62d8179eec0" member_type: VOTER }
I20240503 23:27:40.479184 6375 leader_election.cc:302] T 00000000000000000000000000000000 P 94ce7459b82d4eebac81c62d8179eec0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 94ce7459b82d4eebac81c62d8179eec0; no voters:
I20240503 23:27:40.479732 6375 leader_election.cc:288] T 00000000000000000000000000000000 P 94ce7459b82d4eebac81c62d8179eec0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240503 23:27:40.479897 6378 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 94ce7459b82d4eebac81c62d8179eec0 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 23:27:40.480489 6378 raft_consensus.cc:686] T 00000000000000000000000000000000 P 94ce7459b82d4eebac81c62d8179eec0 [term 1 LEADER]: Becoming Leader. State: Replica: 94ce7459b82d4eebac81c62d8179eec0, State: Running, Role: LEADER
I20240503 23:27:40.480811 6375 sys_catalog.cc:564] T 00000000000000000000000000000000 P 94ce7459b82d4eebac81c62d8179eec0 [sys.catalog]: configured and running, proceeding with master startup.
I20240503 23:27:40.480792 6378 consensus_queue.cc:237] T 00000000000000000000000000000000 P 94ce7459b82d4eebac81c62d8179eec0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "94ce7459b82d4eebac81c62d8179eec0" member_type: VOTER }
I20240503 23:27:40.481829 6379 sys_catalog.cc:455] T 00000000000000000000000000000000 P 94ce7459b82d4eebac81c62d8179eec0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "94ce7459b82d4eebac81c62d8179eec0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "94ce7459b82d4eebac81c62d8179eec0" member_type: VOTER } }
I20240503 23:27:40.481901 6380 sys_catalog.cc:455] T 00000000000000000000000000000000 P 94ce7459b82d4eebac81c62d8179eec0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 94ce7459b82d4eebac81c62d8179eec0. Latest consensus state: current_term: 1 leader_uuid: "94ce7459b82d4eebac81c62d8179eec0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "94ce7459b82d4eebac81c62d8179eec0" member_type: VOTER } }
I20240503 23:27:40.482208 6380 sys_catalog.cc:458] T 00000000000000000000000000000000 P 94ce7459b82d4eebac81c62d8179eec0 [sys.catalog]: This master's current role is: LEADER
I20240503 23:27:40.482880 6379 sys_catalog.cc:458] T 00000000000000000000000000000000 P 94ce7459b82d4eebac81c62d8179eec0 [sys.catalog]: This master's current role is: LEADER
I20240503 23:27:40.483726 6384 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240503 23:27:40.486316 6384 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240503 23:27:40.486776 400 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20240503 23:27:40.490027 6384 catalog_manager.cc:1347] Generated new cluster ID: dad0d4252a634afbbc8cd49c5df89c74
I20240503 23:27:40.490199 6384 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240503 23:27:40.514357 6384 catalog_manager.cc:1370] Generated new certificate authority record
I20240503 23:27:40.515377 6384 catalog_manager.cc:1504] Loading token signing keys...
I20240503 23:27:40.528137 6384 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 94ce7459b82d4eebac81c62d8179eec0: Generated new TSK 0
I20240503 23:27:40.528525 6384 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240503 23:27:40.552502 400 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 23:27:40.556340 6396 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 23:27:40.557135 6397 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 23:27:40.557974 6399 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 23:27:40.558379 400 server_base.cc:1034] running on GCE node
I20240503 23:27:40.558676 400 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240503 23:27:40.558750 400 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240503 23:27:40.558791 400 hybrid_clock.cc:648] HybridClock initialized: now 1714778860558774 us; error 0 us; skew 500 ppm
I20240503 23:27:40.558959 400 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 23:27:40.561489 400 webserver.cc:460] Webserver started at http://127.0.100.1:36505/ using document root <none> and password file <none>
I20240503 23:27:40.561796 400 fs_manager.cc:359] Metadata directory not provided
I20240503 23:27:40.561880 400 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 23:27:40.562014 400 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 23:27:40.562825 400 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZhIiPz/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestHandlingFailedTservers.1714778688451250-400-0/minicluster-data/ts-0-root/instance:
uuid: "3fc1cf80b12b42c2af63d3afec5214a7"
format_stamp: "Formatted at 2024-05-03 23:27:40 on dist-test-slave-jvbp"
I20240503 23:27:40.566036 400 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20240503 23:27:40.568171 6404 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 23:27:40.568646 400 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.002s sys 0.000s
I20240503 23:27:40.568790 400 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZhIiPz/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestHandlingFailedTservers.1714778688451250-400-0/minicluster-data/ts-0-root
uuid: "3fc1cf80b12b42c2af63d3afec5214a7"
format_stamp: "Formatted at 2024-05-03 23:27:40 on dist-test-slave-jvbp"
I20240503 23:27:40.568931 400 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZhIiPz/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestHandlingFailedTservers.1714778688451250-400-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskZhIiPz/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestHandlingFailedTservers.1714778688451250-400-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskZhIiPz/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestHandlingFailedTservers.1714778688451250-400-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 23:27:40.595311 400 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 23:27:40.596271 400 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 23:27:40.597179 400 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 23:27:40.598284 400 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 23:27:40.598373 400 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 23:27:40.598490 400 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 23:27:40.598549 400 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 23:27:40.614591 400 rpc_server.cc:307] RPC server started. Bound to: 127.0.100.1:34273
I20240503 23:27:40.614626 6466 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.100.1:34273 every 8 connection(s)
I20240503 23:27:40.616871 400 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 23:27:40.620769 6471 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 23:27:40.621623 6472 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 23:27:40.622417 6474 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 23:27:40.623337 400 server_base.cc:1034] running on GCE node
I20240503 23:27:40.623773 400 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240503 23:27:40.623878 400 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240503 23:27:40.623953 400 hybrid_clock.cc:648] HybridClock initialized: now 1714778860623925 us; error 0 us; skew 500 ppm
I20240503 23:27:40.624153 400 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 23:27:40.625634 6467 heartbeater.cc:347] Connected to a master server at 127.0.100.62:44317
I20240503 23:27:40.625833 6467 heartbeater.cc:464] Registering TS with master...
I20240503 23:27:40.625974 400 webserver.cc:460] Webserver started at http://127.0.100.2:42189/ using document root <none> and password file <none>
I20240503 23:27:40.626267 400 fs_manager.cc:359] Metadata directory not provided
I20240503 23:27:40.626380 400 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 23:27:40.626421 6467 heartbeater.cc:511] Master 127.0.100.62:44317 requested a full tablet report, sending...
I20240503 23:27:40.626562 400 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 23:27:40.627485 400 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZhIiPz/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestHandlingFailedTservers.1714778688451250-400-0/minicluster-data/ts-1-root/instance:
uuid: "c71e941c84ba45deb326c90968f40308"
format_stamp: "Formatted at 2024-05-03 23:27:40 on dist-test-slave-jvbp"
I20240503 23:27:40.627624 6340 ts_manager.cc:194] Registered new tserver with Master: 3fc1cf80b12b42c2af63d3afec5214a7 (127.0.100.1:34273)
I20240503 23:27:40.629040 6340 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:55740
I20240503 23:27:40.630841 400 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20240503 23:27:40.632835 6479 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 23:27:40.633260 400 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.002s sys 0.000s
I20240503 23:27:40.633404 400 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZhIiPz/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestHandlingFailedTservers.1714778688451250-400-0/minicluster-data/ts-1-root
uuid: "c71e941c84ba45deb326c90968f40308"
format_stamp: "Formatted at 2024-05-03 23:27:40 on dist-test-slave-jvbp"
I20240503 23:27:40.633545 400 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZhIiPz/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestHandlingFailedTservers.1714778688451250-400-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskZhIiPz/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestHandlingFailedTservers.1714778688451250-400-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskZhIiPz/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestHandlingFailedTservers.1714778688451250-400-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 23:27:40.671108 400 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 23:27:40.671842 400 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 23:27:40.672536 400 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 23:27:40.673667 400 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 23:27:40.673759 400 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 23:27:40.673879 400 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 23:27:40.673943 400 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 23:27:40.689082 400 rpc_server.cc:307] RPC server started. Bound to: 127.0.100.2:35323
I20240503 23:27:40.689102 6541 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.100.2:35323 every 8 connection(s)
I20240503 23:27:40.691053 400 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 23:27:40.694962 6545 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 23:27:40.695627 6546 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 23:27:40.696647 400 server_base.cc:1034] running on GCE node
W20240503 23:27:40.698060 6548 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 23:27:40.698671 400 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240503 23:27:40.698783 400 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240503 23:27:40.698861 400 hybrid_clock.cc:648] HybridClock initialized: now 1714778860698833 us; error 0 us; skew 500 ppm
I20240503 23:27:40.699112 400 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 23:27:40.699198 6542 heartbeater.cc:347] Connected to a master server at 127.0.100.62:44317
I20240503 23:27:40.699373 6542 heartbeater.cc:464] Registering TS with master...
I20240503 23:27:40.699812 6542 heartbeater.cc:511] Master 127.0.100.62:44317 requested a full tablet report, sending...
I20240503 23:27:40.700970 400 webserver.cc:460] Webserver started at http://127.0.100.3:34327/ using document root <none> and password file <none>
I20240503 23:27:40.701210 6340 ts_manager.cc:194] Registered new tserver with Master: c71e941c84ba45deb326c90968f40308 (127.0.100.2:35323)
I20240503 23:27:40.701305 400 fs_manager.cc:359] Metadata directory not provided
I20240503 23:27:40.701414 400 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 23:27:40.701577 400 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 23:27:40.702457 6340 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:55744
I20240503 23:27:40.702517 400 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZhIiPz/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestHandlingFailedTservers.1714778688451250-400-0/minicluster-data/ts-2-root/instance:
uuid: "21c0231e5eee4c109d088c02f139c112"
format_stamp: "Formatted at 2024-05-03 23:27:40 on dist-test-slave-jvbp"
I20240503 23:27:40.705492 400 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20240503 23:27:40.707273 6553 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 23:27:40.707675 400 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.002s sys 0.000s
I20240503 23:27:40.707798 400 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZhIiPz/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestHandlingFailedTservers.1714778688451250-400-0/minicluster-data/ts-2-root
uuid: "21c0231e5eee4c109d088c02f139c112"
format_stamp: "Formatted at 2024-05-03 23:27:40 on dist-test-slave-jvbp"
I20240503 23:27:40.707935 400 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZhIiPz/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestHandlingFailedTservers.1714778688451250-400-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-taskZhIiPz/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestHandlingFailedTservers.1714778688451250-400-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-taskZhIiPz/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestHandlingFailedTservers.1714778688451250-400-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 23:27:40.719501 400 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 23:27:40.720228 400 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 23:27:40.720943 400 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 23:27:40.721997 400 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 23:27:40.722081 400 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 23:27:40.722200 400 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 23:27:40.722270 400 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 23:27:40.738436 400 rpc_server.cc:307] RPC server started. Bound to: 127.0.100.3:44345
I20240503 23:27:40.738456 6615 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.100.3:44345 every 8 connection(s)
I20240503 23:27:40.746675 6616 heartbeater.cc:347] Connected to a master server at 127.0.100.62:44317
I20240503 23:27:40.746877 6616 heartbeater.cc:464] Registering TS with master...
I20240503 23:27:40.747328 6616 heartbeater.cc:511] Master 127.0.100.62:44317 requested a full tablet report, sending...
I20240503 23:27:40.748378 6340 ts_manager.cc:194] Registered new tserver with Master: 21c0231e5eee4c109d088c02f139c112 (127.0.100.3:44345)
I20240503 23:27:40.748687 400 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.009147839s
I20240503 23:27:40.749431 6340 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:55746
I20240503 23:27:41.630860 6467 heartbeater.cc:503] Master 127.0.100.62:44317 was elected leader, sending a full tablet report...
I20240503 23:27:41.704048 6542 heartbeater.cc:503] Master 127.0.100.62:44317 was elected leader, sending a full tablet report...
I20240503 23:27:41.750991 6616 heartbeater.cc:503] Master 127.0.100.62:44317 was elected leader, sending a full tablet report...
I20240503 23:27:41.778309 400 test_util.cc:268] Using random seed: 283948391
I20240503 23:27:41.795109 6340 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55754:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "\004\001\000\377\377\377\037\004\001\000\376\377\377?\004\001\000\375\377\377_""\004\001\000\377\377\377\037\004\001\000\376\377\377?\004\001\000\375\377\377_"
indirect_data: """"
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 23:27:41.796098 6340 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-workload in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 23:27:41.828058 6432 tablet_service.cc:1467] Processing CreateTablet for tablet 7441a981eb93446c9d9367af49462e73 (DEFAULT_TABLE table=test-workload [id=493b63a53e0842eda0f947898692001c]), partition=RANGE (key) PARTITION VALUES < 536870911
I20240503 23:27:41.828321 6430 tablet_service.cc:1467] Processing CreateTablet for tablet e9273b9533d14394b454e6c31ae4deb5 (DEFAULT_TABLE table=test-workload [id=493b63a53e0842eda0f947898692001c]), partition=RANGE (key) PARTITION 1073741822 <= VALUES < 1610612733
I20240503 23:27:41.828433 6579 tablet_service.cc:1467] Processing CreateTablet for tablet e9273b9533d14394b454e6c31ae4deb5 (DEFAULT_TABLE table=test-workload [id=493b63a53e0842eda0f947898692001c]), partition=RANGE (key) PARTITION 1073741822 <= VALUES < 1610612733
I20240503 23:27:41.828919 6432 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7441a981eb93446c9d9367af49462e73. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 23:27:41.828835 6580 tablet_service.cc:1467] Processing CreateTablet for tablet ad1b9cc708db4d289c2e847874bf747b (DEFAULT_TABLE table=test-workload [id=493b63a53e0842eda0f947898692001c]), partition=RANGE (key) PARTITION 536870911 <= VALUES < 1073741822
I20240503 23:27:41.829025 6579 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e9273b9533d14394b454e6c31ae4deb5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 23:27:41.829340 6580 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ad1b9cc708db4d289c2e847874bf747b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 23:27:41.828435 6581 tablet_service.cc:1467] Processing CreateTablet for tablet 7441a981eb93446c9d9367af49462e73 (DEFAULT_TABLE table=test-workload [id=493b63a53e0842eda0f947898692001c]), partition=RANGE (key) PARTITION VALUES < 536870911
I20240503 23:27:41.830185 6581 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7441a981eb93446c9d9367af49462e73. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 23:27:41.828920 6429 tablet_service.cc:1467] Processing CreateTablet for tablet fb79ea9cea314c4cab4bed81163a8ded (DEFAULT_TABLE table=test-workload [id=493b63a53e0842eda0f947898692001c]), partition=RANGE (key) PARTITION 1610612733 <= VALUES
I20240503 23:27:41.828058 6431 tablet_service.cc:1467] Processing CreateTablet for tablet ad1b9cc708db4d289c2e847874bf747b (DEFAULT_TABLE table=test-workload [id=493b63a53e0842eda0f947898692001c]), partition=RANGE (key) PARTITION 536870911 <= VALUES < 1073741822
I20240503 23:27:41.833148 6430 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e9273b9533d14394b454e6c31ae4deb5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 23:27:41.833420 6429 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fb79ea9cea314c4cab4bed81163a8ded. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 23:27:41.834417 6578 tablet_service.cc:1467] Processing CreateTablet for tablet fb79ea9cea314c4cab4bed81163a8ded (DEFAULT_TABLE table=test-workload [id=493b63a53e0842eda0f947898692001c]), partition=RANGE (key) PARTITION 1610612733 <= VALUES
I20240503 23:27:41.834959 6578 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fb79ea9cea314c4cab4bed81163a8ded. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 23:27:41.840845 6431 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ad1b9cc708db4d289c2e847874bf747b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 23:27:41.850586 6637 tablet_bootstrap.cc:492] T ad1b9cc708db4d289c2e847874bf747b P 21c0231e5eee4c109d088c02f139c112: Bootstrap starting.
I20240503 23:27:41.853552 6637 tablet_bootstrap.cc:654] T ad1b9cc708db4d289c2e847874bf747b P 21c0231e5eee4c109d088c02f139c112: Neither blocks nor log segments found. Creating new log.
I20240503 23:27:41.854787 6636 tablet_bootstrap.cc:492] T e9273b9533d14394b454e6c31ae4deb5 P 3fc1cf80b12b42c2af63d3afec5214a7: Bootstrap starting.
I20240503 23:27:41.857297 6636 tablet_bootstrap.cc:654] T e9273b9533d14394b454e6c31ae4deb5 P 3fc1cf80b12b42c2af63d3afec5214a7: Neither blocks nor log segments found. Creating new log.
I20240503 23:27:41.857795 6507 tablet_service.cc:1467] Processing CreateTablet for tablet 7441a981eb93446c9d9367af49462e73 (DEFAULT_TABLE table=test-workload [id=493b63a53e0842eda0f947898692001c]), partition=RANGE (key) PARTITION VALUES < 536870911
I20240503 23:27:41.858093 6637 tablet_bootstrap.cc:492] T ad1b9cc708db4d289c2e847874bf747b P 21c0231e5eee4c109d088c02f139c112: No bootstrap required, opened a new log
I20240503 23:27:41.858119 6504 tablet_service.cc:1467] Processing CreateTablet for tablet fb79ea9cea314c4cab4bed81163a8ded (DEFAULT_TABLE table=test-workload [id=493b63a53e0842eda0f947898692001c]), partition=RANGE (key) PARTITION 1610612733 <= VALUES
I20240503 23:27:41.858295 6637 ts_tablet_manager.cc:1367] T ad1b9cc708db4d289c2e847874bf747b P 21c0231e5eee4c109d088c02f139c112: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20240503 23:27:41.858459 6507 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7441a981eb93446c9d9367af49462e73. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 23:27:41.857856 6506 tablet_service.cc:1467] Processing CreateTablet for tablet ad1b9cc708db4d289c2e847874bf747b (DEFAULT_TABLE table=test-workload [id=493b63a53e0842eda0f947898692001c]), partition=RANGE (key) PARTITION 536870911 <= VALUES < 1073741822
I20240503 23:27:41.859081 6505 tablet_service.cc:1467] Processing CreateTablet for tablet e9273b9533d14394b454e6c31ae4deb5 (DEFAULT_TABLE table=test-workload [id=493b63a53e0842eda0f947898692001c]), partition=RANGE (key) PARTITION 1073741822 <= VALUES < 1610612733
I20240503 23:27:41.859198 6506 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ad1b9cc708db4d289c2e847874bf747b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 23:27:41.859768 6505 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e9273b9533d14394b454e6c31ae4deb5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 23:27:41.859650 6637 raft_consensus.cc:348] T ad1b9cc708db4d289c2e847874bf747b P 21c0231e5eee4c109d088c02f139c112 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fc1cf80b12b42c2af63d3afec5214a7" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 34273 } } peers { permanent_uuid: "c71e941c84ba45deb326c90968f40308" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 35323 } } peers { permanent_uuid: "21c0231e5eee4c109d088c02f139c112" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 44345 } }
I20240503 23:27:41.859989 6637 raft_consensus.cc:374] T ad1b9cc708db4d289c2e847874bf747b P 21c0231e5eee4c109d088c02f139c112 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 23:27:41.860128 6637 raft_consensus.cc:729] T ad1b9cc708db4d289c2e847874bf747b P 21c0231e5eee4c109d088c02f139c112 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 21c0231e5eee4c109d088c02f139c112, State: Initialized, Role: FOLLOWER
I20240503 23:27:41.860487 6637 consensus_queue.cc:260] T ad1b9cc708db4d289c2e847874bf747b P 21c0231e5eee4c109d088c02f139c112 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fc1cf80b12b42c2af63d3afec5214a7" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 34273 } } peers { permanent_uuid: "c71e941c84ba45deb326c90968f40308" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 35323 } } peers { permanent_uuid: "21c0231e5eee4c109d088c02f139c112" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 44345 } }
I20240503 23:27:41.864948 6504 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fb79ea9cea314c4cab4bed81163a8ded. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 23:27:41.870035 6636 tablet_bootstrap.cc:492] T e9273b9533d14394b454e6c31ae4deb5 P 3fc1cf80b12b42c2af63d3afec5214a7: No bootstrap required, opened a new log
I20240503 23:27:41.870414 6636 ts_tablet_manager.cc:1367] T e9273b9533d14394b454e6c31ae4deb5 P 3fc1cf80b12b42c2af63d3afec5214a7: Time spent bootstrapping tablet: real 0.016s user 0.005s sys 0.000s
I20240503 23:27:41.871466 6636 raft_consensus.cc:348] T e9273b9533d14394b454e6c31ae4deb5 P 3fc1cf80b12b42c2af63d3afec5214a7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fc1cf80b12b42c2af63d3afec5214a7" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 34273 } } peers { permanent_uuid: "c71e941c84ba45deb326c90968f40308" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 35323 } } peers { permanent_uuid: "21c0231e5eee4c109d088c02f139c112" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 44345 } }
I20240503 23:27:41.871790 6636 raft_consensus.cc:374] T e9273b9533d14394b454e6c31ae4deb5 P 3fc1cf80b12b42c2af63d3afec5214a7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 23:27:41.871922 6636 raft_consensus.cc:729] T e9273b9533d14394b454e6c31ae4deb5 P 3fc1cf80b12b42c2af63d3afec5214a7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3fc1cf80b12b42c2af63d3afec5214a7, State: Initialized, Role: FOLLOWER
I20240503 23:27:41.872310 6636 consensus_queue.cc:260] T e9273b9533d14394b454e6c31ae4deb5 P 3fc1cf80b12b42c2af63d3afec5214a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fc1cf80b12b42c2af63d3afec5214a7" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 34273 } } peers { permanent_uuid: "c71e941c84ba45deb326c90968f40308" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 35323 } } peers { permanent_uuid: "21c0231e5eee4c109d088c02f139c112" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 44345 } }
I20240503 23:27:41.874233 6636 ts_tablet_manager.cc:1398] T e9273b9533d14394b454e6c31ae4deb5 P 3fc1cf80b12b42c2af63d3afec5214a7: Time spent starting tablet: real 0.004s user 0.003s sys 0.000s
I20240503 23:27:41.874701 6636 tablet_bootstrap.cc:492] T ad1b9cc708db4d289c2e847874bf747b P 3fc1cf80b12b42c2af63d3afec5214a7: Bootstrap starting.
I20240503 23:27:41.877050 6636 tablet_bootstrap.cc:654] T ad1b9cc708db4d289c2e847874bf747b P 3fc1cf80b12b42c2af63d3afec5214a7: Neither blocks nor log segments found. Creating new log.
I20240503 23:27:41.879493 6637 ts_tablet_manager.cc:1398] T ad1b9cc708db4d289c2e847874bf747b P 21c0231e5eee4c109d088c02f139c112: Time spent starting tablet: real 0.021s user 0.010s sys 0.008s
I20240503 23:27:41.880065 6637 tablet_bootstrap.cc:492] T 7441a981eb93446c9d9367af49462e73 P 21c0231e5eee4c109d088c02f139c112: Bootstrap starting.
I20240503 23:27:41.878119 6641 tablet_bootstrap.cc:492] T 7441a981eb93446c9d9367af49462e73 P c71e941c84ba45deb326c90968f40308: Bootstrap starting.
I20240503 23:27:41.882647 6637 tablet_bootstrap.cc:654] T 7441a981eb93446c9d9367af49462e73 P 21c0231e5eee4c109d088c02f139c112: Neither blocks nor log segments found. Creating new log.
I20240503 23:27:41.886010 6637 tablet_bootstrap.cc:492] T 7441a981eb93446c9d9367af49462e73 P 21c0231e5eee4c109d088c02f139c112: No bootstrap required, opened a new log
I20240503 23:27:41.886057 6641 tablet_bootstrap.cc:654] T 7441a981eb93446c9d9367af49462e73 P c71e941c84ba45deb326c90968f40308: Neither blocks nor log segments found. Creating new log.
I20240503 23:27:41.886286 6637 ts_tablet_manager.cc:1367] T 7441a981eb93446c9d9367af49462e73 P 21c0231e5eee4c109d088c02f139c112: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.003s
I20240503 23:27:41.886935 6636 tablet_bootstrap.cc:492] T ad1b9cc708db4d289c2e847874bf747b P 3fc1cf80b12b42c2af63d3afec5214a7: No bootstrap required, opened a new log
I20240503 23:27:41.887271 6636 ts_tablet_manager.cc:1367] T ad1b9cc708db4d289c2e847874bf747b P 3fc1cf80b12b42c2af63d3afec5214a7: Time spent bootstrapping tablet: real 0.013s user 0.009s sys 0.000s
I20240503 23:27:41.887177 6637 raft_consensus.cc:348] T 7441a981eb93446c9d9367af49462e73 P 21c0231e5eee4c109d088c02f139c112 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fc1cf80b12b42c2af63d3afec5214a7" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 34273 } } peers { permanent_uuid: "21c0231e5eee4c109d088c02f139c112" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 44345 } } peers { permanent_uuid: "c71e941c84ba45deb326c90968f40308" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 35323 } }
I20240503 23:27:41.887439 6637 raft_consensus.cc:374] T 7441a981eb93446c9d9367af49462e73 P 21c0231e5eee4c109d088c02f139c112 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 23:27:41.887545 6637 raft_consensus.cc:729] T 7441a981eb93446c9d9367af49462e73 P 21c0231e5eee4c109d088c02f139c112 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 21c0231e5eee4c109d088c02f139c112, State: Initialized, Role: FOLLOWER
I20240503 23:27:41.887853 6637 consensus_queue.cc:260] T 7441a981eb93446c9d9367af49462e73 P 21c0231e5eee4c109d088c02f139c112 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fc1cf80b12b42c2af63d3afec5214a7" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 34273 } } peers { permanent_uuid: "21c0231e5eee4c109d088c02f139c112" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 44345 } } peers { permanent_uuid: "c71e941c84ba45deb326c90968f40308" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 35323 } }
I20240503 23:27:41.888345 6637 ts_tablet_manager.cc:1398] T 7441a981eb93446c9d9367af49462e73 P 21c0231e5eee4c109d088c02f139c112: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240503 23:27:41.888746 6637 tablet_bootstrap.cc:492] T fb79ea9cea314c4cab4bed81163a8ded P 21c0231e5eee4c109d088c02f139c112: Bootstrap starting.
I20240503 23:27:41.888526 6636 raft_consensus.cc:348] T ad1b9cc708db4d289c2e847874bf747b P 3fc1cf80b12b42c2af63d3afec5214a7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fc1cf80b12b42c2af63d3afec5214a7" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 34273 } } peers { permanent_uuid: "c71e941c84ba45deb326c90968f40308" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 35323 } } peers { permanent_uuid: "21c0231e5eee4c109d088c02f139c112" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 44345 } }
I20240503 23:27:41.888967 6636 raft_consensus.cc:374] T ad1b9cc708db4d289c2e847874bf747b P 3fc1cf80b12b42c2af63d3afec5214a7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 23:27:41.889122 6636 raft_consensus.cc:729] T ad1b9cc708db4d289c2e847874bf747b P 3fc1cf80b12b42c2af63d3afec5214a7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3fc1cf80b12b42c2af63d3afec5214a7, State: Initialized, Role: FOLLOWER
I20240503 23:27:41.889503 6636 consensus_queue.cc:260] T ad1b9cc708db4d289c2e847874bf747b P 3fc1cf80b12b42c2af63d3afec5214a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fc1cf80b12b42c2af63d3afec5214a7" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 34273 } } peers { permanent_uuid: "c71e941c84ba45deb326c90968f40308" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 35323 } } peers { permanent_uuid: "21c0231e5eee4c109d088c02f139c112" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 44345 } }
I20240503 23:27:41.890373 6636 ts_tablet_manager.cc:1398] T ad1b9cc708db4d289c2e847874bf747b P 3fc1cf80b12b42c2af63d3afec5214a7: Time spent starting tablet: real 0.003s user 0.000s sys 0.004s
I20240503 23:27:41.890935 6637 tablet_bootstrap.cc:654] T fb79ea9cea314c4cab4bed81163a8ded P 21c0231e5eee4c109d088c02f139c112: Neither blocks nor log segments found. Creating new log.
I20240503 23:27:41.891999 6636 tablet_bootstrap.cc:492] T 7441a981eb93446c9d9367af49462e73 P 3fc1cf80b12b42c2af63d3afec5214a7: Bootstrap starting.
I20240503 23:27:41.894691 6636 tablet_bootstrap.cc:654] T 7441a981eb93446c9d9367af49462e73 P 3fc1cf80b12b42c2af63d3afec5214a7: Neither blocks nor log segments found. Creating new log.
I20240503 23:27:41.895251 6641 tablet_bootstrap.cc:492] T 7441a981eb93446c9d9367af49462e73 P c71e941c84ba45deb326c90968f40308: No bootstrap required, opened a new log
I20240503 23:27:41.895524 6641 ts_tablet_manager.cc:1367] T 7441a981eb93446c9d9367af49462e73 P c71e941c84ba45deb326c90968f40308: Time spent bootstrapping tablet: real 0.017s user 0.007s sys 0.004s
I20240503 23:27:41.898662 6636 tablet_bootstrap.cc:492] T 7441a981eb93446c9d9367af49462e73 P 3fc1cf80b12b42c2af63d3afec5214a7: No bootstrap required, opened a new log
I20240503 23:27:41.898979 6636 ts_tablet_manager.cc:1367] T 7441a981eb93446c9d9367af49462e73 P 3fc1cf80b12b42c2af63d3afec5214a7: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20240503 23:27:41.898662 6637 tablet_bootstrap.cc:492] T fb79ea9cea314c4cab4bed81163a8ded P 21c0231e5eee4c109d088c02f139c112: No bootstrap required, opened a new log
I20240503 23:27:41.899441 6641 raft_consensus.cc:348] T 7441a981eb93446c9d9367af49462e73 P c71e941c84ba45deb326c90968f40308 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fc1cf80b12b42c2af63d3afec5214a7" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 34273 } } peers { permanent_uuid: "21c0231e5eee4c109d088c02f139c112" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 44345 } } peers { permanent_uuid: "c71e941c84ba45deb326c90968f40308" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 35323 } }
I20240503 23:27:41.899771 6641 raft_consensus.cc:374] T 7441a981eb93446c9d9367af49462e73 P c71e941c84ba45deb326c90968f40308 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 23:27:41.899876 6637 ts_tablet_manager.cc:1367] T fb79ea9cea314c4cab4bed81163a8ded P 21c0231e5eee4c109d088c02f139c112: Time spent bootstrapping tablet: real 0.011s user 0.002s sys 0.006s
I20240503 23:27:41.899902 6641 raft_consensus.cc:729] T 7441a981eb93446c9d9367af49462e73 P c71e941c84ba45deb326c90968f40308 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c71e941c84ba45deb326c90968f40308, State: Initialized, Role: FOLLOWER
I20240503 23:27:41.900285 6641 consensus_queue.cc:260] T 7441a981eb93446c9d9367af49462e73 P c71e941c84ba45deb326c90968f40308 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fc1cf80b12b42c2af63d3afec5214a7" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 34273 } } peers { permanent_uuid: "21c0231e5eee4c109d088c02f139c112" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 44345 } } peers { permanent_uuid: "c71e941c84ba45deb326c90968f40308" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 35323 } }
I20240503 23:27:41.900750 6636 raft_consensus.cc:348] T 7441a981eb93446c9d9367af49462e73 P 3fc1cf80b12b42c2af63d3afec5214a7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fc1cf80b12b42c2af63d3afec5214a7" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 34273 } } peers { permanent_uuid: "21c0231e5eee4c109d088c02f139c112" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 44345 } } peers { permanent_uuid: "c71e941c84ba45deb326c90968f40308" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 35323 } }
I20240503 23:27:41.900758 6637 raft_consensus.cc:348] T fb79ea9cea314c4cab4bed81163a8ded P 21c0231e5eee4c109d088c02f139c112 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fc1cf80b12b42c2af63d3afec5214a7" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 34273 } } peers { permanent_uuid: "c71e941c84ba45deb326c90968f40308" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 35323 } } peers { permanent_uuid: "21c0231e5eee4c109d088c02f139c112" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 44345 } }
I20240503 23:27:41.901031 6636 raft_consensus.cc:374] T 7441a981eb93446c9d9367af49462e73 P 3fc1cf80b12b42c2af63d3afec5214a7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 23:27:41.901104 6637 raft_consensus.cc:374] T fb79ea9cea314c4cab4bed81163a8ded P 21c0231e5eee4c109d088c02f139c112 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 23:27:41.901163 6636 raft_consensus.cc:729] T 7441a981eb93446c9d9367af49462e73 P 3fc1cf80b12b42c2af63d3afec5214a7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3fc1cf80b12b42c2af63d3afec5214a7, State: Initialized, Role: FOLLOWER
I20240503 23:27:41.901304 6637 raft_consensus.cc:729] T fb79ea9cea314c4cab4bed81163a8ded P 21c0231e5eee4c109d088c02f139c112 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 21c0231e5eee4c109d088c02f139c112, State: Initialized, Role: FOLLOWER
I20240503 23:27:41.901583 6636 consensus_queue.cc:260] T 7441a981eb93446c9d9367af49462e73 P 3fc1cf80b12b42c2af63d3afec5214a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fc1cf80b12b42c2af63d3afec5214a7" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 34273 } } peers { permanent_uuid: "21c0231e5eee4c109d088c02f139c112" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 44345 } } peers { permanent_uuid: "c71e941c84ba45deb326c90968f40308" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 35323 } }
I20240503 23:27:41.901644 6637 consensus_queue.cc:260] T fb79ea9cea314c4cab4bed81163a8ded P 21c0231e5eee4c109d088c02f139c112 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fc1cf80b12b42c2af63d3afec5214a7" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 34273 } } peers { permanent_uuid: "c71e941c84ba45deb326c90968f40308" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 35323 } } peers { permanent_uuid: "21c0231e5eee4c109d088c02f139c112" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 44345 } }
I20240503 23:27:41.902362 6636 ts_tablet_manager.cc:1398] T 7441a981eb93446c9d9367af49462e73 P 3fc1cf80b12b42c2af63d3afec5214a7: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240503 23:27:41.902762 6636 tablet_bootstrap.cc:492] T fb79ea9cea314c4cab4bed81163a8ded P 3fc1cf80b12b42c2af63d3afec5214a7: Bootstrap starting.
I20240503 23:27:41.905193 6636 tablet_bootstrap.cc:654] T fb79ea9cea314c4cab4bed81163a8ded P 3fc1cf80b12b42c2af63d3afec5214a7: Neither blocks nor log segments found. Creating new log.
I20240503 23:27:41.907951 6641 ts_tablet_manager.cc:1398] T 7441a981eb93446c9d9367af49462e73 P c71e941c84ba45deb326c90968f40308: Time spent starting tablet: real 0.012s user 0.009s sys 0.000s
I20240503 23:27:41.908501 6641 tablet_bootstrap.cc:492] T e9273b9533d14394b454e6c31ae4deb5 P c71e941c84ba45deb326c90968f40308: Bootstrap starting.
I20240503 23:27:41.910686 6641 tablet_bootstrap.cc:654] T e9273b9533d14394b454e6c31ae4deb5 P c71e941c84ba45deb326c90968f40308: Neither blocks nor log segments found. Creating new log.
I20240503 23:27:41.917979 6636 tablet_bootstrap.cc:492] T fb79ea9cea314c4cab4bed81163a8ded P 3fc1cf80b12b42c2af63d3afec5214a7: No bootstrap required, opened a new log
I20240503 23:27:41.918177 6636 ts_tablet_manager.cc:1367] T fb79ea9cea314c4cab4bed81163a8ded P 3fc1cf80b12b42c2af63d3afec5214a7: Time spent bootstrapping tablet: real 0.016s user 0.007s sys 0.007s
I20240503 23:27:41.918958 6636 raft_consensus.cc:348] T fb79ea9cea314c4cab4bed81163a8ded P 3fc1cf80b12b42c2af63d3afec5214a7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fc1cf80b12b42c2af63d3afec5214a7" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 34273 } } peers { permanent_uuid: "c71e941c84ba45deb326c90968f40308" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 35323 } } peers { permanent_uuid: "21c0231e5eee4c109d088c02f139c112" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 44345 } }
I20240503 23:27:41.919185 6636 raft_consensus.cc:374] T fb79ea9cea314c4cab4bed81163a8ded P 3fc1cf80b12b42c2af63d3afec5214a7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 23:27:41.919275 6636 raft_consensus.cc:729] T fb79ea9cea314c4cab4bed81163a8ded P 3fc1cf80b12b42c2af63d3afec5214a7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3fc1cf80b12b42c2af63d3afec5214a7, State: Initialized, Role: FOLLOWER
I20240503 23:27:41.919530 6636 consensus_queue.cc:260] T fb79ea9cea314c4cab4bed81163a8ded P 3fc1cf80b12b42c2af63d3afec5214a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fc1cf80b12b42c2af63d3afec5214a7" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 34273 } } peers { permanent_uuid: "c71e941c84ba45deb326c90968f40308" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 35323 } } peers { permanent_uuid: "21c0231e5eee4c109d088c02f139c112" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 44345 } }
I20240503 23:27:41.920084 6641 tablet_bootstrap.cc:492] T e9273b9533d14394b454e6c31ae4deb5 P c71e941c84ba45deb326c90968f40308: No bootstrap required, opened a new log
I20240503 23:27:41.920133 6636 ts_tablet_manager.cc:1398] T fb79ea9cea314c4cab4bed81163a8ded P 3fc1cf80b12b42c2af63d3afec5214a7: Time spent starting tablet: real 0.002s user 0.002s sys 0.001s
I20240503 23:27:41.920329 6641 ts_tablet_manager.cc:1367] T e9273b9533d14394b454e6c31ae4deb5 P c71e941c84ba45deb326c90968f40308: Time spent bootstrapping tablet: real 0.012s user 0.001s sys 0.009s
I20240503 23:27:41.921312 6641 raft_consensus.cc:348] T e9273b9533d14394b454e6c31ae4deb5 P c71e941c84ba45deb326c90968f40308 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fc1cf80b12b42c2af63d3afec5214a7" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 34273 } } peers { permanent_uuid: "c71e941c84ba45deb326c90968f40308" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 35323 } } peers { permanent_uuid: "21c0231e5eee4c109d088c02f139c112" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 44345 } }
I20240503 23:27:41.921564 6641 raft_consensus.cc:374] T e9273b9533d14394b454e6c31ae4deb5 P c71e941c84ba45deb326c90968f40308 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 23:27:41.921662 6641 raft_consensus.cc:729] T e9273b9533d14394b454e6c31ae4deb5 P c71e941c84ba45deb326c90968f40308 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c71e941c84ba45deb326c90968f40308, State: Initialized, Role: FOLLOWER
I20240503 23:27:41.921937 6641 consensus_queue.cc:260] T e9273b9533d14394b454e6c31ae4deb5 P c71e941c84ba45deb326c90968f40308 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fc1cf80b12b42c2af63d3afec5214a7" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 34273 } } peers { permanent_uuid: "c71e941c84ba45deb326c90968f40308" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 35323 } } peers { permanent_uuid: "21c0231e5eee4c109d088c02f139c112" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 44345 } }
I20240503 23:27:41.922626 6641 ts_tablet_manager.cc:1398] T e9273b9533d14394b454e6c31ae4deb5 P c71e941c84ba45deb326c90968f40308: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 23:27:41.923000 6641 tablet_bootstrap.cc:492] T fb79ea9cea314c4cab4bed81163a8ded P c71e941c84ba45deb326c90968f40308: Bootstrap starting.
I20240503 23:27:41.924981 6641 tablet_bootstrap.cc:654] T fb79ea9cea314c4cab4bed81163a8ded P c71e941c84ba45deb326c90968f40308: Neither blocks nor log segments found. Creating new log.
I20240503 23:27:41.925487 6637 ts_tablet_manager.cc:1398] T fb79ea9cea314c4cab4bed81163a8ded P 21c0231e5eee4c109d088c02f139c112: Time spent starting tablet: real 0.025s user 0.024s sys 0.000s
I20240503 23:27:41.925908 6637 tablet_bootstrap.cc:492] T e9273b9533d14394b454e6c31ae4deb5 P 21c0231e5eee4c109d088c02f139c112: Bootstrap starting.
I20240503 23:27:41.927872 6637 tablet_bootstrap.cc:654] T e9273b9533d14394b454e6c31ae4deb5 P 21c0231e5eee4c109d088c02f139c112: Neither blocks nor log segments found. Creating new log.
I20240503 23:27:41.930371 6637 tablet_bootstrap.cc:492] T e9273b9533d14394b454e6c31ae4deb5 P 21c0231e5eee4c109d088c02f139c112: No bootstrap required, opened a new log
I20240503 23:27:41.930371 6641 tablet_bootstrap.cc:492] T fb79ea9cea314c4cab4bed81163a8ded P c71e941c84ba45deb326c90968f40308: No bootstrap required, opened a new log
I20240503 23:27:41.930591 6641 ts_tablet_manager.cc:1367] T fb79ea9cea314c4cab4bed81163a8ded P c71e941c84ba45deb326c90968f40308: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20240503 23:27:41.930591 6637 ts_tablet_manager.cc:1367] T e9273b9533d14394b454e6c31ae4deb5 P 21c0231e5eee4c109d088c02f139c112: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.002s
I20240503 23:27:41.931489 6637 raft_consensus.cc:348] T e9273b9533d14394b454e6c31ae4deb5 P 21c0231e5eee4c109d088c02f139c112 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fc1cf80b12b42c2af63d3afec5214a7" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 34273 } } peers { permanent_uuid: "c71e941c84ba45deb326c90968f40308" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 35323 } } peers { permanent_uuid: "21c0231e5eee4c109d088c02f139c112" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 44345 } }
I20240503 23:27:41.931489 6641 raft_consensus.cc:348] T fb79ea9cea314c4cab4bed81163a8ded P c71e941c84ba45deb326c90968f40308 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fc1cf80b12b42c2af63d3afec5214a7" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 34273 } } peers { permanent_uuid: "c71e941c84ba45deb326c90968f40308" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 35323 } } peers { permanent_uuid: "21c0231e5eee4c109d088c02f139c112" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 44345 } }
I20240503 23:27:41.931769 6641 raft_consensus.cc:374] T fb79ea9cea314c4cab4bed81163a8ded P c71e941c84ba45deb326c90968f40308 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 23:27:41.931769 6637 raft_consensus.cc:374] T e9273b9533d14394b454e6c31ae4deb5 P 21c0231e5eee4c109d088c02f139c112 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 23:27:41.931864 6637 raft_consensus.cc:729] T e9273b9533d14394b454e6c31ae4deb5 P 21c0231e5eee4c109d088c02f139c112 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 21c0231e5eee4c109d088c02f139c112, State: Initialized, Role: FOLLOWER
I20240503 23:27:41.931864 6641 raft_consensus.cc:729] T fb79ea9cea314c4cab4bed81163a8ded P c71e941c84ba45deb326c90968f40308 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c71e941c84ba45deb326c90968f40308, State: Initialized, Role: FOLLOWER
I20240503 23:27:41.932142 6637 consensus_queue.cc:260] T e9273b9533d14394b454e6c31ae4deb5 P 21c0231e5eee4c109d088c02f139c112 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fc1cf80b12b42c2af63d3afec5214a7" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 34273 } } peers { permanent_uuid: "c71e941c84ba45deb326c90968f40308" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 35323 } } peers { permanent_uuid: "21c0231e5eee4c109d088c02f139c112" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 44345 } }
I20240503 23:27:41.932142 6641 consensus_queue.cc:260] T fb79ea9cea314c4cab4bed81163a8ded P c71e941c84ba45deb326c90968f40308 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fc1cf80b12b42c2af63d3afec5214a7" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 34273 } } peers { permanent_uuid: "c71e941c84ba45deb326c90968f40308" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 35323 } } peers { permanent_uuid: "21c0231e5eee4c109d088c02f139c112" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 44345 } }
I20240503 23:27:41.932936 6641 ts_tablet_manager.cc:1398] T fb79ea9cea314c4cab4bed81163a8ded P c71e941c84ba45deb326c90968f40308: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 23:27:41.932936 6637 ts_tablet_manager.cc:1398] T e9273b9533d14394b454e6c31ae4deb5 P 21c0231e5eee4c109d088c02f139c112: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240503 23:27:41.933359 6641 tablet_bootstrap.cc:492] T ad1b9cc708db4d289c2e847874bf747b P c71e941c84ba45deb326c90968f40308: Bootstrap starting.
I20240503 23:27:41.934937 6641 tablet_bootstrap.cc:654] T ad1b9cc708db4d289c2e847874bf747b P c71e941c84ba45deb326c90968f40308: Neither blocks nor log segments found. Creating new log.
I20240503 23:27:41.937964 6641 tablet_bootstrap.cc:492] T ad1b9cc708db4d289c2e847874bf747b P c71e941c84ba45deb326c90968f40308: No bootstrap required, opened a new log
I20240503 23:27:41.938158 6641 ts_tablet_manager.cc:1367] T ad1b9cc708db4d289c2e847874bf747b P c71e941c84ba45deb326c90968f40308: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20240503 23:27:41.938880 6641 raft_consensus.cc:348] T ad1b9cc708db4d289c2e847874bf747b P c71e941c84ba45deb326c90968f40308 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fc1cf80b12b42c2af63d3afec5214a7" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 34273 } } peers { permanent_uuid: "c71e941c84ba45deb326c90968f40308" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 35323 } } peers { permanent_uuid: "21c0231e5eee4c109d088c02f139c112" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 44345 } }
I20240503 23:27:41.939078 6641 raft_consensus.cc:374] T ad1b9cc708db4d289c2e847874bf747b P c71e941c84ba45deb326c90968f40308 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 23:27:41.939152 6641 raft_consensus.cc:729] T ad1b9cc708db4d289c2e847874bf747b P c71e941c84ba45deb326c90968f40308 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c71e941c84ba45deb326c90968f40308, State: Initialized, Role: FOLLOWER
I20240503 23:27:41.939379 6641 consensus_queue.cc:260] T ad1b9cc708db4d289c2e847874bf747b P c71e941c84ba45deb326c90968f40308 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fc1cf80b12b42c2af63d3afec5214a7" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 34273 } } peers { permanent_uuid: "c71e941c84ba45deb326c90968f40308" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 35323 } } peers { permanent_uuid: "21c0231e5eee4c109d088c02f139c112" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 44345 } }
I20240503 23:27:41.939846 6641 ts_tablet_manager.cc:1398] T ad1b9cc708db4d289c2e847874bf747b P c71e941c84ba45deb326c90968f40308: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
W20240503 23:27:41.941643 6543 tablet.cc:2369] T 7441a981eb93446c9d9367af49462e73 P c71e941c84ba45deb326c90968f40308: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240503 23:27:41.956441 6640 raft_consensus.cc:482] T 7441a981eb93446c9d9367af49462e73 P 21c0231e5eee4c109d088c02f139c112 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 23:27:41.956750 6640 raft_consensus.cc:504] T 7441a981eb93446c9d9367af49462e73 P 21c0231e5eee4c109d088c02f139c112 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fc1cf80b12b42c2af63d3afec5214a7" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 34273 } } peers { permanent_uuid: "21c0231e5eee4c109d088c02f139c112" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 44345 } } peers { permanent_uuid: "c71e941c84ba45deb326c90968f40308" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 35323 } }
I20240503 23:27:41.957639 6640 leader_election.cc:288] T 7441a981eb93446c9d9367af49462e73 P 21c0231e5eee4c109d088c02f139c112 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3fc1cf80b12b42c2af63d3afec5214a7 (127.0.100.1:34273), c71e941c84ba45deb326c90968f40308 (127.0.100.2:35323)
I20240503 23:27:41.965016 6517 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7441a981eb93446c9d9367af49462e73" candidate_uuid: "21c0231e5eee4c109d088c02f139c112" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c71e941c84ba45deb326c90968f40308" is_pre_election: true
I20240503 23:27:41.965016 6442 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7441a981eb93446c9d9367af49462e73" candidate_uuid: "21c0231e5eee4c109d088c02f139c112" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3fc1cf80b12b42c2af63d3afec5214a7" is_pre_election: true
I20240503 23:27:41.965420 6442 raft_consensus.cc:2454] T 7441a981eb93446c9d9367af49462e73 P 3fc1cf80b12b42c2af63d3afec5214a7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 21c0231e5eee4c109d088c02f139c112 in term 0.
I20240503 23:27:41.965420 6517 raft_consensus.cc:2454] T 7441a981eb93446c9d9367af49462e73 P c71e941c84ba45deb326c90968f40308 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 21c0231e5eee4c109d088c02f139c112 in term 0.
I20240503 23:27:41.966002 6556 leader_election.cc:302] T 7441a981eb93446c9d9367af49462e73 P 21c0231e5eee4c109d088c02f139c112 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 21c0231e5eee4c109d088c02f139c112, 3fc1cf80b12b42c2af63d3afec5214a7; no voters:
I20240503 23:27:41.966277 6640 raft_consensus.cc:2781] T 7441a981eb93446c9d9367af49462e73 P 21c0231e5eee4c109d088c02f139c112 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 23:27:41.966379 6640 raft_consensus.cc:482] T 7441a981eb93446c9d9367af49462e73 P 21c0231e5eee4c109d088c02f139c112 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 23:27:41.966468 6640 raft_consensus.cc:3037] T 7441a981eb93446c9d9367af49462e73 P 21c0231e5eee4c109d088c02f139c112 [term 0 FOLLOWER]: Advancing to term 1
I20240503 23:27:41.968842 6640 raft_consensus.cc:504] T 7441a981eb93446c9d9367af49462e73 P 21c0231e5eee4c109d088c02f139c112 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fc1cf80b12b42c2af63d3afec5214a7" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 34273 } } peers { permanent_uuid: "21c0231e5eee4c109d088c02f139c112" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 44345 } } peers { permanent_uuid: "c71e941c84ba45deb326c90968f40308" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 35323 } }
I20240503 23:27:41.969581 6640 leader_election.cc:288] T 7441a981eb93446c9d9367af49462e73 P 21c0231e5eee4c109d088c02f139c112 [CANDIDATE]: Term 1 election: Requested vote from peers 3fc1cf80b12b42c2af63d3afec5214a7 (127.0.100.1:34273), c71e941c84ba45deb326c90968f40308 (127.0.100.2:35323)
I20240503 23:27:41.970295 6442 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7441a981eb93446c9d9367af49462e73" candidate_uuid: "21c0231e5eee4c109d088c02f139c112" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3fc1cf80b12b42c2af63d3afec5214a7"
I20240503 23:27:41.970295 6517 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7441a981eb93446c9d9367af49462e73" candidate_uuid: "21c0231e5eee4c109d088c02f139c112" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c71e941c84ba45deb326c90968f40308"
I20240503 23:27:41.970693 6442 raft_consensus.cc:3037] T 7441a981eb93446c9d9367af49462e73 P 3fc1cf80b12b42c2af63d3afec5214a7 [term 0 FOLLOWER]: Advancing to term 1
I20240503 23:27:41.970693 6517 raft_consensus.cc:3037] T 7441a981eb93446c9d9367af49462e73 P c71e941c84ba45deb326c90968f40308 [term 0 FOLLOWER]: Advancing to term 1
I20240503 23:27:41.973083 6442 raft_consensus.cc:2454] T 7441a981eb93446c9d9367af49462e73 P 3fc1cf80b12b42c2af63d3afec5214a7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 21c0231e5eee4c109d088c02f139c112 in term 1.
I20240503 23:27:41.973498 6517 raft_consensus.cc:2454] T 7441a981eb93446c9d9367af49462e73 P c71e941c84ba45deb326c90968f40308 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 21c0231e5eee4c109d088c02f139c112 in term 1.
I20240503 23:27:41.973584 6556 leader_election.cc:302] T 7441a981eb93446c9d9367af49462e73 P 21c0231e5eee4c109d088c02f139c112 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 21c0231e5eee4c109d088c02f139c112, 3fc1cf80b12b42c2af63d3afec5214a7; no voters:
I20240503 23:27:41.974161 6640 raft_consensus.cc:2781] T 7441a981eb93446c9d9367af49462e73 P 21c0231e5eee4c109d088c02f139c112 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 23:27:41.974632 6640 raft_consensus.cc:686] T 7441a981eb93446c9d9367af49462e73 P 21c0231e5eee4c109d088c02f139c112 [term 1 LEADER]: Becoming Leader. State: Replica: 21c0231e5eee4c109d088c02f139c112, State: Running, Role: LEADER
I20240503 23:27:41.974933 6640 consensus_queue.cc:237] T 7441a981eb93446c9d9367af49462e73 P 21c0231e5eee4c109d088c02f139c112 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fc1cf80b12b42c2af63d3afec5214a7" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 34273 } } peers { permanent_uuid: "21c0231e5eee4c109d088c02f139c112" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 44345 } } peers { permanent_uuid: "c71e941c84ba45deb326c90968f40308" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 35323 } }
I20240503 23:27:41.978305 6338 catalog_manager.cc:5484] T 7441a981eb93446c9d9367af49462e73 P 21c0231e5eee4c109d088c02f139c112 reported cstate change: term changed from 0 to 1, leader changed from <none> to 21c0231e5eee4c109d088c02f139c112 (127.0.100.3). New cstate: current_term: 1 leader_uuid: "21c0231e5eee4c109d088c02f139c112" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fc1cf80b12b42c2af63d3afec5214a7" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 34273 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "21c0231e5eee4c109d088c02f139c112" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 44345 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c71e941c84ba45deb326c90968f40308" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 35323 } health_report { overall_health: UNKNOWN } } }
I20240503 23:27:42.047024 6640 raft_consensus.cc:482] T e9273b9533d14394b454e6c31ae4deb5 P 21c0231e5eee4c109d088c02f139c112 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 23:27:42.047230 6640 raft_consensus.cc:504] T e9273b9533d14394b454e6c31ae4deb5 P 21c0231e5eee4c109d088c02f139c112 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fc1cf80b12b42c2af63d3afec5214a7" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 34273 } } peers { permanent_uuid: "c71e941c84ba45deb326c90968f40308" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 35323 } } peers { permanent_uuid: "21c0231e5eee4c109d088c02f139c112" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 44345 } }
I20240503 23:27:42.047849 6640 leader_election.cc:288] T e9273b9533d14394b454e6c31ae4deb5 P 21c0231e5eee4c109d088c02f139c112 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3fc1cf80b12b42c2af63d3afec5214a7 (127.0.100.1:34273), c71e941c84ba45deb326c90968f40308 (127.0.100.2:35323)
I20240503 23:27:42.048264 6640 raft_consensus.cc:482] T ad1b9cc708db4d289c2e847874bf747b P 21c0231e5eee4c109d088c02f139c112 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 23:27:42.048645 6442 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9273b9533d14394b454e6c31ae4deb5" candidate_uuid: "21c0231e5eee4c109d088c02f139c112" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3fc1cf80b12b42c2af63d3afec5214a7" is_pre_election: true
I20240503 23:27:42.048789 6517 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9273b9533d14394b454e6c31ae4deb5" candidate_uuid: "21c0231e5eee4c109d088c02f139c112" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c71e941c84ba45deb326c90968f40308" is_pre_election: true
I20240503 23:27:42.048840 6442 raft_consensus.cc:2454] T e9273b9533d14394b454e6c31ae4deb5 P 3fc1cf80b12b42c2af63d3afec5214a7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 21c0231e5eee4c109d088c02f139c112 in term 0.
I20240503 23:27:42.049058 6517 raft_consensus.cc:2454] T e9273b9533d14394b454e6c31ae4deb5 P c71e941c84ba45deb326c90968f40308 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 21c0231e5eee4c109d088c02f139c112 in term 0.
I20240503 23:27:42.048405 6640 raft_consensus.cc:504] T ad1b9cc708db4d289c2e847874bf747b P 21c0231e5eee4c109d088c02f139c112 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fc1cf80b12b42c2af63d3afec5214a7" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 34273 } } peers { permanent_uuid: "c71e941c84ba45deb326c90968f40308" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 35323 } } peers { permanent_uuid: "21c0231e5eee4c109d088c02f139c112" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 44345 } }
I20240503 23:27:42.049477 6556 leader_election.cc:302] T e9273b9533d14394b454e6c31ae4deb5 P 21c0231e5eee4c109d088c02f139c112 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 21c0231e5eee4c109d088c02f139c112, 3fc1cf80b12b42c2af63d3afec5214a7; no voters:
I20240503 23:27:42.049836 6656 raft_consensus.cc:2781] T e9273b9533d14394b454e6c31ae4deb5 P 21c0231e5eee4c109d088c02f139c112 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 23:27:42.049870 6640 leader_election.cc:288] T ad1b9cc708db4d289c2e847874bf747b P 21c0231e5eee4c109d088c02f139c112 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3fc1cf80b12b42c2af63d3afec5214a7 (127.0.100.1:34273), c71e941c84ba45deb326c90968f40308 (127.0.100.2:35323)
I20240503 23:27:42.050033 6656 raft_consensus.cc:482] T e9273b9533d14394b454e6c31ae4deb5 P 21c0231e5eee4c109d088c02f139c112 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 23:27:42.050158 6656 raft_consensus.cc:3037] T e9273b9533d14394b454e6c31ae4deb5 P 21c0231e5eee4c109d088c02f139c112 [term 0 FOLLOWER]: Advancing to term 1
I20240503 23:27:42.050290 6442 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ad1b9cc708db4d289c2e847874bf747b" candidate_uuid: "21c0231e5eee4c109d088c02f139c112" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3fc1cf80b12b42c2af63d3afec5214a7" is_pre_election: true
I20240503 23:27:42.050498 6442 raft_consensus.cc:2454] T ad1b9cc708db4d289c2e847874bf747b P 3fc1cf80b12b42c2af63d3afec5214a7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 21c0231e5eee4c109d088c02f139c112 in term 0.
I20240503 23:27:42.050611 6517 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ad1b9cc708db4d289c2e847874bf747b" candidate_uuid: "21c0231e5eee4c109d088c02f139c112" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c71e941c84ba45deb326c90968f40308" is_pre_election: true
I20240503 23:27:42.050823 6517 raft_consensus.cc:2454] T ad1b9cc708db4d289c2e847874bf747b P c71e941c84ba45deb326c90968f40308 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 21c0231e5eee4c109d088c02f139c112 in term 0.
I20240503 23:27:42.051100 6556 leader_election.cc:302] T ad1b9cc708db4d289c2e847874bf747b P 21c0231e5eee4c109d088c02f139c112 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 21c0231e5eee4c109d088c02f139c112, 3fc1cf80b12b42c2af63d3afec5214a7; no voters:
I20240503 23:27:42.051398 6640 raft_consensus.cc:2781] T ad1b9cc708db4d289c2e847874bf747b P 21c0231e5eee4c109d088c02f139c112 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 23:27:42.051484 6640 raft_consensus.cc:482] T ad1b9cc708db4d289c2e847874bf747b P 21c0231e5eee4c109d088c02f139c112 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 23:27:42.051570 6640 raft_consensus.cc:3037] T ad1b9cc708db4d289c2e847874bf747b P 21c0231e5eee4c109d088c02f139c112 [term 0 FOLLOWER]: Advancing to term 1
I20240503 23:27:42.052624 6656 raft_consensus.cc:504] T e9273b9533d14394b454e6c31ae4deb5 P 21c0231e5eee4c109d088c02f139c112 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fc1cf80b12b42c2af63d3afec5214a7" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 34273 } } peers { permanent_uuid: "c71e941c84ba45deb326c90968f40308" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 35323 } } peers { permanent_uuid: "21c0231e5eee4c109d088c02f139c112" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 44345 } }
I20240503 23:27:42.053274 6656 leader_election.cc:288] T e9273b9533d14394b454e6c31ae4deb5 P 21c0231e5eee4c109d088c02f139c112 [CANDIDATE]: Term 1 election: Requested vote from peers 3fc1cf80b12b42c2af63d3afec5214a7 (127.0.100.1:34273), c71e941c84ba45deb326c90968f40308 (127.0.100.2:35323)
I20240503 23:27:42.053800 6442 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9273b9533d14394b454e6c31ae4deb5" candidate_uuid: "21c0231e5eee4c109d088c02f139c112" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3fc1cf80b12b42c2af63d3afec5214a7"
I20240503 23:27:42.053958 6442 raft_consensus.cc:3037] T e9273b9533d14394b454e6c31ae4deb5 P 3fc1cf80b12b42c2af63d3afec5214a7 [term 0 FOLLOWER]: Advancing to term 1
I20240503 23:27:42.054018 6517 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9273b9533d14394b454e6c31ae4deb5" candidate_uuid: "21c0231e5eee4c109d088c02f139c112" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c71e941c84ba45deb326c90968f40308"
I20240503 23:27:42.054018 6640 raft_consensus.cc:504] T ad1b9cc708db4d289c2e847874bf747b P 21c0231e5eee4c109d088c02f139c112 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fc1cf80b12b42c2af63d3afec5214a7" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 34273 } } peers { permanent_uuid: "c71e941c84ba45deb326c90968f40308" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 35323 } } peers { permanent_uuid: "21c0231e5eee4c109d088c02f139c112" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 44345 } }
I20240503 23:27:42.054252 6517 raft_consensus.cc:3037] T e9273b9533d14394b454e6c31ae4deb5 P c71e941c84ba45deb326c90968f40308 [term 0 FOLLOWER]: Advancing to term 1
I20240503 23:27:42.054759 6640 leader_election.cc:288] T ad1b9cc708db4d289c2e847874bf747b P 21c0231e5eee4c109d088c02f139c112 [CANDIDATE]: Term 1 election: Requested vote from peers 3fc1cf80b12b42c2af63d3afec5214a7 (127.0.100.1:34273), c71e941c84ba45deb326c90968f40308 (127.0.100.2:35323)
I20240503 23:27:42.055476 6516 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ad1b9cc708db4d289c2e847874bf747b" candidate_uuid: "21c0231e5eee4c109d088c02f139c112" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c71e941c84ba45deb326c90968f40308"
I20240503 23:27:42.055683 6516 raft_consensus.cc:3037] T ad1b9cc708db4d289c2e847874bf747b P c71e941c84ba45deb326c90968f40308 [term 0 FOLLOWER]: Advancing to term 1
I20240503 23:27:42.056692 6517 raft_consensus.cc:2454] T e9273b9533d14394b454e6c31ae4deb5 P c71e941c84ba45deb326c90968f40308 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 21c0231e5eee4c109d088c02f139c112 in term 1.
I20240503 23:27:42.057039 6442 raft_consensus.cc:2454] T e9273b9533d14394b454e6c31ae4deb5 P 3fc1cf80b12b42c2af63d3afec5214a7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 21c0231e5eee4c109d088c02f139c112 in term 1.
I20240503 23:27:42.057200 6556 leader_election.cc:302] T e9273b9533d14394b454e6c31ae4deb5 P 21c0231e5eee4c109d088c02f139c112 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 21c0231e5eee4c109d088c02f139c112, c71e941c84ba45deb326c90968f40308; no voters:
I20240503 23:27:42.057051 6441 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ad1b9cc708db4d289c2e847874bf747b" candidate_uuid: "21c0231e5eee4c109d088c02f139c112" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3fc1cf80b12b42c2af63d3afec5214a7"
I20240503 23:27:42.057358 6441 raft_consensus.cc:3037] T ad1b9cc708db4d289c2e847874bf747b P 3fc1cf80b12b42c2af63d3afec5214a7 [term 0 FOLLOWER]: Advancing to term 1
I20240503 23:27:42.057652 6640 raft_consensus.cc:2781] T e9273b9533d14394b454e6c31ae4deb5 P 21c0231e5eee4c109d088c02f139c112 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 23:27:42.057821 6640 raft_consensus.cc:686] T e9273b9533d14394b454e6c31ae4deb5 P 21c0231e5eee4c109d088c02f139c112 [term 1 LEADER]: Becoming Leader. State: Replica: 21c0231e5eee4c109d088c02f139c112, State: Running, Role: LEADER
I20240503 23:27:42.058161 6640 consensus_queue.cc:237] T e9273b9533d14394b454e6c31ae4deb5 P 21c0231e5eee4c109d088c02f139c112 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fc1cf80b12b42c2af63d3afec5214a7" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 34273 } } peers { permanent_uuid: "c71e941c84ba45deb326c90968f40308" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 35323 } } peers { permanent_uuid: "21c0231e5eee4c109d088c02f139c112" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 44345 } }
I20240503 23:27:42.058960 6516 raft_consensus.cc:2454] T ad1b9cc708db4d289c2e847874bf747b P c71e941c84ba45deb326c90968f40308 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 21c0231e5eee4c109d088c02f139c112 in term 1.
I20240503 23:27:42.059422 6556 leader_election.cc:302] T ad1b9cc708db4d289c2e847874bf747b P 21c0231e5eee4c109d088c02f139c112 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 21c0231e5eee4c109d088c02f139c112, c71e941c84ba45deb326c90968f40308; no voters:
I20240503 23:27:42.059787 6656 raft_consensus.cc:2781] T ad1b9cc708db4d289c2e847874bf747b P 21c0231e5eee4c109d088c02f139c112 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 23:27:42.059820 6441 raft_consensus.cc:2454] T ad1b9cc708db4d289c2e847874bf747b P 3fc1cf80b12b42c2af63d3afec5214a7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 21c0231e5eee4c109d088c02f139c112 in term 1.
I20240503 23:27:42.060586 6656 raft_consensus.cc:686] T ad1b9cc708db4d289c2e847874bf747b P 21c0231e5eee4c109d088c02f139c112 [term 1 LEADER]: Becoming Leader. State: Replica: 21c0231e5eee4c109d088c02f139c112, State: Running, Role: LEADER
I20240503 23:27:42.060873 6656 consensus_queue.cc:237] T ad1b9cc708db4d289c2e847874bf747b P 21c0231e5eee4c109d088c02f139c112 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fc1cf80b12b42c2af63d3afec5214a7" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 34273 } } peers { permanent_uuid: "c71e941c84ba45deb326c90968f40308" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 35323 } } peers { permanent_uuid: "21c0231e5eee4c109d088c02f139c112" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 44345 } }
I20240503 23:27:42.061756 6338 catalog_manager.cc:5484] T e9273b9533d14394b454e6c31ae4deb5 P 21c0231e5eee4c109d088c02f139c112 reported cstate change: term changed from 0 to 1, leader changed from <none> to 21c0231e5eee4c109d088c02f139c112 (127.0.100.3). New cstate: current_term: 1 leader_uuid: "21c0231e5eee4c109d088c02f139c112" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fc1cf80b12b42c2af63d3afec5214a7" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 34273 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c71e941c84ba45deb326c90968f40308" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 35323 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "21c0231e5eee4c109d088c02f139c112" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 44345 } health_report { overall_health: HEALTHY } } }
I20240503 23:27:42.066912 6338 catalog_manager.cc:5484] T ad1b9cc708db4d289c2e847874bf747b P 21c0231e5eee4c109d088c02f139c112 reported cstate change: term changed from 0 to 1, leader changed from <none> to 21c0231e5eee4c109d088c02f139c112 (127.0.100.3). New cstate: current_term: 1 leader_uuid: "21c0231e5eee4c109d088c02f139c112" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fc1cf80b12b42c2af63d3afec5214a7" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 34273 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c71e941c84ba45deb326c90968f40308" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 35323 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "21c0231e5eee4c109d088c02f139c112" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 44345 } health_report { overall_health: HEALTHY } } }
I20240503 23:27:42.229907 6656 raft_consensus.cc:482] T fb79ea9cea314c4cab4bed81163a8ded P 21c0231e5eee4c109d088c02f139c112 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 23:27:42.230163 6656 raft_consensus.cc:504] T fb79ea9cea314c4cab4bed81163a8ded P 21c0231e5eee4c109d088c02f139c112 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fc1cf80b12b42c2af63d3afec5214a7" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 34273 } } peers { permanent_uuid: "c71e941c84ba45deb326c90968f40308" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 35323 } } peers { permanent_uuid: "21c0231e5eee4c109d088c02f139c112" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 44345 } }
I20240503 23:27:42.231047 6656 leader_election.cc:288] T fb79ea9cea314c4cab4bed81163a8ded P 21c0231e5eee4c109d088c02f139c112 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3fc1cf80b12b42c2af63d3afec5214a7 (127.0.100.1:34273), c71e941c84ba45deb326c90968f40308 (127.0.100.2:35323)
I20240503 23:27:42.231837 6441 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb79ea9cea314c4cab4bed81163a8ded" candidate_uuid: "21c0231e5eee4c109d088c02f139c112" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3fc1cf80b12b42c2af63d3afec5214a7" is_pre_election: true
I20240503 23:27:42.231837 6516 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb79ea9cea314c4cab4bed81163a8ded" candidate_uuid: "21c0231e5eee4c109d088c02f139c112" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c71e941c84ba45deb326c90968f40308" is_pre_election: true
I20240503 23:27:42.232085 6516 raft_consensus.cc:2454] T fb79ea9cea314c4cab4bed81163a8ded P c71e941c84ba45deb326c90968f40308 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 21c0231e5eee4c109d088c02f139c112 in term 0.
I20240503 23:27:42.232085 6441 raft_consensus.cc:2454] T fb79ea9cea314c4cab4bed81163a8ded P 3fc1cf80b12b42c2af63d3afec5214a7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 21c0231e5eee4c109d088c02f139c112 in term 0.
I20240503 23:27:42.232600 6556 leader_election.cc:302] T fb79ea9cea314c4cab4bed81163a8ded P 21c0231e5eee4c109d088c02f139c112 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 21c0231e5eee4c109d088c02f139c112, c71e941c84ba45deb326c90968f40308; no voters:
I20240503 23:27:42.232862 6656 raft_consensus.cc:2781] T fb79ea9cea314c4cab4bed81163a8ded P 21c0231e5eee4c109d088c02f139c112 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 23:27:42.232968 6656 raft_consensus.cc:482] T fb79ea9cea314c4cab4bed81163a8ded P 21c0231e5eee4c109d088c02f139c112 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 23:27:42.233078 6656 raft_consensus.cc:3037] T fb79ea9cea314c4cab4bed81163a8ded P 21c0231e5eee4c109d088c02f139c112 [term 0 FOLLOWER]: Advancing to term 1
I20240503 23:27:42.235406 6656 raft_consensus.cc:504] T fb79ea9cea314c4cab4bed81163a8ded P 21c0231e5eee4c109d088c02f139c112 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fc1cf80b12b42c2af63d3afec5214a7" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 34273 } } peers { permanent_uuid: "c71e941c84ba45deb326c90968f40308" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 35323 } } peers { permanent_uuid: "21c0231e5eee4c109d088c02f139c112" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 44345 } }
I20240503 23:27:42.236141 6656 leader_election.cc:288] T fb79ea9cea314c4cab4bed81163a8ded P 21c0231e5eee4c109d088c02f139c112 [CANDIDATE]: Term 1 election: Requested vote from peers 3fc1cf80b12b42c2af63d3afec5214a7 (127.0.100.1:34273), c71e941c84ba45deb326c90968f40308 (127.0.100.2:35323)
I20240503 23:27:42.236831 6516 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb79ea9cea314c4cab4bed81163a8ded" candidate_uuid: "21c0231e5eee4c109d088c02f139c112" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c71e941c84ba45deb326c90968f40308"
I20240503 23:27:42.236832 6441 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb79ea9cea314c4cab4bed81163a8ded" candidate_uuid: "21c0231e5eee4c109d088c02f139c112" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3fc1cf80b12b42c2af63d3afec5214a7"
I20240503 23:27:42.237097 6516 raft_consensus.cc:3037] T fb79ea9cea314c4cab4bed81163a8ded P c71e941c84ba45deb326c90968f40308 [term 0 FOLLOWER]: Advancing to term 1
I20240503 23:27:42.237174 6441 raft_consensus.cc:3037] T fb79ea9cea314c4cab4bed81163a8ded P 3fc1cf80b12b42c2af63d3afec5214a7 [term 0 FOLLOWER]: Advancing to term 1
I20240503 23:27:42.239447 6441 raft_consensus.cc:2454] T fb79ea9cea314c4cab4bed81163a8ded P 3fc1cf80b12b42c2af63d3afec5214a7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 21c0231e5eee4c109d088c02f139c112 in term 1.
I20240503 23:27:42.239447 6516 raft_consensus.cc:2454] T fb79ea9cea314c4cab4bed81163a8ded P c71e941c84ba45deb326c90968f40308 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 21c0231e5eee4c109d088c02f139c112 in term 1.
I20240503 23:27:42.240101 6556 leader_election.cc:302] T fb79ea9cea314c4cab4bed81163a8ded P 21c0231e5eee4c109d088c02f139c112 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 21c0231e5eee4c109d088c02f139c112, c71e941c84ba45deb326c90968f40308; no voters:
I20240503 23:27:42.240479 6656 raft_consensus.cc:2781] T fb79ea9cea314c4cab4bed81163a8ded P 21c0231e5eee4c109d088c02f139c112 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 23:27:42.240689 6656 raft_consensus.cc:686] T fb79ea9cea314c4cab4bed81163a8ded P 21c0231e5eee4c109d088c02f139c112 [term 1 LEADER]: Becoming Leader. State: Replica: 21c0231e5eee4c109d088c02f139c112, State: Running, Role: LEADER
I20240503 23:27:42.240978 6656 consensus_queue.cc:237] T fb79ea9cea314c4cab4bed81163a8ded P 21c0231e5eee4c109d088c02f139c112 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fc1cf80b12b42c2af63d3afec5214a7" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 34273 } } peers { permanent_uuid: "c71e941c84ba45deb326c90968f40308" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 35323 } } peers { permanent_uuid: "21c0231e5eee4c109d088c02f139c112" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 44345 } }
I20240503 23:27:42.244206 6338 catalog_manager.cc:5484] T fb79ea9cea314c4cab4bed81163a8ded P 21c0231e5eee4c109d088c02f139c112 reported cstate change: term changed from 0 to 1, leader changed from <none> to 21c0231e5eee4c109d088c02f139c112 (127.0.100.3). New cstate: current_term: 1 leader_uuid: "21c0231e5eee4c109d088c02f139c112" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fc1cf80b12b42c2af63d3afec5214a7" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 34273 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c71e941c84ba45deb326c90968f40308" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 35323 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "21c0231e5eee4c109d088c02f139c112" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 44345 } health_report { overall_health: HEALTHY } } }
I20240503 23:27:42.272258 400 tablet_server.cc:178] TabletServer@127.0.100.1:0 shutting down...
I20240503 23:27:42.283010 400 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240503 23:27:42.283528 400 tablet_replica.cc:330] T fb79ea9cea314c4cab4bed81163a8ded P 3fc1cf80b12b42c2af63d3afec5214a7: stopping tablet replica
I20240503 23:27:42.283810 400 raft_consensus.cc:2229] T fb79ea9cea314c4cab4bed81163a8ded P 3fc1cf80b12b42c2af63d3afec5214a7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 23:27:42.284040 400 raft_consensus.cc:2258] T fb79ea9cea314c4cab4bed81163a8ded P 3fc1cf80b12b42c2af63d3afec5214a7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 23:27:42.285038 400 tablet_replica.cc:330] T 7441a981eb93446c9d9367af49462e73 P 3fc1cf80b12b42c2af63d3afec5214a7: stopping tablet replica
I20240503 23:27:42.285207 400 raft_consensus.cc:2229] T 7441a981eb93446c9d9367af49462e73 P 3fc1cf80b12b42c2af63d3afec5214a7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 23:27:42.285362 400 raft_consensus.cc:2258] T 7441a981eb93446c9d9367af49462e73 P 3fc1cf80b12b42c2af63d3afec5214a7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 23:27:42.286006 400 tablet_replica.cc:330] T e9273b9533d14394b454e6c31ae4deb5 P 3fc1cf80b12b42c2af63d3afec5214a7: stopping tablet replica
I20240503 23:27:42.286151 400 raft_consensus.cc:2229] T e9273b9533d14394b454e6c31ae4deb5 P 3fc1cf80b12b42c2af63d3afec5214a7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 23:27:42.286276 400 raft_consensus.cc:2258] T e9273b9533d14394b454e6c31ae4deb5 P 3fc1cf80b12b42c2af63d3afec5214a7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 23:27:42.286902 400 tablet_replica.cc:330] T ad1b9cc708db4d289c2e847874bf747b P 3fc1cf80b12b42c2af63d3afec5214a7: stopping tablet replica
I20240503 23:27:42.287025 400 raft_consensus.cc:2229] T ad1b9cc708db4d289c2e847874bf747b P 3fc1cf80b12b42c2af63d3afec5214a7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 23:27:42.287143 400 raft_consensus.cc:2258] T ad1b9cc708db4d289c2e847874bf747b P 3fc1cf80b12b42c2af63d3afec5214a7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 23:27:42.302771 400 tablet_server.cc:194] TabletServer@127.0.100.1:0 shutdown complete.
I20240503 23:27:42.307389 400 tablet_server.cc:178] TabletServer@127.0.100.2:0 shutting down...
I20240503 23:27:42.316475 400 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240503 23:27:42.317080 400 tablet_replica.cc:330] T fb79ea9cea314c4cab4bed81163a8ded P c71e941c84ba45deb326c90968f40308: stopping tablet replica
I20240503 23:27:42.317306 400 raft_consensus.cc:2229] T fb79ea9cea314c4cab4bed81163a8ded P c71e941c84ba45deb326c90968f40308 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 23:27:42.317500 400 raft_consensus.cc:2258] T fb79ea9cea314c4cab4bed81163a8ded P c71e941c84ba45deb326c90968f40308 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 23:27:42.318303 400 tablet_replica.cc:330] T 7441a981eb93446c9d9367af49462e73 P c71e941c84ba45deb326c90968f40308: stopping tablet replica
I20240503 23:27:42.318456 400 raft_consensus.cc:2229] T 7441a981eb93446c9d9367af49462e73 P c71e941c84ba45deb326c90968f40308 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 23:27:42.318595 400 raft_consensus.cc:2258] T 7441a981eb93446c9d9367af49462e73 P c71e941c84ba45deb326c90968f40308 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 23:27:42.319155 400 tablet_replica.cc:330] T ad1b9cc708db4d289c2e847874bf747b P c71e941c84ba45deb326c90968f40308: stopping tablet replica
I20240503 23:27:42.319281 400 raft_consensus.cc:2229] T ad1b9cc708db4d289c2e847874bf747b P c71e941c84ba45deb326c90968f40308 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 23:27:42.319401 400 raft_consensus.cc:2258] T ad1b9cc708db4d289c2e847874bf747b P c71e941c84ba45deb326c90968f40308 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 23:27:42.319911 400 tablet_replica.cc:330] T e9273b9533d14394b454e6c31ae4deb5 P c71e941c84ba45deb326c90968f40308: stopping tablet replica
I20240503 23:27:42.320029 400 raft_consensus.cc:2229] T e9273b9533d14394b454e6c31ae4deb5 P c71e941c84ba45deb326c90968f40308 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 23:27:42.320148 400 raft_consensus.cc:2258] T e9273b9533d14394b454e6c31ae4deb5 P c71e941c84ba45deb326c90968f40308 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 23:27:42.335431 400 tablet_server.cc:194] TabletServer@127.0.100.2:0 shutdown complete.
I20240503 23:27:42.339738 400 tablet_server.cc:178] TabletServer@127.0.100.3:0 shutting down...
I20240503 23:27:42.349512 400 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240503 23:27:42.349987 400 tablet_replica.cc:330] T e9273b9533d14394b454e6c31ae4deb5 P 21c0231e5eee4c109d088c02f139c112: stopping tablet replica
I20240503 23:27:42.350314 400 raft_consensus.cc:2229] T e9273b9533d14394b454e6c31ae4deb5 P 21c0231e5eee4c109d088c02f139c112 [term 1 LEADER]: Raft consensus shutting down.
I20240503 23:27:42.350654 400 pending_rounds.cc:62] T e9273b9533d14394b454e6c31ae4deb5 P 21c0231e5eee4c109d088c02f139c112: Trying to abort 1 pending ops.
I20240503 23:27:42.350814 400 pending_rounds.cc:68] T e9273b9533d14394b454e6c31ae4deb5 P 21c0231e5eee4c109d088c02f139c112: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7023734218993750016 op_type: NO_OP noop_request { }
I20240503 23:27:42.351014 400 raft_consensus.cc:2866] T e9273b9533d14394b454e6c31ae4deb5 P 21c0231e5eee4c109d088c02f139c112 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20240503 23:27:42.351125 400 raft_consensus.cc:2258] T e9273b9533d14394b454e6c31ae4deb5 P 21c0231e5eee4c109d088c02f139c112 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 23:27:42.352237 400 tablet_replica.cc:330] T ad1b9cc708db4d289c2e847874bf747b P 21c0231e5eee4c109d088c02f139c112: stopping tablet replica
I20240503 23:27:42.352442 400 raft_consensus.cc:2229] T ad1b9cc708db4d289c2e847874bf747b P 21c0231e5eee4c109d088c02f139c112 [term 1 LEADER]: Raft consensus shutting down.
I20240503 23:27:42.352752 400 pending_rounds.cc:62] T ad1b9cc708db4d289c2e847874bf747b P 21c0231e5eee4c109d088c02f139c112: Trying to abort 1 pending ops.
I20240503 23:27:42.352824 400 pending_rounds.cc:68] T ad1b9cc708db4d289c2e847874bf747b P 21c0231e5eee4c109d088c02f139c112: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7023734219003854848 op_type: NO_OP noop_request { }
I20240503 23:27:42.352977 400 raft_consensus.cc:2866] T ad1b9cc708db4d289c2e847874bf747b P 21c0231e5eee4c109d088c02f139c112 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20240503 23:27:42.353092 400 raft_consensus.cc:2258] T ad1b9cc708db4d289c2e847874bf747b P 21c0231e5eee4c109d088c02f139c112 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 23:27:42.354009 400 tablet_replica.cc:330] T fb79ea9cea314c4cab4bed81163a8ded P 21c0231e5eee4c109d088c02f139c112: stopping tablet replica
I20240503 23:27:42.354187 400 raft_consensus.cc:2229] T fb79ea9cea314c4cab4bed81163a8ded P 21c0231e5eee4c109d088c02f139c112 [term 1 LEADER]: Raft consensus shutting down.
I20240503 23:27:42.354429 400 pending_rounds.cc:62] T fb79ea9cea314c4cab4bed81163a8ded P 21c0231e5eee4c109d088c02f139c112: Trying to abort 1 pending ops.
I20240503 23:27:42.354493 400 pending_rounds.cc:68] T fb79ea9cea314c4cab4bed81163a8ded P 21c0231e5eee4c109d088c02f139c112: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7023734219741863936 op_type: NO_OP noop_request { }
I20240503 23:27:42.354631 400 raft_consensus.cc:2866] T fb79ea9cea314c4cab4bed81163a8ded P 21c0231e5eee4c109d088c02f139c112 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20240503 23:27:42.354750 400 raft_consensus.cc:2258] T fb79ea9cea314c4cab4bed81163a8ded P 21c0231e5eee4c109d088c02f139c112 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 23:27:42.355624 400 tablet_replica.cc:330] T 7441a981eb93446c9d9367af49462e73 P 21c0231e5eee4c109d088c02f139c112: stopping tablet replica
I20240503 23:27:42.355794 400 raft_consensus.cc:2229] T 7441a981eb93446c9d9367af49462e73 P 21c0231e5eee4c109d088c02f139c112 [term 1 LEADER]: Raft consensus shutting down.
I20240503 23:27:42.356015 400 pending_rounds.cc:62] T 7441a981eb93446c9d9367af49462e73 P 21c0231e5eee4c109d088c02f139c112: Trying to abort 1 pending ops.
I20240503 23:27:42.356079 400 pending_rounds.cc:68] T 7441a981eb93446c9d9367af49462e73 P 21c0231e5eee4c109d088c02f139c112: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7023734218652082176 op_type: NO_OP noop_request { }
I20240503 23:27:42.356236 400 raft_consensus.cc:2866] T 7441a981eb93446c9d9367af49462e73 P 21c0231e5eee4c109d088c02f139c112 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20240503 23:27:42.356346 400 raft_consensus.cc:2258] T 7441a981eb93446c9d9367af49462e73 P 21c0231e5eee4c109d088c02f139c112 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 23:27:42.372362 400 tablet_server.cc:194] TabletServer@127.0.100.3:0 shutdown complete.
I20240503 23:27:42.378002 400 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 23:27:42.395002 6662 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 23:27:42.395994 6663 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 23:27:42.396869 6665 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 23:27:42.397787 400 server_base.cc:1034] running on GCE node
I20240503 23:27:42.398171 400 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240503 23:27:42.398247 400 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240503 23:27:42.398311 400 hybrid_clock.cc:648] HybridClock initialized: now 1714778862398308 us; error 0 us; skew 500 ppm
I20240503 23:27:42.398502 400 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 23:27:42.400223 400 webserver.cc:460] Webserver started at http://127.0.100.4:33573/ using document root <none> and password file <none>
I20240503 23:27:42.400583 400 fs_manager.cc:359] Metadata directory not provided
I20240503 23:27:42.400676 400 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 23:27:42.400846 400 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 23:27:42.401677 400 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZhIiPz/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestHandlingFailedTservers.1714778688451250-400-0/minicluster-data/ts-3-root/instance:
uuid: "c11e09f93341477483d0045fb1165058"
format_stamp: "Formatted at 2024-05-03 23:27:42 on dist-test-slave-jvbp"
I20240503 23:27:42.404896 400 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20240503 23:27:42.407196 6670 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 23:27:42.407801 400 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.001s
I20240503 23:27:42.407963 400 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZhIiPz/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestHandlingFailedTservers.1714778688451250-400-0/minicluster-data/ts-3-root
uuid: "c11e09f93341477483d0045fb1165058"
format_stamp: "Formatted at 2024-05-03 23:27:42 on dist-test-slave-jvbp"
I20240503 23:27:42.408152 400 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZhIiPz/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestHandlingFailedTservers.1714778688451250-400-0/minicluster-data/ts-3-root
metadata directory: /tmp/dist-test-taskZhIiPz/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestHandlingFailedTservers.1714778688451250-400-0/minicluster-data/ts-3-root
1 data directories: /tmp/dist-test-taskZhIiPz/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestHandlingFailedTservers.1714778688451250-400-0/minicluster-data/ts-3-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 23:27:42.427223 400 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 23:27:42.427845 400 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 23:27:42.428516 400 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 23:27:42.429570 400 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 23:27:42.429644 400 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 23:27:42.429728 400 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 23:27:42.429786 400 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 23:27:42.445825 400 rpc_server.cc:307] RPC server started. Bound to: 127.0.100.4:33341
I20240503 23:27:42.445876 6732 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.100.4:33341 every 8 connection(s)
I20240503 23:27:42.453853 6733 heartbeater.cc:347] Connected to a master server at 127.0.100.62:44317
I20240503 23:27:42.454051 6733 heartbeater.cc:464] Registering TS with master...
I20240503 23:27:42.454463 6733 heartbeater.cc:511] Master 127.0.100.62:44317 requested a full tablet report, sending...
I20240503 23:27:42.455556 6338 ts_manager.cc:194] Registered new tserver with Master: c11e09f93341477483d0045fb1165058 (127.0.100.4:33341)
I20240503 23:27:42.456514 6338 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:55758
W20240503 23:27:42.489285 6388 auto_rebalancer.cc:254] failed to send replica move request: Network error: Client connection negotiation failed: client connection to 127.0.100.3:44345: connect: Connection refused (error 111)
W20240503 23:27:42.490732 6326 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.100.3:44345: connect: Connection refused (error 111) [suppressed 7 similar messages]
W20240503 23:27:42.492168 6388 auto_rebalancer.cc:663] Could not move replica: Network error: Client connection negotiation failed: client connection to 127.0.100.3:44345: connect: Connection refused (error 111)
W20240503 23:27:42.492321 6388 auto_rebalancer.cc:264] scheduled replica move failed to complete: Network error: Client connection negotiation failed: client connection to 127.0.100.3:44345: connect: Connection refused (error 111)
W20240503 23:27:42.494092 6388 auto_rebalancer.cc:663] Could not move replica: Network error: Client connection negotiation failed: client connection to 127.0.100.3:44345: connect: Connection refused (error 111)
W20240503 23:27:42.494226 6388 auto_rebalancer.cc:264] scheduled replica move failed to complete: Network error: Client connection negotiation failed: client connection to 127.0.100.3:44345: connect: Connection refused (error 111)
W20240503 23:27:42.495738 6388 auto_rebalancer.cc:663] Could not move replica: Network error: Client connection negotiation failed: client connection to 127.0.100.3:44345: connect: Connection refused (error 111)
W20240503 23:27:42.495855 6388 auto_rebalancer.cc:264] scheduled replica move failed to complete: Network error: Client connection negotiation failed: client connection to 127.0.100.3:44345: connect: Connection refused (error 111)
I20240503 23:27:43.458012 6733 heartbeater.cc:503] Master 127.0.100.62:44317 was elected leader, sending a full tablet report...
W20240503 23:27:43.500455 6388 auto_rebalancer.cc:254] failed to send replica move request: Network error: Client connection negotiation failed: client connection to 127.0.100.3:44345: connect: Connection refused (error 111)
W20240503 23:27:43.505999 6388 auto_rebalancer.cc:663] Could not move replica: Network error: Client connection negotiation failed: client connection to 127.0.100.3:44345: connect: Connection refused (error 111)
W20240503 23:27:43.506143 6388 auto_rebalancer.cc:264] scheduled replica move failed to complete: Network error: Client connection negotiation failed: client connection to 127.0.100.3:44345: connect: Connection refused (error 111)
W20240503 23:27:43.508213 6388 auto_rebalancer.cc:663] Could not move replica: Network error: Client connection negotiation failed: client connection to 127.0.100.3:44345: connect: Connection refused (error 111)
W20240503 23:27:43.508344 6388 auto_rebalancer.cc:264] scheduled replica move failed to complete: Network error: Client connection negotiation failed: client connection to 127.0.100.3:44345: connect: Connection refused (error 111)
W20240503 23:27:43.510195 6388 auto_rebalancer.cc:663] Could not move replica: Network error: Client connection negotiation failed: client connection to 127.0.100.3:44345: connect: Connection refused (error 111)
W20240503 23:27:43.510313 6388 auto_rebalancer.cc:264] scheduled replica move failed to complete: Network error: Client connection negotiation failed: client connection to 127.0.100.3:44345: connect: Connection refused (error 111)
W20240503 23:27:44.514760 6388 auto_rebalancer.cc:254] failed to send replica move request: Network error: Client connection negotiation failed: client connection to 127.0.100.3:44345: connect: Connection refused (error 111)
W20240503 23:27:44.517383 6388 auto_rebalancer.cc:663] Could not move replica: Network error: Client connection negotiation failed: client connection to 127.0.100.3:44345: connect: Connection refused (error 111)
W20240503 23:27:44.517496 6388 auto_rebalancer.cc:264] scheduled replica move failed to complete: Network error: Client connection negotiation failed: client connection to 127.0.100.3:44345: connect: Connection refused (error 111)
W20240503 23:27:44.519294 6388 auto_rebalancer.cc:663] Could not move replica: Network error: Client connection negotiation failed: client connection to 127.0.100.3:44345: connect: Connection refused (error 111)
W20240503 23:27:44.519428 6388 auto_rebalancer.cc:264] scheduled replica move failed to complete: Network error: Client connection negotiation failed: client connection to 127.0.100.3:44345: connect: Connection refused (error 111)
W20240503 23:27:44.521184 6388 auto_rebalancer.cc:663] Could not move replica: Network error: Client connection negotiation failed: client connection to 127.0.100.3:44345: connect: Connection refused (error 111)
W20240503 23:27:44.521292 6388 auto_rebalancer.cc:264] scheduled replica move failed to complete: Network error: Client connection negotiation failed: client connection to 127.0.100.3:44345: connect: Connection refused (error 111)
W20240503 23:27:45.526293 6388 auto_rebalancer.cc:254] failed to send replica move request: Network error: Client connection negotiation failed: client connection to 127.0.100.3:44345: connect: Connection refused (error 111)
W20240503 23:27:45.528828 6388 auto_rebalancer.cc:663] Could not move replica: Network error: Client connection negotiation failed: client connection to 127.0.100.3:44345: connect: Connection refused (error 111)
W20240503 23:27:45.528930 6388 auto_rebalancer.cc:264] scheduled replica move failed to complete: Network error: Client connection negotiation failed: client connection to 127.0.100.3:44345: connect: Connection refused (error 111)
W20240503 23:27:45.530711 6388 auto_rebalancer.cc:663] Could not move replica: Network error: Client connection negotiation failed: client connection to 127.0.100.3:44345: connect: Connection refused (error 111)
W20240503 23:27:45.530797 6388 auto_rebalancer.cc:264] scheduled replica move failed to complete: Network error: Client connection negotiation failed: client connection to 127.0.100.3:44345: connect: Connection refused (error 111)
W20240503 23:27:45.532485 6388 auto_rebalancer.cc:663] Could not move replica: Network error: Client connection negotiation failed: client connection to 127.0.100.3:44345: connect: Connection refused (error 111)
W20240503 23:27:45.532600 6388 auto_rebalancer.cc:264] scheduled replica move failed to complete: Network error: Client connection negotiation failed: client connection to 127.0.100.3:44345: connect: Connection refused (error 111)
W20240503 23:27:46.538527 6388 auto_rebalancer.cc:254] failed to send replica move request: Network error: Client connection negotiation failed: client connection to 127.0.100.3:44345: connect: Connection refused (error 111)
W20240503 23:27:46.541643 6388 auto_rebalancer.cc:663] Could not move replica: Network error: Client connection negotiation failed: client connection to 127.0.100.3:44345: connect: Connection refused (error 111)
W20240503 23:27:46.541781 6388 auto_rebalancer.cc:264] scheduled replica move failed to complete: Network error: Client connection negotiation failed: client connection to 127.0.100.3:44345: connect: Connection refused (error 111)
W20240503 23:27:46.543711 6388 auto_rebalancer.cc:663] Could not move replica: Network error: Client connection negotiation failed: client connection to 127.0.100.3:44345: connect: Connection refused (error 111)
W20240503 23:27:46.543825 6388 auto_rebalancer.cc:264] scheduled replica move failed to complete: Network error: Client connection negotiation failed: client connection to 127.0.100.3:44345: connect: Connection refused (error 111)
W20240503 23:27:46.545723 6388 auto_rebalancer.cc:663] Could not move replica: Network error: Client connection negotiation failed: client connection to 127.0.100.3:44345: connect: Connection refused (error 111)
W20240503 23:27:46.545840 6388 auto_rebalancer.cc:264] scheduled replica move failed to complete: Network error: Client connection negotiation failed: client connection to 127.0.100.3:44345: connect: Connection refused (error 111)
I20240503 23:27:47.522325 400 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 23:27:47.525959 6752 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 23:27:47.526857 6753 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 23:27:47.527741 6755 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 23:27:47.540838 400 server_base.cc:1034] running on GCE node
I20240503 23:27:47.541194 400 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240503 23:27:47.541244 400 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240503 23:27:47.541294 400 hybrid_clock.cc:648] HybridClock initialized: now 1714778867541274 us; error 0 us; skew 500 ppm
I20240503 23:27:47.541464 400 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
W20240503 23:27:47.546804 6388 auto_rebalancer.cc:227] Could not retrieve cluster info: Not found: tserver 3fc1cf80b12b42c2af63d3afec5214a7 not available for placement
I20240503 23:27:47.549306 400 webserver.cc:460] Webserver started at http://127.0.100.1:36505/ using document root <none> and password file <none>
I20240503 23:27:47.549613 400 fs_manager.cc:359] Metadata directory not provided
I20240503 23:27:47.549682 400 fs_manager.cc:362] Using existing metadata directory in first data directory
I20240503 23:27:47.552362 400 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20240503 23:27:47.554294 6760 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 23:27:47.554747 400 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20240503 23:27:47.554877 400 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZhIiPz/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestHandlingFailedTservers.1714778688451250-400-0/minicluster-data/ts-0-root
uuid: "3fc1cf80b12b42c2af63d3afec5214a7"
format_stamp: "Formatted at 2024-05-03 23:27:40 on dist-test-slave-jvbp"
I20240503 23:27:47.555084 400 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZhIiPz/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestHandlingFailedTservers.1714778688451250-400-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskZhIiPz/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestHandlingFailedTservers.1714778688451250-400-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskZhIiPz/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestHandlingFailedTservers.1714778688451250-400-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 23:27:47.568876 400 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 23:27:47.569559 400 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 23:27:47.570240 400 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 23:27:47.571928 6767 ts_tablet_manager.cc:522] Loading tablet metadata (0/4 complete)
I20240503 23:27:47.586005 400 ts_tablet_manager.cc:559] Loaded tablet metadata (4 total tablets, 4 live tablets)
I20240503 23:27:47.586184 400 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.015s user 0.001s sys 0.000s
I20240503 23:27:47.586350 400 ts_tablet_manager.cc:574] Registering tablets (0/4 complete)
I20240503 23:27:47.589304 6767 tablet_bootstrap.cc:492] T ad1b9cc708db4d289c2e847874bf747b P 3fc1cf80b12b42c2af63d3afec5214a7: Bootstrap starting.
I20240503 23:27:47.594311 6767 tablet_bootstrap.cc:492] T ad1b9cc708db4d289c2e847874bf747b P 3fc1cf80b12b42c2af63d3afec5214a7: Bootstrap replayed 1/1 log segments. Stats: ops{read=0 overwritten=0 applied=0 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 23:27:47.594820 6767 tablet_bootstrap.cc:492] T ad1b9cc708db4d289c2e847874bf747b P 3fc1cf80b12b42c2af63d3afec5214a7: Bootstrap complete.
I20240503 23:27:47.595257 6767 ts_tablet_manager.cc:1367] T ad1b9cc708db4d289c2e847874bf747b P 3fc1cf80b12b42c2af63d3afec5214a7: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20240503 23:27:47.595472 400 ts_tablet_manager.cc:590] Registered 4 tablets
I20240503 23:27:47.595564 400 ts_tablet_manager.cc:569] Time spent register tablets: real 0.009s user 0.007s sys 0.000s
I20240503 23:27:47.596087 6767 raft_consensus.cc:348] T ad1b9cc708db4d289c2e847874bf747b P 3fc1cf80b12b42c2af63d3afec5214a7 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fc1cf80b12b42c2af63d3afec5214a7" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 34273 } } peers { permanent_uuid: "c71e941c84ba45deb326c90968f40308" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 35323 } } peers { permanent_uuid: "21c0231e5eee4c109d088c02f139c112" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 44345 } }
I20240503 23:27:47.596313 6767 raft_consensus.cc:729] T ad1b9cc708db4d289c2e847874bf747b P 3fc1cf80b12b42c2af63d3afec5214a7 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3fc1cf80b12b42c2af63d3afec5214a7, State: Initialized, Role: FOLLOWER
I20240503 23:27:47.596668 6767 consensus_queue.cc:260] T ad1b9cc708db4d289c2e847874bf747b P 3fc1cf80b12b42c2af63d3afec5214a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fc1cf80b12b42c2af63d3afec5214a7" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 34273 } } peers { permanent_uuid: "c71e941c84ba45deb326c90968f40308" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 35323 } } peers { permanent_uuid: "21c0231e5eee4c109d088c02f139c112" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 44345 } }
I20240503 23:27:47.597543 6767 ts_tablet_manager.cc:1398] T ad1b9cc708db4d289c2e847874bf747b P 3fc1cf80b12b42c2af63d3afec5214a7: Time spent starting tablet: real 0.002s user 0.004s sys 0.000s
I20240503 23:27:47.597870 6767 tablet_bootstrap.cc:492] T e9273b9533d14394b454e6c31ae4deb5 P 3fc1cf80b12b42c2af63d3afec5214a7: Bootstrap starting.
I20240503 23:27:47.602684 6767 tablet_bootstrap.cc:492] T e9273b9533d14394b454e6c31ae4deb5 P 3fc1cf80b12b42c2af63d3afec5214a7: Bootstrap replayed 1/1 log segments. Stats: ops{read=0 overwritten=0 applied=0 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 23:27:47.603099 6767 tablet_bootstrap.cc:492] T e9273b9533d14394b454e6c31ae4deb5 P 3fc1cf80b12b42c2af63d3afec5214a7: Bootstrap complete.
I20240503 23:27:47.603402 6767 ts_tablet_manager.cc:1367] T e9273b9533d14394b454e6c31ae4deb5 P 3fc1cf80b12b42c2af63d3afec5214a7: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20240503 23:27:47.604349 6767 raft_consensus.cc:348] T e9273b9533d14394b454e6c31ae4deb5 P 3fc1cf80b12b42c2af63d3afec5214a7 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fc1cf80b12b42c2af63d3afec5214a7" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 34273 } } peers { permanent_uuid: "c71e941c84ba45deb326c90968f40308" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 35323 } } peers { permanent_uuid: "21c0231e5eee4c109d088c02f139c112" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 44345 } }
I20240503 23:27:47.604629 6767 raft_consensus.cc:729] T e9273b9533d14394b454e6c31ae4deb5 P 3fc1cf80b12b42c2af63d3afec5214a7 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3fc1cf80b12b42c2af63d3afec5214a7, State: Initialized, Role: FOLLOWER
I20240503 23:27:47.604948 6767 consensus_queue.cc:260] T e9273b9533d14394b454e6c31ae4deb5 P 3fc1cf80b12b42c2af63d3afec5214a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fc1cf80b12b42c2af63d3afec5214a7" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 34273 } } peers { permanent_uuid: "c71e941c84ba45deb326c90968f40308" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 35323 } } peers { permanent_uuid: "21c0231e5eee4c109d088c02f139c112" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 44345 } }
I20240503 23:27:47.605507 6767 ts_tablet_manager.cc:1398] T e9273b9533d14394b454e6c31ae4deb5 P 3fc1cf80b12b42c2af63d3afec5214a7: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
I20240503 23:27:47.605793 6767 tablet_bootstrap.cc:492] T 7441a981eb93446c9d9367af49462e73 P 3fc1cf80b12b42c2af63d3afec5214a7: Bootstrap starting.
I20240503 23:27:47.610821 6767 tablet_bootstrap.cc:492] T 7441a981eb93446c9d9367af49462e73 P 3fc1cf80b12b42c2af63d3afec5214a7: Bootstrap replayed 1/1 log segments. Stats: ops{read=0 overwritten=0 applied=0 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 23:27:47.611330 6767 tablet_bootstrap.cc:492] T 7441a981eb93446c9d9367af49462e73 P 3fc1cf80b12b42c2af63d3afec5214a7: Bootstrap complete.
I20240503 23:27:47.611632 6767 ts_tablet_manager.cc:1367] T 7441a981eb93446c9d9367af49462e73 P 3fc1cf80b12b42c2af63d3afec5214a7: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20240503 23:27:47.612455 6767 raft_consensus.cc:348] T 7441a981eb93446c9d9367af49462e73 P 3fc1cf80b12b42c2af63d3afec5214a7 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fc1cf80b12b42c2af63d3afec5214a7" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 34273 } } peers { permanent_uuid: "21c0231e5eee4c109d088c02f139c112" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 44345 } } peers { permanent_uuid: "c71e941c84ba45deb326c90968f40308" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 35323 } }
I20240503 23:27:47.612797 6767 raft_consensus.cc:729] T 7441a981eb93446c9d9367af49462e73 P 3fc1cf80b12b42c2af63d3afec5214a7 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3fc1cf80b12b42c2af63d3afec5214a7, State: Initialized, Role: FOLLOWER
I20240503 23:27:47.613108 6767 consensus_queue.cc:260] T 7441a981eb93446c9d9367af49462e73 P 3fc1cf80b12b42c2af63d3afec5214a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fc1cf80b12b42c2af63d3afec5214a7" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 34273 } } peers { permanent_uuid: "21c0231e5eee4c109d088c02f139c112" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 44345 } } peers { permanent_uuid: "c71e941c84ba45deb326c90968f40308" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 35323 } }
I20240503 23:27:47.613610 6767 ts_tablet_manager.cc:1398] T 7441a981eb93446c9d9367af49462e73 P 3fc1cf80b12b42c2af63d3afec5214a7: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20240503 23:27:47.613888 6767 tablet_bootstrap.cc:492] T fb79ea9cea314c4cab4bed81163a8ded P 3fc1cf80b12b42c2af63d3afec5214a7: Bootstrap starting.
I20240503 23:27:47.619151 6767 tablet_bootstrap.cc:492] T fb79ea9cea314c4cab4bed81163a8ded P 3fc1cf80b12b42c2af63d3afec5214a7: Bootstrap replayed 1/1 log segments. Stats: ops{read=0 overwritten=0 applied=0 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 23:27:47.619612 6767 tablet_bootstrap.cc:492] T fb79ea9cea314c4cab4bed81163a8ded P 3fc1cf80b12b42c2af63d3afec5214a7: Bootstrap complete.
I20240503 23:27:47.619915 6767 ts_tablet_manager.cc:1367] T fb79ea9cea314c4cab4bed81163a8ded P 3fc1cf80b12b42c2af63d3afec5214a7: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20240503 23:27:47.620893 6767 raft_consensus.cc:348] T fb79ea9cea314c4cab4bed81163a8ded P 3fc1cf80b12b42c2af63d3afec5214a7 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fc1cf80b12b42c2af63d3afec5214a7" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 34273 } } peers { permanent_uuid: "c71e941c84ba45deb326c90968f40308" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 35323 } } peers { permanent_uuid: "21c0231e5eee4c109d088c02f139c112" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 44345 } }
I20240503 23:27:47.621318 6767 raft_consensus.cc:729] T fb79ea9cea314c4cab4bed81163a8ded P 3fc1cf80b12b42c2af63d3afec5214a7 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3fc1cf80b12b42c2af63d3afec5214a7, State: Initialized, Role: FOLLOWER
I20240503 23:27:47.621670 6767 consensus_queue.cc:260] T fb79ea9cea314c4cab4bed81163a8ded P 3fc1cf80b12b42c2af63d3afec5214a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fc1cf80b12b42c2af63d3afec5214a7" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 34273 } } peers { permanent_uuid: "c71e941c84ba45deb326c90968f40308" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 35323 } } peers { permanent_uuid: "21c0231e5eee4c109d088c02f139c112" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 44345 } }
I20240503 23:27:47.622382 6767 ts_tablet_manager.cc:1398] T fb79ea9cea314c4cab4bed81163a8ded P 3fc1cf80b12b42c2af63d3afec5214a7: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20240503 23:27:47.630358 6828 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.100.1:34273 every 8 connection(s)
I20240503 23:27:47.630358 400 rpc_server.cc:307] RPC server started. Bound to: 127.0.100.1:34273
I20240503 23:27:47.632220 400 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 23:27:47.636442 6833 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 23:27:47.637581 6834 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 23:27:47.642479 6829 heartbeater.cc:347] Connected to a master server at 127.0.100.62:44317
I20240503 23:27:47.642800 6829 heartbeater.cc:464] Registering TS with master...
I20240503 23:27:47.643651 6829 heartbeater.cc:511] Master 127.0.100.62:44317 requested a full tablet report, sending...
W20240503 23:27:47.644908 6836 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 23:27:47.645895 6338 ts_manager.cc:194] Re-registered known tserver with Master: 3fc1cf80b12b42c2af63d3afec5214a7 (127.0.100.1:34273)
I20240503 23:27:47.645946 400 server_base.cc:1034] running on GCE node
I20240503 23:27:47.646257 400 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240503 23:27:47.646369 400 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240503 23:27:47.646458 400 hybrid_clock.cc:648] HybridClock initialized: now 1714778867646426 us; error 0 us; skew 500 ppm
I20240503 23:27:47.646795 400 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 23:27:47.648773 6338 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:55766
I20240503 23:27:47.648798 400 webserver.cc:460] Webserver started at http://127.0.100.2:42189/ using document root <none> and password file <none>
I20240503 23:27:47.649153 400 fs_manager.cc:359] Metadata directory not provided
I20240503 23:27:47.649255 400 fs_manager.cc:362] Using existing metadata directory in first data directory
I20240503 23:27:47.651818 400 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.000s sys 0.003s
I20240503 23:27:47.653657 6841 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 23:27:47.654124 400 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20240503 23:27:47.654237 400 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZhIiPz/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestHandlingFailedTservers.1714778688451250-400-0/minicluster-data/ts-1-root
uuid: "c71e941c84ba45deb326c90968f40308"
format_stamp: "Formatted at 2024-05-03 23:27:40 on dist-test-slave-jvbp"
I20240503 23:27:47.654387 400 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZhIiPz/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestHandlingFailedTservers.1714778688451250-400-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskZhIiPz/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestHandlingFailedTservers.1714778688451250-400-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskZhIiPz/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestHandlingFailedTservers.1714778688451250-400-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 23:27:47.670303 400 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 23:27:47.671029 400 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 23:27:47.671718 400 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 23:27:47.682845 400 ts_tablet_manager.cc:559] Loaded tablet metadata (4 total tablets, 4 live tablets)
I20240503 23:27:47.682998 400 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.010s user 0.000s sys 0.001s
I20240503 23:27:47.685588 6848 tablet_bootstrap.cc:492] T ad1b9cc708db4d289c2e847874bf747b P c71e941c84ba45deb326c90968f40308: Bootstrap starting.
I20240503 23:27:47.690219 6848 tablet_bootstrap.cc:492] T ad1b9cc708db4d289c2e847874bf747b P c71e941c84ba45deb326c90968f40308: Bootstrap replayed 1/1 log segments. Stats: ops{read=0 overwritten=0 applied=0 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 23:27:47.690663 6848 tablet_bootstrap.cc:492] T ad1b9cc708db4d289c2e847874bf747b P c71e941c84ba45deb326c90968f40308: Bootstrap complete.
I20240503 23:27:47.690989 6848 ts_tablet_manager.cc:1367] T ad1b9cc708db4d289c2e847874bf747b P c71e941c84ba45deb326c90968f40308: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20240503 23:27:47.691797 400 ts_tablet_manager.cc:590] Registered 4 tablets
I20240503 23:27:47.691893 400 ts_tablet_manager.cc:569] Time spent register tablets: real 0.009s user 0.001s sys 0.004s
I20240503 23:27:47.691776 6848 raft_consensus.cc:348] T ad1b9cc708db4d289c2e847874bf747b P c71e941c84ba45deb326c90968f40308 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fc1cf80b12b42c2af63d3afec5214a7" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 34273 } } peers { permanent_uuid: "c71e941c84ba45deb326c90968f40308" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 35323 } } peers { permanent_uuid: "21c0231e5eee4c109d088c02f139c112" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 44345 } }
I20240503 23:27:47.692025 6848 raft_consensus.cc:729] T ad1b9cc708db4d289c2e847874bf747b P c71e941c84ba45deb326c90968f40308 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: c71e941c84ba45deb326c90968f40308, State: Initialized, Role: FOLLOWER
I20240503 23:27:47.692339 6848 consensus_queue.cc:260] T ad1b9cc708db4d289c2e847874bf747b P c71e941c84ba45deb326c90968f40308 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fc1cf80b12b42c2af63d3afec5214a7" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 34273 } } peers { permanent_uuid: "c71e941c84ba45deb326c90968f40308" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 35323 } } peers { permanent_uuid: "21c0231e5eee4c109d088c02f139c112" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 44345 } }
I20240503 23:27:47.693138 6848 ts_tablet_manager.cc:1398] T ad1b9cc708db4d289c2e847874bf747b P c71e941c84ba45deb326c90968f40308: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20240503 23:27:47.693429 6848 tablet_bootstrap.cc:492] T e9273b9533d14394b454e6c31ae4deb5 P c71e941c84ba45deb326c90968f40308: Bootstrap starting.
I20240503 23:27:47.697875 6848 tablet_bootstrap.cc:492] T e9273b9533d14394b454e6c31ae4deb5 P c71e941c84ba45deb326c90968f40308: Bootstrap replayed 1/1 log segments. Stats: ops{read=0 overwritten=0 applied=0 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 23:27:47.698288 6848 tablet_bootstrap.cc:492] T e9273b9533d14394b454e6c31ae4deb5 P c71e941c84ba45deb326c90968f40308: Bootstrap complete.
I20240503 23:27:47.698803 6848 ts_tablet_manager.cc:1367] T e9273b9533d14394b454e6c31ae4deb5 P c71e941c84ba45deb326c90968f40308: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20240503 23:27:47.699579 6848 raft_consensus.cc:348] T e9273b9533d14394b454e6c31ae4deb5 P c71e941c84ba45deb326c90968f40308 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fc1cf80b12b42c2af63d3afec5214a7" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 34273 } } peers { permanent_uuid: "c71e941c84ba45deb326c90968f40308" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 35323 } } peers { permanent_uuid: "21c0231e5eee4c109d088c02f139c112" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 44345 } }
I20240503 23:27:47.699851 6848 raft_consensus.cc:729] T e9273b9533d14394b454e6c31ae4deb5 P c71e941c84ba45deb326c90968f40308 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: c71e941c84ba45deb326c90968f40308, State: Initialized, Role: FOLLOWER
I20240503 23:27:47.700179 6848 consensus_queue.cc:260] T e9273b9533d14394b454e6c31ae4deb5 P c71e941c84ba45deb326c90968f40308 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fc1cf80b12b42c2af63d3afec5214a7" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 34273 } } peers { permanent_uuid: "c71e941c84ba45deb326c90968f40308" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 35323 } } peers { permanent_uuid: "21c0231e5eee4c109d088c02f139c112" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 44345 } }
I20240503 23:27:47.700806 6848 ts_tablet_manager.cc:1398] T e9273b9533d14394b454e6c31ae4deb5 P c71e941c84ba45deb326c90968f40308: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 23:27:47.701081 6848 tablet_bootstrap.cc:492] T 7441a981eb93446c9d9367af49462e73 P c71e941c84ba45deb326c90968f40308: Bootstrap starting.
I20240503 23:27:47.706234 6848 tablet_bootstrap.cc:492] T 7441a981eb93446c9d9367af49462e73 P c71e941c84ba45deb326c90968f40308: Bootstrap replayed 1/1 log segments. Stats: ops{read=0 overwritten=0 applied=0 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 23:27:47.706631 6848 tablet_bootstrap.cc:492] T 7441a981eb93446c9d9367af49462e73 P c71e941c84ba45deb326c90968f40308: Bootstrap complete.
I20240503 23:27:47.706921 6848 ts_tablet_manager.cc:1367] T 7441a981eb93446c9d9367af49462e73 P c71e941c84ba45deb326c90968f40308: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20240503 23:27:47.707731 6848 raft_consensus.cc:348] T 7441a981eb93446c9d9367af49462e73 P c71e941c84ba45deb326c90968f40308 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fc1cf80b12b42c2af63d3afec5214a7" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 34273 } } peers { permanent_uuid: "21c0231e5eee4c109d088c02f139c112" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 44345 } } peers { permanent_uuid: "c71e941c84ba45deb326c90968f40308" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 35323 } }
I20240503 23:27:47.708154 6848 raft_consensus.cc:729] T 7441a981eb93446c9d9367af49462e73 P c71e941c84ba45deb326c90968f40308 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: c71e941c84ba45deb326c90968f40308, State: Initialized, Role: FOLLOWER
I20240503 23:27:47.708510 6848 consensus_queue.cc:260] T 7441a981eb93446c9d9367af49462e73 P c71e941c84ba45deb326c90968f40308 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fc1cf80b12b42c2af63d3afec5214a7" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 34273 } } peers { permanent_uuid: "21c0231e5eee4c109d088c02f139c112" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 44345 } } peers { permanent_uuid: "c71e941c84ba45deb326c90968f40308" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 35323 } }
I20240503 23:27:47.709846 6848 ts_tablet_manager.cc:1398] T 7441a981eb93446c9d9367af49462e73 P c71e941c84ba45deb326c90968f40308: Time spent starting tablet: real 0.003s user 0.004s sys 0.000s
I20240503 23:27:47.710165 6848 tablet_bootstrap.cc:492] T fb79ea9cea314c4cab4bed81163a8ded P c71e941c84ba45deb326c90968f40308: Bootstrap starting.
I20240503 23:27:47.715193 6848 tablet_bootstrap.cc:492] T fb79ea9cea314c4cab4bed81163a8ded P c71e941c84ba45deb326c90968f40308: Bootstrap replayed 1/1 log segments. Stats: ops{read=0 overwritten=0 applied=0 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 23:27:47.715688 6848 tablet_bootstrap.cc:492] T fb79ea9cea314c4cab4bed81163a8ded P c71e941c84ba45deb326c90968f40308: Bootstrap complete.
I20240503 23:27:47.716034 6848 ts_tablet_manager.cc:1367] T fb79ea9cea314c4cab4bed81163a8ded P c71e941c84ba45deb326c90968f40308: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20240503 23:27:47.716951 6848 raft_consensus.cc:348] T fb79ea9cea314c4cab4bed81163a8ded P c71e941c84ba45deb326c90968f40308 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fc1cf80b12b42c2af63d3afec5214a7" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 34273 } } peers { permanent_uuid: "c71e941c84ba45deb326c90968f40308" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 35323 } } peers { permanent_uuid: "21c0231e5eee4c109d088c02f139c112" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 44345 } }
I20240503 23:27:47.717272 6848 raft_consensus.cc:729] T fb79ea9cea314c4cab4bed81163a8ded P c71e941c84ba45deb326c90968f40308 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: c71e941c84ba45deb326c90968f40308, State: Initialized, Role: FOLLOWER
I20240503 23:27:47.717582 6848 consensus_queue.cc:260] T fb79ea9cea314c4cab4bed81163a8ded P c71e941c84ba45deb326c90968f40308 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fc1cf80b12b42c2af63d3afec5214a7" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 34273 } } peers { permanent_uuid: "c71e941c84ba45deb326c90968f40308" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 35323 } } peers { permanent_uuid: "21c0231e5eee4c109d088c02f139c112" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 44345 } }
I20240503 23:27:47.718153 6848 ts_tablet_manager.cc:1398] T fb79ea9cea314c4cab4bed81163a8ded P c71e941c84ba45deb326c90968f40308: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240503 23:27:47.722112 400 rpc_server.cc:307] RPC server started. Bound to: 127.0.100.2:35323
I20240503 23:27:47.722144 6909 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.100.2:35323 every 8 connection(s)
I20240503 23:27:47.723783 400 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 23:27:47.728444 6916 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 23:27:47.729398 400 server_base.cc:1034] running on GCE node
W20240503 23:27:47.730404 6913 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 23:27:47.731146 6914 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 23:27:47.731659 400 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240503 23:27:47.731766 400 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240503 23:27:47.731848 400 hybrid_clock.cc:648] HybridClock initialized: now 1714778867731814 us; error 0 us; skew 500 ppm
I20240503 23:27:47.731952 6910 heartbeater.cc:347] Connected to a master server at 127.0.100.62:44317
I20240503 23:27:47.732080 400 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 23:27:47.732110 6910 heartbeater.cc:464] Registering TS with master...
I20240503 23:27:47.732672 6910 heartbeater.cc:511] Master 127.0.100.62:44317 requested a full tablet report, sending...
I20240503 23:27:47.734042 400 webserver.cc:460] Webserver started at http://127.0.100.3:34327/ using document root <none> and password file <none>
I20240503 23:27:47.734373 400 fs_manager.cc:359] Metadata directory not provided
I20240503 23:27:47.734472 400 fs_manager.cc:362] Using existing metadata directory in first data directory
I20240503 23:27:47.734628 6338 ts_manager.cc:194] Re-registered known tserver with Master: c71e941c84ba45deb326c90968f40308 (127.0.100.2:35323)
I20240503 23:27:47.737352 6338 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:55772
I20240503 23:27:47.737437 400 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20240503 23:27:47.739653 6921 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 23:27:47.740115 400 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.002s sys 0.000s
I20240503 23:27:47.740252 400 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZhIiPz/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestHandlingFailedTservers.1714778688451250-400-0/minicluster-data/ts-2-root
uuid: "21c0231e5eee4c109d088c02f139c112"
format_stamp: "Formatted at 2024-05-03 23:27:40 on dist-test-slave-jvbp"
I20240503 23:27:47.740415 400 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZhIiPz/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestHandlingFailedTservers.1714778688451250-400-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-taskZhIiPz/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestHandlingFailedTservers.1714778688451250-400-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-taskZhIiPz/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestHandlingFailedTservers.1714778688451250-400-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 23:27:47.766883 400 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 23:27:47.767598 400 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 23:27:47.768311 400 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 23:27:47.779825 400 ts_tablet_manager.cc:559] Loaded tablet metadata (4 total tablets, 4 live tablets)
I20240503 23:27:47.779994 400 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.011s user 0.001s sys 0.000s
I20240503 23:27:47.782621 6928 tablet_bootstrap.cc:492] T ad1b9cc708db4d289c2e847874bf747b P 21c0231e5eee4c109d088c02f139c112: Bootstrap starting.
I20240503 23:27:47.788015 6928 tablet_bootstrap.cc:492] T ad1b9cc708db4d289c2e847874bf747b P 21c0231e5eee4c109d088c02f139c112: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=0 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20240503 23:27:47.788414 400 ts_tablet_manager.cc:590] Registered 4 tablets
I20240503 23:27:47.788527 400 ts_tablet_manager.cc:569] Time spent register tablets: real 0.008s user 0.005s sys 0.000s
I20240503 23:27:47.788928 6928 tablet_bootstrap.cc:492] T ad1b9cc708db4d289c2e847874bf747b P 21c0231e5eee4c109d088c02f139c112: Bootstrap complete.
I20240503 23:27:47.789253 6928 ts_tablet_manager.cc:1367] T ad1b9cc708db4d289c2e847874bf747b P 21c0231e5eee4c109d088c02f139c112: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.004s
I20240503 23:27:47.789974 6928 raft_consensus.cc:348] T ad1b9cc708db4d289c2e847874bf747b P 21c0231e5eee4c109d088c02f139c112 [term 1 FOLLOWER]: Replica starting. Triggering 1 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fc1cf80b12b42c2af63d3afec5214a7" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 34273 } } peers { permanent_uuid: "c71e941c84ba45deb326c90968f40308" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 35323 } } peers { permanent_uuid: "21c0231e5eee4c109d088c02f139c112" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 44345 } }
I20240503 23:27:47.790298 6928 raft_consensus.cc:729] T ad1b9cc708db4d289c2e847874bf747b P 21c0231e5eee4c109d088c02f139c112 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 21c0231e5eee4c109d088c02f139c112, State: Initialized, Role: FOLLOWER
I20240503 23:27:47.790606 6928 consensus_queue.cc:260] T ad1b9cc708db4d289c2e847874bf747b P 21c0231e5eee4c109d088c02f139c112 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fc1cf80b12b42c2af63d3afec5214a7" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 34273 } } peers { permanent_uuid: "c71e941c84ba45deb326c90968f40308" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 35323 } } peers { permanent_uuid: "21c0231e5eee4c109d088c02f139c112" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 44345 } }
I20240503 23:27:47.791661 6928 ts_tablet_manager.cc:1398] T ad1b9cc708db4d289c2e847874bf747b P 21c0231e5eee4c109d088c02f139c112: Time spent starting tablet: real 0.002s user 0.002s sys 0.001s
I20240503 23:27:47.792130 6928 tablet_bootstrap.cc:492] T e9273b9533d14394b454e6c31ae4deb5 P 21c0231e5eee4c109d088c02f139c112: Bootstrap starting.
I20240503 23:27:47.797773 6928 tablet_bootstrap.cc:492] T e9273b9533d14394b454e6c31ae4deb5 P 21c0231e5eee4c109d088c02f139c112: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=0 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20240503 23:27:47.798317 6928 tablet_bootstrap.cc:492] T e9273b9533d14394b454e6c31ae4deb5 P 21c0231e5eee4c109d088c02f139c112: Bootstrap complete.
I20240503 23:27:47.798658 6928 ts_tablet_manager.cc:1367] T e9273b9533d14394b454e6c31ae4deb5 P 21c0231e5eee4c109d088c02f139c112: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20240503 23:27:47.799785 6928 raft_consensus.cc:348] T e9273b9533d14394b454e6c31ae4deb5 P 21c0231e5eee4c109d088c02f139c112 [term 1 FOLLOWER]: Replica starting. Triggering 1 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fc1cf80b12b42c2af63d3afec5214a7" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 34273 } } peers { permanent_uuid: "c71e941c84ba45deb326c90968f40308" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 35323 } } peers { permanent_uuid: "21c0231e5eee4c109d088c02f139c112" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 44345 } }
I20240503 23:27:47.800119 6928 raft_consensus.cc:729] T e9273b9533d14394b454e6c31ae4deb5 P 21c0231e5eee4c109d088c02f139c112 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 21c0231e5eee4c109d088c02f139c112, State: Initialized, Role: FOLLOWER
I20240503 23:27:47.800432 6928 consensus_queue.cc:260] T e9273b9533d14394b454e6c31ae4deb5 P 21c0231e5eee4c109d088c02f139c112 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fc1cf80b12b42c2af63d3afec5214a7" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 34273 } } peers { permanent_uuid: "c71e941c84ba45deb326c90968f40308" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 35323 } } peers { permanent_uuid: "21c0231e5eee4c109d088c02f139c112" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 44345 } }
I20240503 23:27:47.801102 6928 ts_tablet_manager.cc:1398] T e9273b9533d14394b454e6c31ae4deb5 P 21c0231e5eee4c109d088c02f139c112: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 23:27:47.801515 6928 tablet_bootstrap.cc:492] T 7441a981eb93446c9d9367af49462e73 P 21c0231e5eee4c109d088c02f139c112: Bootstrap starting.
I20240503 23:27:47.807045 6928 tablet_bootstrap.cc:492] T 7441a981eb93446c9d9367af49462e73 P 21c0231e5eee4c109d088c02f139c112: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=0 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20240503 23:27:47.807724 6928 tablet_bootstrap.cc:492] T 7441a981eb93446c9d9367af49462e73 P 21c0231e5eee4c109d088c02f139c112: Bootstrap complete.
I20240503 23:27:47.808120 6928 ts_tablet_manager.cc:1367] T 7441a981eb93446c9d9367af49462e73 P 21c0231e5eee4c109d088c02f139c112: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20240503 23:27:47.809059 6928 raft_consensus.cc:348] T 7441a981eb93446c9d9367af49462e73 P 21c0231e5eee4c109d088c02f139c112 [term 1 FOLLOWER]: Replica starting. Triggering 1 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fc1cf80b12b42c2af63d3afec5214a7" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 34273 } } peers { permanent_uuid: "21c0231e5eee4c109d088c02f139c112" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 44345 } } peers { permanent_uuid: "c71e941c84ba45deb326c90968f40308" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 35323 } }
I20240503 23:27:47.809383 6928 raft_consensus.cc:729] T 7441a981eb93446c9d9367af49462e73 P 21c0231e5eee4c109d088c02f139c112 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 21c0231e5eee4c109d088c02f139c112, State: Initialized, Role: FOLLOWER
I20240503 23:27:47.809666 6928 consensus_queue.cc:260] T 7441a981eb93446c9d9367af49462e73 P 21c0231e5eee4c109d088c02f139c112 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fc1cf80b12b42c2af63d3afec5214a7" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 34273 } } peers { permanent_uuid: "21c0231e5eee4c109d088c02f139c112" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 44345 } } peers { permanent_uuid: "c71e941c84ba45deb326c90968f40308" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 35323 } }
I20240503 23:27:47.810159 6928 ts_tablet_manager.cc:1398] T 7441a981eb93446c9d9367af49462e73 P 21c0231e5eee4c109d088c02f139c112: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240503 23:27:47.810436 6928 tablet_bootstrap.cc:492] T fb79ea9cea314c4cab4bed81163a8ded P 21c0231e5eee4c109d088c02f139c112: Bootstrap starting.
I20240503 23:27:47.812918 400 rpc_server.cc:307] RPC server started. Bound to: 127.0.100.3:44345
I20240503 23:27:47.812969 6988 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.100.3:44345 every 8 connection(s)
I20240503 23:27:47.815630 6928 tablet_bootstrap.cc:492] T fb79ea9cea314c4cab4bed81163a8ded P 21c0231e5eee4c109d088c02f139c112: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=0 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20240503 23:27:47.816169 6928 tablet_bootstrap.cc:492] T fb79ea9cea314c4cab4bed81163a8ded P 21c0231e5eee4c109d088c02f139c112: Bootstrap complete.
I20240503 23:27:47.816587 6928 ts_tablet_manager.cc:1367] T fb79ea9cea314c4cab4bed81163a8ded P 21c0231e5eee4c109d088c02f139c112: Time spent bootstrapping tablet: real 0.006s user 0.003s sys 0.000s
I20240503 23:27:47.817485 6928 raft_consensus.cc:348] T fb79ea9cea314c4cab4bed81163a8ded P 21c0231e5eee4c109d088c02f139c112 [term 1 FOLLOWER]: Replica starting. Triggering 1 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fc1cf80b12b42c2af63d3afec5214a7" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 34273 } } peers { permanent_uuid: "c71e941c84ba45deb326c90968f40308" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 35323 } } peers { permanent_uuid: "21c0231e5eee4c109d088c02f139c112" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 44345 } }
I20240503 23:27:47.817922 6928 raft_consensus.cc:729] T fb79ea9cea314c4cab4bed81163a8ded P 21c0231e5eee4c109d088c02f139c112 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 21c0231e5eee4c109d088c02f139c112, State: Initialized, Role: FOLLOWER
I20240503 23:27:47.818280 6928 consensus_queue.cc:260] T fb79ea9cea314c4cab4bed81163a8ded P 21c0231e5eee4c109d088c02f139c112 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fc1cf80b12b42c2af63d3afec5214a7" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 34273 } } peers { permanent_uuid: "c71e941c84ba45deb326c90968f40308" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 35323 } } peers { permanent_uuid: "21c0231e5eee4c109d088c02f139c112" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 44345 } }
I20240503 23:27:47.818948 6928 ts_tablet_manager.cc:1398] T fb79ea9cea314c4cab4bed81163a8ded P 21c0231e5eee4c109d088c02f139c112: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240503 23:27:47.822877 6990 heartbeater.cc:347] Connected to a master server at 127.0.100.62:44317
I20240503 23:27:47.823081 6990 heartbeater.cc:464] Registering TS with master...
I20240503 23:27:47.823531 6990 heartbeater.cc:511] Master 127.0.100.62:44317 requested a full tablet report, sending...
I20240503 23:27:47.825016 6338 ts_manager.cc:194] Re-registered known tserver with Master: 21c0231e5eee4c109d088c02f139c112 (127.0.100.3:44345)
I20240503 23:27:47.826998 6338 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:55778
I20240503 23:27:47.828786 6990 heartbeater.cc:503] Master 127.0.100.62:44317 was elected leader, sending a full tablet report...
W20240503 23:27:48.558478 6388 auto_rebalancer.cc:254] failed to send replica move request: Illegal state: Replica 21c0231e5eee4c109d088c02f139c112 is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fc1cf80b12b42c2af63d3afec5214a7" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 34273 } } peers { permanent_uuid: "c71e941c84ba45deb326c90968f40308" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 35323 } } peers { permanent_uuid: "21c0231e5eee4c109d088c02f139c112" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 44345 } } }
W20240503 23:27:48.560853 6388 auto_rebalancer.cc:663] Could not move replica: Incomplete: tablet fb79ea9cea314c4cab4bed81163a8ded, TS 3fc1cf80b12b42c2af63d3afec5214a7 -> TS c11e09f93341477483d0045fb1165058 move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fc1cf80b12b42c2af63d3afec5214a7" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 34273 } } peers { permanent_uuid: "c71e941c84ba45deb326c90968f40308" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 35323 } } peers { permanent_uuid: "21c0231e5eee4c109d088c02f139c112" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 44345 } }
W20240503 23:27:48.561030 6388 auto_rebalancer.cc:264] scheduled replica move failed to complete: Incomplete: tablet fb79ea9cea314c4cab4bed81163a8ded, TS 3fc1cf80b12b42c2af63d3afec5214a7 -> TS c11e09f93341477483d0045fb1165058 move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fc1cf80b12b42c2af63d3afec5214a7" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 34273 } } peers { permanent_uuid: "c71e941c84ba45deb326c90968f40308" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 35323 } } peers { permanent_uuid: "21c0231e5eee4c109d088c02f139c112" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 44345 } }
W20240503 23:27:48.563032 6388 auto_rebalancer.cc:663] Could not move replica: Incomplete: tablet 7441a981eb93446c9d9367af49462e73, TS 21c0231e5eee4c109d088c02f139c112 -> TS c11e09f93341477483d0045fb1165058 move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fc1cf80b12b42c2af63d3afec5214a7" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 34273 } } peers { permanent_uuid: "21c0231e5eee4c109d088c02f139c112" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 44345 } } peers { permanent_uuid: "c71e941c84ba45deb326c90968f40308" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 35323 } }
W20240503 23:27:48.563184 6388 auto_rebalancer.cc:264] scheduled replica move failed to complete: Incomplete: tablet 7441a981eb93446c9d9367af49462e73, TS 21c0231e5eee4c109d088c02f139c112 -> TS c11e09f93341477483d0045fb1165058 move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fc1cf80b12b42c2af63d3afec5214a7" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 34273 } } peers { permanent_uuid: "21c0231e5eee4c109d088c02f139c112" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 44345 } } peers { permanent_uuid: "c71e941c84ba45deb326c90968f40308" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 35323 } }
W20240503 23:27:48.564841 6388 auto_rebalancer.cc:663] Could not move replica: Incomplete: tablet e9273b9533d14394b454e6c31ae4deb5, TS c71e941c84ba45deb326c90968f40308 -> TS c11e09f93341477483d0045fb1165058 move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fc1cf80b12b42c2af63d3afec5214a7" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 34273 } } peers { permanent_uuid: "c71e941c84ba45deb326c90968f40308" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 35323 } } peers { permanent_uuid: "21c0231e5eee4c109d088c02f139c112" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 44345 } }
W20240503 23:27:48.564991 6388 auto_rebalancer.cc:264] scheduled replica move failed to complete: Incomplete: tablet e9273b9533d14394b454e6c31ae4deb5, TS c71e941c84ba45deb326c90968f40308 -> TS c11e09f93341477483d0045fb1165058 move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fc1cf80b12b42c2af63d3afec5214a7" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 34273 } } peers { permanent_uuid: "c71e941c84ba45deb326c90968f40308" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 35323 } } peers { permanent_uuid: "21c0231e5eee4c109d088c02f139c112" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 44345 } }
I20240503 23:27:48.651175 6829 heartbeater.cc:503] Master 127.0.100.62:44317 was elected leader, sending a full tablet report...
I20240503 23:27:48.739379 6910 heartbeater.cc:503] Master 127.0.100.62:44317 was elected leader, sending a full tablet report...
I20240503 23:27:48.970705 6995 raft_consensus.cc:482] T fb79ea9cea314c4cab4bed81163a8ded P c71e941c84ba45deb326c90968f40308 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 23:27:48.970912 6995 raft_consensus.cc:504] T fb79ea9cea314c4cab4bed81163a8ded P c71e941c84ba45deb326c90968f40308 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fc1cf80b12b42c2af63d3afec5214a7" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 34273 } } peers { permanent_uuid: "c71e941c84ba45deb326c90968f40308" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 35323 } } peers { permanent_uuid: "21c0231e5eee4c109d088c02f139c112" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 44345 } }
I20240503 23:27:48.971834 6995 leader_election.cc:288] T fb79ea9cea314c4cab4bed81163a8ded P c71e941c84ba45deb326c90968f40308 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3fc1cf80b12b42c2af63d3afec5214a7 (127.0.100.1:34273), 21c0231e5eee4c109d088c02f139c112 (127.0.100.3:44345)
I20240503 23:27:48.980662 6804 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb79ea9cea314c4cab4bed81163a8ded" candidate_uuid: "c71e941c84ba45deb326c90968f40308" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3fc1cf80b12b42c2af63d3afec5214a7" is_pre_election: true
I20240503 23:27:48.980664 6963 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb79ea9cea314c4cab4bed81163a8ded" candidate_uuid: "c71e941c84ba45deb326c90968f40308" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "21c0231e5eee4c109d088c02f139c112" is_pre_election: true
I20240503 23:27:48.981079 6804 raft_consensus.cc:2454] T fb79ea9cea314c4cab4bed81163a8ded P 3fc1cf80b12b42c2af63d3afec5214a7 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c71e941c84ba45deb326c90968f40308 in term 1.
I20240503 23:27:48.981227 6963 raft_consensus.cc:2396] T fb79ea9cea314c4cab4bed81163a8ded P 21c0231e5eee4c109d088c02f139c112 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate c71e941c84ba45deb326c90968f40308 for term 2 because replica has last-logged OpId of term: 1 index: 1, which is greater than that of the candidate, which has last-logged OpId of term: 0 index: 0.
I20240503 23:27:48.981773 6844 leader_election.cc:302] T fb79ea9cea314c4cab4bed81163a8ded P c71e941c84ba45deb326c90968f40308 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3fc1cf80b12b42c2af63d3afec5214a7, c71e941c84ba45deb326c90968f40308; no voters: 21c0231e5eee4c109d088c02f139c112
I20240503 23:27:48.982110 6995 raft_consensus.cc:2781] T fb79ea9cea314c4cab4bed81163a8ded P c71e941c84ba45deb326c90968f40308 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240503 23:27:48.982239 6995 raft_consensus.cc:482] T fb79ea9cea314c4cab4bed81163a8ded P c71e941c84ba45deb326c90968f40308 [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 23:27:48.982336 6995 raft_consensus.cc:3037] T fb79ea9cea314c4cab4bed81163a8ded P c71e941c84ba45deb326c90968f40308 [term 1 FOLLOWER]: Advancing to term 2
I20240503 23:27:48.984973 6995 raft_consensus.cc:504] T fb79ea9cea314c4cab4bed81163a8ded P c71e941c84ba45deb326c90968f40308 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fc1cf80b12b42c2af63d3afec5214a7" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 34273 } } peers { permanent_uuid: "c71e941c84ba45deb326c90968f40308" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 35323 } } peers { permanent_uuid: "21c0231e5eee4c109d088c02f139c112" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 44345 } }
I20240503 23:27:48.985739 6995 leader_election.cc:288] T fb79ea9cea314c4cab4bed81163a8ded P c71e941c84ba45deb326c90968f40308 [CANDIDATE]: Term 2 election: Requested vote from peers 3fc1cf80b12b42c2af63d3afec5214a7 (127.0.100.1:34273), 21c0231e5eee4c109d088c02f139c112 (127.0.100.3:44345)
I20240503 23:27:48.986389 6963 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb79ea9cea314c4cab4bed81163a8ded" candidate_uuid: "c71e941c84ba45deb326c90968f40308" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "21c0231e5eee4c109d088c02f139c112"
I20240503 23:27:48.986393 6804 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb79ea9cea314c4cab4bed81163a8ded" candidate_uuid: "c71e941c84ba45deb326c90968f40308" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3fc1cf80b12b42c2af63d3afec5214a7"
I20240503 23:27:48.986675 6963 raft_consensus.cc:3037] T fb79ea9cea314c4cab4bed81163a8ded P 21c0231e5eee4c109d088c02f139c112 [term 1 FOLLOWER]: Advancing to term 2
I20240503 23:27:48.986701 6804 raft_consensus.cc:3037] T fb79ea9cea314c4cab4bed81163a8ded P 3fc1cf80b12b42c2af63d3afec5214a7 [term 1 FOLLOWER]: Advancing to term 2
I20240503 23:27:48.989394 6804 raft_consensus.cc:2454] T fb79ea9cea314c4cab4bed81163a8ded P 3fc1cf80b12b42c2af63d3afec5214a7 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c71e941c84ba45deb326c90968f40308 in term 2.
I20240503 23:27:48.989534 6963 raft_consensus.cc:2396] T fb79ea9cea314c4cab4bed81163a8ded P 21c0231e5eee4c109d088c02f139c112 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate c71e941c84ba45deb326c90968f40308 for term 2 because replica has last-logged OpId of term: 1 index: 1, which is greater than that of the candidate, which has last-logged OpId of term: 0 index: 0.
I20240503 23:27:48.990016 6844 leader_election.cc:302] T fb79ea9cea314c4cab4bed81163a8ded P c71e941c84ba45deb326c90968f40308 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3fc1cf80b12b42c2af63d3afec5214a7, c71e941c84ba45deb326c90968f40308; no voters:
I20240503 23:27:48.990484 6995 raft_consensus.cc:2781] T fb79ea9cea314c4cab4bed81163a8ded P c71e941c84ba45deb326c90968f40308 [term 2 FOLLOWER]: Leader election won for term 2
I20240503 23:27:48.991026 6995 raft_consensus.cc:686] T fb79ea9cea314c4cab4bed81163a8ded P c71e941c84ba45deb326c90968f40308 [term 2 LEADER]: Becoming Leader. State: Replica: c71e941c84ba45deb326c90968f40308, State: Running, Role: LEADER
I20240503 23:27:48.991393 6995 consensus_queue.cc:237] T fb79ea9cea314c4cab4bed81163a8ded P c71e941c84ba45deb326c90968f40308 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fc1cf80b12b42c2af63d3afec5214a7" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 34273 } } peers { permanent_uuid: "c71e941c84ba45deb326c90968f40308" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 35323 } } peers { permanent_uuid: "21c0231e5eee4c109d088c02f139c112" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 44345 } }
I20240503 23:27:48.994765 6338 catalog_manager.cc:5484] T fb79ea9cea314c4cab4bed81163a8ded P c71e941c84ba45deb326c90968f40308 reported cstate change: term changed from 1 to 2, leader changed from 21c0231e5eee4c109d088c02f139c112 (127.0.100.3) to c71e941c84ba45deb326c90968f40308 (127.0.100.2). New cstate: current_term: 2 leader_uuid: "c71e941c84ba45deb326c90968f40308" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fc1cf80b12b42c2af63d3afec5214a7" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 34273 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c71e941c84ba45deb326c90968f40308" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 35323 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "21c0231e5eee4c109d088c02f139c112" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 44345 } health_report { overall_health: UNKNOWN } } }
I20240503 23:27:49.093245 7005 raft_consensus.cc:482] T 7441a981eb93446c9d9367af49462e73 P 21c0231e5eee4c109d088c02f139c112 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 23:27:49.093495 7005 raft_consensus.cc:504] T 7441a981eb93446c9d9367af49462e73 P 21c0231e5eee4c109d088c02f139c112 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fc1cf80b12b42c2af63d3afec5214a7" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 34273 } } peers { permanent_uuid: "21c0231e5eee4c109d088c02f139c112" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 44345 } } peers { permanent_uuid: "c71e941c84ba45deb326c90968f40308" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 35323 } }
I20240503 23:27:49.094484 7005 leader_election.cc:288] T 7441a981eb93446c9d9367af49462e73 P 21c0231e5eee4c109d088c02f139c112 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3fc1cf80b12b42c2af63d3afec5214a7 (127.0.100.1:34273), c71e941c84ba45deb326c90968f40308 (127.0.100.2:35323)
I20240503 23:27:49.103595 6804 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7441a981eb93446c9d9367af49462e73" candidate_uuid: "21c0231e5eee4c109d088c02f139c112" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3fc1cf80b12b42c2af63d3afec5214a7" is_pre_election: true
I20240503 23:27:49.104027 6804 raft_consensus.cc:2454] T 7441a981eb93446c9d9367af49462e73 P 3fc1cf80b12b42c2af63d3afec5214a7 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 21c0231e5eee4c109d088c02f139c112 in term 1.
I20240503 23:27:49.104344 7005 raft_consensus.cc:482] T fb79ea9cea314c4cab4bed81163a8ded P 21c0231e5eee4c109d088c02f139c112 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 23:27:49.104869 6924 leader_election.cc:302] T 7441a981eb93446c9d9367af49462e73 P 21c0231e5eee4c109d088c02f139c112 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 21c0231e5eee4c109d088c02f139c112, 3fc1cf80b12b42c2af63d3afec5214a7; no voters:
I20240503 23:27:49.104838 7005 raft_consensus.cc:504] T fb79ea9cea314c4cab4bed81163a8ded P 21c0231e5eee4c109d088c02f139c112 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fc1cf80b12b42c2af63d3afec5214a7" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 34273 } } peers { permanent_uuid: "c71e941c84ba45deb326c90968f40308" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 35323 } } peers { permanent_uuid: "21c0231e5eee4c109d088c02f139c112" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 44345 } }
I20240503 23:27:49.106251 7005 leader_election.cc:288] T fb79ea9cea314c4cab4bed81163a8ded P 21c0231e5eee4c109d088c02f139c112 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 3fc1cf80b12b42c2af63d3afec5214a7 (127.0.100.1:34273), c71e941c84ba45deb326c90968f40308 (127.0.100.2:35323)
I20240503 23:27:49.106604 7005 raft_consensus.cc:2781] T 7441a981eb93446c9d9367af49462e73 P 21c0231e5eee4c109d088c02f139c112 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240503 23:27:49.106779 7005 raft_consensus.cc:482] T 7441a981eb93446c9d9367af49462e73 P 21c0231e5eee4c109d088c02f139c112 [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 23:27:49.106990 7005 raft_consensus.cc:3037] T 7441a981eb93446c9d9367af49462e73 P 21c0231e5eee4c109d088c02f139c112 [term 1 FOLLOWER]: Advancing to term 2
I20240503 23:27:49.107394 6884 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7441a981eb93446c9d9367af49462e73" candidate_uuid: "21c0231e5eee4c109d088c02f139c112" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c71e941c84ba45deb326c90968f40308" is_pre_election: true
I20240503 23:27:49.107929 6884 raft_consensus.cc:2454] T 7441a981eb93446c9d9367af49462e73 P c71e941c84ba45deb326c90968f40308 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 21c0231e5eee4c109d088c02f139c112 in term 1.
I20240503 23:27:49.110002 6804 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb79ea9cea314c4cab4bed81163a8ded" candidate_uuid: "21c0231e5eee4c109d088c02f139c112" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3fc1cf80b12b42c2af63d3afec5214a7" is_pre_election: true
I20240503 23:27:49.110301 6804 raft_consensus.cc:2454] T fb79ea9cea314c4cab4bed81163a8ded P 3fc1cf80b12b42c2af63d3afec5214a7 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 21c0231e5eee4c109d088c02f139c112 in term 2.
I20240503 23:27:49.110944 6884 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb79ea9cea314c4cab4bed81163a8ded" candidate_uuid: "21c0231e5eee4c109d088c02f139c112" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c71e941c84ba45deb326c90968f40308" is_pre_election: true
I20240503 23:27:49.111061 6924 leader_election.cc:302] T fb79ea9cea314c4cab4bed81163a8ded P 21c0231e5eee4c109d088c02f139c112 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 21c0231e5eee4c109d088c02f139c112, 3fc1cf80b12b42c2af63d3afec5214a7; no voters:
I20240503 23:27:49.110999 7005 raft_consensus.cc:504] T 7441a981eb93446c9d9367af49462e73 P 21c0231e5eee4c109d088c02f139c112 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fc1cf80b12b42c2af63d3afec5214a7" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 34273 } } peers { permanent_uuid: "21c0231e5eee4c109d088c02f139c112" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 44345 } } peers { permanent_uuid: "c71e941c84ba45deb326c90968f40308" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 35323 } }
I20240503 23:27:49.111752 7005 leader_election.cc:288] T 7441a981eb93446c9d9367af49462e73 P 21c0231e5eee4c109d088c02f139c112 [CANDIDATE]: Term 2 election: Requested vote from peers 3fc1cf80b12b42c2af63d3afec5214a7 (127.0.100.1:34273), c71e941c84ba45deb326c90968f40308 (127.0.100.2:35323)
I20240503 23:27:49.111950 7005 raft_consensus.cc:2781] T fb79ea9cea314c4cab4bed81163a8ded P 21c0231e5eee4c109d088c02f139c112 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20240503 23:27:49.112077 7005 raft_consensus.cc:482] T fb79ea9cea314c4cab4bed81163a8ded P 21c0231e5eee4c109d088c02f139c112 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 23:27:49.112191 7005 raft_consensus.cc:3037] T fb79ea9cea314c4cab4bed81163a8ded P 21c0231e5eee4c109d088c02f139c112 [term 2 FOLLOWER]: Advancing to term 3
I20240503 23:27:49.112275 6804 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7441a981eb93446c9d9367af49462e73" candidate_uuid: "21c0231e5eee4c109d088c02f139c112" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3fc1cf80b12b42c2af63d3afec5214a7"
I20240503 23:27:49.112507 6804 raft_consensus.cc:3037] T 7441a981eb93446c9d9367af49462e73 P 3fc1cf80b12b42c2af63d3afec5214a7 [term 1 FOLLOWER]: Advancing to term 2
I20240503 23:27:49.115725 6804 raft_consensus.cc:2454] T 7441a981eb93446c9d9367af49462e73 P 3fc1cf80b12b42c2af63d3afec5214a7 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 21c0231e5eee4c109d088c02f139c112 in term 2.
I20240503 23:27:49.115731 7005 raft_consensus.cc:504] T fb79ea9cea314c4cab4bed81163a8ded P 21c0231e5eee4c109d088c02f139c112 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fc1cf80b12b42c2af63d3afec5214a7" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 34273 } } peers { permanent_uuid: "c71e941c84ba45deb326c90968f40308" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 35323 } } peers { permanent_uuid: "21c0231e5eee4c109d088c02f139c112" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 44345 } }
I20240503 23:27:49.116583 6884 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7441a981eb93446c9d9367af49462e73" candidate_uuid: "21c0231e5eee4c109d088c02f139c112" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c71e941c84ba45deb326c90968f40308"
I20240503 23:27:49.116647 6924 leader_election.cc:302] T 7441a981eb93446c9d9367af49462e73 P 21c0231e5eee4c109d088c02f139c112 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 21c0231e5eee4c109d088c02f139c112, 3fc1cf80b12b42c2af63d3afec5214a7; no voters:
I20240503 23:27:49.116775 7005 leader_election.cc:288] T fb79ea9cea314c4cab4bed81163a8ded P 21c0231e5eee4c109d088c02f139c112 [CANDIDATE]: Term 3 election: Requested vote from peers 3fc1cf80b12b42c2af63d3afec5214a7 (127.0.100.1:34273), c71e941c84ba45deb326c90968f40308 (127.0.100.2:35323)
I20240503 23:27:49.116829 6884 raft_consensus.cc:3037] T 7441a981eb93446c9d9367af49462e73 P c71e941c84ba45deb326c90968f40308 [term 1 FOLLOWER]: Advancing to term 2
I20240503 23:27:49.117030 7009 raft_consensus.cc:2781] T 7441a981eb93446c9d9367af49462e73 P 21c0231e5eee4c109d088c02f139c112 [term 2 FOLLOWER]: Leader election won for term 2
I20240503 23:27:49.117231 7009 raft_consensus.cc:686] T 7441a981eb93446c9d9367af49462e73 P 21c0231e5eee4c109d088c02f139c112 [term 2 LEADER]: Becoming Leader. State: Replica: 21c0231e5eee4c109d088c02f139c112, State: Running, Role: LEADER
I20240503 23:27:49.117585 7009 consensus_queue.cc:237] T 7441a981eb93446c9d9367af49462e73 P 21c0231e5eee4c109d088c02f139c112 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fc1cf80b12b42c2af63d3afec5214a7" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 34273 } } peers { permanent_uuid: "21c0231e5eee4c109d088c02f139c112" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 44345 } } peers { permanent_uuid: "c71e941c84ba45deb326c90968f40308" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 35323 } }
I20240503 23:27:49.117653 6883 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb79ea9cea314c4cab4bed81163a8ded" candidate_uuid: "21c0231e5eee4c109d088c02f139c112" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c71e941c84ba45deb326c90968f40308"
I20240503 23:27:49.118188 6804 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb79ea9cea314c4cab4bed81163a8ded" candidate_uuid: "21c0231e5eee4c109d088c02f139c112" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3fc1cf80b12b42c2af63d3afec5214a7"
I20240503 23:27:49.118356 6804 raft_consensus.cc:3037] T fb79ea9cea314c4cab4bed81163a8ded P 3fc1cf80b12b42c2af63d3afec5214a7 [term 2 FOLLOWER]: Advancing to term 3
I20240503 23:27:49.120398 6884 raft_consensus.cc:2454] T 7441a981eb93446c9d9367af49462e73 P c71e941c84ba45deb326c90968f40308 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 21c0231e5eee4c109d088c02f139c112 in term 2.
I20240503 23:27:49.120919 6804 raft_consensus.cc:2454] T fb79ea9cea314c4cab4bed81163a8ded P 3fc1cf80b12b42c2af63d3afec5214a7 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 21c0231e5eee4c109d088c02f139c112 in term 3.
I20240503 23:27:49.121668 6924 leader_election.cc:302] T fb79ea9cea314c4cab4bed81163a8ded P 21c0231e5eee4c109d088c02f139c112 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 21c0231e5eee4c109d088c02f139c112, 3fc1cf80b12b42c2af63d3afec5214a7; no voters: c71e941c84ba45deb326c90968f40308
I20240503 23:27:49.122016 7009 raft_consensus.cc:2781] T fb79ea9cea314c4cab4bed81163a8ded P 21c0231e5eee4c109d088c02f139c112 [term 3 FOLLOWER]: Leader election won for term 3
I20240503 23:27:49.122274 7009 raft_consensus.cc:686] T fb79ea9cea314c4cab4bed81163a8ded P 21c0231e5eee4c109d088c02f139c112 [term 3 LEADER]: Becoming Leader. State: Replica: 21c0231e5eee4c109d088c02f139c112, State: Running, Role: LEADER
I20240503 23:27:49.122378 6338 catalog_manager.cc:5484] T 7441a981eb93446c9d9367af49462e73 P 21c0231e5eee4c109d088c02f139c112 reported cstate change: term changed from 1 to 2. New cstate: current_term: 2 leader_uuid: "21c0231e5eee4c109d088c02f139c112" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fc1cf80b12b42c2af63d3afec5214a7" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 34273 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "21c0231e5eee4c109d088c02f139c112" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 44345 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c71e941c84ba45deb326c90968f40308" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 35323 } health_report { overall_health: UNKNOWN } } }
I20240503 23:27:49.122589 7009 consensus_queue.cc:237] T fb79ea9cea314c4cab4bed81163a8ded P 21c0231e5eee4c109d088c02f139c112 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fc1cf80b12b42c2af63d3afec5214a7" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 34273 } } peers { permanent_uuid: "c71e941c84ba45deb326c90968f40308" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 35323 } } peers { permanent_uuid: "21c0231e5eee4c109d088c02f139c112" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 44345 } }
I20240503 23:27:49.127689 6338 catalog_manager.cc:5484] T fb79ea9cea314c4cab4bed81163a8ded P 21c0231e5eee4c109d088c02f139c112 reported cstate change: term changed from 2 to 3, leader changed from c71e941c84ba45deb326c90968f40308 (127.0.100.2) to 21c0231e5eee4c109d088c02f139c112 (127.0.100.3). New cstate: current_term: 3 leader_uuid: "21c0231e5eee4c109d088c02f139c112" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fc1cf80b12b42c2af63d3afec5214a7" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 34273 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c71e941c84ba45deb326c90968f40308" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 35323 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "21c0231e5eee4c109d088c02f139c112" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 44345 } health_report { overall_health: HEALTHY } } }
I20240503 23:27:49.177526 6995 raft_consensus.cc:482] T e9273b9533d14394b454e6c31ae4deb5 P c71e941c84ba45deb326c90968f40308 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 23:27:49.177796 6995 raft_consensus.cc:504] T e9273b9533d14394b454e6c31ae4deb5 P c71e941c84ba45deb326c90968f40308 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fc1cf80b12b42c2af63d3afec5214a7" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 34273 } } peers { permanent_uuid: "c71e941c84ba45deb326c90968f40308" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 35323 } } peers { permanent_uuid: "21c0231e5eee4c109d088c02f139c112" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 44345 } }
I20240503 23:27:49.178812 6995 leader_election.cc:288] T e9273b9533d14394b454e6c31ae4deb5 P c71e941c84ba45deb326c90968f40308 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3fc1cf80b12b42c2af63d3afec5214a7 (127.0.100.1:34273), 21c0231e5eee4c109d088c02f139c112 (127.0.100.3:44345)
I20240503 23:27:49.179533 6963 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9273b9533d14394b454e6c31ae4deb5" candidate_uuid: "c71e941c84ba45deb326c90968f40308" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "21c0231e5eee4c109d088c02f139c112" is_pre_election: true
I20240503 23:27:49.179534 6804 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9273b9533d14394b454e6c31ae4deb5" candidate_uuid: "c71e941c84ba45deb326c90968f40308" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3fc1cf80b12b42c2af63d3afec5214a7" is_pre_election: true
I20240503 23:27:49.179852 6963 raft_consensus.cc:2396] T e9273b9533d14394b454e6c31ae4deb5 P 21c0231e5eee4c109d088c02f139c112 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate c71e941c84ba45deb326c90968f40308 for term 2 because replica has last-logged OpId of term: 1 index: 1, which is greater than that of the candidate, which has last-logged OpId of term: 0 index: 0.
I20240503 23:27:49.179823 6804 raft_consensus.cc:2454] T e9273b9533d14394b454e6c31ae4deb5 P 3fc1cf80b12b42c2af63d3afec5214a7 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c71e941c84ba45deb326c90968f40308 in term 1.
I20240503 23:27:49.180313 6844 leader_election.cc:302] T e9273b9533d14394b454e6c31ae4deb5 P c71e941c84ba45deb326c90968f40308 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3fc1cf80b12b42c2af63d3afec5214a7, c71e941c84ba45deb326c90968f40308; no voters: 21c0231e5eee4c109d088c02f139c112
I20240503 23:27:49.180660 6995 raft_consensus.cc:2781] T e9273b9533d14394b454e6c31ae4deb5 P c71e941c84ba45deb326c90968f40308 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240503 23:27:49.180780 6995 raft_consensus.cc:482] T e9273b9533d14394b454e6c31ae4deb5 P c71e941c84ba45deb326c90968f40308 [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 23:27:49.180876 6995 raft_consensus.cc:3037] T e9273b9533d14394b454e6c31ae4deb5 P c71e941c84ba45deb326c90968f40308 [term 1 FOLLOWER]: Advancing to term 2
I20240503 23:27:49.183218 6995 raft_consensus.cc:504] T e9273b9533d14394b454e6c31ae4deb5 P c71e941c84ba45deb326c90968f40308 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fc1cf80b12b42c2af63d3afec5214a7" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 34273 } } peers { permanent_uuid: "c71e941c84ba45deb326c90968f40308" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 35323 } } peers { permanent_uuid: "21c0231e5eee4c109d088c02f139c112" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 44345 } }
I20240503 23:27:49.184136 6995 leader_election.cc:288] T e9273b9533d14394b454e6c31ae4deb5 P c71e941c84ba45deb326c90968f40308 [CANDIDATE]: Term 2 election: Requested vote from peers 3fc1cf80b12b42c2af63d3afec5214a7 (127.0.100.1:34273), 21c0231e5eee4c109d088c02f139c112 (127.0.100.3:44345)
I20240503 23:27:49.184796 6963 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9273b9533d14394b454e6c31ae4deb5" candidate_uuid: "c71e941c84ba45deb326c90968f40308" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "21c0231e5eee4c109d088c02f139c112"
I20240503 23:27:49.184830 6804 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9273b9533d14394b454e6c31ae4deb5" candidate_uuid: "c71e941c84ba45deb326c90968f40308" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3fc1cf80b12b42c2af63d3afec5214a7"
I20240503 23:27:49.185030 6963 raft_consensus.cc:3037] T e9273b9533d14394b454e6c31ae4deb5 P 21c0231e5eee4c109d088c02f139c112 [term 1 FOLLOWER]: Advancing to term 2
I20240503 23:27:49.185117 6804 raft_consensus.cc:3037] T e9273b9533d14394b454e6c31ae4deb5 P 3fc1cf80b12b42c2af63d3afec5214a7 [term 1 FOLLOWER]: Advancing to term 2
I20240503 23:27:49.187649 6963 raft_consensus.cc:2396] T e9273b9533d14394b454e6c31ae4deb5 P 21c0231e5eee4c109d088c02f139c112 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate c71e941c84ba45deb326c90968f40308 for term 2 because replica has last-logged OpId of term: 1 index: 1, which is greater than that of the candidate, which has last-logged OpId of term: 0 index: 0.
I20240503 23:27:49.187626 6804 raft_consensus.cc:2454] T e9273b9533d14394b454e6c31ae4deb5 P 3fc1cf80b12b42c2af63d3afec5214a7 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c71e941c84ba45deb326c90968f40308 in term 2.
I20240503 23:27:49.188261 6844 leader_election.cc:302] T e9273b9533d14394b454e6c31ae4deb5 P c71e941c84ba45deb326c90968f40308 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3fc1cf80b12b42c2af63d3afec5214a7, c71e941c84ba45deb326c90968f40308; no voters:
I20240503 23:27:49.188673 6995 raft_consensus.cc:2781] T e9273b9533d14394b454e6c31ae4deb5 P c71e941c84ba45deb326c90968f40308 [term 2 FOLLOWER]: Leader election won for term 2
I20240503 23:27:49.188843 6995 raft_consensus.cc:686] T e9273b9533d14394b454e6c31ae4deb5 P c71e941c84ba45deb326c90968f40308 [term 2 LEADER]: Becoming Leader. State: Replica: c71e941c84ba45deb326c90968f40308, State: Running, Role: LEADER
I20240503 23:27:49.189217 6995 consensus_queue.cc:237] T e9273b9533d14394b454e6c31ae4deb5 P c71e941c84ba45deb326c90968f40308 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fc1cf80b12b42c2af63d3afec5214a7" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 34273 } } peers { permanent_uuid: "c71e941c84ba45deb326c90968f40308" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 35323 } } peers { permanent_uuid: "21c0231e5eee4c109d088c02f139c112" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 44345 } }
I20240503 23:27:49.192405 6338 catalog_manager.cc:5484] T e9273b9533d14394b454e6c31ae4deb5 P c71e941c84ba45deb326c90968f40308 reported cstate change: term changed from 1 to 2, leader changed from 21c0231e5eee4c109d088c02f139c112 (127.0.100.3) to c71e941c84ba45deb326c90968f40308 (127.0.100.2). New cstate: current_term: 2 leader_uuid: "c71e941c84ba45deb326c90968f40308" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fc1cf80b12b42c2af63d3afec5214a7" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 34273 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c71e941c84ba45deb326c90968f40308" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 35323 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "21c0231e5eee4c109d088c02f139c112" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 44345 } health_report { overall_health: UNKNOWN } } }
I20240503 23:27:49.306345 7013 raft_consensus.cc:482] T ad1b9cc708db4d289c2e847874bf747b P 3fc1cf80b12b42c2af63d3afec5214a7 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 23:27:49.306617 7013 raft_consensus.cc:504] T ad1b9cc708db4d289c2e847874bf747b P 3fc1cf80b12b42c2af63d3afec5214a7 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fc1cf80b12b42c2af63d3afec5214a7" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 34273 } } peers { permanent_uuid: "c71e941c84ba45deb326c90968f40308" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 35323 } } peers { permanent_uuid: "21c0231e5eee4c109d088c02f139c112" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 44345 } }
I20240503 23:27:49.307734 7013 leader_election.cc:288] T ad1b9cc708db4d289c2e847874bf747b P 3fc1cf80b12b42c2af63d3afec5214a7 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c71e941c84ba45deb326c90968f40308 (127.0.100.2:35323), 21c0231e5eee4c109d088c02f139c112 (127.0.100.3:44345)
I20240503 23:27:49.316430 6884 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ad1b9cc708db4d289c2e847874bf747b" candidate_uuid: "3fc1cf80b12b42c2af63d3afec5214a7" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c71e941c84ba45deb326c90968f40308" is_pre_election: true
I20240503 23:27:49.316447 6963 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ad1b9cc708db4d289c2e847874bf747b" candidate_uuid: "3fc1cf80b12b42c2af63d3afec5214a7" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "21c0231e5eee4c109d088c02f139c112" is_pre_election: true
I20240503 23:27:49.316792 6884 raft_consensus.cc:2454] T ad1b9cc708db4d289c2e847874bf747b P c71e941c84ba45deb326c90968f40308 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3fc1cf80b12b42c2af63d3afec5214a7 in term 1.
I20240503 23:27:49.316865 6963 raft_consensus.cc:2396] T ad1b9cc708db4d289c2e847874bf747b P 21c0231e5eee4c109d088c02f139c112 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3fc1cf80b12b42c2af63d3afec5214a7 for term 2 because replica has last-logged OpId of term: 1 index: 1, which is greater than that of the candidate, which has last-logged OpId of term: 0 index: 0.
I20240503 23:27:49.317284 6763 leader_election.cc:302] T ad1b9cc708db4d289c2e847874bf747b P 3fc1cf80b12b42c2af63d3afec5214a7 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3fc1cf80b12b42c2af63d3afec5214a7, c71e941c84ba45deb326c90968f40308; no voters:
I20240503 23:27:49.317641 7013 raft_consensus.cc:2781] T ad1b9cc708db4d289c2e847874bf747b P 3fc1cf80b12b42c2af63d3afec5214a7 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240503 23:27:49.317740 7013 raft_consensus.cc:482] T ad1b9cc708db4d289c2e847874bf747b P 3fc1cf80b12b42c2af63d3afec5214a7 [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 23:27:49.317844 7013 raft_consensus.cc:3037] T ad1b9cc708db4d289c2e847874bf747b P 3fc1cf80b12b42c2af63d3afec5214a7 [term 1 FOLLOWER]: Advancing to term 2
I20240503 23:27:49.320482 7013 raft_consensus.cc:504] T ad1b9cc708db4d289c2e847874bf747b P 3fc1cf80b12b42c2af63d3afec5214a7 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fc1cf80b12b42c2af63d3afec5214a7" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 34273 } } peers { permanent_uuid: "c71e941c84ba45deb326c90968f40308" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 35323 } } peers { permanent_uuid: "21c0231e5eee4c109d088c02f139c112" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 44345 } }
I20240503 23:27:49.321329 7013 leader_election.cc:288] T ad1b9cc708db4d289c2e847874bf747b P 3fc1cf80b12b42c2af63d3afec5214a7 [CANDIDATE]: Term 2 election: Requested vote from peers c71e941c84ba45deb326c90968f40308 (127.0.100.2:35323), 21c0231e5eee4c109d088c02f139c112 (127.0.100.3:44345)
I20240503 23:27:49.321930 6884 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ad1b9cc708db4d289c2e847874bf747b" candidate_uuid: "3fc1cf80b12b42c2af63d3afec5214a7" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c71e941c84ba45deb326c90968f40308"
I20240503 23:27:49.322011 6963 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ad1b9cc708db4d289c2e847874bf747b" candidate_uuid: "3fc1cf80b12b42c2af63d3afec5214a7" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "21c0231e5eee4c109d088c02f139c112"
I20240503 23:27:49.322223 6884 raft_consensus.cc:3037] T ad1b9cc708db4d289c2e847874bf747b P c71e941c84ba45deb326c90968f40308 [term 1 FOLLOWER]: Advancing to term 2
I20240503 23:27:49.322248 6963 raft_consensus.cc:3037] T ad1b9cc708db4d289c2e847874bf747b P 21c0231e5eee4c109d088c02f139c112 [term 1 FOLLOWER]: Advancing to term 2
I20240503 23:27:49.324805 6884 raft_consensus.cc:2454] T ad1b9cc708db4d289c2e847874bf747b P c71e941c84ba45deb326c90968f40308 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3fc1cf80b12b42c2af63d3afec5214a7 in term 2.
I20240503 23:27:49.324913 6963 raft_consensus.cc:2396] T ad1b9cc708db4d289c2e847874bf747b P 21c0231e5eee4c109d088c02f139c112 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 3fc1cf80b12b42c2af63d3afec5214a7 for term 2 because replica has last-logged OpId of term: 1 index: 1, which is greater than that of the candidate, which has last-logged OpId of term: 0 index: 0.
I20240503 23:27:49.325456 6763 leader_election.cc:302] T ad1b9cc708db4d289c2e847874bf747b P 3fc1cf80b12b42c2af63d3afec5214a7 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3fc1cf80b12b42c2af63d3afec5214a7, c71e941c84ba45deb326c90968f40308; no voters: 21c0231e5eee4c109d088c02f139c112
I20240503 23:27:49.325867 7013 raft_consensus.cc:2781] T ad1b9cc708db4d289c2e847874bf747b P 3fc1cf80b12b42c2af63d3afec5214a7 [term 2 FOLLOWER]: Leader election won for term 2
I20240503 23:27:49.326452 7013 raft_consensus.cc:686] T ad1b9cc708db4d289c2e847874bf747b P 3fc1cf80b12b42c2af63d3afec5214a7 [term 2 LEADER]: Becoming Leader. State: Replica: 3fc1cf80b12b42c2af63d3afec5214a7, State: Running, Role: LEADER
I20240503 23:27:49.326826 7013 consensus_queue.cc:237] T ad1b9cc708db4d289c2e847874bf747b P 3fc1cf80b12b42c2af63d3afec5214a7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fc1cf80b12b42c2af63d3afec5214a7" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 34273 } } peers { permanent_uuid: "c71e941c84ba45deb326c90968f40308" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 35323 } } peers { permanent_uuid: "21c0231e5eee4c109d088c02f139c112" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 44345 } }
I20240503 23:27:49.330199 6338 catalog_manager.cc:5484] T ad1b9cc708db4d289c2e847874bf747b P 3fc1cf80b12b42c2af63d3afec5214a7 reported cstate change: term changed from 1 to 2, leader changed from 21c0231e5eee4c109d088c02f139c112 (127.0.100.3) to 3fc1cf80b12b42c2af63d3afec5214a7 (127.0.100.1). New cstate: current_term: 2 leader_uuid: "3fc1cf80b12b42c2af63d3afec5214a7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fc1cf80b12b42c2af63d3afec5214a7" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 34273 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c71e941c84ba45deb326c90968f40308" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 35323 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "21c0231e5eee4c109d088c02f139c112" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 44345 } health_report { overall_health: UNKNOWN } } }
I20240503 23:27:49.390888 6963 raft_consensus.cc:1226] T fb79ea9cea314c4cab4bed81163a8ded P 21c0231e5eee4c109d088c02f139c112 [term 3 LEADER]: Rejecting Update request from peer c71e941c84ba45deb326c90968f40308 for earlier term 2. Current term is 3. Ops: []
I20240503 23:27:49.391791 6995 consensus_queue.cc:1046] T fb79ea9cea314c4cab4bed81163a8ded P c71e941c84ba45deb326c90968f40308 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "21c0231e5eee4c109d088c02f139c112" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 44345 }, Status: INVALID_TERM, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 23:27:49.392097 6995 consensus_queue.cc:1225] T fb79ea9cea314c4cab4bed81163a8ded P c71e941c84ba45deb326c90968f40308 [LEADER]: Peer 21c0231e5eee4c109d088c02f139c112 log is divergent from this leader: its last log entry 3.2 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 0
I20240503 23:27:49.392251 6999 raft_consensus.cc:3032] T fb79ea9cea314c4cab4bed81163a8ded P c71e941c84ba45deb326c90968f40308 [term 2 LEADER]: Stepping down as leader of term 2
I20240503 23:27:49.392338 6999 raft_consensus.cc:729] T fb79ea9cea314c4cab4bed81163a8ded P c71e941c84ba45deb326c90968f40308 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: c71e941c84ba45deb326c90968f40308, State: Running, Role: LEADER
I20240503 23:27:49.392680 6999 consensus_queue.cc:260] T fb79ea9cea314c4cab4bed81163a8ded P c71e941c84ba45deb326c90968f40308 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 2.1, Last appended by leader: 1, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fc1cf80b12b42c2af63d3afec5214a7" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 34273 } } peers { permanent_uuid: "c71e941c84ba45deb326c90968f40308" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 35323 } } peers { permanent_uuid: "21c0231e5eee4c109d088c02f139c112" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 44345 } }
I20240503 23:27:49.393024 6999 raft_consensus.cc:3037] T fb79ea9cea314c4cab4bed81163a8ded P c71e941c84ba45deb326c90968f40308 [term 2 FOLLOWER]: Advancing to term 3
I20240503 23:27:49.552893 6804 raft_consensus.cc:1261] T 7441a981eb93446c9d9367af49462e73 P 3fc1cf80b12b42c2af63d3afec5214a7 [term 2 FOLLOWER]: Refusing update from remote peer 21c0231e5eee4c109d088c02f139c112: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240503 23:27:49.553668 7009 consensus_queue.cc:1035] T 7441a981eb93446c9d9367af49462e73 P 21c0231e5eee4c109d088c02f139c112 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3fc1cf80b12b42c2af63d3afec5214a7" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 34273 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 23:27:49.559725 6884 raft_consensus.cc:1261] T 7441a981eb93446c9d9367af49462e73 P c71e941c84ba45deb326c90968f40308 [term 2 FOLLOWER]: Refusing update from remote peer 21c0231e5eee4c109d088c02f139c112: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240503 23:27:49.560613 7009 consensus_queue.cc:1035] T 7441a981eb93446c9d9367af49462e73 P 21c0231e5eee4c109d088c02f139c112 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c71e941c84ba45deb326c90968f40308" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 35323 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 23:27:49.563688 7005 raft_consensus.cc:482] T ad1b9cc708db4d289c2e847874bf747b P 21c0231e5eee4c109d088c02f139c112 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 23:27:49.563922 7005 raft_consensus.cc:504] T ad1b9cc708db4d289c2e847874bf747b P 21c0231e5eee4c109d088c02f139c112 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fc1cf80b12b42c2af63d3afec5214a7" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 34273 } } peers { permanent_uuid: "c71e941c84ba45deb326c90968f40308" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 35323 } } peers { permanent_uuid: "21c0231e5eee4c109d088c02f139c112" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 44345 } }
I20240503 23:27:49.564719 7005 leader_election.cc:288] T ad1b9cc708db4d289c2e847874bf747b P 21c0231e5eee4c109d088c02f139c112 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 3fc1cf80b12b42c2af63d3afec5214a7 (127.0.100.1:34273), c71e941c84ba45deb326c90968f40308 (127.0.100.2:35323)
I20240503 23:27:49.565692 6884 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ad1b9cc708db4d289c2e847874bf747b" candidate_uuid: "21c0231e5eee4c109d088c02f139c112" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c71e941c84ba45deb326c90968f40308" is_pre_election: true
I20240503 23:27:49.566020 6884 raft_consensus.cc:2454] T ad1b9cc708db4d289c2e847874bf747b P c71e941c84ba45deb326c90968f40308 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 21c0231e5eee4c109d088c02f139c112 in term 2.
I20240503 23:27:49.566573 6924 leader_election.cc:302] T ad1b9cc708db4d289c2e847874bf747b P 21c0231e5eee4c109d088c02f139c112 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 21c0231e5eee4c109d088c02f139c112, c71e941c84ba45deb326c90968f40308; no voters:
I20240503 23:27:49.566936 7005 raft_consensus.cc:2781] T ad1b9cc708db4d289c2e847874bf747b P 21c0231e5eee4c109d088c02f139c112 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20240503 23:27:49.567096 7005 raft_consensus.cc:482] T ad1b9cc708db4d289c2e847874bf747b P 21c0231e5eee4c109d088c02f139c112 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 23:27:49.567304 7005 raft_consensus.cc:3037] T ad1b9cc708db4d289c2e847874bf747b P 21c0231e5eee4c109d088c02f139c112 [term 2 FOLLOWER]: Advancing to term 3
I20240503 23:27:49.565802 6804 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ad1b9cc708db4d289c2e847874bf747b" candidate_uuid: "21c0231e5eee4c109d088c02f139c112" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3fc1cf80b12b42c2af63d3afec5214a7" is_pre_election: true
I20240503 23:27:49.571319 7005 raft_consensus.cc:504] T ad1b9cc708db4d289c2e847874bf747b P 21c0231e5eee4c109d088c02f139c112 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fc1cf80b12b42c2af63d3afec5214a7" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 34273 } } peers { permanent_uuid: "c71e941c84ba45deb326c90968f40308" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 35323 } } peers { permanent_uuid: "21c0231e5eee4c109d088c02f139c112" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 44345 } }
I20240503 23:27:49.571542 6963 consensus_queue.cc:237] T 7441a981eb93446c9d9367af49462e73 P 21c0231e5eee4c109d088c02f139c112 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3fc1cf80b12b42c2af63d3afec5214a7" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 34273 } } peers { permanent_uuid: "21c0231e5eee4c109d088c02f139c112" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 44345 } } peers { permanent_uuid: "c71e941c84ba45deb326c90968f40308" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 35323 } attrs { replace: true } } peers { permanent_uuid: "c11e09f93341477483d0045fb1165058" member_type: NON_VOTER last_known_addr { host: "127.0.100.4" port: 33341 } attrs { promote: true } }
I20240503 23:27:49.572047 7005 leader_election.cc:288] T ad1b9cc708db4d289c2e847874bf747b P 21c0231e5eee4c109d088c02f139c112 [CANDIDATE]: Term 3 election: Requested vote from peers 3fc1cf80b12b42c2af63d3afec5214a7 (127.0.100.1:34273), c71e941c84ba45deb326c90968f40308 (127.0.100.2:35323)
I20240503 23:27:49.572533 6804 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ad1b9cc708db4d289c2e847874bf747b" candidate_uuid: "21c0231e5eee4c109d088c02f139c112" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3fc1cf80b12b42c2af63d3afec5214a7"
I20240503 23:27:49.572685 6884 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ad1b9cc708db4d289c2e847874bf747b" candidate_uuid: "21c0231e5eee4c109d088c02f139c112" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c71e941c84ba45deb326c90968f40308"
I20240503 23:27:49.572907 6884 raft_consensus.cc:3037] T ad1b9cc708db4d289c2e847874bf747b P c71e941c84ba45deb326c90968f40308 [term 2 FOLLOWER]: Advancing to term 3
I20240503 23:27:49.574505 6883 raft_consensus.cc:1261] T 7441a981eb93446c9d9367af49462e73 P c71e941c84ba45deb326c90968f40308 [term 2 FOLLOWER]: Refusing update from remote peer 21c0231e5eee4c109d088c02f139c112: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20240503 23:27:49.574513 6804 raft_consensus.cc:1261] T 7441a981eb93446c9d9367af49462e73 P 3fc1cf80b12b42c2af63d3afec5214a7 [term 2 FOLLOWER]: Refusing update from remote peer 21c0231e5eee4c109d088c02f139c112: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20240503 23:27:49.575201 7005 consensus_queue.cc:1035] T 7441a981eb93446c9d9367af49462e73 P 21c0231e5eee4c109d088c02f139c112 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c71e941c84ba45deb326c90968f40308" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 35323 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 23:27:49.575510 7009 consensus_queue.cc:1035] T 7441a981eb93446c9d9367af49462e73 P 21c0231e5eee4c109d088c02f139c112 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3fc1cf80b12b42c2af63d3afec5214a7" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 34273 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 23:27:49.577785 6884 raft_consensus.cc:2454] T ad1b9cc708db4d289c2e847874bf747b P c71e941c84ba45deb326c90968f40308 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 21c0231e5eee4c109d088c02f139c112 in term 3.
I20240503 23:27:49.578415 6924 leader_election.cc:302] T ad1b9cc708db4d289c2e847874bf747b P 21c0231e5eee4c109d088c02f139c112 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 21c0231e5eee4c109d088c02f139c112, c71e941c84ba45deb326c90968f40308; no voters: 3fc1cf80b12b42c2af63d3afec5214a7
I20240503 23:27:49.579669 7009 raft_consensus.cc:2781] T ad1b9cc708db4d289c2e847874bf747b P 21c0231e5eee4c109d088c02f139c112 [term 3 FOLLOWER]: Leader election won for term 3
I20240503 23:27:49.579865 7009 raft_consensus.cc:686] T ad1b9cc708db4d289c2e847874bf747b P 21c0231e5eee4c109d088c02f139c112 [term 3 LEADER]: Becoming Leader. State: Replica: 21c0231e5eee4c109d088c02f139c112, State: Running, Role: LEADER
I20240503 23:27:49.580179 7009 consensus_queue.cc:237] T ad1b9cc708db4d289c2e847874bf747b P 21c0231e5eee4c109d088c02f139c112 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fc1cf80b12b42c2af63d3afec5214a7" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 34273 } } peers { permanent_uuid: "c71e941c84ba45deb326c90968f40308" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 35323 } } peers { permanent_uuid: "21c0231e5eee4c109d088c02f139c112" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 44345 } }
I20240503 23:27:49.579918 6804 raft_consensus.cc:2932] T 7441a981eb93446c9d9367af49462e73 P 3fc1cf80b12b42c2af63d3afec5214a7 [term 2 FOLLOWER]: Committing config change with OpId 2.3: config changed from index -1 to 3, NON_VOTER c11e09f93341477483d0045fb1165058 (127.0.100.4) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3fc1cf80b12b42c2af63d3afec5214a7" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 34273 } } peers { permanent_uuid: "21c0231e5eee4c109d088c02f139c112" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 44345 } } peers { permanent_uuid: "c71e941c84ba45deb326c90968f40308" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 35323 } attrs { replace: true } } peers { permanent_uuid: "c11e09f93341477483d0045fb1165058" member_type: NON_VOTER last_known_addr { host: "127.0.100.4" port: 33341 } attrs { promote: true } } }
I20240503 23:27:49.586668 6339 catalog_manager.cc:5484] T 7441a981eb93446c9d9367af49462e73 P 3fc1cf80b12b42c2af63d3afec5214a7 reported cstate change: config changed from index -1 to 3, NON_VOTER c11e09f93341477483d0045fb1165058 (127.0.100.4) added. New cstate: current_term: 2 leader_uuid: "21c0231e5eee4c109d088c02f139c112" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3fc1cf80b12b42c2af63d3afec5214a7" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 34273 } } peers { permanent_uuid: "21c0231e5eee4c109d088c02f139c112" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 44345 } } peers { permanent_uuid: "c71e941c84ba45deb326c90968f40308" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 35323 } attrs { replace: true } } peers { permanent_uuid: "c11e09f93341477483d0045fb1165058" member_type: NON_VOTER last_known_addr { host: "127.0.100.4" port: 33341 } attrs { promote: true } } }
I20240503 23:27:49.586804 6340 catalog_manager.cc:5484] T ad1b9cc708db4d289c2e847874bf747b P 21c0231e5eee4c109d088c02f139c112 reported cstate change: term changed from 2 to 3, leader changed from 3fc1cf80b12b42c2af63d3afec5214a7 (127.0.100.1) to 21c0231e5eee4c109d088c02f139c112 (127.0.100.3). New cstate: current_term: 3 leader_uuid: "21c0231e5eee4c109d088c02f139c112" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fc1cf80b12b42c2af63d3afec5214a7" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 34273 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c71e941c84ba45deb326c90968f40308" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 35323 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "21c0231e5eee4c109d088c02f139c112" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 44345 } health_report { overall_health: HEALTHY } } }
I20240503 23:27:49.588497 7020 raft_consensus.cc:2932] T 7441a981eb93446c9d9367af49462e73 P 21c0231e5eee4c109d088c02f139c112 [term 2 LEADER]: Committing config change with OpId 2.3: config changed from index -1 to 3, NON_VOTER c11e09f93341477483d0045fb1165058 (127.0.100.4) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3fc1cf80b12b42c2af63d3afec5214a7" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 34273 } } peers { permanent_uuid: "21c0231e5eee4c109d088c02f139c112" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 44345 } } peers { permanent_uuid: "c71e941c84ba45deb326c90968f40308" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 35323 } attrs { replace: true } } peers { permanent_uuid: "c11e09f93341477483d0045fb1165058" member_type: NON_VOTER last_known_addr { host: "127.0.100.4" port: 33341 } attrs { promote: true } } }
W20240503 23:27:49.595568 6388 auto_rebalancer.cc:254] failed to send replica move request: Illegal state: Leader has not yet committed an operation in its own term
I20240503 23:27:49.594655 6884 raft_consensus.cc:2932] T 7441a981eb93446c9d9367af49462e73 P c71e941c84ba45deb326c90968f40308 [term 2 FOLLOWER]: Committing config change with OpId 2.3: config changed from index -1 to 3, NON_VOTER c11e09f93341477483d0045fb1165058 (127.0.100.4) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3fc1cf80b12b42c2af63d3afec5214a7" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 34273 } } peers { permanent_uuid: "21c0231e5eee4c109d088c02f139c112" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 44345 } } peers { permanent_uuid: "c71e941c84ba45deb326c90968f40308" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 35323 } attrs { replace: true } } peers { permanent_uuid: "c11e09f93341477483d0045fb1165058" member_type: NON_VOTER last_known_addr { host: "127.0.100.4" port: 33341 } attrs { promote: true } } }
W20240503 23:27:49.598982 6388 auto_rebalancer.cc:663] Could not move replica: Incomplete: tablet fb79ea9cea314c4cab4bed81163a8ded, TS 3fc1cf80b12b42c2af63d3afec5214a7 -> TS c11e09f93341477483d0045fb1165058 move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fc1cf80b12b42c2af63d3afec5214a7" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 34273 } } peers { permanent_uuid: "c71e941c84ba45deb326c90968f40308" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 35323 } } peers { permanent_uuid: "21c0231e5eee4c109d088c02f139c112" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 44345 } }
W20240503 23:27:49.599118 6388 auto_rebalancer.cc:264] scheduled replica move failed to complete: Incomplete: tablet fb79ea9cea314c4cab4bed81163a8ded, TS 3fc1cf80b12b42c2af63d3afec5214a7 -> TS c11e09f93341477483d0045fb1165058 move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fc1cf80b12b42c2af63d3afec5214a7" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 34273 } } peers { permanent_uuid: "c71e941c84ba45deb326c90968f40308" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 35323 } } peers { permanent_uuid: "21c0231e5eee4c109d088c02f139c112" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 44345 } }
W20240503 23:27:49.599788 6924 consensus_peers.cc:485] T 7441a981eb93446c9d9367af49462e73 P 21c0231e5eee4c109d088c02f139c112 -> Peer c11e09f93341477483d0045fb1165058 (127.0.100.4:33341): Couldn't send request to peer c11e09f93341477483d0045fb1165058. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 7441a981eb93446c9d9367af49462e73. This is attempt 1: this message will repeat every 5th retry.
W20240503 23:27:49.608482 6388 auto_rebalancer.cc:663] Could not move replica: Incomplete: tablet e9273b9533d14394b454e6c31ae4deb5, TS 21c0231e5eee4c109d088c02f139c112 -> TS c11e09f93341477483d0045fb1165058 move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fc1cf80b12b42c2af63d3afec5214a7" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 34273 } } peers { permanent_uuid: "c71e941c84ba45deb326c90968f40308" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 35323 } } peers { permanent_uuid: "21c0231e5eee4c109d088c02f139c112" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 44345 } }
W20240503 23:27:49.608635 6388 auto_rebalancer.cc:264] scheduled replica move failed to complete: Incomplete: tablet e9273b9533d14394b454e6c31ae4deb5, TS 21c0231e5eee4c109d088c02f139c112 -> TS c11e09f93341477483d0045fb1165058 move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fc1cf80b12b42c2af63d3afec5214a7" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 34273 } } peers { permanent_uuid: "c71e941c84ba45deb326c90968f40308" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 35323 } } peers { permanent_uuid: "21c0231e5eee4c109d088c02f139c112" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 44345 } }
I20240503 23:27:49.674926 7020 raft_consensus.cc:482] T e9273b9533d14394b454e6c31ae4deb5 P 21c0231e5eee4c109d088c02f139c112 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 23:27:49.675146 7020 raft_consensus.cc:504] T e9273b9533d14394b454e6c31ae4deb5 P 21c0231e5eee4c109d088c02f139c112 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fc1cf80b12b42c2af63d3afec5214a7" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 34273 } } peers { permanent_uuid: "c71e941c84ba45deb326c90968f40308" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 35323 } } peers { permanent_uuid: "21c0231e5eee4c109d088c02f139c112" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 44345 } }
I20240503 23:27:49.675885 7020 leader_election.cc:288] T e9273b9533d14394b454e6c31ae4deb5 P 21c0231e5eee4c109d088c02f139c112 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 3fc1cf80b12b42c2af63d3afec5214a7 (127.0.100.1:34273), c71e941c84ba45deb326c90968f40308 (127.0.100.2:35323)
I20240503 23:27:49.676453 6884 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9273b9533d14394b454e6c31ae4deb5" candidate_uuid: "21c0231e5eee4c109d088c02f139c112" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c71e941c84ba45deb326c90968f40308" is_pre_election: true
I20240503 23:27:49.676458 6804 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9273b9533d14394b454e6c31ae4deb5" candidate_uuid: "21c0231e5eee4c109d088c02f139c112" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3fc1cf80b12b42c2af63d3afec5214a7" is_pre_election: true
I20240503 23:27:49.676797 6804 raft_consensus.cc:2454] T e9273b9533d14394b454e6c31ae4deb5 P 3fc1cf80b12b42c2af63d3afec5214a7 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 21c0231e5eee4c109d088c02f139c112 in term 2.
I20240503 23:27:49.677243 6924 leader_election.cc:302] T e9273b9533d14394b454e6c31ae4deb5 P 21c0231e5eee4c109d088c02f139c112 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 21c0231e5eee4c109d088c02f139c112, 3fc1cf80b12b42c2af63d3afec5214a7; no voters:
I20240503 23:27:49.677562 7020 raft_consensus.cc:2781] T e9273b9533d14394b454e6c31ae4deb5 P 21c0231e5eee4c109d088c02f139c112 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20240503 23:27:49.677649 7020 raft_consensus.cc:482] T e9273b9533d14394b454e6c31ae4deb5 P 21c0231e5eee4c109d088c02f139c112 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 23:27:49.677733 7020 raft_consensus.cc:3037] T e9273b9533d14394b454e6c31ae4deb5 P 21c0231e5eee4c109d088c02f139c112 [term 2 FOLLOWER]: Advancing to term 3
I20240503 23:27:49.680961 7020 raft_consensus.cc:504] T e9273b9533d14394b454e6c31ae4deb5 P 21c0231e5eee4c109d088c02f139c112 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fc1cf80b12b42c2af63d3afec5214a7" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 34273 } } peers { permanent_uuid: "c71e941c84ba45deb326c90968f40308" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 35323 } } peers { permanent_uuid: "21c0231e5eee4c109d088c02f139c112" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 44345 } }
I20240503 23:27:49.681752 7020 leader_election.cc:288] T e9273b9533d14394b454e6c31ae4deb5 P 21c0231e5eee4c109d088c02f139c112 [CANDIDATE]: Term 3 election: Requested vote from peers 3fc1cf80b12b42c2af63d3afec5214a7 (127.0.100.1:34273), c71e941c84ba45deb326c90968f40308 (127.0.100.2:35323)
I20240503 23:27:49.682137 6804 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9273b9533d14394b454e6c31ae4deb5" candidate_uuid: "21c0231e5eee4c109d088c02f139c112" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3fc1cf80b12b42c2af63d3afec5214a7"
I20240503 23:27:49.682216 6884 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9273b9533d14394b454e6c31ae4deb5" candidate_uuid: "21c0231e5eee4c109d088c02f139c112" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c71e941c84ba45deb326c90968f40308"
I20240503 23:27:49.682324 6804 raft_consensus.cc:3037] T e9273b9533d14394b454e6c31ae4deb5 P 3fc1cf80b12b42c2af63d3afec5214a7 [term 2 FOLLOWER]: Advancing to term 3
I20240503 23:27:49.684688 6804 raft_consensus.cc:2454] T e9273b9533d14394b454e6c31ae4deb5 P 3fc1cf80b12b42c2af63d3afec5214a7 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 21c0231e5eee4c109d088c02f139c112 in term 3.
I20240503 23:27:49.685109 6924 leader_election.cc:302] T e9273b9533d14394b454e6c31ae4deb5 P 21c0231e5eee4c109d088c02f139c112 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 21c0231e5eee4c109d088c02f139c112, 3fc1cf80b12b42c2af63d3afec5214a7; no voters: c71e941c84ba45deb326c90968f40308
I20240503 23:27:49.685462 7020 raft_consensus.cc:2781] T e9273b9533d14394b454e6c31ae4deb5 P 21c0231e5eee4c109d088c02f139c112 [term 3 FOLLOWER]: Leader election won for term 3
I20240503 23:27:49.685624 7020 raft_consensus.cc:686] T e9273b9533d14394b454e6c31ae4deb5 P 21c0231e5eee4c109d088c02f139c112 [term 3 LEADER]: Becoming Leader. State: Replica: 21c0231e5eee4c109d088c02f139c112, State: Running, Role: LEADER
I20240503 23:27:49.685892 7020 consensus_queue.cc:237] T e9273b9533d14394b454e6c31ae4deb5 P 21c0231e5eee4c109d088c02f139c112 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fc1cf80b12b42c2af63d3afec5214a7" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 34273 } } peers { permanent_uuid: "c71e941c84ba45deb326c90968f40308" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 35323 } } peers { permanent_uuid: "21c0231e5eee4c109d088c02f139c112" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 44345 } }
I20240503 23:27:49.689369 6340 catalog_manager.cc:5484] T e9273b9533d14394b454e6c31ae4deb5 P 21c0231e5eee4c109d088c02f139c112 reported cstate change: term changed from 2 to 3, leader changed from c71e941c84ba45deb326c90968f40308 (127.0.100.2) to 21c0231e5eee4c109d088c02f139c112 (127.0.100.3). New cstate: current_term: 3 leader_uuid: "21c0231e5eee4c109d088c02f139c112" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fc1cf80b12b42c2af63d3afec5214a7" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 34273 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c71e941c84ba45deb326c90968f40308" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 35323 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "21c0231e5eee4c109d088c02f139c112" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 44345 } health_report { overall_health: HEALTHY } } }
I20240503 23:27:49.698305 6804 raft_consensus.cc:1261] T fb79ea9cea314c4cab4bed81163a8ded P 3fc1cf80b12b42c2af63d3afec5214a7 [term 3 FOLLOWER]: Refusing update from remote peer 21c0231e5eee4c109d088c02f139c112: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20240503 23:27:49.698885 7020 consensus_queue.cc:1035] T fb79ea9cea314c4cab4bed81163a8ded P 21c0231e5eee4c109d088c02f139c112 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3fc1cf80b12b42c2af63d3afec5214a7" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 34273 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 23:27:49.703655 6884 raft_consensus.cc:1261] T fb79ea9cea314c4cab4bed81163a8ded P c71e941c84ba45deb326c90968f40308 [term 3 FOLLOWER]: Refusing update from remote peer 21c0231e5eee4c109d088c02f139c112: Log matching property violated. Preceding OpId in replica: term: 2 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20240503 23:27:49.704394 7025 consensus_queue.cc:1035] T fb79ea9cea314c4cab4bed81163a8ded P 21c0231e5eee4c109d088c02f139c112 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c71e941c84ba45deb326c90968f40308" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 35323 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 23:27:49.704712 7025 consensus_queue.cc:1225] T fb79ea9cea314c4cab4bed81163a8ded P 21c0231e5eee4c109d088c02f139c112 [LEADER]: Peer c71e941c84ba45deb326c90968f40308 log is divergent from this leader: its last log entry 2.1 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 0
I20240503 23:27:49.705920 6884 pending_rounds.cc:76] T fb79ea9cea314c4cab4bed81163a8ded P c71e941c84ba45deb326c90968f40308: Aborting all ops after (but not including) 0
I20240503 23:27:49.706020 6884 pending_rounds.cc:97] T fb79ea9cea314c4cab4bed81163a8ded P c71e941c84ba45deb326c90968f40308: Aborting uncommitted NO_OP operation due to leader change: 2.1
I20240503 23:27:49.706102 6884 raft_consensus.cc:2866] T fb79ea9cea314c4cab4bed81163a8ded P c71e941c84ba45deb326c90968f40308 [term 3 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20240503 23:27:49.723408 6963 raft_consensus.cc:1226] T ad1b9cc708db4d289c2e847874bf747b P 21c0231e5eee4c109d088c02f139c112 [term 3 LEADER]: Rejecting Update request from peer 3fc1cf80b12b42c2af63d3afec5214a7 for earlier term 2. Current term is 3. Ops: []
I20240503 23:27:49.724182 7013 consensus_queue.cc:1046] T ad1b9cc708db4d289c2e847874bf747b P 3fc1cf80b12b42c2af63d3afec5214a7 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "21c0231e5eee4c109d088c02f139c112" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 44345 }, Status: INVALID_TERM, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 23:27:49.724445 7013 consensus_queue.cc:1225] T ad1b9cc708db4d289c2e847874bf747b P 3fc1cf80b12b42c2af63d3afec5214a7 [LEADER]: Peer 21c0231e5eee4c109d088c02f139c112 log is divergent from this leader: its last log entry 3.2 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 0
I20240503 23:27:49.724571 7016 raft_consensus.cc:3032] T ad1b9cc708db4d289c2e847874bf747b P 3fc1cf80b12b42c2af63d3afec5214a7 [term 2 LEADER]: Stepping down as leader of term 2
I20240503 23:27:49.724682 7016 raft_consensus.cc:729] T ad1b9cc708db4d289c2e847874bf747b P 3fc1cf80b12b42c2af63d3afec5214a7 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 3fc1cf80b12b42c2af63d3afec5214a7, State: Running, Role: LEADER
I20240503 23:27:49.724949 7016 consensus_queue.cc:260] T ad1b9cc708db4d289c2e847874bf747b P 3fc1cf80b12b42c2af63d3afec5214a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 2.1, Last appended by leader: 1, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fc1cf80b12b42c2af63d3afec5214a7" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 34273 } } peers { permanent_uuid: "c71e941c84ba45deb326c90968f40308" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 35323 } } peers { permanent_uuid: "21c0231e5eee4c109d088c02f139c112" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 44345 } }
I20240503 23:27:49.725324 7016 raft_consensus.cc:3037] T ad1b9cc708db4d289c2e847874bf747b P 3fc1cf80b12b42c2af63d3afec5214a7 [term 2 FOLLOWER]: Advancing to term 3
I20240503 23:27:49.803117 6963 raft_consensus.cc:1226] T e9273b9533d14394b454e6c31ae4deb5 P 21c0231e5eee4c109d088c02f139c112 [term 3 LEADER]: Rejecting Update request from peer c71e941c84ba45deb326c90968f40308 for earlier term 2. Current term is 3. Ops: []
I20240503 23:27:49.803771 6999 consensus_queue.cc:1046] T e9273b9533d14394b454e6c31ae4deb5 P c71e941c84ba45deb326c90968f40308 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "21c0231e5eee4c109d088c02f139c112" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 44345 }, Status: INVALID_TERM, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 23:27:49.804095 6999 consensus_queue.cc:1225] T e9273b9533d14394b454e6c31ae4deb5 P c71e941c84ba45deb326c90968f40308 [LEADER]: Peer 21c0231e5eee4c109d088c02f139c112 log is divergent from this leader: its last log entry 3.2 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 0
I20240503 23:27:49.804222 6999 raft_consensus.cc:3032] T e9273b9533d14394b454e6c31ae4deb5 P c71e941c84ba45deb326c90968f40308 [term 2 LEADER]: Stepping down as leader of term 2
I20240503 23:27:49.804291 6999 raft_consensus.cc:729] T e9273b9533d14394b454e6c31ae4deb5 P c71e941c84ba45deb326c90968f40308 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: c71e941c84ba45deb326c90968f40308, State: Running, Role: LEADER
I20240503 23:27:49.804594 6999 consensus_queue.cc:260] T e9273b9533d14394b454e6c31ae4deb5 P c71e941c84ba45deb326c90968f40308 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 2.1, Last appended by leader: 1, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fc1cf80b12b42c2af63d3afec5214a7" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 34273 } } peers { permanent_uuid: "c71e941c84ba45deb326c90968f40308" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 35323 } } peers { permanent_uuid: "21c0231e5eee4c109d088c02f139c112" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 44345 } }
I20240503 23:27:49.804939 6999 raft_consensus.cc:3037] T e9273b9533d14394b454e6c31ae4deb5 P c71e941c84ba45deb326c90968f40308 [term 2 FOLLOWER]: Advancing to term 3
I20240503 23:27:50.015769 7031 ts_tablet_manager.cc:902] T 7441a981eb93446c9d9367af49462e73 P c11e09f93341477483d0045fb1165058: Initiating tablet copy from peer 21c0231e5eee4c109d088c02f139c112 (127.0.100.3:44345)
I20240503 23:27:50.016485 7031 tablet_copy_client.cc:316] T 7441a981eb93446c9d9367af49462e73 P c11e09f93341477483d0045fb1165058: tablet copy: Beginning tablet copy session from remote peer at address 127.0.100.3:44345
I20240503 23:27:50.023792 6973 tablet_copy_service.cc:140] P 21c0231e5eee4c109d088c02f139c112: Received BeginTabletCopySession request for tablet 7441a981eb93446c9d9367af49462e73 from peer c11e09f93341477483d0045fb1165058 ({username='slave'} at 127.0.0.1:60112)
I20240503 23:27:50.024062 6973 tablet_copy_service.cc:161] P 21c0231e5eee4c109d088c02f139c112: Beginning new tablet copy session on tablet 7441a981eb93446c9d9367af49462e73 from peer c11e09f93341477483d0045fb1165058 at {username='slave'} at 127.0.0.1:60112: session id = c11e09f93341477483d0045fb1165058-7441a981eb93446c9d9367af49462e73
I20240503 23:27:50.027038 6973 tablet_copy_source_session.cc:205] T 7441a981eb93446c9d9367af49462e73 P 21c0231e5eee4c109d088c02f139c112: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 23:27:50.028105 6804 raft_consensus.cc:1261] T ad1b9cc708db4d289c2e847874bf747b P 3fc1cf80b12b42c2af63d3afec5214a7 [term 3 FOLLOWER]: Refusing update from remote peer 21c0231e5eee4c109d088c02f139c112: Log matching property violated. Preceding OpId in replica: term: 2 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20240503 23:27:50.028506 7031 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7441a981eb93446c9d9367af49462e73. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 23:27:50.028775 7020 consensus_queue.cc:1035] T ad1b9cc708db4d289c2e847874bf747b P 21c0231e5eee4c109d088c02f139c112 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3fc1cf80b12b42c2af63d3afec5214a7" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 34273 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 23:27:50.029008 7020 consensus_queue.cc:1225] T ad1b9cc708db4d289c2e847874bf747b P 21c0231e5eee4c109d088c02f139c112 [LEADER]: Peer 3fc1cf80b12b42c2af63d3afec5214a7 log is divergent from this leader: its last log entry 2.1 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 0
I20240503 23:27:50.030051 6804 pending_rounds.cc:76] T ad1b9cc708db4d289c2e847874bf747b P 3fc1cf80b12b42c2af63d3afec5214a7: Aborting all ops after (but not including) 0
I20240503 23:27:50.030118 6804 pending_rounds.cc:97] T ad1b9cc708db4d289c2e847874bf747b P 3fc1cf80b12b42c2af63d3afec5214a7: Aborting uncommitted NO_OP operation due to leader change: 2.1
I20240503 23:27:50.030174 6804 raft_consensus.cc:2866] T ad1b9cc708db4d289c2e847874bf747b P 3fc1cf80b12b42c2af63d3afec5214a7 [term 3 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20240503 23:27:50.034174 6884 raft_consensus.cc:1261] T ad1b9cc708db4d289c2e847874bf747b P c71e941c84ba45deb326c90968f40308 [term 3 FOLLOWER]: Refusing update from remote peer 21c0231e5eee4c109d088c02f139c112: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20240503 23:27:50.035351 7031 tablet_copy_client.cc:791] T 7441a981eb93446c9d9367af49462e73 P c11e09f93341477483d0045fb1165058: tablet copy: Starting download of 0 data blocks...
I20240503 23:27:50.035634 7031 tablet_copy_client.cc:655] T 7441a981eb93446c9d9367af49462e73 P c11e09f93341477483d0045fb1165058: tablet copy: Starting download of 1 WAL segments...
I20240503 23:27:50.035831 7005 consensus_queue.cc:1035] T ad1b9cc708db4d289c2e847874bf747b P 21c0231e5eee4c109d088c02f139c112 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c71e941c84ba45deb326c90968f40308" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 35323 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 23:27:50.042114 7031 tablet_copy_client.cc:531] T 7441a981eb93446c9d9367af49462e73 P c11e09f93341477483d0045fb1165058: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 23:27:50.046053 7031 tablet_bootstrap.cc:492] T 7441a981eb93446c9d9367af49462e73 P c11e09f93341477483d0045fb1165058: Bootstrap starting.
I20240503 23:27:50.052382 7031 tablet_bootstrap.cc:492] T 7441a981eb93446c9d9367af49462e73 P c11e09f93341477483d0045fb1165058: Bootstrap replayed 1/1 log segments. Stats: ops{read=3 overwritten=0 applied=3 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 23:27:50.052788 7031 tablet_bootstrap.cc:492] T 7441a981eb93446c9d9367af49462e73 P c11e09f93341477483d0045fb1165058: Bootstrap complete.
I20240503 23:27:50.053050 7031 ts_tablet_manager.cc:1367] T 7441a981eb93446c9d9367af49462e73 P c11e09f93341477483d0045fb1165058: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.002s
I20240503 23:27:50.053754 7031 raft_consensus.cc:348] T 7441a981eb93446c9d9367af49462e73 P c11e09f93341477483d0045fb1165058 [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3fc1cf80b12b42c2af63d3afec5214a7" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 34273 } } peers { permanent_uuid: "21c0231e5eee4c109d088c02f139c112" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 44345 } } peers { permanent_uuid: "c71e941c84ba45deb326c90968f40308" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 35323 } attrs { replace: true } } peers { permanent_uuid: "c11e09f93341477483d0045fb1165058" member_type: NON_VOTER last_known_addr { host: "127.0.100.4" port: 33341 } attrs { promote: true } }
I20240503 23:27:50.053969 7031 raft_consensus.cc:729] T 7441a981eb93446c9d9367af49462e73 P c11e09f93341477483d0045fb1165058 [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: c11e09f93341477483d0045fb1165058, State: Initialized, Role: LEARNER
I20240503 23:27:50.054178 7031 consensus_queue.cc:260] T 7441a981eb93446c9d9367af49462e73 P c11e09f93341477483d0045fb1165058 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 2.3, Last appended by leader: 3, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3fc1cf80b12b42c2af63d3afec5214a7" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 34273 } } peers { permanent_uuid: "21c0231e5eee4c109d088c02f139c112" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 44345 } } peers { permanent_uuid: "c71e941c84ba45deb326c90968f40308" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 35323 } attrs { replace: true } } peers { permanent_uuid: "c11e09f93341477483d0045fb1165058" member_type: NON_VOTER last_known_addr { host: "127.0.100.4" port: 33341 } attrs { promote: true } }
I20240503 23:27:50.055157 7031 ts_tablet_manager.cc:1398] T 7441a981eb93446c9d9367af49462e73 P c11e09f93341477483d0045fb1165058: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240503 23:27:50.055802 6973 tablet_copy_service.cc:342] P 21c0231e5eee4c109d088c02f139c112: Request end of tablet copy session c11e09f93341477483d0045fb1165058-7441a981eb93446c9d9367af49462e73 received from {username='slave'} at 127.0.0.1:60112
I20240503 23:27:50.055927 6973 tablet_copy_service.cc:433] P 21c0231e5eee4c109d088c02f139c112: ending tablet copy session c11e09f93341477483d0045fb1165058-7441a981eb93446c9d9367af49462e73 on tablet 7441a981eb93446c9d9367af49462e73 with peer c11e09f93341477483d0045fb1165058
I20240503 23:27:50.267927 6804 raft_consensus.cc:1261] T e9273b9533d14394b454e6c31ae4deb5 P 3fc1cf80b12b42c2af63d3afec5214a7 [term 3 FOLLOWER]: Refusing update from remote peer 21c0231e5eee4c109d088c02f139c112: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20240503 23:27:50.268597 7009 consensus_queue.cc:1035] T e9273b9533d14394b454e6c31ae4deb5 P 21c0231e5eee4c109d088c02f139c112 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3fc1cf80b12b42c2af63d3afec5214a7" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 34273 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 23:27:50.274904 6884 raft_consensus.cc:1261] T e9273b9533d14394b454e6c31ae4deb5 P c71e941c84ba45deb326c90968f40308 [term 3 FOLLOWER]: Refusing update from remote peer 21c0231e5eee4c109d088c02f139c112: Log matching property violated. Preceding OpId in replica: term: 2 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20240503 23:27:50.276144 7009 consensus_queue.cc:1035] T e9273b9533d14394b454e6c31ae4deb5 P 21c0231e5eee4c109d088c02f139c112 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c71e941c84ba45deb326c90968f40308" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 35323 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 23:27:50.276491 7009 consensus_queue.cc:1225] T e9273b9533d14394b454e6c31ae4deb5 P 21c0231e5eee4c109d088c02f139c112 [LEADER]: Peer c71e941c84ba45deb326c90968f40308 log is divergent from this leader: its last log entry 2.1 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 0
I20240503 23:27:50.277654 6884 pending_rounds.cc:76] T e9273b9533d14394b454e6c31ae4deb5 P c71e941c84ba45deb326c90968f40308: Aborting all ops after (but not including) 0
I20240503 23:27:50.277731 6884 pending_rounds.cc:97] T e9273b9533d14394b454e6c31ae4deb5 P c71e941c84ba45deb326c90968f40308: Aborting uncommitted NO_OP operation due to leader change: 2.1
I20240503 23:27:50.277793 6884 raft_consensus.cc:2866] T e9273b9533d14394b454e6c31ae4deb5 P c71e941c84ba45deb326c90968f40308 [term 3 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20240503 23:27:50.435428 6708 raft_consensus.cc:1203] T 7441a981eb93446c9d9367af49462e73 P c11e09f93341477483d0045fb1165058 [term 2 LEARNER]: Deduplicated request from leader. Original: 2.2->[2.3-2.3] Dedup: 2.3->[]
I20240503 23:27:50.854368 7005 raft_consensus.cc:1050] T 7441a981eb93446c9d9367af49462e73 P 21c0231e5eee4c109d088c02f139c112: attempting to promote NON_VOTER c11e09f93341477483d0045fb1165058 to VOTER
I20240503 23:27:50.855276 7005 consensus_queue.cc:237] T 7441a981eb93446c9d9367af49462e73 P 21c0231e5eee4c109d088c02f139c112 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.3, Last appended by leader: 1, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fc1cf80b12b42c2af63d3afec5214a7" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 34273 } } peers { permanent_uuid: "21c0231e5eee4c109d088c02f139c112" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 44345 } } peers { permanent_uuid: "c71e941c84ba45deb326c90968f40308" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 35323 } attrs { replace: true } } peers { permanent_uuid: "c11e09f93341477483d0045fb1165058" member_type: VOTER last_known_addr { host: "127.0.100.4" port: 33341 } attrs { promote: false } }
I20240503 23:27:50.857733 6884 raft_consensus.cc:1261] T 7441a981eb93446c9d9367af49462e73 P c71e941c84ba45deb326c90968f40308 [term 2 FOLLOWER]: Refusing update from remote peer 21c0231e5eee4c109d088c02f139c112: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240503 23:27:50.857740 6708 raft_consensus.cc:1261] T 7441a981eb93446c9d9367af49462e73 P c11e09f93341477483d0045fb1165058 [term 2 LEARNER]: Refusing update from remote peer 21c0231e5eee4c109d088c02f139c112: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240503 23:27:50.858456 6804 raft_consensus.cc:1261] T 7441a981eb93446c9d9367af49462e73 P 3fc1cf80b12b42c2af63d3afec5214a7 [term 2 FOLLOWER]: Refusing update from remote peer 21c0231e5eee4c109d088c02f139c112: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240503 23:27:50.858887 7020 consensus_queue.cc:1035] T 7441a981eb93446c9d9367af49462e73 P 21c0231e5eee4c109d088c02f139c112 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c71e941c84ba45deb326c90968f40308" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 35323 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 23:27:50.859248 7009 consensus_queue.cc:1035] T 7441a981eb93446c9d9367af49462e73 P 21c0231e5eee4c109d088c02f139c112 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c11e09f93341477483d0045fb1165058" member_type: VOTER last_known_addr { host: "127.0.100.4" port: 33341 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 23:27:50.859454 7005 consensus_queue.cc:1035] T 7441a981eb93446c9d9367af49462e73 P 21c0231e5eee4c109d088c02f139c112 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3fc1cf80b12b42c2af63d3afec5214a7" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 34273 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 23:27:50.862776 7020 raft_consensus.cc:2932] T 7441a981eb93446c9d9367af49462e73 P 21c0231e5eee4c109d088c02f139c112 [term 2 LEADER]: Committing config change with OpId 2.4: config changed from index 3 to 4, c11e09f93341477483d0045fb1165058 (127.0.100.4) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fc1cf80b12b42c2af63d3afec5214a7" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 34273 } } peers { permanent_uuid: "21c0231e5eee4c109d088c02f139c112" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 44345 } } peers { permanent_uuid: "c71e941c84ba45deb326c90968f40308" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 35323 } attrs { replace: true } } peers { permanent_uuid: "c11e09f93341477483d0045fb1165058" member_type: VOTER last_known_addr { host: "127.0.100.4" port: 33341 } attrs { promote: false } } }
I20240503 23:27:50.863389 6884 raft_consensus.cc:2932] T 7441a981eb93446c9d9367af49462e73 P c71e941c84ba45deb326c90968f40308 [term 2 FOLLOWER]: Committing config change with OpId 2.4: config changed from index 3 to 4, c11e09f93341477483d0045fb1165058 (127.0.100.4) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fc1cf80b12b42c2af63d3afec5214a7" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 34273 } } peers { permanent_uuid: "21c0231e5eee4c109d088c02f139c112" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 44345 } } peers { permanent_uuid: "c71e941c84ba45deb326c90968f40308" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 35323 } attrs { replace: true } } peers { permanent_uuid: "c11e09f93341477483d0045fb1165058" member_type: VOTER last_known_addr { host: "127.0.100.4" port: 33341 } attrs { promote: false } } }
I20240503 23:27:50.867730 6708 raft_consensus.cc:2932] T 7441a981eb93446c9d9367af49462e73 P c11e09f93341477483d0045fb1165058 [term 2 FOLLOWER]: Committing config change with OpId 2.4: config changed from index 3 to 4, c11e09f93341477483d0045fb1165058 (127.0.100.4) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fc1cf80b12b42c2af63d3afec5214a7" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 34273 } } peers { permanent_uuid: "21c0231e5eee4c109d088c02f139c112" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 44345 } } peers { permanent_uuid: "c71e941c84ba45deb326c90968f40308" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 35323 } attrs { replace: true } } peers { permanent_uuid: "c11e09f93341477483d0045fb1165058" member_type: VOTER last_known_addr { host: "127.0.100.4" port: 33341 } attrs { promote: false } } }
I20240503 23:27:50.867908 6804 raft_consensus.cc:2932] T 7441a981eb93446c9d9367af49462e73 P 3fc1cf80b12b42c2af63d3afec5214a7 [term 2 FOLLOWER]: Committing config change with OpId 2.4: config changed from index 3 to 4, c11e09f93341477483d0045fb1165058 (127.0.100.4) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fc1cf80b12b42c2af63d3afec5214a7" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 34273 } } peers { permanent_uuid: "21c0231e5eee4c109d088c02f139c112" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 44345 } } peers { permanent_uuid: "c71e941c84ba45deb326c90968f40308" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 35323 } attrs { replace: true } } peers { permanent_uuid: "c11e09f93341477483d0045fb1165058" member_type: VOTER last_known_addr { host: "127.0.100.4" port: 33341 } attrs { promote: false } } }
I20240503 23:27:50.868256 6340 catalog_manager.cc:5484] T 7441a981eb93446c9d9367af49462e73 P 21c0231e5eee4c109d088c02f139c112 reported cstate change: config changed from index 3 to 4, c11e09f93341477483d0045fb1165058 (127.0.100.4) changed from NON_VOTER to VOTER. New cstate: current_term: 2 leader_uuid: "21c0231e5eee4c109d088c02f139c112" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fc1cf80b12b42c2af63d3afec5214a7" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 34273 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "21c0231e5eee4c109d088c02f139c112" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 44345 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c71e941c84ba45deb326c90968f40308" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 35323 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c11e09f93341477483d0045fb1165058" member_type: VOTER last_known_addr { host: "127.0.100.4" port: 33341 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 23:27:50.886977 6963 consensus_queue.cc:237] T 7441a981eb93446c9d9367af49462e73 P 21c0231e5eee4c109d088c02f139c112 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.4, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3fc1cf80b12b42c2af63d3afec5214a7" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 34273 } } peers { permanent_uuid: "21c0231e5eee4c109d088c02f139c112" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 44345 } } peers { permanent_uuid: "c11e09f93341477483d0045fb1165058" member_type: VOTER last_known_addr { host: "127.0.100.4" port: 33341 } attrs { promote: false } }
I20240503 23:27:50.889328 6708 raft_consensus.cc:1261] T 7441a981eb93446c9d9367af49462e73 P c11e09f93341477483d0045fb1165058 [term 2 FOLLOWER]: Refusing update from remote peer 21c0231e5eee4c109d088c02f139c112: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240503 23:27:50.889380 6804 raft_consensus.cc:1261] T 7441a981eb93446c9d9367af49462e73 P 3fc1cf80b12b42c2af63d3afec5214a7 [term 2 FOLLOWER]: Refusing update from remote peer 21c0231e5eee4c109d088c02f139c112: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240503 23:27:50.890069 7009 consensus_queue.cc:1035] T 7441a981eb93446c9d9367af49462e73 P 21c0231e5eee4c109d088c02f139c112 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3fc1cf80b12b42c2af63d3afec5214a7" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 34273 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 23:27:50.890414 7020 consensus_queue.cc:1035] T 7441a981eb93446c9d9367af49462e73 P 21c0231e5eee4c109d088c02f139c112 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c11e09f93341477483d0045fb1165058" member_type: VOTER last_known_addr { host: "127.0.100.4" port: 33341 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 23:27:50.892851 7005 raft_consensus.cc:2932] T 7441a981eb93446c9d9367af49462e73 P 21c0231e5eee4c109d088c02f139c112 [term 2 LEADER]: Committing config change with OpId 2.5: config changed from index 4 to 5, VOTER c71e941c84ba45deb326c90968f40308 (127.0.100.2) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3fc1cf80b12b42c2af63d3afec5214a7" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 34273 } } peers { permanent_uuid: "21c0231e5eee4c109d088c02f139c112" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 44345 } } peers { permanent_uuid: "c11e09f93341477483d0045fb1165058" member_type: VOTER last_known_addr { host: "127.0.100.4" port: 33341 } attrs { promote: false } } }
I20240503 23:27:50.893479 6708 raft_consensus.cc:2932] T 7441a981eb93446c9d9367af49462e73 P c11e09f93341477483d0045fb1165058 [term 2 FOLLOWER]: Committing config change with OpId 2.5: config changed from index 4 to 5, VOTER c71e941c84ba45deb326c90968f40308 (127.0.100.2) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3fc1cf80b12b42c2af63d3afec5214a7" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 34273 } } peers { permanent_uuid: "21c0231e5eee4c109d088c02f139c112" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 44345 } } peers { permanent_uuid: "c11e09f93341477483d0045fb1165058" member_type: VOTER last_known_addr { host: "127.0.100.4" port: 33341 } attrs { promote: false } } }
I20240503 23:27:50.893492 6804 raft_consensus.cc:2932] T 7441a981eb93446c9d9367af49462e73 P 3fc1cf80b12b42c2af63d3afec5214a7 [term 2 FOLLOWER]: Committing config change with OpId 2.5: config changed from index 4 to 5, VOTER c71e941c84ba45deb326c90968f40308 (127.0.100.2) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3fc1cf80b12b42c2af63d3afec5214a7" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 34273 } } peers { permanent_uuid: "21c0231e5eee4c109d088c02f139c112" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 44345 } } peers { permanent_uuid: "c11e09f93341477483d0045fb1165058" member_type: VOTER last_known_addr { host: "127.0.100.4" port: 33341 } attrs { promote: false } } }
I20240503 23:27:50.896117 6385 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 7441a981eb93446c9d9367af49462e73 with cas_config_opid_index 4: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 23:27:50.897657 6337 catalog_manager.cc:5484] T 7441a981eb93446c9d9367af49462e73 P 21c0231e5eee4c109d088c02f139c112 reported cstate change: config changed from index 4 to 5, VOTER c71e941c84ba45deb326c90968f40308 (127.0.100.2) evicted. New cstate: current_term: 2 leader_uuid: "21c0231e5eee4c109d088c02f139c112" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3fc1cf80b12b42c2af63d3afec5214a7" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 34273 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "21c0231e5eee4c109d088c02f139c112" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 44345 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c11e09f93341477483d0045fb1165058" member_type: VOTER last_known_addr { host: "127.0.100.4" port: 33341 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 23:27:50.907173 6873 tablet_service.cc:1514] Processing DeleteTablet for tablet 7441a981eb93446c9d9367af49462e73 with delete_type TABLET_DATA_TOMBSTONED (TS c71e941c84ba45deb326c90968f40308 not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:53256
I20240503 23:27:50.907994 7052 tablet_replica.cc:330] stopping tablet replica
I20240503 23:27:50.908192 7052 raft_consensus.cc:2229] T 7441a981eb93446c9d9367af49462e73 P c71e941c84ba45deb326c90968f40308 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 23:27:50.908440 7052 raft_consensus.cc:2258] T 7441a981eb93446c9d9367af49462e73 P c71e941c84ba45deb326c90968f40308 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 23:27:50.909401 7052 ts_tablet_manager.cc:1875] T 7441a981eb93446c9d9367af49462e73 P c71e941c84ba45deb326c90968f40308: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 23:27:50.914383 7052 ts_tablet_manager.cc:1888] T 7441a981eb93446c9d9367af49462e73 P c71e941c84ba45deb326c90968f40308: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.4
I20240503 23:27:50.914527 7052 log.cc:1196] T 7441a981eb93446c9d9367af49462e73 P c71e941c84ba45deb326c90968f40308: Deleting WAL directory at /tmp/dist-test-taskZhIiPz/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestHandlingFailedTservers.1714778688451250-400-0/minicluster-data/ts-1-root/wals/7441a981eb93446c9d9367af49462e73
I20240503 23:27:50.915343 6326 catalog_manager.cc:4830] TS c71e941c84ba45deb326c90968f40308 (127.0.100.2:35323): tablet 7441a981eb93446c9d9367af49462e73 (table test-workload [id=493b63a53e0842eda0f947898692001c]) successfully deleted
I20240503 23:27:51.855892 400 tablet_server.cc:178] TabletServer@127.0.100.1:34273 shutting down...
I20240503 23:27:51.868162 400 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240503 23:27:51.868613 400 tablet_replica.cc:330] stopping tablet replica
I20240503 23:27:51.868952 400 raft_consensus.cc:2229] T fb79ea9cea314c4cab4bed81163a8ded P 3fc1cf80b12b42c2af63d3afec5214a7 [term 3 FOLLOWER]: Raft consensus shutting down.
I20240503 23:27:51.869230 400 raft_consensus.cc:2258] T fb79ea9cea314c4cab4bed81163a8ded P 3fc1cf80b12b42c2af63d3afec5214a7 [term 3 FOLLOWER]: Raft consensus is shut down!
I20240503 23:27:51.870244 400 tablet_replica.cc:330] stopping tablet replica
I20240503 23:27:51.870479 400 raft_consensus.cc:2229] T 7441a981eb93446c9d9367af49462e73 P 3fc1cf80b12b42c2af63d3afec5214a7 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 23:27:51.870836 400 raft_consensus.cc:2258] T 7441a981eb93446c9d9367af49462e73 P 3fc1cf80b12b42c2af63d3afec5214a7 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 23:27:51.871874 400 tablet_replica.cc:330] stopping tablet replica
I20240503 23:27:51.872090 400 raft_consensus.cc:2229] T ad1b9cc708db4d289c2e847874bf747b P 3fc1cf80b12b42c2af63d3afec5214a7 [term 3 FOLLOWER]: Raft consensus shutting down.
I20240503 23:27:51.872275 400 raft_consensus.cc:2258] T ad1b9cc708db4d289c2e847874bf747b P 3fc1cf80b12b42c2af63d3afec5214a7 [term 3 FOLLOWER]: Raft consensus is shut down!
I20240503 23:27:51.873049 400 tablet_replica.cc:330] stopping tablet replica
I20240503 23:27:51.873245 400 raft_consensus.cc:2229] T e9273b9533d14394b454e6c31ae4deb5 P 3fc1cf80b12b42c2af63d3afec5214a7 [term 3 FOLLOWER]: Raft consensus shutting down.
I20240503 23:27:51.873414 400 raft_consensus.cc:2258] T e9273b9533d14394b454e6c31ae4deb5 P 3fc1cf80b12b42c2af63d3afec5214a7 [term 3 FOLLOWER]: Raft consensus is shut down!
I20240503 23:27:51.890157 400 tablet_server.cc:194] TabletServer@127.0.100.1:34273 shutdown complete.
I20240503 23:27:51.894881 400 tablet_server.cc:178] TabletServer@127.0.100.2:35323 shutting down...
I20240503 23:27:51.903334 6963 consensus_queue.cc:237] T ad1b9cc708db4d289c2e847874bf747b P 21c0231e5eee4c109d088c02f139c112 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3fc1cf80b12b42c2af63d3afec5214a7" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 34273 } attrs { replace: true } } peers { permanent_uuid: "c71e941c84ba45deb326c90968f40308" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 35323 } } peers { permanent_uuid: "21c0231e5eee4c109d088c02f139c112" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 44345 } } peers { permanent_uuid: "c11e09f93341477483d0045fb1165058" member_type: NON_VOTER last_known_addr { host: "127.0.100.4" port: 33341 } attrs { promote: true } }
I20240503 23:27:51.919914 400 ts_tablet_manager.cc:1470] Shutting down tablet manager...
W20240503 23:27:51.919881 6924 consensus_peers.cc:485] T ad1b9cc708db4d289c2e847874bf747b P 21c0231e5eee4c109d088c02f139c112 -> Peer c11e09f93341477483d0045fb1165058 (127.0.100.4:33341): Couldn't send request to peer c11e09f93341477483d0045fb1165058. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: ad1b9cc708db4d289c2e847874bf747b. This is attempt 1: this message will repeat every 5th retry.
W20240503 23:27:51.920151 6924 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.100.1:34273: connect: Connection refused (error 111) [suppressed 14 similar messages]
I20240503 23:27:51.920594 400 tablet_replica.cc:330] stopping tablet replica
I20240503 23:27:51.920923 400 raft_consensus.cc:2229] T fb79ea9cea314c4cab4bed81163a8ded P c71e941c84ba45deb326c90968f40308 [term 3 FOLLOWER]: Raft consensus shutting down.
I20240503 23:27:51.921195 400 raft_consensus.cc:2258] T fb79ea9cea314c4cab4bed81163a8ded P c71e941c84ba45deb326c90968f40308 [term 3 FOLLOWER]: Raft consensus is shut down!
I20240503 23:27:51.922014 400 tablet_replica.cc:330] stopping tablet replica
W20240503 23:27:51.922125 6924 consensus_peers.cc:485] T ad1b9cc708db4d289c2e847874bf747b P 21c0231e5eee4c109d088c02f139c112 -> Peer 3fc1cf80b12b42c2af63d3afec5214a7 (127.0.100.1:34273): Couldn't send request to peer 3fc1cf80b12b42c2af63d3afec5214a7. Status: Network error: Client connection negotiation failed: client connection to 127.0.100.1:34273: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 23:27:51.922173 400 raft_consensus.cc:2229] T ad1b9cc708db4d289c2e847874bf747b P c71e941c84ba45deb326c90968f40308 [term 3 FOLLOWER]: Raft consensus shutting down.
I20240503 23:27:51.922380 400 raft_consensus.cc:2258] T ad1b9cc708db4d289c2e847874bf747b P c71e941c84ba45deb326c90968f40308 [term 3 FOLLOWER]: Raft consensus is shut down!
W20240503 23:27:51.922376 6924 consensus_peers.cc:485] T ad1b9cc708db4d289c2e847874bf747b P 21c0231e5eee4c109d088c02f139c112 -> Peer c71e941c84ba45deb326c90968f40308 (127.0.100.2:35323): Couldn't send request to peer c71e941c84ba45deb326c90968f40308. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20240503 23:27:51.922959 400 tablet_replica.cc:330] stopping tablet replica
I20240503 23:27:51.923086 400 raft_consensus.cc:2229] T e9273b9533d14394b454e6c31ae4deb5 P c71e941c84ba45deb326c90968f40308 [term 3 FOLLOWER]: Raft consensus shutting down.
I20240503 23:27:51.923238 400 raft_consensus.cc:2258] T e9273b9533d14394b454e6c31ae4deb5 P c71e941c84ba45deb326c90968f40308 [term 3 FOLLOWER]: Raft consensus is shut down!
W20240503 23:27:52.009302 6924 consensus_peers.cc:485] T e9273b9533d14394b454e6c31ae4deb5 P 21c0231e5eee4c109d088c02f139c112 -> Peer c71e941c84ba45deb326c90968f40308 (127.0.100.2:35323): Couldn't send request to peer c71e941c84ba45deb326c90968f40308. Status: Network error: Client connection negotiation failed: client connection to 127.0.100.2:35323: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 23:27:52.036530 6924 consensus_peers.cc:485] T e9273b9533d14394b454e6c31ae4deb5 P 21c0231e5eee4c109d088c02f139c112 -> Peer 3fc1cf80b12b42c2af63d3afec5214a7 (127.0.100.1:34273): Couldn't send request to peer 3fc1cf80b12b42c2af63d3afec5214a7. Status: Network error: Client connection negotiation failed: client connection to 127.0.100.1:34273: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 23:27:52.150974 400 tablet_server.cc:194] TabletServer@127.0.100.2:35323 shutdown complete.
I20240503 23:27:52.155439 400 tablet_server.cc:178] TabletServer@127.0.100.3:44345 shutting down...
I20240503 23:27:52.165673 400 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240503 23:27:52.165988 400 tablet_replica.cc:330] stopping tablet replica
I20240503 23:27:52.166206 400 raft_consensus.cc:2229] T fb79ea9cea314c4cab4bed81163a8ded P 21c0231e5eee4c109d088c02f139c112 [term 3 LEADER]: Raft consensus shutting down.
I20240503 23:27:52.166592 400 raft_consensus.cc:2258] T fb79ea9cea314c4cab4bed81163a8ded P 21c0231e5eee4c109d088c02f139c112 [term 3 FOLLOWER]: Raft consensus is shut down!
I20240503 23:27:52.167315 400 tablet_replica.cc:330] stopping tablet replica
I20240503 23:27:52.167460 400 raft_consensus.cc:2229] T 7441a981eb93446c9d9367af49462e73 P 21c0231e5eee4c109d088c02f139c112 [term 2 LEADER]: Raft consensus shutting down.
I20240503 23:27:52.167737 400 raft_consensus.cc:2258] T 7441a981eb93446c9d9367af49462e73 P 21c0231e5eee4c109d088c02f139c112 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 23:27:52.168282 400 tablet_replica.cc:330] stopping tablet replica
I20240503 23:27:52.168396 400 raft_consensus.cc:2229] T ad1b9cc708db4d289c2e847874bf747b P 21c0231e5eee4c109d088c02f139c112 [term 3 LEADER]: Raft consensus shutting down.
I20240503 23:27:52.168751 400 pending_rounds.cc:62] T ad1b9cc708db4d289c2e847874bf747b P 21c0231e5eee4c109d088c02f139c112: Trying to abort 1 pending ops.
I20240503 23:27:52.168804 400 pending_rounds.cc:68] T ad1b9cc708db4d289c2e847874bf747b P 21c0231e5eee4c109d088c02f139c112: Aborting op as it isn't in flight: id { term: 3 index: 3 } timestamp: 7023734259314163712 op_type: CHANGE_CONFIG_OP change_config_record { tablet_id: "ad1b9cc708db4d289c2e847874bf747b" old_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fc1cf80b12b42c2af63d3afec5214a7" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 34273 } } peers { permanent_uuid: "c71e941c84ba45deb326c90968f40308" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 35323 } } peers { permanent_uuid: "21c0231e5eee4c109d088c02f139c112" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 44345 } } } new_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3fc1cf80b12b42c2af63d3afec5214a7" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 34273 } attrs { replace: true } } peers { permanent_uuid: "c71e941c84ba45deb326c90968f40308" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 35323 } } peers { permanent_uuid: "21c0231e5eee4c109d088c02f139c112" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 44345 } } peers { permanent_uuid: "c11e09f93341477483d0045fb1165058" member_type: NON_VOTER last_known_addr { host: "127.0.100.4" port: 33341 } attrs { promote: true } } } }
I20240503 23:27:52.169116 400 raft_consensus.cc:2894] T ad1b9cc708db4d289c2e847874bf747b P 21c0231e5eee4c109d088c02f139c112 [term 3 LEADER]: Aborting config change with OpId 3.3: Aborted: Op aborted
I20240503 23:27:52.169225 400 raft_consensus.cc:2866] T ad1b9cc708db4d289c2e847874bf747b P 21c0231e5eee4c109d088c02f139c112 [term 3 LEADER]: CHANGE_CONFIG_OP replication failed: Aborted: Op aborted
I20240503 23:27:52.169528 400 raft_consensus.cc:2258] T ad1b9cc708db4d289c2e847874bf747b P 21c0231e5eee4c109d088c02f139c112 [term 3 FOLLOWER]: Raft consensus is shut down!
W20240503 23:27:52.169998 6388 auto_rebalancer.cc:254] failed to send replica move request: Aborted: Op aborted
I20240503 23:27:52.170302 400 tablet_replica.cc:330] stopping tablet replica
I20240503 23:27:52.170436 400 raft_consensus.cc:2229] T e9273b9533d14394b454e6c31ae4deb5 P 21c0231e5eee4c109d088c02f139c112 [term 3 LEADER]: Raft consensus shutting down.
I20240503 23:27:52.170699 400 raft_consensus.cc:2258] T e9273b9533d14394b454e6c31ae4deb5 P 21c0231e5eee4c109d088c02f139c112 [term 3 FOLLOWER]: Raft consensus is shut down!
W20240503 23:27:52.172425 6388 auto_rebalancer.cc:663] Could not move replica: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20240503 23:27:52.172523 6388 auto_rebalancer.cc:264] scheduled replica move failed to complete: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20240503 23:27:52.174288 6925 connection.cc:441] server connection from 127.0.0.1:43076 torn down before Call kudu.consensus.ConsensusService.GetConsensusState from 127.0.0.1:43076 (request call id 1061) could send its response
W20240503 23:27:52.174686 6388 auto_rebalancer.cc:663] Could not move replica: Network error: recv got EOF from 127.0.100.3:44345 (error 108)
W20240503 23:27:52.174779 6388 auto_rebalancer.cc:264] scheduled replica move failed to complete: Network error: recv got EOF from 127.0.100.3:44345 (error 108)
I20240503 23:27:52.186401 400 tablet_server.cc:194] TabletServer@127.0.100.3:44345 shutdown complete.
I20240503 23:27:52.190973 400 tablet_server.cc:178] TabletServer@127.0.100.4:0 shutting down...
I20240503 23:27:52.201773 400 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240503 23:27:52.202109 400 tablet_replica.cc:330] T 7441a981eb93446c9d9367af49462e73 P c11e09f93341477483d0045fb1165058: stopping tablet replica
I20240503 23:27:52.202311 400 raft_consensus.cc:2229] T 7441a981eb93446c9d9367af49462e73 P c11e09f93341477483d0045fb1165058 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 23:27:52.202625 400 raft_consensus.cc:2258] T 7441a981eb93446c9d9367af49462e73 P c11e09f93341477483d0045fb1165058 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 23:27:52.217062 400 tablet_server.cc:194] TabletServer@127.0.100.4:0 shutdown complete.
I20240503 23:27:52.231262 400 master.cc:537] Master@127.0.100.62:44317 shutting down...
I20240503 23:27:52.241073 400 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 94ce7459b82d4eebac81c62d8179eec0 [term 1 LEADER]: Raft consensus shutting down.
I20240503 23:27:52.241420 400 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 94ce7459b82d4eebac81c62d8179eec0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 23:27:52.241564 400 tablet_replica.cc:330] T 00000000000000000000000000000000 P 94ce7459b82d4eebac81c62d8179eec0: stopping tablet replica
I20240503 23:27:52.255911 400 master.cc:552] Master@127.0.100.62:44317 shutdown complete.
[ OK ] AutoRebalancerTest.TestHandlingFailedTservers (11853 ms)
[ RUN ] AutoRebalancerTest.TestDeletedTables
I20240503 23:27:52.273655 400 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.0.100.62:43625
I20240503 23:27:52.274317 400 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 23:27:52.277683 7057 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 23:27:52.278491 7058 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 23:27:52.279368 7060 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 23:27:52.280419 400 server_base.cc:1034] running on GCE node
I20240503 23:27:52.280757 400 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240503 23:27:52.280817 400 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240503 23:27:52.280864 400 hybrid_clock.cc:648] HybridClock initialized: now 1714778872280857 us; error 0 us; skew 500 ppm
I20240503 23:27:52.281059 400 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 23:27:52.282696 400 webserver.cc:460] Webserver started at http://127.0.100.62:41095/ using document root <none> and password file <none>
I20240503 23:27:52.282975 400 fs_manager.cc:359] Metadata directory not provided
I20240503 23:27:52.283041 400 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 23:27:52.283191 400 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 23:27:52.283969 400 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZhIiPz/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestDeletedTables.1714778688451250-400-0/minicluster-data/master-0-root/instance:
uuid: "3e719c8272094dccadee8e7490c155c9"
format_stamp: "Formatted at 2024-05-03 23:27:52 on dist-test-slave-jvbp"
I20240503 23:27:52.287132 400 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.002s
I20240503 23:27:52.289194 7065 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 23:27:52.289597 400 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.001s
I20240503 23:27:52.289716 400 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZhIiPz/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestDeletedTables.1714778688451250-400-0/minicluster-data/master-0-root
uuid: "3e719c8272094dccadee8e7490c155c9"
format_stamp: "Formatted at 2024-05-03 23:27:52 on dist-test-slave-jvbp"
I20240503 23:27:52.289840 400 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZhIiPz/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestDeletedTables.1714778688451250-400-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskZhIiPz/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestDeletedTables.1714778688451250-400-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskZhIiPz/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestDeletedTables.1714778688451250-400-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 23:27:52.297573 400 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 23:27:52.298239 400 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 23:27:52.311728 400 rpc_server.cc:307] RPC server started. Bound to: 127.0.100.62:43625
I20240503 23:27:52.311738 7116 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.100.62:43625 every 8 connection(s)
I20240503 23:27:52.313324 7117 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 23:27:52.318336 7117 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3e719c8272094dccadee8e7490c155c9: Bootstrap starting.
I20240503 23:27:52.319753 7117 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3e719c8272094dccadee8e7490c155c9: Neither blocks nor log segments found. Creating new log.
I20240503 23:27:52.321532 7117 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3e719c8272094dccadee8e7490c155c9: No bootstrap required, opened a new log
I20240503 23:27:52.322254 7117 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3e719c8272094dccadee8e7490c155c9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3e719c8272094dccadee8e7490c155c9" member_type: VOTER }
I20240503 23:27:52.322413 7117 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3e719c8272094dccadee8e7490c155c9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 23:27:52.322450 7117 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3e719c8272094dccadee8e7490c155c9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3e719c8272094dccadee8e7490c155c9, State: Initialized, Role: FOLLOWER
I20240503 23:27:52.322659 7117 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3e719c8272094dccadee8e7490c155c9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3e719c8272094dccadee8e7490c155c9" member_type: VOTER }
I20240503 23:27:52.322780 7117 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3e719c8272094dccadee8e7490c155c9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 23:27:52.322826 7117 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3e719c8272094dccadee8e7490c155c9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 23:27:52.322881 7117 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3e719c8272094dccadee8e7490c155c9 [term 0 FOLLOWER]: Advancing to term 1
I20240503 23:27:52.325007 7117 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3e719c8272094dccadee8e7490c155c9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3e719c8272094dccadee8e7490c155c9" member_type: VOTER }
I20240503 23:27:52.325217 7117 leader_election.cc:302] T 00000000000000000000000000000000 P 3e719c8272094dccadee8e7490c155c9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3e719c8272094dccadee8e7490c155c9; no voters:
I20240503 23:27:52.325668 7117 leader_election.cc:288] T 00000000000000000000000000000000 P 3e719c8272094dccadee8e7490c155c9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240503 23:27:52.325891 7120 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3e719c8272094dccadee8e7490c155c9 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 23:27:52.326417 7120 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3e719c8272094dccadee8e7490c155c9 [term 1 LEADER]: Becoming Leader. State: Replica: 3e719c8272094dccadee8e7490c155c9, State: Running, Role: LEADER
I20240503 23:27:52.326725 7117 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3e719c8272094dccadee8e7490c155c9 [sys.catalog]: configured and running, proceeding with master startup.
I20240503 23:27:52.326681 7120 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3e719c8272094dccadee8e7490c155c9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3e719c8272094dccadee8e7490c155c9" member_type: VOTER }
I20240503 23:27:52.327819 7122 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3e719c8272094dccadee8e7490c155c9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3e719c8272094dccadee8e7490c155c9. Latest consensus state: current_term: 1 leader_uuid: "3e719c8272094dccadee8e7490c155c9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3e719c8272094dccadee8e7490c155c9" member_type: VOTER } }
I20240503 23:27:52.327989 7122 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3e719c8272094dccadee8e7490c155c9 [sys.catalog]: This master's current role is: LEADER
I20240503 23:27:52.327780 7121 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3e719c8272094dccadee8e7490c155c9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3e719c8272094dccadee8e7490c155c9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3e719c8272094dccadee8e7490c155c9" member_type: VOTER } }
I20240503 23:27:52.328260 7121 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3e719c8272094dccadee8e7490c155c9 [sys.catalog]: This master's current role is: LEADER
I20240503 23:27:52.329172 7125 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240503 23:27:52.330924 7125 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240503 23:27:52.332832 400 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20240503 23:27:52.334980 7125 catalog_manager.cc:1347] Generated new cluster ID: f60db596156c48d2816f415781498e49
I20240503 23:27:52.335107 7125 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240503 23:27:52.343816 7125 catalog_manager.cc:1370] Generated new certificate authority record
I20240503 23:27:52.344934 7125 catalog_manager.cc:1504] Loading token signing keys...
I20240503 23:27:52.353727 7125 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 3e719c8272094dccadee8e7490c155c9: Generated new TSK 0
I20240503 23:27:52.354082 7125 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240503 23:27:52.366164 400 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 23:27:52.370299 7138 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 23:27:52.371245 7139 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 23:27:52.372079 7141 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 23:27:52.373065 400 server_base.cc:1034] running on GCE node
I20240503 23:27:52.373399 400 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240503 23:27:52.373464 400 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240503 23:27:52.373492 400 hybrid_clock.cc:648] HybridClock initialized: now 1714778872373488 us; error 0 us; skew 500 ppm
I20240503 23:27:52.373682 400 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 23:27:52.375320 400 webserver.cc:460] Webserver started at http://127.0.100.1:34279/ using document root <none> and password file <none>
I20240503 23:27:52.375608 400 fs_manager.cc:359] Metadata directory not provided
I20240503 23:27:52.375680 400 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 23:27:52.375808 400 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 23:27:52.376631 400 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZhIiPz/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestDeletedTables.1714778688451250-400-0/minicluster-data/ts-0-root/instance:
uuid: "9e3668e37348496ca15ab4da57534a53"
format_stamp: "Formatted at 2024-05-03 23:27:52 on dist-test-slave-jvbp"
I20240503 23:27:52.379621 400 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.003s
I20240503 23:27:52.381616 7146 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 23:27:52.382068 400 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20240503 23:27:52.382174 400 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZhIiPz/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestDeletedTables.1714778688451250-400-0/minicluster-data/ts-0-root
uuid: "9e3668e37348496ca15ab4da57534a53"
format_stamp: "Formatted at 2024-05-03 23:27:52 on dist-test-slave-jvbp"
I20240503 23:27:52.382328 400 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZhIiPz/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestDeletedTables.1714778688451250-400-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskZhIiPz/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestDeletedTables.1714778688451250-400-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskZhIiPz/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestDeletedTables.1714778688451250-400-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 23:27:52.403208 400 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 23:27:52.403991 400 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 23:27:52.404765 400 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 23:27:52.405926 400 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 23:27:52.405995 400 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 23:27:52.406095 400 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 23:27:52.406136 400 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 23:27:52.420660 400 rpc_server.cc:307] RPC server started. Bound to: 127.0.100.1:41837
I20240503 23:27:52.420701 7208 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.100.1:41837 every 8 connection(s)
I20240503 23:27:52.422510 400 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 23:27:52.426723 7213 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 23:27:52.427649 7214 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 23:27:52.428579 7216 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 23:27:52.429452 400 server_base.cc:1034] running on GCE node
I20240503 23:27:52.429716 400 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240503 23:27:52.429805 400 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240503 23:27:52.429857 400 hybrid_clock.cc:648] HybridClock initialized: now 1714778872429852 us; error 0 us; skew 500 ppm
I20240503 23:27:52.430027 400 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 23:27:52.430312 7209 heartbeater.cc:347] Connected to a master server at 127.0.100.62:43625
I20240503 23:27:52.430469 7209 heartbeater.cc:464] Registering TS with master...
I20240503 23:27:52.430994 7209 heartbeater.cc:511] Master 127.0.100.62:43625 requested a full tablet report, sending...
I20240503 23:27:52.431919 400 webserver.cc:460] Webserver started at http://127.0.100.2:45947/ using document root <none> and password file <none>
I20240503 23:27:52.432168 400 fs_manager.cc:359] Metadata directory not provided
I20240503 23:27:52.432188 7082 ts_manager.cc:194] Registered new tserver with Master: 9e3668e37348496ca15ab4da57534a53 (127.0.100.1:41837)
I20240503 23:27:52.432252 400 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 23:27:52.432401 400 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 23:27:52.433115 400 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZhIiPz/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestDeletedTables.1714778688451250-400-0/minicluster-data/ts-1-root/instance:
uuid: "ad2dd25e957d4536bdc22a0df2b7263f"
format_stamp: "Formatted at 2024-05-03 23:27:52 on dist-test-slave-jvbp"
I20240503 23:27:52.433533 7082 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:34330
I20240503 23:27:52.436208 400 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.003s
I20240503 23:27:52.438110 7221 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 23:27:52.438571 400 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20240503 23:27:52.438673 400 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZhIiPz/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestDeletedTables.1714778688451250-400-0/minicluster-data/ts-1-root
uuid: "ad2dd25e957d4536bdc22a0df2b7263f"
format_stamp: "Formatted at 2024-05-03 23:27:52 on dist-test-slave-jvbp"
I20240503 23:27:52.438781 400 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZhIiPz/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestDeletedTables.1714778688451250-400-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskZhIiPz/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestDeletedTables.1714778688451250-400-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskZhIiPz/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestDeletedTables.1714778688451250-400-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 23:27:52.458626 400 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 23:27:52.459205 400 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 23:27:52.459820 400 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 23:27:52.460817 400 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 23:27:52.460884 400 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 23:27:52.460983 400 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 23:27:52.461023 400 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 23:27:52.475211 400 rpc_server.cc:307] RPC server started. Bound to: 127.0.100.2:38457
I20240503 23:27:52.475230 7283 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.100.2:38457 every 8 connection(s)
I20240503 23:27:52.477010 400 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 23:27:52.481127 7287 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 23:27:52.482170 7288 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 23:27:52.483071 7290 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 23:27:52.483901 400 server_base.cc:1034] running on GCE node
I20240503 23:27:52.484212 400 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240503 23:27:52.484298 400 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240503 23:27:52.484351 400 hybrid_clock.cc:648] HybridClock initialized: now 1714778872484345 us; error 0 us; skew 500 ppm
I20240503 23:27:52.484530 400 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 23:27:52.484947 7284 heartbeater.cc:347] Connected to a master server at 127.0.100.62:43625
I20240503 23:27:52.485091 7284 heartbeater.cc:464] Registering TS with master...
I20240503 23:27:52.485615 7284 heartbeater.cc:511] Master 127.0.100.62:43625 requested a full tablet report, sending...
I20240503 23:27:52.486255 400 webserver.cc:460] Webserver started at http://127.0.100.3:45939/ using document root <none> and password file <none>
I20240503 23:27:52.486547 400 fs_manager.cc:359] Metadata directory not provided
I20240503 23:27:52.486617 400 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 23:27:52.486688 7082 ts_manager.cc:194] Registered new tserver with Master: ad2dd25e957d4536bdc22a0df2b7263f (127.0.100.2:38457)
I20240503 23:27:52.486757 400 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 23:27:52.487427 400 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZhIiPz/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestDeletedTables.1714778688451250-400-0/minicluster-data/ts-2-root/instance:
uuid: "63d49bebfcb9422b87754fb258e16a97"
format_stamp: "Formatted at 2024-05-03 23:27:52 on dist-test-slave-jvbp"
I20240503 23:27:52.487550 7082 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:34338
I20240503 23:27:52.490753 400 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20240503 23:27:52.492664 7295 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 23:27:52.493083 400 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.001s
I20240503 23:27:52.493211 400 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZhIiPz/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestDeletedTables.1714778688451250-400-0/minicluster-data/ts-2-root
uuid: "63d49bebfcb9422b87754fb258e16a97"
format_stamp: "Formatted at 2024-05-03 23:27:52 on dist-test-slave-jvbp"
I20240503 23:27:52.493328 400 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZhIiPz/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestDeletedTables.1714778688451250-400-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-taskZhIiPz/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestDeletedTables.1714778688451250-400-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-taskZhIiPz/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestDeletedTables.1714778688451250-400-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 23:27:52.506911 400 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 23:27:52.507596 400 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 23:27:52.508224 400 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 23:27:52.509189 400 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 23:27:52.509250 400 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 23:27:52.509351 400 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 23:27:52.509397 400 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 23:27:52.523380 400 rpc_server.cc:307] RPC server started. Bound to: 127.0.100.3:41385
I20240503 23:27:52.523392 7357 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.100.3:41385 every 8 connection(s)
I20240503 23:27:52.531560 7358 heartbeater.cc:347] Connected to a master server at 127.0.100.62:43625
I20240503 23:27:52.531752 7358 heartbeater.cc:464] Registering TS with master...
I20240503 23:27:52.532143 7358 heartbeater.cc:511] Master 127.0.100.62:43625 requested a full tablet report, sending...
I20240503 23:27:52.533130 7082 ts_manager.cc:194] Registered new tserver with Master: 63d49bebfcb9422b87754fb258e16a97 (127.0.100.3:41385)
I20240503 23:27:52.533551 400 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.009090873s
I20240503 23:27:52.534071 7082 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:34344
I20240503 23:27:53.435273 7209 heartbeater.cc:503] Master 127.0.100.62:43625 was elected leader, sending a full tablet report...
I20240503 23:27:53.489212 7284 heartbeater.cc:503] Master 127.0.100.62:43625 was elected leader, sending a full tablet report...
I20240503 23:27:53.535617 7358 heartbeater.cc:503] Master 127.0.100.62:43625 was elected leader, sending a full tablet report...
I20240503 23:27:53.562999 400 test_util.cc:268] Using random seed: 295733099
I20240503 23:27:53.587802 7082 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34356:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "\004\001\000\377\377\377\037\004\001\000\376\377\377?\004\001\000\375\377\377_""\004\001\000\377\377\377\037\004\001\000\376\377\377?\004\001\000\375\377\377_"
indirect_data: """"
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 23:27:53.588675 7082 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-workload in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 23:27:53.631884 7249 tablet_service.cc:1467] Processing CreateTablet for tablet fe71c54bf82048669579628b317224e7 (DEFAULT_TABLE table=test-workload [id=23b3c6b2809b4e18a2ef0440c24e7d9e]), partition=RANGE (key) PARTITION VALUES < 536870911
I20240503 23:27:53.632230 7320 tablet_service.cc:1467] Processing CreateTablet for tablet 27c38e75002d42688eac2cb1f13c66d4 (DEFAULT_TABLE table=test-workload [id=23b3c6b2809b4e18a2ef0440c24e7d9e]), partition=RANGE (key) PARTITION 1610612733 <= VALUES
I20240503 23:27:53.632659 7249 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fe71c54bf82048669579628b317224e7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 23:27:53.632601 7322 tablet_service.cc:1467] Processing CreateTablet for tablet 28c504576e8147d2ab6418920edf5e2f (DEFAULT_TABLE table=test-workload [id=23b3c6b2809b4e18a2ef0440c24e7d9e]), partition=RANGE (key) PARTITION 536870911 <= VALUES < 1073741822
I20240503 23:27:53.632774 7320 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 27c38e75002d42688eac2cb1f13c66d4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 23:27:53.633054 7322 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 28c504576e8147d2ab6418920edf5e2f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 23:27:53.637118 7247 tablet_service.cc:1467] Processing CreateTablet for tablet 95f3282b21394b11807125a214e8697a (DEFAULT_TABLE table=test-workload [id=23b3c6b2809b4e18a2ef0440c24e7d9e]), partition=RANGE (key) PARTITION 1073741822 <= VALUES < 1610612733
I20240503 23:27:53.637777 7247 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 95f3282b21394b11807125a214e8697a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 23:27:53.639210 7171 tablet_service.cc:1467] Processing CreateTablet for tablet 27c38e75002d42688eac2cb1f13c66d4 (DEFAULT_TABLE table=test-workload [id=23b3c6b2809b4e18a2ef0440c24e7d9e]), partition=RANGE (key) PARTITION 1610612733 <= VALUES
I20240503 23:27:53.632254 7248 tablet_service.cc:1467] Processing CreateTablet for tablet 28c504576e8147d2ab6418920edf5e2f (DEFAULT_TABLE table=test-workload [id=23b3c6b2809b4e18a2ef0440c24e7d9e]), partition=RANGE (key) PARTITION 536870911 <= VALUES < 1073741822
I20240503 23:27:53.639981 7248 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 28c504576e8147d2ab6418920edf5e2f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 23:27:53.641036 7173 tablet_service.cc:1467] Processing CreateTablet for tablet 28c504576e8147d2ab6418920edf5e2f (DEFAULT_TABLE table=test-workload [id=23b3c6b2809b4e18a2ef0440c24e7d9e]), partition=RANGE (key) PARTITION 536870911 <= VALUES < 1073741822
I20240503 23:27:53.642062 7174 tablet_service.cc:1467] Processing CreateTablet for tablet fe71c54bf82048669579628b317224e7 (DEFAULT_TABLE table=test-workload [id=23b3c6b2809b4e18a2ef0440c24e7d9e]), partition=RANGE (key) PARTITION VALUES < 536870911
I20240503 23:27:53.643013 7246 tablet_service.cc:1467] Processing CreateTablet for tablet 27c38e75002d42688eac2cb1f13c66d4 (DEFAULT_TABLE table=test-workload [id=23b3c6b2809b4e18a2ef0440c24e7d9e]), partition=RANGE (key) PARTITION 1610612733 <= VALUES
I20240503 23:27:53.643582 7246 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 27c38e75002d42688eac2cb1f13c66d4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 23:27:53.639987 7171 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 27c38e75002d42688eac2cb1f13c66d4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 23:27:53.645098 7321 tablet_service.cc:1467] Processing CreateTablet for tablet 95f3282b21394b11807125a214e8697a (DEFAULT_TABLE table=test-workload [id=23b3c6b2809b4e18a2ef0440c24e7d9e]), partition=RANGE (key) PARTITION 1073741822 <= VALUES < 1610612733
I20240503 23:27:53.645757 7321 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 95f3282b21394b11807125a214e8697a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 23:27:53.649253 7379 tablet_bootstrap.cc:492] T 95f3282b21394b11807125a214e8697a P ad2dd25e957d4536bdc22a0df2b7263f: Bootstrap starting.
I20240503 23:27:53.631884 7323 tablet_service.cc:1467] Processing CreateTablet for tablet fe71c54bf82048669579628b317224e7 (DEFAULT_TABLE table=test-workload [id=23b3c6b2809b4e18a2ef0440c24e7d9e]), partition=RANGE (key) PARTITION VALUES < 536870911
I20240503 23:27:53.651019 7323 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fe71c54bf82048669579628b317224e7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 23:27:53.652364 7379 tablet_bootstrap.cc:654] T 95f3282b21394b11807125a214e8697a P ad2dd25e957d4536bdc22a0df2b7263f: Neither blocks nor log segments found. Creating new log.
I20240503 23:27:53.652763 7172 tablet_service.cc:1467] Processing CreateTablet for tablet 95f3282b21394b11807125a214e8697a (DEFAULT_TABLE table=test-workload [id=23b3c6b2809b4e18a2ef0440c24e7d9e]), partition=RANGE (key) PARTITION 1073741822 <= VALUES < 1610612733
I20240503 23:27:53.653568 7172 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 95f3282b21394b11807125a214e8697a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 23:27:53.657792 7174 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fe71c54bf82048669579628b317224e7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 23:27:53.663666 7173 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 28c504576e8147d2ab6418920edf5e2f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 23:27:53.673223 7381 tablet_bootstrap.cc:492] T 27c38e75002d42688eac2cb1f13c66d4 P 9e3668e37348496ca15ab4da57534a53: Bootstrap starting.
I20240503 23:27:53.675513 7381 tablet_bootstrap.cc:654] T 27c38e75002d42688eac2cb1f13c66d4 P 9e3668e37348496ca15ab4da57534a53: Neither blocks nor log segments found. Creating new log.
I20240503 23:27:53.691251 7378 tablet_bootstrap.cc:492] T 27c38e75002d42688eac2cb1f13c66d4 P 63d49bebfcb9422b87754fb258e16a97: Bootstrap starting.
I20240503 23:27:53.691675 7379 tablet_bootstrap.cc:492] T 95f3282b21394b11807125a214e8697a P ad2dd25e957d4536bdc22a0df2b7263f: No bootstrap required, opened a new log
I20240503 23:27:53.691845 7379 ts_tablet_manager.cc:1367] T 95f3282b21394b11807125a214e8697a P ad2dd25e957d4536bdc22a0df2b7263f: Time spent bootstrapping tablet: real 0.043s user 0.011s sys 0.011s
I20240503 23:27:53.692793 7379 raft_consensus.cc:348] T 95f3282b21394b11807125a214e8697a P ad2dd25e957d4536bdc22a0df2b7263f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e3668e37348496ca15ab4da57534a53" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 41837 } } peers { permanent_uuid: "ad2dd25e957d4536bdc22a0df2b7263f" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 38457 } } peers { permanent_uuid: "63d49bebfcb9422b87754fb258e16a97" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 41385 } }
I20240503 23:27:53.693109 7379 raft_consensus.cc:374] T 95f3282b21394b11807125a214e8697a P ad2dd25e957d4536bdc22a0df2b7263f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 23:27:53.693243 7379 raft_consensus.cc:729] T 95f3282b21394b11807125a214e8697a P ad2dd25e957d4536bdc22a0df2b7263f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ad2dd25e957d4536bdc22a0df2b7263f, State: Initialized, Role: FOLLOWER
I20240503 23:27:53.693311 7378 tablet_bootstrap.cc:654] T 27c38e75002d42688eac2cb1f13c66d4 P 63d49bebfcb9422b87754fb258e16a97: Neither blocks nor log segments found. Creating new log.
I20240503 23:27:53.693620 7379 consensus_queue.cc:260] T 95f3282b21394b11807125a214e8697a P ad2dd25e957d4536bdc22a0df2b7263f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e3668e37348496ca15ab4da57534a53" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 41837 } } peers { permanent_uuid: "ad2dd25e957d4536bdc22a0df2b7263f" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 38457 } } peers { permanent_uuid: "63d49bebfcb9422b87754fb258e16a97" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 41385 } }
I20240503 23:27:53.694036 7381 tablet_bootstrap.cc:492] T 27c38e75002d42688eac2cb1f13c66d4 P 9e3668e37348496ca15ab4da57534a53: No bootstrap required, opened a new log
I20240503 23:27:53.694253 7381 ts_tablet_manager.cc:1367] T 27c38e75002d42688eac2cb1f13c66d4 P 9e3668e37348496ca15ab4da57534a53: Time spent bootstrapping tablet: real 0.021s user 0.005s sys 0.000s
I20240503 23:27:53.695204 7381 raft_consensus.cc:348] T 27c38e75002d42688eac2cb1f13c66d4 P 9e3668e37348496ca15ab4da57534a53 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e3668e37348496ca15ab4da57534a53" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 41837 } } peers { permanent_uuid: "ad2dd25e957d4536bdc22a0df2b7263f" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 38457 } } peers { permanent_uuid: "63d49bebfcb9422b87754fb258e16a97" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 41385 } }
I20240503 23:27:53.695459 7381 raft_consensus.cc:374] T 27c38e75002d42688eac2cb1f13c66d4 P 9e3668e37348496ca15ab4da57534a53 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 23:27:53.695574 7381 raft_consensus.cc:729] T 27c38e75002d42688eac2cb1f13c66d4 P 9e3668e37348496ca15ab4da57534a53 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9e3668e37348496ca15ab4da57534a53, State: Initialized, Role: FOLLOWER
I20240503 23:27:53.695943 7381 consensus_queue.cc:260] T 27c38e75002d42688eac2cb1f13c66d4 P 9e3668e37348496ca15ab4da57534a53 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e3668e37348496ca15ab4da57534a53" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 41837 } } peers { permanent_uuid: "ad2dd25e957d4536bdc22a0df2b7263f" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 38457 } } peers { permanent_uuid: "63d49bebfcb9422b87754fb258e16a97" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 41385 } }
I20240503 23:27:53.696735 7379 ts_tablet_manager.cc:1398] T 95f3282b21394b11807125a214e8697a P ad2dd25e957d4536bdc22a0df2b7263f: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20240503 23:27:53.697074 7379 tablet_bootstrap.cc:492] T 28c504576e8147d2ab6418920edf5e2f P ad2dd25e957d4536bdc22a0df2b7263f: Bootstrap starting.
I20240503 23:27:53.697182 7378 tablet_bootstrap.cc:492] T 27c38e75002d42688eac2cb1f13c66d4 P 63d49bebfcb9422b87754fb258e16a97: No bootstrap required, opened a new log
I20240503 23:27:53.697393 7378 ts_tablet_manager.cc:1367] T 27c38e75002d42688eac2cb1f13c66d4 P 63d49bebfcb9422b87754fb258e16a97: Time spent bootstrapping tablet: real 0.006s user 0.006s sys 0.000s
I20240503 23:27:53.698298 7378 raft_consensus.cc:348] T 27c38e75002d42688eac2cb1f13c66d4 P 63d49bebfcb9422b87754fb258e16a97 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e3668e37348496ca15ab4da57534a53" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 41837 } } peers { permanent_uuid: "ad2dd25e957d4536bdc22a0df2b7263f" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 38457 } } peers { permanent_uuid: "63d49bebfcb9422b87754fb258e16a97" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 41385 } }
I20240503 23:27:53.698616 7378 raft_consensus.cc:374] T 27c38e75002d42688eac2cb1f13c66d4 P 63d49bebfcb9422b87754fb258e16a97 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 23:27:53.698755 7378 raft_consensus.cc:729] T 27c38e75002d42688eac2cb1f13c66d4 P 63d49bebfcb9422b87754fb258e16a97 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 63d49bebfcb9422b87754fb258e16a97, State: Initialized, Role: FOLLOWER
I20240503 23:27:53.698931 7379 tablet_bootstrap.cc:654] T 28c504576e8147d2ab6418920edf5e2f P ad2dd25e957d4536bdc22a0df2b7263f: Neither blocks nor log segments found. Creating new log.
I20240503 23:27:53.699189 7378 consensus_queue.cc:260] T 27c38e75002d42688eac2cb1f13c66d4 P 63d49bebfcb9422b87754fb258e16a97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e3668e37348496ca15ab4da57534a53" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 41837 } } peers { permanent_uuid: "ad2dd25e957d4536bdc22a0df2b7263f" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 38457 } } peers { permanent_uuid: "63d49bebfcb9422b87754fb258e16a97" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 41385 } }
I20240503 23:27:53.703146 7381 ts_tablet_manager.cc:1398] T 27c38e75002d42688eac2cb1f13c66d4 P 9e3668e37348496ca15ab4da57534a53: Time spent starting tablet: real 0.009s user 0.003s sys 0.005s
I20240503 23:27:53.703907 7381 tablet_bootstrap.cc:492] T fe71c54bf82048669579628b317224e7 P 9e3668e37348496ca15ab4da57534a53: Bootstrap starting.
I20240503 23:27:53.705875 7381 tablet_bootstrap.cc:654] T fe71c54bf82048669579628b317224e7 P 9e3668e37348496ca15ab4da57534a53: Neither blocks nor log segments found. Creating new log.
I20240503 23:27:53.706452 7378 ts_tablet_manager.cc:1398] T 27c38e75002d42688eac2cb1f13c66d4 P 63d49bebfcb9422b87754fb258e16a97: Time spent starting tablet: real 0.009s user 0.004s sys 0.004s
I20240503 23:27:53.707012 7379 tablet_bootstrap.cc:492] T 28c504576e8147d2ab6418920edf5e2f P ad2dd25e957d4536bdc22a0df2b7263f: No bootstrap required, opened a new log
I20240503 23:27:53.707188 7379 ts_tablet_manager.cc:1367] T 28c504576e8147d2ab6418920edf5e2f P ad2dd25e957d4536bdc22a0df2b7263f: Time spent bootstrapping tablet: real 0.010s user 0.002s sys 0.007s
I20240503 23:27:53.707330 7378 tablet_bootstrap.cc:492] T 28c504576e8147d2ab6418920edf5e2f P 63d49bebfcb9422b87754fb258e16a97: Bootstrap starting.
I20240503 23:27:53.708091 7379 raft_consensus.cc:348] T 28c504576e8147d2ab6418920edf5e2f P ad2dd25e957d4536bdc22a0df2b7263f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e3668e37348496ca15ab4da57534a53" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 41837 } } peers { permanent_uuid: "ad2dd25e957d4536bdc22a0df2b7263f" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 38457 } } peers { permanent_uuid: "63d49bebfcb9422b87754fb258e16a97" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 41385 } }
I20240503 23:27:53.708374 7379 raft_consensus.cc:374] T 28c504576e8147d2ab6418920edf5e2f P ad2dd25e957d4536bdc22a0df2b7263f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 23:27:53.708458 7379 raft_consensus.cc:729] T 28c504576e8147d2ab6418920edf5e2f P ad2dd25e957d4536bdc22a0df2b7263f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ad2dd25e957d4536bdc22a0df2b7263f, State: Initialized, Role: FOLLOWER
I20240503 23:27:53.708770 7379 consensus_queue.cc:260] T 28c504576e8147d2ab6418920edf5e2f P ad2dd25e957d4536bdc22a0df2b7263f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e3668e37348496ca15ab4da57534a53" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 41837 } } peers { permanent_uuid: "ad2dd25e957d4536bdc22a0df2b7263f" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 38457 } } peers { permanent_uuid: "63d49bebfcb9422b87754fb258e16a97" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 41385 } }
I20240503 23:27:53.709121 7378 tablet_bootstrap.cc:654] T 28c504576e8147d2ab6418920edf5e2f P 63d49bebfcb9422b87754fb258e16a97: Neither blocks nor log segments found. Creating new log.
I20240503 23:27:53.709448 7379 ts_tablet_manager.cc:1398] T 28c504576e8147d2ab6418920edf5e2f P ad2dd25e957d4536bdc22a0df2b7263f: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 23:27:53.710366 7379 tablet_bootstrap.cc:492] T fe71c54bf82048669579628b317224e7 P ad2dd25e957d4536bdc22a0df2b7263f: Bootstrap starting.
I20240503 23:27:53.712849 7379 tablet_bootstrap.cc:654] T fe71c54bf82048669579628b317224e7 P ad2dd25e957d4536bdc22a0df2b7263f: Neither blocks nor log segments found. Creating new log.
I20240503 23:27:53.714536 7378 tablet_bootstrap.cc:492] T 28c504576e8147d2ab6418920edf5e2f P 63d49bebfcb9422b87754fb258e16a97: No bootstrap required, opened a new log
I20240503 23:27:53.714740 7378 ts_tablet_manager.cc:1367] T 28c504576e8147d2ab6418920edf5e2f P 63d49bebfcb9422b87754fb258e16a97: Time spent bootstrapping tablet: real 0.007s user 0.006s sys 0.000s
I20240503 23:27:53.714983 7379 tablet_bootstrap.cc:492] T fe71c54bf82048669579628b317224e7 P ad2dd25e957d4536bdc22a0df2b7263f: No bootstrap required, opened a new log
I20240503 23:27:53.715139 7379 ts_tablet_manager.cc:1367] T fe71c54bf82048669579628b317224e7 P ad2dd25e957d4536bdc22a0df2b7263f: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.004s
I20240503 23:27:53.715706 7378 raft_consensus.cc:348] T 28c504576e8147d2ab6418920edf5e2f P 63d49bebfcb9422b87754fb258e16a97 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e3668e37348496ca15ab4da57534a53" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 41837 } } peers { permanent_uuid: "ad2dd25e957d4536bdc22a0df2b7263f" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 38457 } } peers { permanent_uuid: "63d49bebfcb9422b87754fb258e16a97" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 41385 } }
I20240503 23:27:53.715818 7379 raft_consensus.cc:348] T fe71c54bf82048669579628b317224e7 P ad2dd25e957d4536bdc22a0df2b7263f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad2dd25e957d4536bdc22a0df2b7263f" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 38457 } } peers { permanent_uuid: "9e3668e37348496ca15ab4da57534a53" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 41837 } } peers { permanent_uuid: "63d49bebfcb9422b87754fb258e16a97" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 41385 } }
I20240503 23:27:53.715979 7378 raft_consensus.cc:374] T 28c504576e8147d2ab6418920edf5e2f P 63d49bebfcb9422b87754fb258e16a97 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 23:27:53.715984 7379 raft_consensus.cc:374] T fe71c54bf82048669579628b317224e7 P ad2dd25e957d4536bdc22a0df2b7263f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 23:27:53.716084 7378 raft_consensus.cc:729] T 28c504576e8147d2ab6418920edf5e2f P 63d49bebfcb9422b87754fb258e16a97 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 63d49bebfcb9422b87754fb258e16a97, State: Initialized, Role: FOLLOWER
I20240503 23:27:53.716120 7379 raft_consensus.cc:729] T fe71c54bf82048669579628b317224e7 P ad2dd25e957d4536bdc22a0df2b7263f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ad2dd25e957d4536bdc22a0df2b7263f, State: Initialized, Role: FOLLOWER
I20240503 23:27:53.716423 7379 consensus_queue.cc:260] T fe71c54bf82048669579628b317224e7 P ad2dd25e957d4536bdc22a0df2b7263f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad2dd25e957d4536bdc22a0df2b7263f" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 38457 } } peers { permanent_uuid: "9e3668e37348496ca15ab4da57534a53" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 41837 } } peers { permanent_uuid: "63d49bebfcb9422b87754fb258e16a97" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 41385 } }
I20240503 23:27:53.716423 7378 consensus_queue.cc:260] T 28c504576e8147d2ab6418920edf5e2f P 63d49bebfcb9422b87754fb258e16a97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e3668e37348496ca15ab4da57534a53" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 41837 } } peers { permanent_uuid: "ad2dd25e957d4536bdc22a0df2b7263f" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 38457 } } peers { permanent_uuid: "63d49bebfcb9422b87754fb258e16a97" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 41385 } }
I20240503 23:27:53.717321 7378 ts_tablet_manager.cc:1398] T 28c504576e8147d2ab6418920edf5e2f P 63d49bebfcb9422b87754fb258e16a97: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20240503 23:27:53.718469 7378 tablet_bootstrap.cc:492] T fe71c54bf82048669579628b317224e7 P 63d49bebfcb9422b87754fb258e16a97: Bootstrap starting.
I20240503 23:27:53.720975 7378 tablet_bootstrap.cc:654] T fe71c54bf82048669579628b317224e7 P 63d49bebfcb9422b87754fb258e16a97: Neither blocks nor log segments found. Creating new log.
I20240503 23:27:53.729856 7378 tablet_bootstrap.cc:492] T fe71c54bf82048669579628b317224e7 P 63d49bebfcb9422b87754fb258e16a97: No bootstrap required, opened a new log
I20240503 23:27:53.730201 7378 ts_tablet_manager.cc:1367] T fe71c54bf82048669579628b317224e7 P 63d49bebfcb9422b87754fb258e16a97: Time spent bootstrapping tablet: real 0.012s user 0.011s sys 0.000s
I20240503 23:27:53.731637 7378 raft_consensus.cc:348] T fe71c54bf82048669579628b317224e7 P 63d49bebfcb9422b87754fb258e16a97 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad2dd25e957d4536bdc22a0df2b7263f" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 38457 } } peers { permanent_uuid: "9e3668e37348496ca15ab4da57534a53" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 41837 } } peers { permanent_uuid: "63d49bebfcb9422b87754fb258e16a97" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 41385 } }
I20240503 23:27:53.731985 7378 raft_consensus.cc:374] T fe71c54bf82048669579628b317224e7 P 63d49bebfcb9422b87754fb258e16a97 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 23:27:53.732088 7378 raft_consensus.cc:729] T fe71c54bf82048669579628b317224e7 P 63d49bebfcb9422b87754fb258e16a97 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 63d49bebfcb9422b87754fb258e16a97, State: Initialized, Role: FOLLOWER
I20240503 23:27:53.732376 7378 consensus_queue.cc:260] T fe71c54bf82048669579628b317224e7 P 63d49bebfcb9422b87754fb258e16a97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad2dd25e957d4536bdc22a0df2b7263f" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 38457 } } peers { permanent_uuid: "9e3668e37348496ca15ab4da57534a53" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 41837 } } peers { permanent_uuid: "63d49bebfcb9422b87754fb258e16a97" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 41385 } }
I20240503 23:27:53.733457 7379 ts_tablet_manager.cc:1398] T fe71c54bf82048669579628b317224e7 P ad2dd25e957d4536bdc22a0df2b7263f: Time spent starting tablet: real 0.018s user 0.003s sys 0.015s
I20240503 23:27:53.733704 7381 tablet_bootstrap.cc:492] T fe71c54bf82048669579628b317224e7 P 9e3668e37348496ca15ab4da57534a53: No bootstrap required, opened a new log
I20240503 23:27:53.733867 7379 tablet_bootstrap.cc:492] T 27c38e75002d42688eac2cb1f13c66d4 P ad2dd25e957d4536bdc22a0df2b7263f: Bootstrap starting.
I20240503 23:27:53.733880 7381 ts_tablet_manager.cc:1367] T fe71c54bf82048669579628b317224e7 P 9e3668e37348496ca15ab4da57534a53: Time spent bootstrapping tablet: real 0.030s user 0.014s sys 0.014s
I20240503 23:27:53.734107 7378 ts_tablet_manager.cc:1398] T fe71c54bf82048669579628b317224e7 P 63d49bebfcb9422b87754fb258e16a97: Time spent starting tablet: real 0.004s user 0.003s sys 0.000s
I20240503 23:27:53.734406 7378 tablet_bootstrap.cc:492] T 95f3282b21394b11807125a214e8697a P 63d49bebfcb9422b87754fb258e16a97: Bootstrap starting.
I20240503 23:27:53.734930 7381 raft_consensus.cc:348] T fe71c54bf82048669579628b317224e7 P 9e3668e37348496ca15ab4da57534a53 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad2dd25e957d4536bdc22a0df2b7263f" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 38457 } } peers { permanent_uuid: "9e3668e37348496ca15ab4da57534a53" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 41837 } } peers { permanent_uuid: "63d49bebfcb9422b87754fb258e16a97" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 41385 } }
I20240503 23:27:53.735200 7381 raft_consensus.cc:374] T fe71c54bf82048669579628b317224e7 P 9e3668e37348496ca15ab4da57534a53 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 23:27:53.735301 7381 raft_consensus.cc:729] T fe71c54bf82048669579628b317224e7 P 9e3668e37348496ca15ab4da57534a53 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9e3668e37348496ca15ab4da57534a53, State: Initialized, Role: FOLLOWER
I20240503 23:27:53.735634 7381 consensus_queue.cc:260] T fe71c54bf82048669579628b317224e7 P 9e3668e37348496ca15ab4da57534a53 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad2dd25e957d4536bdc22a0df2b7263f" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 38457 } } peers { permanent_uuid: "9e3668e37348496ca15ab4da57534a53" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 41837 } } peers { permanent_uuid: "63d49bebfcb9422b87754fb258e16a97" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 41385 } }
I20240503 23:27:53.735826 7379 tablet_bootstrap.cc:654] T 27c38e75002d42688eac2cb1f13c66d4 P ad2dd25e957d4536bdc22a0df2b7263f: Neither blocks nor log segments found. Creating new log.
I20240503 23:27:53.736094 7378 tablet_bootstrap.cc:654] T 95f3282b21394b11807125a214e8697a P 63d49bebfcb9422b87754fb258e16a97: Neither blocks nor log segments found. Creating new log.
I20240503 23:27:53.736918 7381 ts_tablet_manager.cc:1398] T fe71c54bf82048669579628b317224e7 P 9e3668e37348496ca15ab4da57534a53: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20240503 23:27:53.737277 7381 tablet_bootstrap.cc:492] T 28c504576e8147d2ab6418920edf5e2f P 9e3668e37348496ca15ab4da57534a53: Bootstrap starting.
I20240503 23:27:53.740633 7381 tablet_bootstrap.cc:654] T 28c504576e8147d2ab6418920edf5e2f P 9e3668e37348496ca15ab4da57534a53: Neither blocks nor log segments found. Creating new log.
I20240503 23:27:53.745143 7379 tablet_bootstrap.cc:492] T 27c38e75002d42688eac2cb1f13c66d4 P ad2dd25e957d4536bdc22a0df2b7263f: No bootstrap required, opened a new log
I20240503 23:27:53.745368 7378 tablet_bootstrap.cc:492] T 95f3282b21394b11807125a214e8697a P 63d49bebfcb9422b87754fb258e16a97: No bootstrap required, opened a new log
I20240503 23:27:53.745390 7379 ts_tablet_manager.cc:1367] T 27c38e75002d42688eac2cb1f13c66d4 P ad2dd25e957d4536bdc22a0df2b7263f: Time spent bootstrapping tablet: real 0.012s user 0.009s sys 0.001s
I20240503 23:27:53.745576 7378 ts_tablet_manager.cc:1367] T 95f3282b21394b11807125a214e8697a P 63d49bebfcb9422b87754fb258e16a97: Time spent bootstrapping tablet: real 0.011s user 0.003s sys 0.004s
I20240503 23:27:53.746640 7379 raft_consensus.cc:348] T 27c38e75002d42688eac2cb1f13c66d4 P ad2dd25e957d4536bdc22a0df2b7263f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e3668e37348496ca15ab4da57534a53" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 41837 } } peers { permanent_uuid: "ad2dd25e957d4536bdc22a0df2b7263f" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 38457 } } peers { permanent_uuid: "63d49bebfcb9422b87754fb258e16a97" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 41385 } }
I20240503 23:27:53.746925 7379 raft_consensus.cc:374] T 27c38e75002d42688eac2cb1f13c66d4 P ad2dd25e957d4536bdc22a0df2b7263f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 23:27:53.747047 7379 raft_consensus.cc:729] T 27c38e75002d42688eac2cb1f13c66d4 P ad2dd25e957d4536bdc22a0df2b7263f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ad2dd25e957d4536bdc22a0df2b7263f, State: Initialized, Role: FOLLOWER
I20240503 23:27:53.747097 7378 raft_consensus.cc:348] T 95f3282b21394b11807125a214e8697a P 63d49bebfcb9422b87754fb258e16a97 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e3668e37348496ca15ab4da57534a53" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 41837 } } peers { permanent_uuid: "ad2dd25e957d4536bdc22a0df2b7263f" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 38457 } } peers { permanent_uuid: "63d49bebfcb9422b87754fb258e16a97" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 41385 } }
I20240503 23:27:53.747314 7378 raft_consensus.cc:374] T 95f3282b21394b11807125a214e8697a P 63d49bebfcb9422b87754fb258e16a97 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 23:27:53.747413 7378 raft_consensus.cc:729] T 95f3282b21394b11807125a214e8697a P 63d49bebfcb9422b87754fb258e16a97 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 63d49bebfcb9422b87754fb258e16a97, State: Initialized, Role: FOLLOWER
I20240503 23:27:53.747444 7379 consensus_queue.cc:260] T 27c38e75002d42688eac2cb1f13c66d4 P ad2dd25e957d4536bdc22a0df2b7263f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e3668e37348496ca15ab4da57534a53" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 41837 } } peers { permanent_uuid: "ad2dd25e957d4536bdc22a0df2b7263f" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 38457 } } peers { permanent_uuid: "63d49bebfcb9422b87754fb258e16a97" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 41385 } }
I20240503 23:27:53.747709 7381 tablet_bootstrap.cc:492] T 28c504576e8147d2ab6418920edf5e2f P 9e3668e37348496ca15ab4da57534a53: No bootstrap required, opened a new log
I20240503 23:27:53.747927 7381 ts_tablet_manager.cc:1367] T 28c504576e8147d2ab6418920edf5e2f P 9e3668e37348496ca15ab4da57534a53: Time spent bootstrapping tablet: real 0.011s user 0.006s sys 0.002s
I20240503 23:27:53.747783 7378 consensus_queue.cc:260] T 95f3282b21394b11807125a214e8697a P 63d49bebfcb9422b87754fb258e16a97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e3668e37348496ca15ab4da57534a53" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 41837 } } peers { permanent_uuid: "ad2dd25e957d4536bdc22a0df2b7263f" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 38457 } } peers { permanent_uuid: "63d49bebfcb9422b87754fb258e16a97" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 41385 } }
I20240503 23:27:53.748307 7379 ts_tablet_manager.cc:1398] T 27c38e75002d42688eac2cb1f13c66d4 P ad2dd25e957d4536bdc22a0df2b7263f: Time spent starting tablet: real 0.003s user 0.000s sys 0.001s
I20240503 23:27:53.748451 7378 ts_tablet_manager.cc:1398] T 95f3282b21394b11807125a214e8697a P 63d49bebfcb9422b87754fb258e16a97: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240503 23:27:53.748996 7381 raft_consensus.cc:348] T 28c504576e8147d2ab6418920edf5e2f P 9e3668e37348496ca15ab4da57534a53 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e3668e37348496ca15ab4da57534a53" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 41837 } } peers { permanent_uuid: "ad2dd25e957d4536bdc22a0df2b7263f" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 38457 } } peers { permanent_uuid: "63d49bebfcb9422b87754fb258e16a97" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 41385 } }
I20240503 23:27:53.749210 7381 raft_consensus.cc:374] T 28c504576e8147d2ab6418920edf5e2f P 9e3668e37348496ca15ab4da57534a53 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 23:27:53.749305 7381 raft_consensus.cc:729] T 28c504576e8147d2ab6418920edf5e2f P 9e3668e37348496ca15ab4da57534a53 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9e3668e37348496ca15ab4da57534a53, State: Initialized, Role: FOLLOWER
I20240503 23:27:53.749662 7381 consensus_queue.cc:260] T 28c504576e8147d2ab6418920edf5e2f P 9e3668e37348496ca15ab4da57534a53 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e3668e37348496ca15ab4da57534a53" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 41837 } } peers { permanent_uuid: "ad2dd25e957d4536bdc22a0df2b7263f" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 38457 } } peers { permanent_uuid: "63d49bebfcb9422b87754fb258e16a97" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 41385 } }
I20240503 23:27:53.756974 7385 raft_consensus.cc:482] T fe71c54bf82048669579628b317224e7 P 9e3668e37348496ca15ab4da57534a53 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 23:27:53.757300 7385 raft_consensus.cc:504] T fe71c54bf82048669579628b317224e7 P 9e3668e37348496ca15ab4da57534a53 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad2dd25e957d4536bdc22a0df2b7263f" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 38457 } } peers { permanent_uuid: "9e3668e37348496ca15ab4da57534a53" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 41837 } } peers { permanent_uuid: "63d49bebfcb9422b87754fb258e16a97" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 41385 } }
I20240503 23:27:53.758504 7385 leader_election.cc:288] T fe71c54bf82048669579628b317224e7 P 9e3668e37348496ca15ab4da57534a53 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ad2dd25e957d4536bdc22a0df2b7263f (127.0.100.2:38457), 63d49bebfcb9422b87754fb258e16a97 (127.0.100.3:41385)
I20240503 23:27:53.760388 7386 raft_consensus.cc:482] T 27c38e75002d42688eac2cb1f13c66d4 P 63d49bebfcb9422b87754fb258e16a97 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 23:27:53.760596 7386 raft_consensus.cc:504] T 27c38e75002d42688eac2cb1f13c66d4 P 63d49bebfcb9422b87754fb258e16a97 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e3668e37348496ca15ab4da57534a53" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 41837 } } peers { permanent_uuid: "ad2dd25e957d4536bdc22a0df2b7263f" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 38457 } } peers { permanent_uuid: "63d49bebfcb9422b87754fb258e16a97" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 41385 } }
I20240503 23:27:53.761467 7386 leader_election.cc:288] T 27c38e75002d42688eac2cb1f13c66d4 P 63d49bebfcb9422b87754fb258e16a97 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9e3668e37348496ca15ab4da57534a53 (127.0.100.1:41837), ad2dd25e957d4536bdc22a0df2b7263f (127.0.100.2:38457)
I20240503 23:27:53.775324 7381 ts_tablet_manager.cc:1398] T 28c504576e8147d2ab6418920edf5e2f P 9e3668e37348496ca15ab4da57534a53: Time spent starting tablet: real 0.027s user 0.020s sys 0.003s
I20240503 23:27:53.779268 7381 tablet_bootstrap.cc:492] T 95f3282b21394b11807125a214e8697a P 9e3668e37348496ca15ab4da57534a53: Bootstrap starting.
I20240503 23:27:53.781380 7381 tablet_bootstrap.cc:654] T 95f3282b21394b11807125a214e8697a P 9e3668e37348496ca15ab4da57534a53: Neither blocks nor log segments found. Creating new log.
I20240503 23:27:53.797361 7333 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe71c54bf82048669579628b317224e7" candidate_uuid: "9e3668e37348496ca15ab4da57534a53" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63d49bebfcb9422b87754fb258e16a97" is_pre_election: true
I20240503 23:27:53.797781 7333 raft_consensus.cc:2454] T fe71c54bf82048669579628b317224e7 P 63d49bebfcb9422b87754fb258e16a97 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9e3668e37348496ca15ab4da57534a53 in term 0.
I20240503 23:27:53.798391 7147 leader_election.cc:302] T fe71c54bf82048669579628b317224e7 P 9e3668e37348496ca15ab4da57534a53 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 63d49bebfcb9422b87754fb258e16a97, 9e3668e37348496ca15ab4da57534a53; no voters:
I20240503 23:27:53.798837 7385 raft_consensus.cc:2781] T fe71c54bf82048669579628b317224e7 P 9e3668e37348496ca15ab4da57534a53 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 23:27:53.798952 7385 raft_consensus.cc:482] T fe71c54bf82048669579628b317224e7 P 9e3668e37348496ca15ab4da57534a53 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 23:27:53.799049 7385 raft_consensus.cc:3037] T fe71c54bf82048669579628b317224e7 P 9e3668e37348496ca15ab4da57534a53 [term 0 FOLLOWER]: Advancing to term 1
I20240503 23:27:53.802340 7381 tablet_bootstrap.cc:492] T 95f3282b21394b11807125a214e8697a P 9e3668e37348496ca15ab4da57534a53: No bootstrap required, opened a new log
I20240503 23:27:53.802543 7381 ts_tablet_manager.cc:1367] T 95f3282b21394b11807125a214e8697a P 9e3668e37348496ca15ab4da57534a53: Time spent bootstrapping tablet: real 0.023s user 0.007s sys 0.016s
I20240503 23:27:53.803442 7385 raft_consensus.cc:504] T fe71c54bf82048669579628b317224e7 P 9e3668e37348496ca15ab4da57534a53 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad2dd25e957d4536bdc22a0df2b7263f" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 38457 } } peers { permanent_uuid: "9e3668e37348496ca15ab4da57534a53" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 41837 } } peers { permanent_uuid: "63d49bebfcb9422b87754fb258e16a97" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 41385 } }
I20240503 23:27:53.803408 7381 raft_consensus.cc:348] T 95f3282b21394b11807125a214e8697a P 9e3668e37348496ca15ab4da57534a53 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e3668e37348496ca15ab4da57534a53" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 41837 } } peers { permanent_uuid: "ad2dd25e957d4536bdc22a0df2b7263f" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 38457 } } peers { permanent_uuid: "63d49bebfcb9422b87754fb258e16a97" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 41385 } }
I20240503 23:27:53.803709 7381 raft_consensus.cc:374] T 95f3282b21394b11807125a214e8697a P 9e3668e37348496ca15ab4da57534a53 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 23:27:53.803649 7259 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "27c38e75002d42688eac2cb1f13c66d4" candidate_uuid: "63d49bebfcb9422b87754fb258e16a97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ad2dd25e957d4536bdc22a0df2b7263f" is_pre_election: true
I20240503 23:27:53.803766 7381 raft_consensus.cc:729] T 95f3282b21394b11807125a214e8697a P 9e3668e37348496ca15ab4da57534a53 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9e3668e37348496ca15ab4da57534a53, State: Initialized, Role: FOLLOWER
I20240503 23:27:53.803889 7259 raft_consensus.cc:2454] T 27c38e75002d42688eac2cb1f13c66d4 P ad2dd25e957d4536bdc22a0df2b7263f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 63d49bebfcb9422b87754fb258e16a97 in term 0.
I20240503 23:27:53.804021 7381 consensus_queue.cc:260] T 95f3282b21394b11807125a214e8697a P 9e3668e37348496ca15ab4da57534a53 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e3668e37348496ca15ab4da57534a53" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 41837 } } peers { permanent_uuid: "ad2dd25e957d4536bdc22a0df2b7263f" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 38457 } } peers { permanent_uuid: "63d49bebfcb9422b87754fb258e16a97" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 41385 } }
I20240503 23:27:53.804661 7299 leader_election.cc:302] T 27c38e75002d42688eac2cb1f13c66d4 P 63d49bebfcb9422b87754fb258e16a97 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 63d49bebfcb9422b87754fb258e16a97, ad2dd25e957d4536bdc22a0df2b7263f; no voters:
I20240503 23:27:53.804991 7386 raft_consensus.cc:2781] T 27c38e75002d42688eac2cb1f13c66d4 P 63d49bebfcb9422b87754fb258e16a97 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 23:27:53.805094 7386 raft_consensus.cc:482] T 27c38e75002d42688eac2cb1f13c66d4 P 63d49bebfcb9422b87754fb258e16a97 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 23:27:53.805176 7386 raft_consensus.cc:3037] T 27c38e75002d42688eac2cb1f13c66d4 P 63d49bebfcb9422b87754fb258e16a97 [term 0 FOLLOWER]: Advancing to term 1
I20240503 23:27:53.808763 7259 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe71c54bf82048669579628b317224e7" candidate_uuid: "9e3668e37348496ca15ab4da57534a53" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ad2dd25e957d4536bdc22a0df2b7263f" is_pre_election: true
I20240503 23:27:53.809054 7259 raft_consensus.cc:2454] T fe71c54bf82048669579628b317224e7 P ad2dd25e957d4536bdc22a0df2b7263f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9e3668e37348496ca15ab4da57534a53 in term 0.
I20240503 23:27:53.809413 7258 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe71c54bf82048669579628b317224e7" candidate_uuid: "9e3668e37348496ca15ab4da57534a53" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ad2dd25e957d4536bdc22a0df2b7263f"
I20240503 23:27:53.809564 7258 raft_consensus.cc:3037] T fe71c54bf82048669579628b317224e7 P ad2dd25e957d4536bdc22a0df2b7263f [term 0 FOLLOWER]: Advancing to term 1
I20240503 23:27:53.810515 7333 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe71c54bf82048669579628b317224e7" candidate_uuid: "9e3668e37348496ca15ab4da57534a53" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63d49bebfcb9422b87754fb258e16a97"
I20240503 23:27:53.810745 7333 raft_consensus.cc:3037] T fe71c54bf82048669579628b317224e7 P 63d49bebfcb9422b87754fb258e16a97 [term 0 FOLLOWER]: Advancing to term 1
I20240503 23:27:53.814211 7258 raft_consensus.cc:2454] T fe71c54bf82048669579628b317224e7 P ad2dd25e957d4536bdc22a0df2b7263f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9e3668e37348496ca15ab4da57534a53 in term 1.
I20240503 23:27:53.814635 7333 raft_consensus.cc:2454] T fe71c54bf82048669579628b317224e7 P 63d49bebfcb9422b87754fb258e16a97 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9e3668e37348496ca15ab4da57534a53 in term 1.
I20240503 23:27:53.815241 7147 leader_election.cc:302] T fe71c54bf82048669579628b317224e7 P 9e3668e37348496ca15ab4da57534a53 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 63d49bebfcb9422b87754fb258e16a97, 9e3668e37348496ca15ab4da57534a53; no voters:
I20240503 23:27:53.809578 7385 leader_election.cc:288] T fe71c54bf82048669579628b317224e7 P 9e3668e37348496ca15ab4da57534a53 [CANDIDATE]: Term 1 election: Requested vote from peers ad2dd25e957d4536bdc22a0df2b7263f (127.0.100.2:38457), 63d49bebfcb9422b87754fb258e16a97 (127.0.100.3:41385)
I20240503 23:27:53.816120 7385 raft_consensus.cc:482] T 27c38e75002d42688eac2cb1f13c66d4 P 9e3668e37348496ca15ab4da57534a53 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 23:27:53.816346 7385 raft_consensus.cc:504] T 27c38e75002d42688eac2cb1f13c66d4 P 9e3668e37348496ca15ab4da57534a53 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e3668e37348496ca15ab4da57534a53" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 41837 } } peers { permanent_uuid: "ad2dd25e957d4536bdc22a0df2b7263f" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 38457 } } peers { permanent_uuid: "63d49bebfcb9422b87754fb258e16a97" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 41385 } }
I20240503 23:27:53.815927 7386 raft_consensus.cc:504] T 27c38e75002d42688eac2cb1f13c66d4 P 63d49bebfcb9422b87754fb258e16a97 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e3668e37348496ca15ab4da57534a53" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 41837 } } peers { permanent_uuid: "ad2dd25e957d4536bdc22a0df2b7263f" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 38457 } } peers { permanent_uuid: "63d49bebfcb9422b87754fb258e16a97" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 41385 } }
I20240503 23:27:53.818120 7258 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "27c38e75002d42688eac2cb1f13c66d4" candidate_uuid: "9e3668e37348496ca15ab4da57534a53" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ad2dd25e957d4536bdc22a0df2b7263f" is_pre_election: true
I20240503 23:27:53.818571 7258 raft_consensus.cc:2454] T 27c38e75002d42688eac2cb1f13c66d4 P ad2dd25e957d4536bdc22a0df2b7263f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9e3668e37348496ca15ab4da57534a53 in term 0.
I20240503 23:27:53.819146 7150 leader_election.cc:302] T 27c38e75002d42688eac2cb1f13c66d4 P 9e3668e37348496ca15ab4da57534a53 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9e3668e37348496ca15ab4da57534a53, ad2dd25e957d4536bdc22a0df2b7263f; no voters:
I20240503 23:27:53.819429 7386 leader_election.cc:288] T 27c38e75002d42688eac2cb1f13c66d4 P 63d49bebfcb9422b87754fb258e16a97 [CANDIDATE]: Term 1 election: Requested vote from peers 9e3668e37348496ca15ab4da57534a53 (127.0.100.1:41837), ad2dd25e957d4536bdc22a0df2b7263f (127.0.100.2:38457)
I20240503 23:27:53.819948 7385 leader_election.cc:288] T 27c38e75002d42688eac2cb1f13c66d4 P 9e3668e37348496ca15ab4da57534a53 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ad2dd25e957d4536bdc22a0df2b7263f (127.0.100.2:38457), 63d49bebfcb9422b87754fb258e16a97 (127.0.100.3:41385)
I20240503 23:27:53.820178 7385 raft_consensus.cc:2781] T 27c38e75002d42688eac2cb1f13c66d4 P 9e3668e37348496ca15ab4da57534a53 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 23:27:53.820315 7385 raft_consensus.cc:482] T 27c38e75002d42688eac2cb1f13c66d4 P 9e3668e37348496ca15ab4da57534a53 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 23:27:53.820477 7385 raft_consensus.cc:3037] T 27c38e75002d42688eac2cb1f13c66d4 P 9e3668e37348496ca15ab4da57534a53 [term 0 FOLLOWER]: Advancing to term 1
I20240503 23:27:53.820791 7184 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "27c38e75002d42688eac2cb1f13c66d4" candidate_uuid: "63d49bebfcb9422b87754fb258e16a97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9e3668e37348496ca15ab4da57534a53" is_pre_election: true
I20240503 23:27:53.821456 7183 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "27c38e75002d42688eac2cb1f13c66d4" candidate_uuid: "63d49bebfcb9422b87754fb258e16a97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9e3668e37348496ca15ab4da57534a53"
I20240503 23:27:53.821568 7258 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "27c38e75002d42688eac2cb1f13c66d4" candidate_uuid: "63d49bebfcb9422b87754fb258e16a97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ad2dd25e957d4536bdc22a0df2b7263f"
I20240503 23:27:53.821835 7258 raft_consensus.cc:3037] T 27c38e75002d42688eac2cb1f13c66d4 P ad2dd25e957d4536bdc22a0df2b7263f [term 0 FOLLOWER]: Advancing to term 1
I20240503 23:27:53.820178 7402 raft_consensus.cc:2781] T fe71c54bf82048669579628b317224e7 P 9e3668e37348496ca15ab4da57534a53 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 23:27:53.822311 7402 raft_consensus.cc:686] T fe71c54bf82048669579628b317224e7 P 9e3668e37348496ca15ab4da57534a53 [term 1 LEADER]: Becoming Leader. State: Replica: 9e3668e37348496ca15ab4da57534a53, State: Running, Role: LEADER
I20240503 23:27:53.822647 7402 consensus_queue.cc:237] T fe71c54bf82048669579628b317224e7 P 9e3668e37348496ca15ab4da57534a53 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad2dd25e957d4536bdc22a0df2b7263f" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 38457 } } peers { permanent_uuid: "9e3668e37348496ca15ab4da57534a53" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 41837 } } peers { permanent_uuid: "63d49bebfcb9422b87754fb258e16a97" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 41385 } }
I20240503 23:27:53.824810 7385 raft_consensus.cc:504] T 27c38e75002d42688eac2cb1f13c66d4 P 9e3668e37348496ca15ab4da57534a53 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e3668e37348496ca15ab4da57534a53" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 41837 } } peers { permanent_uuid: "ad2dd25e957d4536bdc22a0df2b7263f" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 38457 } } peers { permanent_uuid: "63d49bebfcb9422b87754fb258e16a97" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 41385 } }
I20240503 23:27:53.825350 7381 ts_tablet_manager.cc:1398] T 95f3282b21394b11807125a214e8697a P 9e3668e37348496ca15ab4da57534a53: Time spent starting tablet: real 0.023s user 0.001s sys 0.017s
I20240503 23:27:53.825439 7184 raft_consensus.cc:2379] T 27c38e75002d42688eac2cb1f13c66d4 P 9e3668e37348496ca15ab4da57534a53 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 63d49bebfcb9422b87754fb258e16a97 in current term 1: Already voted for candidate 9e3668e37348496ca15ab4da57534a53 in this term.
I20240503 23:27:53.826694 7385 leader_election.cc:288] T 27c38e75002d42688eac2cb1f13c66d4 P 9e3668e37348496ca15ab4da57534a53 [CANDIDATE]: Term 1 election: Requested vote from peers ad2dd25e957d4536bdc22a0df2b7263f (127.0.100.2:38457), 63d49bebfcb9422b87754fb258e16a97 (127.0.100.3:41385)
I20240503 23:27:53.826891 7259 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "27c38e75002d42688eac2cb1f13c66d4" candidate_uuid: "9e3668e37348496ca15ab4da57534a53" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ad2dd25e957d4536bdc22a0df2b7263f"
I20240503 23:27:53.827764 7258 raft_consensus.cc:2454] T 27c38e75002d42688eac2cb1f13c66d4 P ad2dd25e957d4536bdc22a0df2b7263f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 63d49bebfcb9422b87754fb258e16a97 in term 1.
I20240503 23:27:53.828141 7299 leader_election.cc:302] T 27c38e75002d42688eac2cb1f13c66d4 P 63d49bebfcb9422b87754fb258e16a97 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 63d49bebfcb9422b87754fb258e16a97, ad2dd25e957d4536bdc22a0df2b7263f; no voters: 9e3668e37348496ca15ab4da57534a53
I20240503 23:27:53.828467 7386 raft_consensus.cc:2781] T 27c38e75002d42688eac2cb1f13c66d4 P 63d49bebfcb9422b87754fb258e16a97 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 23:27:53.829089 7386 raft_consensus.cc:686] T 27c38e75002d42688eac2cb1f13c66d4 P 63d49bebfcb9422b87754fb258e16a97 [term 1 LEADER]: Becoming Leader. State: Replica: 63d49bebfcb9422b87754fb258e16a97, State: Running, Role: LEADER
I20240503 23:27:53.829494 7333 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "27c38e75002d42688eac2cb1f13c66d4" candidate_uuid: "9e3668e37348496ca15ab4da57534a53" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63d49bebfcb9422b87754fb258e16a97" is_pre_election: true
I20240503 23:27:53.829416 7386 consensus_queue.cc:237] T 27c38e75002d42688eac2cb1f13c66d4 P 63d49bebfcb9422b87754fb258e16a97 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e3668e37348496ca15ab4da57534a53" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 41837 } } peers { permanent_uuid: "ad2dd25e957d4536bdc22a0df2b7263f" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 38457 } } peers { permanent_uuid: "63d49bebfcb9422b87754fb258e16a97" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 41385 } }
I20240503 23:27:53.829649 7332 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "27c38e75002d42688eac2cb1f13c66d4" candidate_uuid: "9e3668e37348496ca15ab4da57534a53" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63d49bebfcb9422b87754fb258e16a97"
I20240503 23:27:53.829511 7081 catalog_manager.cc:5484] T fe71c54bf82048669579628b317224e7 P 9e3668e37348496ca15ab4da57534a53 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9e3668e37348496ca15ab4da57534a53 (127.0.100.1). New cstate: current_term: 1 leader_uuid: "9e3668e37348496ca15ab4da57534a53" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad2dd25e957d4536bdc22a0df2b7263f" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 38457 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9e3668e37348496ca15ab4da57534a53" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 41837 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "63d49bebfcb9422b87754fb258e16a97" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 41385 } health_report { overall_health: UNKNOWN } } }
I20240503 23:27:53.830394 7147 leader_election.cc:302] T 27c38e75002d42688eac2cb1f13c66d4 P 9e3668e37348496ca15ab4da57534a53 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9e3668e37348496ca15ab4da57534a53; no voters: 63d49bebfcb9422b87754fb258e16a97, ad2dd25e957d4536bdc22a0df2b7263f
I20240503 23:27:53.830704 7404 raft_consensus.cc:2726] T 27c38e75002d42688eac2cb1f13c66d4 P 9e3668e37348496ca15ab4da57534a53 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20240503 23:27:53.832928 7082 catalog_manager.cc:5484] T 27c38e75002d42688eac2cb1f13c66d4 P 63d49bebfcb9422b87754fb258e16a97 reported cstate change: term changed from 0 to 1, leader changed from <none> to 63d49bebfcb9422b87754fb258e16a97 (127.0.100.3). New cstate: current_term: 1 leader_uuid: "63d49bebfcb9422b87754fb258e16a97" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e3668e37348496ca15ab4da57534a53" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 41837 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ad2dd25e957d4536bdc22a0df2b7263f" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 38457 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "63d49bebfcb9422b87754fb258e16a97" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 41385 } health_report { overall_health: HEALTHY } } }
I20240503 23:27:53.894757 7404 raft_consensus.cc:482] T 95f3282b21394b11807125a214e8697a P 9e3668e37348496ca15ab4da57534a53 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 23:27:53.895015 7404 raft_consensus.cc:504] T 95f3282b21394b11807125a214e8697a P 9e3668e37348496ca15ab4da57534a53 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e3668e37348496ca15ab4da57534a53" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 41837 } } peers { permanent_uuid: "ad2dd25e957d4536bdc22a0df2b7263f" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 38457 } } peers { permanent_uuid: "63d49bebfcb9422b87754fb258e16a97" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 41385 } }
I20240503 23:27:53.895812 7404 leader_election.cc:288] T 95f3282b21394b11807125a214e8697a P 9e3668e37348496ca15ab4da57534a53 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ad2dd25e957d4536bdc22a0df2b7263f (127.0.100.2:38457), 63d49bebfcb9422b87754fb258e16a97 (127.0.100.3:41385)
I20240503 23:27:53.896485 7332 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "95f3282b21394b11807125a214e8697a" candidate_uuid: "9e3668e37348496ca15ab4da57534a53" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63d49bebfcb9422b87754fb258e16a97" is_pre_election: true
I20240503 23:27:53.896489 7258 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "95f3282b21394b11807125a214e8697a" candidate_uuid: "9e3668e37348496ca15ab4da57534a53" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ad2dd25e957d4536bdc22a0df2b7263f" is_pre_election: true
I20240503 23:27:53.896797 7258 raft_consensus.cc:2454] T 95f3282b21394b11807125a214e8697a P ad2dd25e957d4536bdc22a0df2b7263f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9e3668e37348496ca15ab4da57534a53 in term 0.
I20240503 23:27:53.896798 7332 raft_consensus.cc:2454] T 95f3282b21394b11807125a214e8697a P 63d49bebfcb9422b87754fb258e16a97 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9e3668e37348496ca15ab4da57534a53 in term 0.
I20240503 23:27:53.897253 7147 leader_election.cc:302] T 95f3282b21394b11807125a214e8697a P 9e3668e37348496ca15ab4da57534a53 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 63d49bebfcb9422b87754fb258e16a97, 9e3668e37348496ca15ab4da57534a53; no voters:
I20240503 23:27:53.897600 7404 raft_consensus.cc:2781] T 95f3282b21394b11807125a214e8697a P 9e3668e37348496ca15ab4da57534a53 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 23:27:53.897706 7404 raft_consensus.cc:482] T 95f3282b21394b11807125a214e8697a P 9e3668e37348496ca15ab4da57534a53 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 23:27:53.897787 7404 raft_consensus.cc:3037] T 95f3282b21394b11807125a214e8697a P 9e3668e37348496ca15ab4da57534a53 [term 0 FOLLOWER]: Advancing to term 1
I20240503 23:27:53.899971 7404 raft_consensus.cc:504] T 95f3282b21394b11807125a214e8697a P 9e3668e37348496ca15ab4da57534a53 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e3668e37348496ca15ab4da57534a53" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 41837 } } peers { permanent_uuid: "ad2dd25e957d4536bdc22a0df2b7263f" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 38457 } } peers { permanent_uuid: "63d49bebfcb9422b87754fb258e16a97" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 41385 } }
I20240503 23:27:53.900715 7404 leader_election.cc:288] T 95f3282b21394b11807125a214e8697a P 9e3668e37348496ca15ab4da57534a53 [CANDIDATE]: Term 1 election: Requested vote from peers ad2dd25e957d4536bdc22a0df2b7263f (127.0.100.2:38457), 63d49bebfcb9422b87754fb258e16a97 (127.0.100.3:41385)
I20240503 23:27:53.901234 7332 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "95f3282b21394b11807125a214e8697a" candidate_uuid: "9e3668e37348496ca15ab4da57534a53" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63d49bebfcb9422b87754fb258e16a97"
I20240503 23:27:53.901237 7258 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "95f3282b21394b11807125a214e8697a" candidate_uuid: "9e3668e37348496ca15ab4da57534a53" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ad2dd25e957d4536bdc22a0df2b7263f"
I20240503 23:27:53.901505 7332 raft_consensus.cc:3037] T 95f3282b21394b11807125a214e8697a P 63d49bebfcb9422b87754fb258e16a97 [term 0 FOLLOWER]: Advancing to term 1
I20240503 23:27:53.901510 7258 raft_consensus.cc:3037] T 95f3282b21394b11807125a214e8697a P ad2dd25e957d4536bdc22a0df2b7263f [term 0 FOLLOWER]: Advancing to term 1
I20240503 23:27:53.903863 7332 raft_consensus.cc:2454] T 95f3282b21394b11807125a214e8697a P 63d49bebfcb9422b87754fb258e16a97 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9e3668e37348496ca15ab4da57534a53 in term 1.
I20240503 23:27:53.904006 7258 raft_consensus.cc:2454] T 95f3282b21394b11807125a214e8697a P ad2dd25e957d4536bdc22a0df2b7263f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9e3668e37348496ca15ab4da57534a53 in term 1.
I20240503 23:27:53.904317 7147 leader_election.cc:302] T 95f3282b21394b11807125a214e8697a P 9e3668e37348496ca15ab4da57534a53 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 63d49bebfcb9422b87754fb258e16a97, 9e3668e37348496ca15ab4da57534a53; no voters:
I20240503 23:27:53.904614 7404 raft_consensus.cc:2781] T 95f3282b21394b11807125a214e8697a P 9e3668e37348496ca15ab4da57534a53 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 23:27:53.904757 7404 raft_consensus.cc:686] T 95f3282b21394b11807125a214e8697a P 9e3668e37348496ca15ab4da57534a53 [term 1 LEADER]: Becoming Leader. State: Replica: 9e3668e37348496ca15ab4da57534a53, State: Running, Role: LEADER
I20240503 23:27:53.905033 7404 consensus_queue.cc:237] T 95f3282b21394b11807125a214e8697a P 9e3668e37348496ca15ab4da57534a53 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e3668e37348496ca15ab4da57534a53" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 41837 } } peers { permanent_uuid: "ad2dd25e957d4536bdc22a0df2b7263f" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 38457 } } peers { permanent_uuid: "63d49bebfcb9422b87754fb258e16a97" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 41385 } }
I20240503 23:27:53.908290 7082 catalog_manager.cc:5484] T 95f3282b21394b11807125a214e8697a P 9e3668e37348496ca15ab4da57534a53 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9e3668e37348496ca15ab4da57534a53 (127.0.100.1). New cstate: current_term: 1 leader_uuid: "9e3668e37348496ca15ab4da57534a53" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e3668e37348496ca15ab4da57534a53" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 41837 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ad2dd25e957d4536bdc22a0df2b7263f" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 38457 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "63d49bebfcb9422b87754fb258e16a97" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 41385 } health_report { overall_health: UNKNOWN } } }
I20240503 23:27:53.987471 7404 raft_consensus.cc:482] T 28c504576e8147d2ab6418920edf5e2f P 9e3668e37348496ca15ab4da57534a53 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 23:27:53.987754 7404 raft_consensus.cc:504] T 28c504576e8147d2ab6418920edf5e2f P 9e3668e37348496ca15ab4da57534a53 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e3668e37348496ca15ab4da57534a53" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 41837 } } peers { permanent_uuid: "ad2dd25e957d4536bdc22a0df2b7263f" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 38457 } } peers { permanent_uuid: "63d49bebfcb9422b87754fb258e16a97" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 41385 } }
I20240503 23:27:53.988627 7404 leader_election.cc:288] T 28c504576e8147d2ab6418920edf5e2f P 9e3668e37348496ca15ab4da57534a53 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ad2dd25e957d4536bdc22a0df2b7263f (127.0.100.2:38457), 63d49bebfcb9422b87754fb258e16a97 (127.0.100.3:41385)
I20240503 23:27:53.989295 7332 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "28c504576e8147d2ab6418920edf5e2f" candidate_uuid: "9e3668e37348496ca15ab4da57534a53" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63d49bebfcb9422b87754fb258e16a97" is_pre_election: true
I20240503 23:27:53.989295 7258 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "28c504576e8147d2ab6418920edf5e2f" candidate_uuid: "9e3668e37348496ca15ab4da57534a53" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ad2dd25e957d4536bdc22a0df2b7263f" is_pre_election: true
I20240503 23:27:53.989607 7258 raft_consensus.cc:2454] T 28c504576e8147d2ab6418920edf5e2f P ad2dd25e957d4536bdc22a0df2b7263f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9e3668e37348496ca15ab4da57534a53 in term 0.
I20240503 23:27:53.989609 7332 raft_consensus.cc:2454] T 28c504576e8147d2ab6418920edf5e2f P 63d49bebfcb9422b87754fb258e16a97 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9e3668e37348496ca15ab4da57534a53 in term 0.
I20240503 23:27:53.990111 7147 leader_election.cc:302] T 28c504576e8147d2ab6418920edf5e2f P 9e3668e37348496ca15ab4da57534a53 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 63d49bebfcb9422b87754fb258e16a97, 9e3668e37348496ca15ab4da57534a53, ad2dd25e957d4536bdc22a0df2b7263f; no voters:
I20240503 23:27:53.990406 7404 raft_consensus.cc:2781] T 28c504576e8147d2ab6418920edf5e2f P 9e3668e37348496ca15ab4da57534a53 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 23:27:53.990489 7404 raft_consensus.cc:482] T 28c504576e8147d2ab6418920edf5e2f P 9e3668e37348496ca15ab4da57534a53 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 23:27:53.990569 7404 raft_consensus.cc:3037] T 28c504576e8147d2ab6418920edf5e2f P 9e3668e37348496ca15ab4da57534a53 [term 0 FOLLOWER]: Advancing to term 1
I20240503 23:27:53.993007 7404 raft_consensus.cc:504] T 28c504576e8147d2ab6418920edf5e2f P 9e3668e37348496ca15ab4da57534a53 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e3668e37348496ca15ab4da57534a53" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 41837 } } peers { permanent_uuid: "ad2dd25e957d4536bdc22a0df2b7263f" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 38457 } } peers { permanent_uuid: "63d49bebfcb9422b87754fb258e16a97" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 41385 } }
I20240503 23:27:53.993767 7404 leader_election.cc:288] T 28c504576e8147d2ab6418920edf5e2f P 9e3668e37348496ca15ab4da57534a53 [CANDIDATE]: Term 1 election: Requested vote from peers ad2dd25e957d4536bdc22a0df2b7263f (127.0.100.2:38457), 63d49bebfcb9422b87754fb258e16a97 (127.0.100.3:41385)
I20240503 23:27:53.994364 7332 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "28c504576e8147d2ab6418920edf5e2f" candidate_uuid: "9e3668e37348496ca15ab4da57534a53" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63d49bebfcb9422b87754fb258e16a97"
I20240503 23:27:53.994364 7258 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "28c504576e8147d2ab6418920edf5e2f" candidate_uuid: "9e3668e37348496ca15ab4da57534a53" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ad2dd25e957d4536bdc22a0df2b7263f"
I20240503 23:27:53.994633 7332 raft_consensus.cc:3037] T 28c504576e8147d2ab6418920edf5e2f P 63d49bebfcb9422b87754fb258e16a97 [term 0 FOLLOWER]: Advancing to term 1
I20240503 23:27:53.994674 7258 raft_consensus.cc:3037] T 28c504576e8147d2ab6418920edf5e2f P ad2dd25e957d4536bdc22a0df2b7263f [term 0 FOLLOWER]: Advancing to term 1
I20240503 23:27:53.997814 7258 raft_consensus.cc:2454] T 28c504576e8147d2ab6418920edf5e2f P ad2dd25e957d4536bdc22a0df2b7263f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9e3668e37348496ca15ab4da57534a53 in term 1.
I20240503 23:27:53.997814 7332 raft_consensus.cc:2454] T 28c504576e8147d2ab6418920edf5e2f P 63d49bebfcb9422b87754fb258e16a97 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9e3668e37348496ca15ab4da57534a53 in term 1.
I20240503 23:27:53.998528 7147 leader_election.cc:302] T 28c504576e8147d2ab6418920edf5e2f P 9e3668e37348496ca15ab4da57534a53 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 63d49bebfcb9422b87754fb258e16a97, 9e3668e37348496ca15ab4da57534a53, ad2dd25e957d4536bdc22a0df2b7263f; no voters:
I20240503 23:27:53.998867 7404 raft_consensus.cc:2781] T 28c504576e8147d2ab6418920edf5e2f P 9e3668e37348496ca15ab4da57534a53 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 23:27:53.999027 7404 raft_consensus.cc:686] T 28c504576e8147d2ab6418920edf5e2f P 9e3668e37348496ca15ab4da57534a53 [term 1 LEADER]: Becoming Leader. State: Replica: 9e3668e37348496ca15ab4da57534a53, State: Running, Role: LEADER
I20240503 23:27:53.999301 7404 consensus_queue.cc:237] T 28c504576e8147d2ab6418920edf5e2f P 9e3668e37348496ca15ab4da57534a53 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e3668e37348496ca15ab4da57534a53" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 41837 } } peers { permanent_uuid: "ad2dd25e957d4536bdc22a0df2b7263f" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 38457 } } peers { permanent_uuid: "63d49bebfcb9422b87754fb258e16a97" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 41385 } }
I20240503 23:27:54.002311 7082 catalog_manager.cc:5484] T 28c504576e8147d2ab6418920edf5e2f P 9e3668e37348496ca15ab4da57534a53 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9e3668e37348496ca15ab4da57534a53 (127.0.100.1). New cstate: current_term: 1 leader_uuid: "9e3668e37348496ca15ab4da57534a53" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e3668e37348496ca15ab4da57534a53" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 41837 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ad2dd25e957d4536bdc22a0df2b7263f" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 38457 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "63d49bebfcb9422b87754fb258e16a97" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 41385 } health_report { overall_health: UNKNOWN } } }
I20240503 23:27:54.031160 400 test_util.cc:268] Using random seed: 296201216
I20240503 23:27:54.045887 7082 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34370:
name: "dugtrio"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "\004\001\000\377\377\377\037\004\001\000\376\377\377?\004\001\000\375\377\377_""\004\001\000\377\377\377\037\004\001\000\376\377\377?\004\001\000\375\377\377_"
indirect_data: """"
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 23:27:54.046806 7082 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table dugtrio in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 23:27:54.057261 7172 tablet_service.cc:1467] Processing CreateTablet for tablet da50d2c38f2b4b1aa91ca1d5b92d32f8 (DEFAULT_TABLE table=dugtrio [id=1e717799f69545008dc79566448dcf9f]), partition=RANGE (key) PARTITION VALUES < 536870911
I20240503 23:27:54.057587 7320 tablet_service.cc:1467] Processing CreateTablet for tablet da50d2c38f2b4b1aa91ca1d5b92d32f8 (DEFAULT_TABLE table=dugtrio [id=1e717799f69545008dc79566448dcf9f]), partition=RANGE (key) PARTITION VALUES < 536870911
I20240503 23:27:54.057653 7173 tablet_service.cc:1467] Processing CreateTablet for tablet 8269340eb54c455c95d80065a480be0f (DEFAULT_TABLE table=dugtrio [id=1e717799f69545008dc79566448dcf9f]), partition=RANGE (key) PARTITION 536870911 <= VALUES < 1073741822
I20240503 23:27:54.057972 7320 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet da50d2c38f2b4b1aa91ca1d5b92d32f8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 23:27:54.057356 7246 tablet_service.cc:1467] Processing CreateTablet for tablet da50d2c38f2b4b1aa91ca1d5b92d32f8 (DEFAULT_TABLE table=dugtrio [id=1e717799f69545008dc79566448dcf9f]), partition=RANGE (key) PARTITION VALUES < 536870911
I20240503 23:27:54.058533 7246 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet da50d2c38f2b4b1aa91ca1d5b92d32f8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 23:27:54.057950 7172 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet da50d2c38f2b4b1aa91ca1d5b92d32f8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 23:27:54.059001 7249 tablet_service.cc:1467] Processing CreateTablet for tablet 8269340eb54c455c95d80065a480be0f (DEFAULT_TABLE table=dugtrio [id=1e717799f69545008dc79566448dcf9f]), partition=RANGE (key) PARTITION 536870911 <= VALUES < 1073741822
I20240503 23:27:54.059358 7249 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8269340eb54c455c95d80065a480be0f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 23:27:54.060695 7173 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8269340eb54c455c95d80065a480be0f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 23:27:54.065701 7381 tablet_bootstrap.cc:492] T da50d2c38f2b4b1aa91ca1d5b92d32f8 P 9e3668e37348496ca15ab4da57534a53: Bootstrap starting.
I20240503 23:27:54.065783 7379 tablet_bootstrap.cc:492] T da50d2c38f2b4b1aa91ca1d5b92d32f8 P ad2dd25e957d4536bdc22a0df2b7263f: Bootstrap starting.
I20240503 23:27:54.067550 7381 tablet_bootstrap.cc:654] T da50d2c38f2b4b1aa91ca1d5b92d32f8 P 9e3668e37348496ca15ab4da57534a53: Neither blocks nor log segments found. Creating new log.
I20240503 23:27:54.067652 7379 tablet_bootstrap.cc:654] T da50d2c38f2b4b1aa91ca1d5b92d32f8 P ad2dd25e957d4536bdc22a0df2b7263f: Neither blocks nor log segments found. Creating new log.
I20240503 23:27:54.073068 7321 tablet_service.cc:1467] Processing CreateTablet for tablet 8269340eb54c455c95d80065a480be0f (DEFAULT_TABLE table=dugtrio [id=1e717799f69545008dc79566448dcf9f]), partition=RANGE (key) PARTITION 536870911 <= VALUES < 1073741822
I20240503 23:27:54.073503 7248 tablet_service.cc:1467] Processing CreateTablet for tablet 6cfc00700ec04e688a83c434db278210 (DEFAULT_TABLE table=dugtrio [id=1e717799f69545008dc79566448dcf9f]), partition=RANGE (key) PARTITION 1610612733 <= VALUES
I20240503 23:27:54.073673 7321 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8269340eb54c455c95d80065a480be0f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 23:27:54.073865 7248 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6cfc00700ec04e688a83c434db278210. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 23:27:54.074139 7246 tablet_service.cc:1467] Processing CreateTablet for tablet 01ffeaa637ee4aa7a172de6b3e87e2e7 (DEFAULT_TABLE table=dugtrio [id=1e717799f69545008dc79566448dcf9f]), partition=RANGE (key) PARTITION 1073741822 <= VALUES < 1610612733
I20240503 23:27:54.074628 7246 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 01ffeaa637ee4aa7a172de6b3e87e2e7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 23:27:54.075304 7173 tablet_service.cc:1467] Processing CreateTablet for tablet 01ffeaa637ee4aa7a172de6b3e87e2e7 (DEFAULT_TABLE table=dugtrio [id=1e717799f69545008dc79566448dcf9f]), partition=RANGE (key) PARTITION 1073741822 <= VALUES < 1610612733
I20240503 23:27:54.075726 7173 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 01ffeaa637ee4aa7a172de6b3e87e2e7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 23:27:54.076941 7323 tablet_service.cc:1467] Processing CreateTablet for tablet 01ffeaa637ee4aa7a172de6b3e87e2e7 (DEFAULT_TABLE table=dugtrio [id=1e717799f69545008dc79566448dcf9f]), partition=RANGE (key) PARTITION 1073741822 <= VALUES < 1610612733
I20240503 23:27:54.077371 7323 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 01ffeaa637ee4aa7a172de6b3e87e2e7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 23:27:54.081455 7322 tablet_service.cc:1467] Processing CreateTablet for tablet 6cfc00700ec04e688a83c434db278210 (DEFAULT_TABLE table=dugtrio [id=1e717799f69545008dc79566448dcf9f]), partition=RANGE (key) PARTITION 1610612733 <= VALUES
I20240503 23:27:54.082170 7322 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6cfc00700ec04e688a83c434db278210. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 23:27:54.084327 7378 tablet_bootstrap.cc:492] T da50d2c38f2b4b1aa91ca1d5b92d32f8 P 63d49bebfcb9422b87754fb258e16a97: Bootstrap starting.
I20240503 23:27:54.085497 7172 tablet_service.cc:1467] Processing CreateTablet for tablet 6cfc00700ec04e688a83c434db278210 (DEFAULT_TABLE table=dugtrio [id=1e717799f69545008dc79566448dcf9f]), partition=RANGE (key) PARTITION 1610612733 <= VALUES
I20240503 23:27:54.087800 7172 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6cfc00700ec04e688a83c434db278210. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 23:27:54.090448 7378 tablet_bootstrap.cc:654] T da50d2c38f2b4b1aa91ca1d5b92d32f8 P 63d49bebfcb9422b87754fb258e16a97: Neither blocks nor log segments found. Creating new log.
I20240503 23:27:54.104028 7381 tablet_bootstrap.cc:492] T da50d2c38f2b4b1aa91ca1d5b92d32f8 P 9e3668e37348496ca15ab4da57534a53: No bootstrap required, opened a new log
I20240503 23:27:54.104326 7381 ts_tablet_manager.cc:1367] T da50d2c38f2b4b1aa91ca1d5b92d32f8 P 9e3668e37348496ca15ab4da57534a53: Time spent bootstrapping tablet: real 0.039s user 0.007s sys 0.007s
I20240503 23:27:54.105448 7381 raft_consensus.cc:348] T da50d2c38f2b4b1aa91ca1d5b92d32f8 P 9e3668e37348496ca15ab4da57534a53 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e3668e37348496ca15ab4da57534a53" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 41837 } } peers { permanent_uuid: "ad2dd25e957d4536bdc22a0df2b7263f" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 38457 } } peers { permanent_uuid: "63d49bebfcb9422b87754fb258e16a97" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 41385 } }
I20240503 23:27:54.105744 7381 raft_consensus.cc:374] T da50d2c38f2b4b1aa91ca1d5b92d32f8 P 9e3668e37348496ca15ab4da57534a53 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 23:27:54.105810 7379 tablet_bootstrap.cc:492] T da50d2c38f2b4b1aa91ca1d5b92d32f8 P ad2dd25e957d4536bdc22a0df2b7263f: No bootstrap required, opened a new log
I20240503 23:27:54.105837 7381 raft_consensus.cc:729] T da50d2c38f2b4b1aa91ca1d5b92d32f8 P 9e3668e37348496ca15ab4da57534a53 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9e3668e37348496ca15ab4da57534a53, State: Initialized, Role: FOLLOWER
I20240503 23:27:54.105950 7379 ts_tablet_manager.cc:1367] T da50d2c38f2b4b1aa91ca1d5b92d32f8 P ad2dd25e957d4536bdc22a0df2b7263f: Time spent bootstrapping tablet: real 0.040s user 0.002s sys 0.012s
I20240503 23:27:54.106148 7381 consensus_queue.cc:260] T da50d2c38f2b4b1aa91ca1d5b92d32f8 P 9e3668e37348496ca15ab4da57534a53 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e3668e37348496ca15ab4da57534a53" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 41837 } } peers { permanent_uuid: "ad2dd25e957d4536bdc22a0df2b7263f" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 38457 } } peers { permanent_uuid: "63d49bebfcb9422b87754fb258e16a97" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 41385 } }
I20240503 23:27:54.106333 7378 tablet_bootstrap.cc:492] T da50d2c38f2b4b1aa91ca1d5b92d32f8 P 63d49bebfcb9422b87754fb258e16a97: No bootstrap required, opened a new log
I20240503 23:27:54.106472 7378 ts_tablet_manager.cc:1367] T da50d2c38f2b4b1aa91ca1d5b92d32f8 P 63d49bebfcb9422b87754fb258e16a97: Time spent bootstrapping tablet: real 0.022s user 0.008s sys 0.000s
I20240503 23:27:54.106758 7381 ts_tablet_manager.cc:1398] T da50d2c38f2b4b1aa91ca1d5b92d32f8 P 9e3668e37348496ca15ab4da57534a53: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
I20240503 23:27:54.107113 7381 tablet_bootstrap.cc:492] T 8269340eb54c455c95d80065a480be0f P 9e3668e37348496ca15ab4da57534a53: Bootstrap starting.
I20240503 23:27:54.107028 7379 raft_consensus.cc:348] T da50d2c38f2b4b1aa91ca1d5b92d32f8 P ad2dd25e957d4536bdc22a0df2b7263f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e3668e37348496ca15ab4da57534a53" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 41837 } } peers { permanent_uuid: "ad2dd25e957d4536bdc22a0df2b7263f" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 38457 } } peers { permanent_uuid: "63d49bebfcb9422b87754fb258e16a97" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 41385 } }
I20240503 23:27:54.107313 7379 raft_consensus.cc:374] T da50d2c38f2b4b1aa91ca1d5b92d32f8 P ad2dd25e957d4536bdc22a0df2b7263f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 23:27:54.107188 7378 raft_consensus.cc:348] T da50d2c38f2b4b1aa91ca1d5b92d32f8 P 63d49bebfcb9422b87754fb258e16a97 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e3668e37348496ca15ab4da57534a53" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 41837 } } peers { permanent_uuid: "ad2dd25e957d4536bdc22a0df2b7263f" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 38457 } } peers { permanent_uuid: "63d49bebfcb9422b87754fb258e16a97" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 41385 } }
I20240503 23:27:54.107368 7379 raft_consensus.cc:729] T da50d2c38f2b4b1aa91ca1d5b92d32f8 P ad2dd25e957d4536bdc22a0df2b7263f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ad2dd25e957d4536bdc22a0df2b7263f, State: Initialized, Role: FOLLOWER
I20240503 23:27:54.107388 7378 raft_consensus.cc:374] T da50d2c38f2b4b1aa91ca1d5b92d32f8 P 63d49bebfcb9422b87754fb258e16a97 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 23:27:54.107445 7378 raft_consensus.cc:729] T da50d2c38f2b4b1aa91ca1d5b92d32f8 P 63d49bebfcb9422b87754fb258e16a97 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 63d49bebfcb9422b87754fb258e16a97, State: Initialized, Role: FOLLOWER
I20240503 23:27:54.107707 7378 consensus_queue.cc:260] T da50d2c38f2b4b1aa91ca1d5b92d32f8 P 63d49bebfcb9422b87754fb258e16a97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e3668e37348496ca15ab4da57534a53" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 41837 } } peers { permanent_uuid: "ad2dd25e957d4536bdc22a0df2b7263f" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 38457 } } peers { permanent_uuid: "63d49bebfcb9422b87754fb258e16a97" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 41385 } }
I20240503 23:27:54.107810 7379 consensus_queue.cc:260] T da50d2c38f2b4b1aa91ca1d5b92d32f8 P ad2dd25e957d4536bdc22a0df2b7263f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e3668e37348496ca15ab4da57534a53" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 41837 } } peers { permanent_uuid: "ad2dd25e957d4536bdc22a0df2b7263f" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 38457 } } peers { permanent_uuid: "63d49bebfcb9422b87754fb258e16a97" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 41385 } }
I20240503 23:27:54.108996 7379 ts_tablet_manager.cc:1398] T da50d2c38f2b4b1aa91ca1d5b92d32f8 P ad2dd25e957d4536bdc22a0df2b7263f: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20240503 23:27:54.109050 7381 tablet_bootstrap.cc:654] T 8269340eb54c455c95d80065a480be0f P 9e3668e37348496ca15ab4da57534a53: Neither blocks nor log segments found. Creating new log.
I20240503 23:27:54.109457 7379 tablet_bootstrap.cc:492] T 8269340eb54c455c95d80065a480be0f P ad2dd25e957d4536bdc22a0df2b7263f: Bootstrap starting.
I20240503 23:27:54.111128 7379 tablet_bootstrap.cc:654] T 8269340eb54c455c95d80065a480be0f P ad2dd25e957d4536bdc22a0df2b7263f: Neither blocks nor log segments found. Creating new log.
I20240503 23:27:54.113417 7378 ts_tablet_manager.cc:1398] T da50d2c38f2b4b1aa91ca1d5b92d32f8 P 63d49bebfcb9422b87754fb258e16a97: Time spent starting tablet: real 0.007s user 0.002s sys 0.000s
I20240503 23:27:54.114197 7379 tablet_bootstrap.cc:492] T 8269340eb54c455c95d80065a480be0f P ad2dd25e957d4536bdc22a0df2b7263f: No bootstrap required, opened a new log
I20240503 23:27:54.114369 7379 ts_tablet_manager.cc:1367] T 8269340eb54c455c95d80065a480be0f P ad2dd25e957d4536bdc22a0df2b7263f: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.002s
I20240503 23:27:54.115321 7378 tablet_bootstrap.cc:492] T 01ffeaa637ee4aa7a172de6b3e87e2e7 P 63d49bebfcb9422b87754fb258e16a97: Bootstrap starting.
I20240503 23:27:54.115208 7379 raft_consensus.cc:348] T 8269340eb54c455c95d80065a480be0f P ad2dd25e957d4536bdc22a0df2b7263f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e3668e37348496ca15ab4da57534a53" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 41837 } } peers { permanent_uuid: "ad2dd25e957d4536bdc22a0df2b7263f" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 38457 } } peers { permanent_uuid: "63d49bebfcb9422b87754fb258e16a97" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 41385 } }
I20240503 23:27:54.115422 7379 raft_consensus.cc:374] T 8269340eb54c455c95d80065a480be0f P ad2dd25e957d4536bdc22a0df2b7263f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 23:27:54.115520 7379 raft_consensus.cc:729] T 8269340eb54c455c95d80065a480be0f P ad2dd25e957d4536bdc22a0df2b7263f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ad2dd25e957d4536bdc22a0df2b7263f, State: Initialized, Role: FOLLOWER
I20240503 23:27:54.115818 7379 consensus_queue.cc:260] T 8269340eb54c455c95d80065a480be0f P ad2dd25e957d4536bdc22a0df2b7263f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e3668e37348496ca15ab4da57534a53" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 41837 } } peers { permanent_uuid: "ad2dd25e957d4536bdc22a0df2b7263f" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 38457 } } peers { permanent_uuid: "63d49bebfcb9422b87754fb258e16a97" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 41385 } }
I20240503 23:27:54.116533 7379 ts_tablet_manager.cc:1398] T 8269340eb54c455c95d80065a480be0f P ad2dd25e957d4536bdc22a0df2b7263f: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240503 23:27:54.116899 7379 tablet_bootstrap.cc:492] T 6cfc00700ec04e688a83c434db278210 P ad2dd25e957d4536bdc22a0df2b7263f: Bootstrap starting.
I20240503 23:27:54.117265 7381 tablet_bootstrap.cc:492] T 8269340eb54c455c95d80065a480be0f P 9e3668e37348496ca15ab4da57534a53: No bootstrap required, opened a new log
I20240503 23:27:54.117435 7381 ts_tablet_manager.cc:1367] T 8269340eb54c455c95d80065a480be0f P 9e3668e37348496ca15ab4da57534a53: Time spent bootstrapping tablet: real 0.010s user 0.004s sys 0.005s
I20240503 23:27:54.118146 7378 tablet_bootstrap.cc:654] T 01ffeaa637ee4aa7a172de6b3e87e2e7 P 63d49bebfcb9422b87754fb258e16a97: Neither blocks nor log segments found. Creating new log.
I20240503 23:27:54.118475 7379 tablet_bootstrap.cc:654] T 6cfc00700ec04e688a83c434db278210 P ad2dd25e957d4536bdc22a0df2b7263f: Neither blocks nor log segments found. Creating new log.
I20240503 23:27:54.118361 7381 raft_consensus.cc:348] T 8269340eb54c455c95d80065a480be0f P 9e3668e37348496ca15ab4da57534a53 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e3668e37348496ca15ab4da57534a53" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 41837 } } peers { permanent_uuid: "ad2dd25e957d4536bdc22a0df2b7263f" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 38457 } } peers { permanent_uuid: "63d49bebfcb9422b87754fb258e16a97" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 41385 } }
I20240503 23:27:54.118608 7381 raft_consensus.cc:374] T 8269340eb54c455c95d80065a480be0f P 9e3668e37348496ca15ab4da57534a53 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 23:27:54.118710 7381 raft_consensus.cc:729] T 8269340eb54c455c95d80065a480be0f P 9e3668e37348496ca15ab4da57534a53 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9e3668e37348496ca15ab4da57534a53, State: Initialized, Role: FOLLOWER
I20240503 23:27:54.119010 7381 consensus_queue.cc:260] T 8269340eb54c455c95d80065a480be0f P 9e3668e37348496ca15ab4da57534a53 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e3668e37348496ca15ab4da57534a53" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 41837 } } peers { permanent_uuid: "ad2dd25e957d4536bdc22a0df2b7263f" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 38457 } } peers { permanent_uuid: "63d49bebfcb9422b87754fb258e16a97" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 41385 } }
I20240503 23:27:54.119850 7381 ts_tablet_manager.cc:1398] T 8269340eb54c455c95d80065a480be0f P 9e3668e37348496ca15ab4da57534a53: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20240503 23:27:54.120215 7381 tablet_bootstrap.cc:492] T 01ffeaa637ee4aa7a172de6b3e87e2e7 P 9e3668e37348496ca15ab4da57534a53: Bootstrap starting.
I20240503 23:27:54.120743 7379 tablet_bootstrap.cc:492] T 6cfc00700ec04e688a83c434db278210 P ad2dd25e957d4536bdc22a0df2b7263f: No bootstrap required, opened a new log
I20240503 23:27:54.120944 7379 ts_tablet_manager.cc:1367] T 6cfc00700ec04e688a83c434db278210 P ad2dd25e957d4536bdc22a0df2b7263f: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20240503 23:27:54.122056 7379 raft_consensus.cc:348] T 6cfc00700ec04e688a83c434db278210 P ad2dd25e957d4536bdc22a0df2b7263f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e3668e37348496ca15ab4da57534a53" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 41837 } } peers { permanent_uuid: "ad2dd25e957d4536bdc22a0df2b7263f" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 38457 } } peers { permanent_uuid: "63d49bebfcb9422b87754fb258e16a97" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 41385 } }
I20240503 23:27:54.122363 7379 raft_consensus.cc:374] T 6cfc00700ec04e688a83c434db278210 P ad2dd25e957d4536bdc22a0df2b7263f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 23:27:54.122484 7379 raft_consensus.cc:729] T 6cfc00700ec04e688a83c434db278210 P ad2dd25e957d4536bdc22a0df2b7263f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ad2dd25e957d4536bdc22a0df2b7263f, State: Initialized, Role: FOLLOWER
I20240503 23:27:54.122869 7381 tablet_bootstrap.cc:654] T 01ffeaa637ee4aa7a172de6b3e87e2e7 P 9e3668e37348496ca15ab4da57534a53: Neither blocks nor log segments found. Creating new log.
I20240503 23:27:54.122835 7379 consensus_queue.cc:260] T 6cfc00700ec04e688a83c434db278210 P ad2dd25e957d4536bdc22a0df2b7263f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e3668e37348496ca15ab4da57534a53" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 41837 } } peers { permanent_uuid: "ad2dd25e957d4536bdc22a0df2b7263f" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 38457 } } peers { permanent_uuid: "63d49bebfcb9422b87754fb258e16a97" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 41385 } }
I20240503 23:27:54.123876 7379 ts_tablet_manager.cc:1398] T 6cfc00700ec04e688a83c434db278210 P ad2dd25e957d4536bdc22a0df2b7263f: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240503 23:27:54.124279 7379 tablet_bootstrap.cc:492] T 01ffeaa637ee4aa7a172de6b3e87e2e7 P ad2dd25e957d4536bdc22a0df2b7263f: Bootstrap starting.
I20240503 23:27:54.125496 7381 tablet_bootstrap.cc:492] T 01ffeaa637ee4aa7a172de6b3e87e2e7 P 9e3668e37348496ca15ab4da57534a53: No bootstrap required, opened a new log
I20240503 23:27:54.125804 7381 ts_tablet_manager.cc:1367] T 01ffeaa637ee4aa7a172de6b3e87e2e7 P 9e3668e37348496ca15ab4da57534a53: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20240503 23:27:54.126649 7379 tablet_bootstrap.cc:654] T 01ffeaa637ee4aa7a172de6b3e87e2e7 P ad2dd25e957d4536bdc22a0df2b7263f: Neither blocks nor log segments found. Creating new log.
I20240503 23:27:54.127261 7381 raft_consensus.cc:348] T 01ffeaa637ee4aa7a172de6b3e87e2e7 P 9e3668e37348496ca15ab4da57534a53 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e3668e37348496ca15ab4da57534a53" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 41837 } } peers { permanent_uuid: "ad2dd25e957d4536bdc22a0df2b7263f" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 38457 } } peers { permanent_uuid: "63d49bebfcb9422b87754fb258e16a97" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 41385 } }
I20240503 23:27:54.127882 7381 raft_consensus.cc:374] T 01ffeaa637ee4aa7a172de6b3e87e2e7 P 9e3668e37348496ca15ab4da57534a53 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 23:27:54.128048 7381 raft_consensus.cc:729] T 01ffeaa637ee4aa7a172de6b3e87e2e7 P 9e3668e37348496ca15ab4da57534a53 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9e3668e37348496ca15ab4da57534a53, State: Initialized, Role: FOLLOWER
I20240503 23:27:54.128526 7381 consensus_queue.cc:260] T 01ffeaa637ee4aa7a172de6b3e87e2e7 P 9e3668e37348496ca15ab4da57534a53 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e3668e37348496ca15ab4da57534a53" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 41837 } } peers { permanent_uuid: "ad2dd25e957d4536bdc22a0df2b7263f" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 38457 } } peers { permanent_uuid: "63d49bebfcb9422b87754fb258e16a97" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 41385 } }
I20240503 23:27:54.129308 7381 ts_tablet_manager.cc:1398] T 01ffeaa637ee4aa7a172de6b3e87e2e7 P 9e3668e37348496ca15ab4da57534a53: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20240503 23:27:54.129362 7379 tablet_bootstrap.cc:492] T 01ffeaa637ee4aa7a172de6b3e87e2e7 P ad2dd25e957d4536bdc22a0df2b7263f: No bootstrap required, opened a new log
I20240503 23:27:54.129362 7378 tablet_bootstrap.cc:492] T 01ffeaa637ee4aa7a172de6b3e87e2e7 P 63d49bebfcb9422b87754fb258e16a97: No bootstrap required, opened a new log
I20240503 23:27:54.129624 7379 ts_tablet_manager.cc:1367] T 01ffeaa637ee4aa7a172de6b3e87e2e7 P ad2dd25e957d4536bdc22a0df2b7263f: Time spent bootstrapping tablet: real 0.005s user 0.005s sys 0.000s
I20240503 23:27:54.129621 7378 ts_tablet_manager.cc:1367] T 01ffeaa637ee4aa7a172de6b3e87e2e7 P 63d49bebfcb9422b87754fb258e16a97: Time spent bootstrapping tablet: real 0.014s user 0.006s sys 0.000s
I20240503 23:27:54.130458 7381 tablet_bootstrap.cc:492] T 6cfc00700ec04e688a83c434db278210 P 9e3668e37348496ca15ab4da57534a53: Bootstrap starting.
I20240503 23:27:54.130645 7378 raft_consensus.cc:348] T 01ffeaa637ee4aa7a172de6b3e87e2e7 P 63d49bebfcb9422b87754fb258e16a97 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e3668e37348496ca15ab4da57534a53" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 41837 } } peers { permanent_uuid: "ad2dd25e957d4536bdc22a0df2b7263f" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 38457 } } peers { permanent_uuid: "63d49bebfcb9422b87754fb258e16a97" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 41385 } }
I20240503 23:27:54.130645 7379 raft_consensus.cc:348] T 01ffeaa637ee4aa7a172de6b3e87e2e7 P ad2dd25e957d4536bdc22a0df2b7263f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e3668e37348496ca15ab4da57534a53" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 41837 } } peers { permanent_uuid: "ad2dd25e957d4536bdc22a0df2b7263f" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 38457 } } peers { permanent_uuid: "63d49bebfcb9422b87754fb258e16a97" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 41385 } }
I20240503 23:27:54.130887 7378 raft_consensus.cc:374] T 01ffeaa637ee4aa7a172de6b3e87e2e7 P 63d49bebfcb9422b87754fb258e16a97 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 23:27:54.130931 7379 raft_consensus.cc:374] T 01ffeaa637ee4aa7a172de6b3e87e2e7 P ad2dd25e957d4536bdc22a0df2b7263f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 23:27:54.130985 7378 raft_consensus.cc:729] T 01ffeaa637ee4aa7a172de6b3e87e2e7 P 63d49bebfcb9422b87754fb258e16a97 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 63d49bebfcb9422b87754fb258e16a97, State: Initialized, Role: FOLLOWER
I20240503 23:27:54.131023 7379 raft_consensus.cc:729] T 01ffeaa637ee4aa7a172de6b3e87e2e7 P ad2dd25e957d4536bdc22a0df2b7263f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ad2dd25e957d4536bdc22a0df2b7263f, State: Initialized, Role: FOLLOWER
I20240503 23:27:54.131289 7379 consensus_queue.cc:260] T 01ffeaa637ee4aa7a172de6b3e87e2e7 P ad2dd25e957d4536bdc22a0df2b7263f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e3668e37348496ca15ab4da57534a53" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 41837 } } peers { permanent_uuid: "ad2dd25e957d4536bdc22a0df2b7263f" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 38457 } } peers { permanent_uuid: "63d49bebfcb9422b87754fb258e16a97" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 41385 } }
I20240503 23:27:54.131983 7379 ts_tablet_manager.cc:1398] T 01ffeaa637ee4aa7a172de6b3e87e2e7 P ad2dd25e957d4536bdc22a0df2b7263f: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240503 23:27:54.132426 7381 tablet_bootstrap.cc:654] T 6cfc00700ec04e688a83c434db278210 P 9e3668e37348496ca15ab4da57534a53: Neither blocks nor log segments found. Creating new log.
I20240503 23:27:54.131289 7378 consensus_queue.cc:260] T 01ffeaa637ee4aa7a172de6b3e87e2e7 P 63d49bebfcb9422b87754fb258e16a97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e3668e37348496ca15ab4da57534a53" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 41837 } } peers { permanent_uuid: "ad2dd25e957d4536bdc22a0df2b7263f" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 38457 } } peers { permanent_uuid: "63d49bebfcb9422b87754fb258e16a97" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 41385 } }
I20240503 23:27:54.132946 7378 ts_tablet_manager.cc:1398] T 01ffeaa637ee4aa7a172de6b3e87e2e7 P 63d49bebfcb9422b87754fb258e16a97: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240503 23:27:54.139037 7378 tablet_bootstrap.cc:492] T 6cfc00700ec04e688a83c434db278210 P 63d49bebfcb9422b87754fb258e16a97: Bootstrap starting.
I20240503 23:27:54.140826 7378 tablet_bootstrap.cc:654] T 6cfc00700ec04e688a83c434db278210 P 63d49bebfcb9422b87754fb258e16a97: Neither blocks nor log segments found. Creating new log.
I20240503 23:27:54.142966 7381 tablet_bootstrap.cc:492] T 6cfc00700ec04e688a83c434db278210 P 9e3668e37348496ca15ab4da57534a53: No bootstrap required, opened a new log
I20240503 23:27:54.142966 7378 tablet_bootstrap.cc:492] T 6cfc00700ec04e688a83c434db278210 P 63d49bebfcb9422b87754fb258e16a97: No bootstrap required, opened a new log
I20240503 23:27:54.143277 7378 ts_tablet_manager.cc:1367] T 6cfc00700ec04e688a83c434db278210 P 63d49bebfcb9422b87754fb258e16a97: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.004s
I20240503 23:27:54.143277 7381 ts_tablet_manager.cc:1367] T 6cfc00700ec04e688a83c434db278210 P 9e3668e37348496ca15ab4da57534a53: Time spent bootstrapping tablet: real 0.013s user 0.007s sys 0.004s
I20240503 23:27:54.144256 7381 raft_consensus.cc:348] T 6cfc00700ec04e688a83c434db278210 P 9e3668e37348496ca15ab4da57534a53 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e3668e37348496ca15ab4da57534a53" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 41837 } } peers { permanent_uuid: "ad2dd25e957d4536bdc22a0df2b7263f" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 38457 } } peers { permanent_uuid: "63d49bebfcb9422b87754fb258e16a97" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 41385 } }
I20240503 23:27:54.144255 7378 raft_consensus.cc:348] T 6cfc00700ec04e688a83c434db278210 P 63d49bebfcb9422b87754fb258e16a97 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e3668e37348496ca15ab4da57534a53" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 41837 } } peers { permanent_uuid: "ad2dd25e957d4536bdc22a0df2b7263f" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 38457 } } peers { permanent_uuid: "63d49bebfcb9422b87754fb258e16a97" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 41385 } }
I20240503 23:27:54.144601 7381 raft_consensus.cc:374] T 6cfc00700ec04e688a83c434db278210 P 9e3668e37348496ca15ab4da57534a53 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 23:27:54.144616 7378 raft_consensus.cc:374] T 6cfc00700ec04e688a83c434db278210 P 63d49bebfcb9422b87754fb258e16a97 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 23:27:54.144739 7378 raft_consensus.cc:729] T 6cfc00700ec04e688a83c434db278210 P 63d49bebfcb9422b87754fb258e16a97 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 63d49bebfcb9422b87754fb258e16a97, State: Initialized, Role: FOLLOWER
I20240503 23:27:54.144737 7381 raft_consensus.cc:729] T 6cfc00700ec04e688a83c434db278210 P 9e3668e37348496ca15ab4da57534a53 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9e3668e37348496ca15ab4da57534a53, State: Initialized, Role: FOLLOWER
I20240503 23:27:54.145067 7378 consensus_queue.cc:260] T 6cfc00700ec04e688a83c434db278210 P 63d49bebfcb9422b87754fb258e16a97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e3668e37348496ca15ab4da57534a53" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 41837 } } peers { permanent_uuid: "ad2dd25e957d4536bdc22a0df2b7263f" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 38457 } } peers { permanent_uuid: "63d49bebfcb9422b87754fb258e16a97" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 41385 } }
I20240503 23:27:54.145071 7381 consensus_queue.cc:260] T 6cfc00700ec04e688a83c434db278210 P 9e3668e37348496ca15ab4da57534a53 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e3668e37348496ca15ab4da57534a53" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 41837 } } peers { permanent_uuid: "ad2dd25e957d4536bdc22a0df2b7263f" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 38457 } } peers { permanent_uuid: "63d49bebfcb9422b87754fb258e16a97" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 41385 } }
I20240503 23:27:54.145709 7378 ts_tablet_manager.cc:1398] T 6cfc00700ec04e688a83c434db278210 P 63d49bebfcb9422b87754fb258e16a97: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240503 23:27:54.145751 7381 ts_tablet_manager.cc:1398] T 6cfc00700ec04e688a83c434db278210 P 9e3668e37348496ca15ab4da57534a53: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240503 23:27:54.146080 7378 tablet_bootstrap.cc:492] T 8269340eb54c455c95d80065a480be0f P 63d49bebfcb9422b87754fb258e16a97: Bootstrap starting.
I20240503 23:27:54.147907 7378 tablet_bootstrap.cc:654] T 8269340eb54c455c95d80065a480be0f P 63d49bebfcb9422b87754fb258e16a97: Neither blocks nor log segments found. Creating new log.
I20240503 23:27:54.150141 7378 tablet_bootstrap.cc:492] T 8269340eb54c455c95d80065a480be0f P 63d49bebfcb9422b87754fb258e16a97: No bootstrap required, opened a new log
I20240503 23:27:54.150350 7378 ts_tablet_manager.cc:1367] T 8269340eb54c455c95d80065a480be0f P 63d49bebfcb9422b87754fb258e16a97: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.001s
I20240503 23:27:54.151352 7378 raft_consensus.cc:348] T 8269340eb54c455c95d80065a480be0f P 63d49bebfcb9422b87754fb258e16a97 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e3668e37348496ca15ab4da57534a53" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 41837 } } peers { permanent_uuid: "ad2dd25e957d4536bdc22a0df2b7263f" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 38457 } } peers { permanent_uuid: "63d49bebfcb9422b87754fb258e16a97" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 41385 } }
I20240503 23:27:54.151650 7378 raft_consensus.cc:374] T 8269340eb54c455c95d80065a480be0f P 63d49bebfcb9422b87754fb258e16a97 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 23:27:54.151755 7378 raft_consensus.cc:729] T 8269340eb54c455c95d80065a480be0f P 63d49bebfcb9422b87754fb258e16a97 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 63d49bebfcb9422b87754fb258e16a97, State: Initialized, Role: FOLLOWER
I20240503 23:27:54.151988 7378 consensus_queue.cc:260] T 8269340eb54c455c95d80065a480be0f P 63d49bebfcb9422b87754fb258e16a97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e3668e37348496ca15ab4da57534a53" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 41837 } } peers { permanent_uuid: "ad2dd25e957d4536bdc22a0df2b7263f" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 38457 } } peers { permanent_uuid: "63d49bebfcb9422b87754fb258e16a97" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 41385 } }
I20240503 23:27:54.152616 7378 ts_tablet_manager.cc:1398] T 8269340eb54c455c95d80065a480be0f P 63d49bebfcb9422b87754fb258e16a97: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20240503 23:27:54.158164 7391 raft_consensus.cc:482] T 6cfc00700ec04e688a83c434db278210 P ad2dd25e957d4536bdc22a0df2b7263f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 23:27:54.158383 7391 raft_consensus.cc:504] T 6cfc00700ec04e688a83c434db278210 P ad2dd25e957d4536bdc22a0df2b7263f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e3668e37348496ca15ab4da57534a53" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 41837 } } peers { permanent_uuid: "ad2dd25e957d4536bdc22a0df2b7263f" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 38457 } } peers { permanent_uuid: "63d49bebfcb9422b87754fb258e16a97" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 41385 } }
I20240503 23:27:54.159229 7391 leader_election.cc:288] T 6cfc00700ec04e688a83c434db278210 P ad2dd25e957d4536bdc22a0df2b7263f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9e3668e37348496ca15ab4da57534a53 (127.0.100.1:41837), 63d49bebfcb9422b87754fb258e16a97 (127.0.100.3:41385)
I20240503 23:27:54.167414 7332 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6cfc00700ec04e688a83c434db278210" candidate_uuid: "ad2dd25e957d4536bdc22a0df2b7263f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63d49bebfcb9422b87754fb258e16a97" is_pre_election: true
I20240503 23:27:54.167433 7184 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6cfc00700ec04e688a83c434db278210" candidate_uuid: "ad2dd25e957d4536bdc22a0df2b7263f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9e3668e37348496ca15ab4da57534a53" is_pre_election: true
I20240503 23:27:54.167753 7184 raft_consensus.cc:2454] T 6cfc00700ec04e688a83c434db278210 P 9e3668e37348496ca15ab4da57534a53 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ad2dd25e957d4536bdc22a0df2b7263f in term 0.
I20240503 23:27:54.167762 7332 raft_consensus.cc:2454] T 6cfc00700ec04e688a83c434db278210 P 63d49bebfcb9422b87754fb258e16a97 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ad2dd25e957d4536bdc22a0df2b7263f in term 0.
I20240503 23:27:54.168262 7224 leader_election.cc:302] T 6cfc00700ec04e688a83c434db278210 P ad2dd25e957d4536bdc22a0df2b7263f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 63d49bebfcb9422b87754fb258e16a97, 9e3668e37348496ca15ab4da57534a53, ad2dd25e957d4536bdc22a0df2b7263f; no voters:
I20240503 23:27:54.168614 7391 raft_consensus.cc:2781] T 6cfc00700ec04e688a83c434db278210 P ad2dd25e957d4536bdc22a0df2b7263f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 23:27:54.168738 7391 raft_consensus.cc:482] T 6cfc00700ec04e688a83c434db278210 P ad2dd25e957d4536bdc22a0df2b7263f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 23:27:54.168829 7391 raft_consensus.cc:3037] T 6cfc00700ec04e688a83c434db278210 P ad2dd25e957d4536bdc22a0df2b7263f [term 0 FOLLOWER]: Advancing to term 1
I20240503 23:27:54.171087 7391 raft_consensus.cc:504] T 6cfc00700ec04e688a83c434db278210 P ad2dd25e957d4536bdc22a0df2b7263f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e3668e37348496ca15ab4da57534a53" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 41837 } } peers { permanent_uuid: "ad2dd25e957d4536bdc22a0df2b7263f" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 38457 } } peers { permanent_uuid: "63d49bebfcb9422b87754fb258e16a97" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 41385 } }
I20240503 23:27:54.171803 7391 leader_election.cc:288] T 6cfc00700ec04e688a83c434db278210 P ad2dd25e957d4536bdc22a0df2b7263f [CANDIDATE]: Term 1 election: Requested vote from peers 9e3668e37348496ca15ab4da57534a53 (127.0.100.1:41837), 63d49bebfcb9422b87754fb258e16a97 (127.0.100.3:41385)
I20240503 23:27:54.172399 7332 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6cfc00700ec04e688a83c434db278210" candidate_uuid: "ad2dd25e957d4536bdc22a0df2b7263f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63d49bebfcb9422b87754fb258e16a97"
I20240503 23:27:54.172395 7184 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6cfc00700ec04e688a83c434db278210" candidate_uuid: "ad2dd25e957d4536bdc22a0df2b7263f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9e3668e37348496ca15ab4da57534a53"
I20240503 23:27:54.172730 7184 raft_consensus.cc:3037] T 6cfc00700ec04e688a83c434db278210 P 9e3668e37348496ca15ab4da57534a53 [term 0 FOLLOWER]: Advancing to term 1
I20240503 23:27:54.172739 7332 raft_consensus.cc:3037] T 6cfc00700ec04e688a83c434db278210 P 63d49bebfcb9422b87754fb258e16a97 [term 0 FOLLOWER]: Advancing to term 1
I20240503 23:27:54.175928 7184 raft_consensus.cc:2454] T 6cfc00700ec04e688a83c434db278210 P 9e3668e37348496ca15ab4da57534a53 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ad2dd25e957d4536bdc22a0df2b7263f in term 1.
I20240503 23:27:54.175928 7332 raft_consensus.cc:2454] T 6cfc00700ec04e688a83c434db278210 P 63d49bebfcb9422b87754fb258e16a97 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ad2dd25e957d4536bdc22a0df2b7263f in term 1.
I20240503 23:27:54.176565 7224 leader_election.cc:302] T 6cfc00700ec04e688a83c434db278210 P ad2dd25e957d4536bdc22a0df2b7263f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 63d49bebfcb9422b87754fb258e16a97, 9e3668e37348496ca15ab4da57534a53, ad2dd25e957d4536bdc22a0df2b7263f; no voters:
I20240503 23:27:54.176935 7391 raft_consensus.cc:2781] T 6cfc00700ec04e688a83c434db278210 P ad2dd25e957d4536bdc22a0df2b7263f [term 1 FOLLOWER]: Leader election won for term 1
I20240503 23:27:54.177089 7391 raft_consensus.cc:686] T 6cfc00700ec04e688a83c434db278210 P ad2dd25e957d4536bdc22a0df2b7263f [term 1 LEADER]: Becoming Leader. State: Replica: ad2dd25e957d4536bdc22a0df2b7263f, State: Running, Role: LEADER
I20240503 23:27:54.177393 7391 consensus_queue.cc:237] T 6cfc00700ec04e688a83c434db278210 P ad2dd25e957d4536bdc22a0df2b7263f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e3668e37348496ca15ab4da57534a53" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 41837 } } peers { permanent_uuid: "ad2dd25e957d4536bdc22a0df2b7263f" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 38457 } } peers { permanent_uuid: "63d49bebfcb9422b87754fb258e16a97" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 41385 } }
I20240503 23:27:54.180430 7082 catalog_manager.cc:5484] T 6cfc00700ec04e688a83c434db278210 P ad2dd25e957d4536bdc22a0df2b7263f reported cstate change: term changed from 0 to 1, leader changed from <none> to ad2dd25e957d4536bdc22a0df2b7263f (127.0.100.2). New cstate: current_term: 1 leader_uuid: "ad2dd25e957d4536bdc22a0df2b7263f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e3668e37348496ca15ab4da57534a53" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 41837 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ad2dd25e957d4536bdc22a0df2b7263f" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 38457 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "63d49bebfcb9422b87754fb258e16a97" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 41385 } health_report { overall_health: UNKNOWN } } }
I20240503 23:27:54.189661 7404 raft_consensus.cc:482] T 8269340eb54c455c95d80065a480be0f P 9e3668e37348496ca15ab4da57534a53 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 23:27:54.190035 7404 raft_consensus.cc:504] T 8269340eb54c455c95d80065a480be0f P 9e3668e37348496ca15ab4da57534a53 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e3668e37348496ca15ab4da57534a53" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 41837 } } peers { permanent_uuid: "ad2dd25e957d4536bdc22a0df2b7263f" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 38457 } } peers { permanent_uuid: "63d49bebfcb9422b87754fb258e16a97" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 41385 } }
I20240503 23:27:54.190971 7404 leader_election.cc:288] T 8269340eb54c455c95d80065a480be0f P 9e3668e37348496ca15ab4da57534a53 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ad2dd25e957d4536bdc22a0df2b7263f (127.0.100.2:38457), 63d49bebfcb9422b87754fb258e16a97 (127.0.100.3:41385)
I20240503 23:27:54.191555 7332 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8269340eb54c455c95d80065a480be0f" candidate_uuid: "9e3668e37348496ca15ab4da57534a53" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63d49bebfcb9422b87754fb258e16a97" is_pre_election: true
I20240503 23:27:54.191772 7332 raft_consensus.cc:2454] T 8269340eb54c455c95d80065a480be0f P 63d49bebfcb9422b87754fb258e16a97 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9e3668e37348496ca15ab4da57534a53 in term 0.
I20240503 23:27:54.191726 7258 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8269340eb54c455c95d80065a480be0f" candidate_uuid: "9e3668e37348496ca15ab4da57534a53" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ad2dd25e957d4536bdc22a0df2b7263f" is_pre_election: true
I20240503 23:27:54.191967 7258 raft_consensus.cc:2454] T 8269340eb54c455c95d80065a480be0f P ad2dd25e957d4536bdc22a0df2b7263f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9e3668e37348496ca15ab4da57534a53 in term 0.
I20240503 23:27:54.192145 7147 leader_election.cc:302] T 8269340eb54c455c95d80065a480be0f P 9e3668e37348496ca15ab4da57534a53 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 63d49bebfcb9422b87754fb258e16a97, 9e3668e37348496ca15ab4da57534a53; no voters:
I20240503 23:27:54.192453 7404 raft_consensus.cc:2781] T 8269340eb54c455c95d80065a480be0f P 9e3668e37348496ca15ab4da57534a53 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 23:27:54.192535 7404 raft_consensus.cc:482] T 8269340eb54c455c95d80065a480be0f P 9e3668e37348496ca15ab4da57534a53 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 23:27:54.192705 7404 raft_consensus.cc:3037] T 8269340eb54c455c95d80065a480be0f P 9e3668e37348496ca15ab4da57534a53 [term 0 FOLLOWER]: Advancing to term 1
I20240503 23:27:54.195163 7404 raft_consensus.cc:504] T 8269340eb54c455c95d80065a480be0f P 9e3668e37348496ca15ab4da57534a53 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e3668e37348496ca15ab4da57534a53" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 41837 } } peers { permanent_uuid: "ad2dd25e957d4536bdc22a0df2b7263f" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 38457 } } peers { permanent_uuid: "63d49bebfcb9422b87754fb258e16a97" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 41385 } }
I20240503 23:27:54.195981 7404 leader_election.cc:288] T 8269340eb54c455c95d80065a480be0f P 9e3668e37348496ca15ab4da57534a53 [CANDIDATE]: Term 1 election: Requested vote from peers ad2dd25e957d4536bdc22a0df2b7263f (127.0.100.2:38457), 63d49bebfcb9422b87754fb258e16a97 (127.0.100.3:41385)
I20240503 23:27:54.196619 7332 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8269340eb54c455c95d80065a480be0f" candidate_uuid: "9e3668e37348496ca15ab4da57534a53" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63d49bebfcb9422b87754fb258e16a97"
I20240503 23:27:54.196614 7258 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8269340eb54c455c95d80065a480be0f" candidate_uuid: "9e3668e37348496ca15ab4da57534a53" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ad2dd25e957d4536bdc22a0df2b7263f"
I20240503 23:27:54.196923 7332 raft_consensus.cc:3037] T 8269340eb54c455c95d80065a480be0f P 63d49bebfcb9422b87754fb258e16a97 [term 0 FOLLOWER]: Advancing to term 1
I20240503 23:27:54.196923 7258 raft_consensus.cc:3037] T 8269340eb54c455c95d80065a480be0f P ad2dd25e957d4536bdc22a0df2b7263f [term 0 FOLLOWER]: Advancing to term 1
I20240503 23:27:54.199412 7332 raft_consensus.cc:2454] T 8269340eb54c455c95d80065a480be0f P 63d49bebfcb9422b87754fb258e16a97 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9e3668e37348496ca15ab4da57534a53 in term 1.
I20240503 23:27:54.199434 7258 raft_consensus.cc:2454] T 8269340eb54c455c95d80065a480be0f P ad2dd25e957d4536bdc22a0df2b7263f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9e3668e37348496ca15ab4da57534a53 in term 1.
I20240503 23:27:54.199836 7150 leader_election.cc:302] T 8269340eb54c455c95d80065a480be0f P 9e3668e37348496ca15ab4da57534a53 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9e3668e37348496ca15ab4da57534a53, ad2dd25e957d4536bdc22a0df2b7263f; no voters:
I20240503 23:27:54.200174 7404 raft_consensus.cc:2781] T 8269340eb54c455c95d80065a480be0f P 9e3668e37348496ca15ab4da57534a53 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 23:27:54.200364 7404 raft_consensus.cc:686] T 8269340eb54c455c95d80065a480be0f P 9e3668e37348496ca15ab4da57534a53 [term 1 LEADER]: Becoming Leader. State: Replica: 9e3668e37348496ca15ab4da57534a53, State: Running, Role: LEADER
I20240503 23:27:54.200726 7404 consensus_queue.cc:237] T 8269340eb54c455c95d80065a480be0f P 9e3668e37348496ca15ab4da57534a53 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e3668e37348496ca15ab4da57534a53" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 41837 } } peers { permanent_uuid: "ad2dd25e957d4536bdc22a0df2b7263f" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 38457 } } peers { permanent_uuid: "63d49bebfcb9422b87754fb258e16a97" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 41385 } }
I20240503 23:27:54.203768 7082 catalog_manager.cc:5484] T 8269340eb54c455c95d80065a480be0f P 9e3668e37348496ca15ab4da57534a53 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9e3668e37348496ca15ab4da57534a53 (127.0.100.1). New cstate: current_term: 1 leader_uuid: "9e3668e37348496ca15ab4da57534a53" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e3668e37348496ca15ab4da57534a53" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 41837 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ad2dd25e957d4536bdc22a0df2b7263f" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 38457 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "63d49bebfcb9422b87754fb258e16a97" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 41385 } health_report { overall_health: UNKNOWN } } }
I20240503 23:27:54.277164 7404 raft_consensus.cc:482] T 01ffeaa637ee4aa7a172de6b3e87e2e7 P 9e3668e37348496ca15ab4da57534a53 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 23:27:54.277467 7404 raft_consensus.cc:504] T 01ffeaa637ee4aa7a172de6b3e87e2e7 P 9e3668e37348496ca15ab4da57534a53 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e3668e37348496ca15ab4da57534a53" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 41837 } } peers { permanent_uuid: "ad2dd25e957d4536bdc22a0df2b7263f" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 38457 } } peers { permanent_uuid: "63d49bebfcb9422b87754fb258e16a97" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 41385 } }
I20240503 23:27:54.278389 7404 leader_election.cc:288] T 01ffeaa637ee4aa7a172de6b3e87e2e7 P 9e3668e37348496ca15ab4da57534a53 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ad2dd25e957d4536bdc22a0df2b7263f (127.0.100.2:38457), 63d49bebfcb9422b87754fb258e16a97 (127.0.100.3:41385)
I20240503 23:27:54.278986 7258 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01ffeaa637ee4aa7a172de6b3e87e2e7" candidate_uuid: "9e3668e37348496ca15ab4da57534a53" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ad2dd25e957d4536bdc22a0df2b7263f" is_pre_election: true
I20240503 23:27:54.278983 7332 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01ffeaa637ee4aa7a172de6b3e87e2e7" candidate_uuid: "9e3668e37348496ca15ab4da57534a53" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63d49bebfcb9422b87754fb258e16a97" is_pre_election: true
I20240503 23:27:54.279353 7258 raft_consensus.cc:2454] T 01ffeaa637ee4aa7a172de6b3e87e2e7 P ad2dd25e957d4536bdc22a0df2b7263f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9e3668e37348496ca15ab4da57534a53 in term 0.
I20240503 23:27:54.279402 7332 raft_consensus.cc:2454] T 01ffeaa637ee4aa7a172de6b3e87e2e7 P 63d49bebfcb9422b87754fb258e16a97 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9e3668e37348496ca15ab4da57534a53 in term 0.
I20240503 23:27:54.279966 7147 leader_election.cc:302] T 01ffeaa637ee4aa7a172de6b3e87e2e7 P 9e3668e37348496ca15ab4da57534a53 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 63d49bebfcb9422b87754fb258e16a97, 9e3668e37348496ca15ab4da57534a53; no voters:
I20240503 23:27:54.280303 7404 raft_consensus.cc:2781] T 01ffeaa637ee4aa7a172de6b3e87e2e7 P 9e3668e37348496ca15ab4da57534a53 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 23:27:54.280411 7404 raft_consensus.cc:482] T 01ffeaa637ee4aa7a172de6b3e87e2e7 P 9e3668e37348496ca15ab4da57534a53 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 23:27:54.280504 7404 raft_consensus.cc:3037] T 01ffeaa637ee4aa7a172de6b3e87e2e7 P 9e3668e37348496ca15ab4da57534a53 [term 0 FOLLOWER]: Advancing to term 1
I20240503 23:27:54.283035 7404 raft_consensus.cc:504] T 01ffeaa637ee4aa7a172de6b3e87e2e7 P 9e3668e37348496ca15ab4da57534a53 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e3668e37348496ca15ab4da57534a53" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 41837 } } peers { permanent_uuid: "ad2dd25e957d4536bdc22a0df2b7263f" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 38457 } } peers { permanent_uuid: "63d49bebfcb9422b87754fb258e16a97" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 41385 } }
I20240503 23:27:54.283797 7404 leader_election.cc:288] T 01ffeaa637ee4aa7a172de6b3e87e2e7 P 9e3668e37348496ca15ab4da57534a53 [CANDIDATE]: Term 1 election: Requested vote from peers ad2dd25e957d4536bdc22a0df2b7263f (127.0.100.2:38457), 63d49bebfcb9422b87754fb258e16a97 (127.0.100.3:41385)
I20240503 23:27:54.284395 7258 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01ffeaa637ee4aa7a172de6b3e87e2e7" candidate_uuid: "9e3668e37348496ca15ab4da57534a53" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ad2dd25e957d4536bdc22a0df2b7263f"
I20240503 23:27:54.284402 7332 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01ffeaa637ee4aa7a172de6b3e87e2e7" candidate_uuid: "9e3668e37348496ca15ab4da57534a53" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63d49bebfcb9422b87754fb258e16a97"
I20240503 23:27:54.284693 7258 raft_consensus.cc:3037] T 01ffeaa637ee4aa7a172de6b3e87e2e7 P ad2dd25e957d4536bdc22a0df2b7263f [term 0 FOLLOWER]: Advancing to term 1
I20240503 23:27:54.284750 7332 raft_consensus.cc:3037] T 01ffeaa637ee4aa7a172de6b3e87e2e7 P 63d49bebfcb9422b87754fb258e16a97 [term 0 FOLLOWER]: Advancing to term 1
I20240503 23:27:54.287104 7332 raft_consensus.cc:2454] T 01ffeaa637ee4aa7a172de6b3e87e2e7 P 63d49bebfcb9422b87754fb258e16a97 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9e3668e37348496ca15ab4da57534a53 in term 1.
I20240503 23:27:54.287104 7258 raft_consensus.cc:2454] T 01ffeaa637ee4aa7a172de6b3e87e2e7 P ad2dd25e957d4536bdc22a0df2b7263f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9e3668e37348496ca15ab4da57534a53 in term 1.
I20240503 23:27:54.287798 7147 leader_election.cc:302] T 01ffeaa637ee4aa7a172de6b3e87e2e7 P 9e3668e37348496ca15ab4da57534a53 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 63d49bebfcb9422b87754fb258e16a97, 9e3668e37348496ca15ab4da57534a53; no voters:
I20240503 23:27:54.288182 7404 raft_consensus.cc:2781] T 01ffeaa637ee4aa7a172de6b3e87e2e7 P 9e3668e37348496ca15ab4da57534a53 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 23:27:54.288352 7404 raft_consensus.cc:686] T 01ffeaa637ee4aa7a172de6b3e87e2e7 P 9e3668e37348496ca15ab4da57534a53 [term 1 LEADER]: Becoming Leader. State: Replica: 9e3668e37348496ca15ab4da57534a53, State: Running, Role: LEADER
I20240503 23:27:54.288753 7404 consensus_queue.cc:237] T 01ffeaa637ee4aa7a172de6b3e87e2e7 P 9e3668e37348496ca15ab4da57534a53 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e3668e37348496ca15ab4da57534a53" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 41837 } } peers { permanent_uuid: "ad2dd25e957d4536bdc22a0df2b7263f" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 38457 } } peers { permanent_uuid: "63d49bebfcb9422b87754fb258e16a97" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 41385 } }
I20240503 23:27:54.291833 7082 catalog_manager.cc:5484] T 01ffeaa637ee4aa7a172de6b3e87e2e7 P 9e3668e37348496ca15ab4da57534a53 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9e3668e37348496ca15ab4da57534a53 (127.0.100.1). New cstate: current_term: 1 leader_uuid: "9e3668e37348496ca15ab4da57534a53" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e3668e37348496ca15ab4da57534a53" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 41837 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ad2dd25e957d4536bdc22a0df2b7263f" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 38457 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "63d49bebfcb9422b87754fb258e16a97" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 41385 } health_report { overall_health: UNKNOWN } } }
I20240503 23:27:54.322813 7404 consensus_queue.cc:1035] T fe71c54bf82048669579628b317224e7 P 9e3668e37348496ca15ab4da57534a53 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ad2dd25e957d4536bdc22a0df2b7263f" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 38457 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 23:27:54.328362 7404 consensus_queue.cc:1035] T fe71c54bf82048669579628b317224e7 P 9e3668e37348496ca15ab4da57534a53 [LEADER]: Connected to new peer: Peer: permanent_uuid: "63d49bebfcb9422b87754fb258e16a97" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 41385 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240503 23:27:54.331478 7130 auto_rebalancer.cc:227] Could not retrieve cluster info: Service unavailable: Tablet not running
I20240503 23:27:54.333366 7386 consensus_queue.cc:1035] T 27c38e75002d42688eac2cb1f13c66d4 P 63d49bebfcb9422b87754fb258e16a97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9e3668e37348496ca15ab4da57534a53" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 41837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 23:27:54.338310 7438 consensus_queue.cc:1035] T 27c38e75002d42688eac2cb1f13c66d4 P 63d49bebfcb9422b87754fb258e16a97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ad2dd25e957d4536bdc22a0df2b7263f" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 38457 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 23:27:54.342873 7404 consensus_queue.cc:1035] T 95f3282b21394b11807125a214e8697a P 9e3668e37348496ca15ab4da57534a53 [LEADER]: Connected to new peer: Peer: permanent_uuid: "63d49bebfcb9422b87754fb258e16a97" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 41385 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 23:27:54.347126 7404 consensus_queue.cc:1035] T 95f3282b21394b11807125a214e8697a P 9e3668e37348496ca15ab4da57534a53 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ad2dd25e957d4536bdc22a0df2b7263f" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 38457 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 23:27:54.426707 7386 raft_consensus.cc:482] T da50d2c38f2b4b1aa91ca1d5b92d32f8 P 63d49bebfcb9422b87754fb258e16a97 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 23:27:54.426992 7386 raft_consensus.cc:504] T da50d2c38f2b4b1aa91ca1d5b92d32f8 P 63d49bebfcb9422b87754fb258e16a97 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e3668e37348496ca15ab4da57534a53" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 41837 } } peers { permanent_uuid: "ad2dd25e957d4536bdc22a0df2b7263f" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 38457 } } peers { permanent_uuid: "63d49bebfcb9422b87754fb258e16a97" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 41385 } }
I20240503 23:27:54.427891 7386 leader_election.cc:288] T da50d2c38f2b4b1aa91ca1d5b92d32f8 P 63d49bebfcb9422b87754fb258e16a97 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9e3668e37348496ca15ab4da57534a53 (127.0.100.1:41837), ad2dd25e957d4536bdc22a0df2b7263f (127.0.100.2:38457)
I20240503 23:27:54.428490 7184 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da50d2c38f2b4b1aa91ca1d5b92d32f8" candidate_uuid: "63d49bebfcb9422b87754fb258e16a97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9e3668e37348496ca15ab4da57534a53" is_pre_election: true
I20240503 23:27:54.428506 7258 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da50d2c38f2b4b1aa91ca1d5b92d32f8" candidate_uuid: "63d49bebfcb9422b87754fb258e16a97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ad2dd25e957d4536bdc22a0df2b7263f" is_pre_election: true
I20240503 23:27:54.428817 7184 raft_consensus.cc:2454] T da50d2c38f2b4b1aa91ca1d5b92d32f8 P 9e3668e37348496ca15ab4da57534a53 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 63d49bebfcb9422b87754fb258e16a97 in term 0.
I20240503 23:27:54.428834 7258 raft_consensus.cc:2454] T da50d2c38f2b4b1aa91ca1d5b92d32f8 P ad2dd25e957d4536bdc22a0df2b7263f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 63d49bebfcb9422b87754fb258e16a97 in term 0.
I20240503 23:27:54.429356 7299 leader_election.cc:302] T da50d2c38f2b4b1aa91ca1d5b92d32f8 P 63d49bebfcb9422b87754fb258e16a97 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 63d49bebfcb9422b87754fb258e16a97, ad2dd25e957d4536bdc22a0df2b7263f; no voters:
I20240503 23:27:54.429688 7386 raft_consensus.cc:2781] T da50d2c38f2b4b1aa91ca1d5b92d32f8 P 63d49bebfcb9422b87754fb258e16a97 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 23:27:54.429808 7386 raft_consensus.cc:482] T da50d2c38f2b4b1aa91ca1d5b92d32f8 P 63d49bebfcb9422b87754fb258e16a97 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 23:27:54.429906 7386 raft_consensus.cc:3037] T da50d2c38f2b4b1aa91ca1d5b92d32f8 P 63d49bebfcb9422b87754fb258e16a97 [term 0 FOLLOWER]: Advancing to term 1
I20240503 23:27:54.432333 7386 raft_consensus.cc:504] T da50d2c38f2b4b1aa91ca1d5b92d32f8 P 63d49bebfcb9422b87754fb258e16a97 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e3668e37348496ca15ab4da57534a53" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 41837 } } peers { permanent_uuid: "ad2dd25e957d4536bdc22a0df2b7263f" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 38457 } } peers { permanent_uuid: "63d49bebfcb9422b87754fb258e16a97" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 41385 } }
I20240503 23:27:54.433176 7386 leader_election.cc:288] T da50d2c38f2b4b1aa91ca1d5b92d32f8 P 63d49bebfcb9422b87754fb258e16a97 [CANDIDATE]: Term 1 election: Requested vote from peers 9e3668e37348496ca15ab4da57534a53 (127.0.100.1:41837), ad2dd25e957d4536bdc22a0df2b7263f (127.0.100.2:38457)
I20240503 23:27:54.433759 7184 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da50d2c38f2b4b1aa91ca1d5b92d32f8" candidate_uuid: "63d49bebfcb9422b87754fb258e16a97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9e3668e37348496ca15ab4da57534a53"
I20240503 23:27:54.434006 7184 raft_consensus.cc:3037] T da50d2c38f2b4b1aa91ca1d5b92d32f8 P 9e3668e37348496ca15ab4da57534a53 [term 0 FOLLOWER]: Advancing to term 1
I20240503 23:27:54.434651 7258 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da50d2c38f2b4b1aa91ca1d5b92d32f8" candidate_uuid: "63d49bebfcb9422b87754fb258e16a97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ad2dd25e957d4536bdc22a0df2b7263f"
I20240503 23:27:54.434842 7258 raft_consensus.cc:3037] T da50d2c38f2b4b1aa91ca1d5b92d32f8 P ad2dd25e957d4536bdc22a0df2b7263f [term 0 FOLLOWER]: Advancing to term 1
I20240503 23:27:54.436522 7184 raft_consensus.cc:2454] T da50d2c38f2b4b1aa91ca1d5b92d32f8 P 9e3668e37348496ca15ab4da57534a53 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 63d49bebfcb9422b87754fb258e16a97 in term 1.
I20240503 23:27:54.437134 7298 leader_election.cc:302] T da50d2c38f2b4b1aa91ca1d5b92d32f8 P 63d49bebfcb9422b87754fb258e16a97 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 63d49bebfcb9422b87754fb258e16a97, 9e3668e37348496ca15ab4da57534a53; no voters:
I20240503 23:27:54.437199 7258 raft_consensus.cc:2454] T da50d2c38f2b4b1aa91ca1d5b92d32f8 P ad2dd25e957d4536bdc22a0df2b7263f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 63d49bebfcb9422b87754fb258e16a97 in term 1.
I20240503 23:27:54.437476 7386 raft_consensus.cc:2781] T da50d2c38f2b4b1aa91ca1d5b92d32f8 P 63d49bebfcb9422b87754fb258e16a97 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 23:27:54.437666 7386 raft_consensus.cc:686] T da50d2c38f2b4b1aa91ca1d5b92d32f8 P 63d49bebfcb9422b87754fb258e16a97 [term 1 LEADER]: Becoming Leader. State: Replica: 63d49bebfcb9422b87754fb258e16a97, State: Running, Role: LEADER
I20240503 23:27:54.438033 7386 consensus_queue.cc:237] T da50d2c38f2b4b1aa91ca1d5b92d32f8 P 63d49bebfcb9422b87754fb258e16a97 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e3668e37348496ca15ab4da57534a53" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 41837 } } peers { permanent_uuid: "ad2dd25e957d4536bdc22a0df2b7263f" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 38457 } } peers { permanent_uuid: "63d49bebfcb9422b87754fb258e16a97" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 41385 } }
I20240503 23:27:54.441166 7079 catalog_manager.cc:5484] T da50d2c38f2b4b1aa91ca1d5b92d32f8 P 63d49bebfcb9422b87754fb258e16a97 reported cstate change: term changed from 0 to 1, leader changed from <none> to 63d49bebfcb9422b87754fb258e16a97 (127.0.100.3). New cstate: current_term: 1 leader_uuid: "63d49bebfcb9422b87754fb258e16a97" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e3668e37348496ca15ab4da57534a53" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 41837 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ad2dd25e957d4536bdc22a0df2b7263f" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 38457 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "63d49bebfcb9422b87754fb258e16a97" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 41385 } health_report { overall_health: HEALTHY } } }
I20240503 23:27:54.481552 7442 consensus_queue.cc:1035] T 28c504576e8147d2ab6418920edf5e2f P 9e3668e37348496ca15ab4da57534a53 [LEADER]: Connected to new peer: Peer: permanent_uuid: "63d49bebfcb9422b87754fb258e16a97" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 41385 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 23:27:54.487067 7385 consensus_queue.cc:1035] T 28c504576e8147d2ab6418920edf5e2f P 9e3668e37348496ca15ab4da57534a53 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ad2dd25e957d4536bdc22a0df2b7263f" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 38457 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 23:27:54.530390 400 tablet_server.cc:178] TabletServer@127.0.100.1:0 shutting down...
I20240503 23:27:54.540087 400 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240503 23:27:54.540699 400 tablet_replica.cc:330] T 6cfc00700ec04e688a83c434db278210 P 9e3668e37348496ca15ab4da57534a53: stopping tablet replica
I20240503 23:27:54.540910 400 raft_consensus.cc:2229] T 6cfc00700ec04e688a83c434db278210 P 9e3668e37348496ca15ab4da57534a53 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 23:27:54.541122 400 raft_consensus.cc:2258] T 6cfc00700ec04e688a83c434db278210 P 9e3668e37348496ca15ab4da57534a53 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 23:27:54.541929 400 tablet_replica.cc:330] T 01ffeaa637ee4aa7a172de6b3e87e2e7 P 9e3668e37348496ca15ab4da57534a53: stopping tablet replica
I20240503 23:27:54.542109 400 raft_consensus.cc:2229] T 01ffeaa637ee4aa7a172de6b3e87e2e7 P 9e3668e37348496ca15ab4da57534a53 [term 1 LEADER]: Raft consensus shutting down.
I20240503 23:27:54.542344 400 pending_rounds.cc:62] T 01ffeaa637ee4aa7a172de6b3e87e2e7 P 9e3668e37348496ca15ab4da57534a53: Trying to abort 1 pending ops.
I20240503 23:27:54.542415 400 pending_rounds.cc:68] T 01ffeaa637ee4aa7a172de6b3e87e2e7 P 9e3668e37348496ca15ab4da57534a53: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7023734269089669120 op_type: NO_OP noop_request { }
I20240503 23:27:54.542568 400 raft_consensus.cc:2866] T 01ffeaa637ee4aa7a172de6b3e87e2e7 P 9e3668e37348496ca15ab4da57534a53 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20240503 23:27:54.542665 400 raft_consensus.cc:2258] T 01ffeaa637ee4aa7a172de6b3e87e2e7 P 9e3668e37348496ca15ab4da57534a53 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 23:27:54.543510 400 tablet_replica.cc:330] T fe71c54bf82048669579628b317224e7 P 9e3668e37348496ca15ab4da57534a53: stopping tablet replica
I20240503 23:27:54.543674 400 raft_consensus.cc:2229] T fe71c54bf82048669579628b317224e7 P 9e3668e37348496ca15ab4da57534a53 [term 1 LEADER]: Raft consensus shutting down.
I20240503 23:27:54.543984 400 raft_consensus.cc:2258] T fe71c54bf82048669579628b317224e7 P 9e3668e37348496ca15ab4da57534a53 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 23:27:54.544736 400 tablet_replica.cc:330] T 95f3282b21394b11807125a214e8697a P 9e3668e37348496ca15ab4da57534a53: stopping tablet replica
I20240503 23:27:54.544883 400 raft_consensus.cc:2229] T 95f3282b21394b11807125a214e8697a P 9e3668e37348496ca15ab4da57534a53 [term 1 LEADER]: Raft consensus shutting down.
I20240503 23:27:54.545171 400 raft_consensus.cc:2258] T 95f3282b21394b11807125a214e8697a P 9e3668e37348496ca15ab4da57534a53 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 23:27:54.545883 400 tablet_replica.cc:330] T 27c38e75002d42688eac2cb1f13c66d4 P 9e3668e37348496ca15ab4da57534a53: stopping tablet replica
I20240503 23:27:54.546032 400 raft_consensus.cc:2229] T 27c38e75002d42688eac2cb1f13c66d4 P 9e3668e37348496ca15ab4da57534a53 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 23:27:54.546164 400 raft_consensus.cc:2258] T 27c38e75002d42688eac2cb1f13c66d4 P 9e3668e37348496ca15ab4da57534a53 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 23:27:54.546787 400 tablet_replica.cc:330] T 28c504576e8147d2ab6418920edf5e2f P 9e3668e37348496ca15ab4da57534a53: stopping tablet replica
I20240503 23:27:54.546909 400 raft_consensus.cc:2229] T 28c504576e8147d2ab6418920edf5e2f P 9e3668e37348496ca15ab4da57534a53 [term 1 LEADER]: Raft consensus shutting down.
I20240503 23:27:54.547191 400 raft_consensus.cc:2258] T 28c504576e8147d2ab6418920edf5e2f P 9e3668e37348496ca15ab4da57534a53 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 23:27:54.547765 400 tablet_replica.cc:330] T 8269340eb54c455c95d80065a480be0f P 9e3668e37348496ca15ab4da57534a53: stopping tablet replica
I20240503 23:27:54.547909 400 raft_consensus.cc:2229] T 8269340eb54c455c95d80065a480be0f P 9e3668e37348496ca15ab4da57534a53 [term 1 LEADER]: Raft consensus shutting down.
I20240503 23:27:54.548079 400 pending_rounds.cc:62] T 8269340eb54c455c95d80065a480be0f P 9e3668e37348496ca15ab4da57534a53: Trying to abort 1 pending ops.
I20240503 23:27:54.548153 400 pending_rounds.cc:68] T 8269340eb54c455c95d80065a480be0f P 9e3668e37348496ca15ab4da57534a53: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7023734268728868864 op_type: NO_OP noop_request { }
I20240503 23:27:54.548297 400 raft_consensus.cc:2866] T 8269340eb54c455c95d80065a480be0f P 9e3668e37348496ca15ab4da57534a53 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20240503 23:27:54.548427 400 raft_consensus.cc:2258] T 8269340eb54c455c95d80065a480be0f P 9e3668e37348496ca15ab4da57534a53 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 23:27:54.549203 400 tablet_replica.cc:330] T da50d2c38f2b4b1aa91ca1d5b92d32f8 P 9e3668e37348496ca15ab4da57534a53: stopping tablet replica
I20240503 23:27:54.549336 400 raft_consensus.cc:2229] T da50d2c38f2b4b1aa91ca1d5b92d32f8 P 9e3668e37348496ca15ab4da57534a53 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 23:27:54.549451 400 raft_consensus.cc:2258] T da50d2c38f2b4b1aa91ca1d5b92d32f8 P 9e3668e37348496ca15ab4da57534a53 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 23:27:54.565758 400 tablet_server.cc:194] TabletServer@127.0.100.1:0 shutdown complete.
I20240503 23:27:54.575018 7391 consensus_queue.cc:1035] T 6cfc00700ec04e688a83c434db278210 P ad2dd25e957d4536bdc22a0df2b7263f [LEADER]: Connected to new peer: Peer: permanent_uuid: "63d49bebfcb9422b87754fb258e16a97" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 41385 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240503 23:27:54.581215 7224 consensus_peers.cc:485] T 6cfc00700ec04e688a83c434db278210 P ad2dd25e957d4536bdc22a0df2b7263f -> Peer 9e3668e37348496ca15ab4da57534a53 (127.0.100.1:41837): Couldn't send request to peer 9e3668e37348496ca15ab4da57534a53. Status: Network error: Client connection negotiation failed: client connection to 127.0.100.1:41837: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 23:27:54.766366 7298 consensus_peers.cc:485] T 27c38e75002d42688eac2cb1f13c66d4 P 63d49bebfcb9422b87754fb258e16a97 -> Peer 9e3668e37348496ca15ab4da57534a53 (127.0.100.1:41837): Couldn't send request to peer 9e3668e37348496ca15ab4da57534a53. Status: Network error: Client connection negotiation failed: client connection to 127.0.100.1:41837: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 23:27:54.935357 7386 consensus_queue.cc:1035] T da50d2c38f2b4b1aa91ca1d5b92d32f8 P 63d49bebfcb9422b87754fb258e16a97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ad2dd25e957d4536bdc22a0df2b7263f" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 38457 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240503 23:27:54.941431 7298 consensus_peers.cc:485] T da50d2c38f2b4b1aa91ca1d5b92d32f8 P 63d49bebfcb9422b87754fb258e16a97 -> Peer 9e3668e37348496ca15ab4da57534a53 (127.0.100.1:41837): Couldn't send request to peer 9e3668e37348496ca15ab4da57534a53. Status: Network error: Client connection negotiation failed: client connection to 127.0.100.1:41837: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 23:27:55.515928 7448 consensus_queue.cc:579] T 6cfc00700ec04e688a83c434db278210 P ad2dd25e957d4536bdc22a0df2b7263f [LEADER]: Leader has been unable to successfully communicate with peer 9e3668e37348496ca15ab4da57534a53 for more than 1 seconds (1.338s)
I20240503 23:27:55.583892 7082 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:34356:
table { table_name: "dugtrio" } modify_external_catalogs: true
I20240503 23:27:55.584210 7082 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:34356:
table { table_name: "dugtrio" } modify_external_catalogs: true
I20240503 23:27:55.591226 7082 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 3e719c8272094dccadee8e7490c155c9: Sending DeleteTablet for 3 replicas of tablet da50d2c38f2b4b1aa91ca1d5b92d32f8
I20240503 23:27:55.591835 7082 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 3e719c8272094dccadee8e7490c155c9: Sending DeleteTablet for 3 replicas of tablet 8269340eb54c455c95d80065a480be0f
I20240503 23:27:55.592244 7082 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 3e719c8272094dccadee8e7490c155c9: Sending DeleteTablet for 3 replicas of tablet 01ffeaa637ee4aa7a172de6b3e87e2e7
I20240503 23:27:55.592406 7246 tablet_service.cc:1514] Processing DeleteTablet for tablet da50d2c38f2b4b1aa91ca1d5b92d32f8 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 23:27:55 UTC) from {username='slave'} at 127.0.0.1:48548
I20240503 23:27:55.592582 7321 tablet_service.cc:1514] Processing DeleteTablet for tablet da50d2c38f2b4b1aa91ca1d5b92d32f8 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 23:27:55 UTC) from {username='slave'} at 127.0.0.1:45946
I20240503 23:27:55.592626 7082 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 3e719c8272094dccadee8e7490c155c9: Sending DeleteTablet for 3 replicas of tablet 6cfc00700ec04e688a83c434db278210
I20240503 23:27:55.593529 7464 tablet_replica.cc:330] T da50d2c38f2b4b1aa91ca1d5b92d32f8 P ad2dd25e957d4536bdc22a0df2b7263f: stopping tablet replica
I20240503 23:27:55.593690 7465 tablet_replica.cc:330] T da50d2c38f2b4b1aa91ca1d5b92d32f8 P 63d49bebfcb9422b87754fb258e16a97: stopping tablet replica
I20240503 23:27:55.593799 7323 tablet_service.cc:1514] Processing DeleteTablet for tablet 01ffeaa637ee4aa7a172de6b3e87e2e7 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 23:27:55 UTC) from {username='slave'} at 127.0.0.1:45946
I20240503 23:27:55.593957 7464 raft_consensus.cc:2229] T da50d2c38f2b4b1aa91ca1d5b92d32f8 P ad2dd25e957d4536bdc22a0df2b7263f [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 23:27:55.594215 7464 raft_consensus.cc:2258] T da50d2c38f2b4b1aa91ca1d5b92d32f8 P ad2dd25e957d4536bdc22a0df2b7263f [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 23:27:55.594475 7246 tablet_service.cc:1514] Processing DeleteTablet for tablet 8269340eb54c455c95d80065a480be0f with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 23:27:55 UTC) from {username='slave'} at 127.0.0.1:48548
I20240503 23:27:55.594498 7249 tablet_service.cc:1514] Processing DeleteTablet for tablet 6cfc00700ec04e688a83c434db278210 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 23:27:55 UTC) from {username='slave'} at 127.0.0.1:48548
I20240503 23:27:55.594763 7465 raft_consensus.cc:2229] T da50d2c38f2b4b1aa91ca1d5b92d32f8 P 63d49bebfcb9422b87754fb258e16a97 [term 1 LEADER]: Raft consensus shutting down.
I20240503 23:27:55.594732 7248 tablet_service.cc:1514] Processing DeleteTablet for tablet 01ffeaa637ee4aa7a172de6b3e87e2e7 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 23:27:55 UTC) from {username='slave'} at 127.0.0.1:48548
I20240503 23:27:55.593803 7320 tablet_service.cc:1514] Processing DeleteTablet for tablet 6cfc00700ec04e688a83c434db278210 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 23:27:55 UTC) from {username='slave'} at 127.0.0.1:45946
I20240503 23:27:55.593709 7322 tablet_service.cc:1514] Processing DeleteTablet for tablet 8269340eb54c455c95d80065a480be0f with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 23:27:55 UTC) from {username='slave'} at 127.0.0.1:45946
I20240503 23:27:55.595052 400 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20240503 23:27:55.595249 7465 raft_consensus.cc:2258] T da50d2c38f2b4b1aa91ca1d5b92d32f8 P 63d49bebfcb9422b87754fb258e16a97 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 23:27:55.595654 7464 ts_tablet_manager.cc:1875] T da50d2c38f2b4b1aa91ca1d5b92d32f8 P ad2dd25e957d4536bdc22a0df2b7263f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 23:27:55.596450 7465 ts_tablet_manager.cc:1875] T da50d2c38f2b4b1aa91ca1d5b92d32f8 P 63d49bebfcb9422b87754fb258e16a97: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 23:27:55.606168 7465 ts_tablet_manager.cc:1888] T da50d2c38f2b4b1aa91ca1d5b92d32f8 P 63d49bebfcb9422b87754fb258e16a97: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240503 23:27:55.607340 7465 log.cc:1196] T da50d2c38f2b4b1aa91ca1d5b92d32f8 P 63d49bebfcb9422b87754fb258e16a97: Deleting WAL directory at /tmp/dist-test-taskZhIiPz/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestDeletedTables.1714778688451250-400-0/minicluster-data/ts-2-root/wals/da50d2c38f2b4b1aa91ca1d5b92d32f8
I20240503 23:27:55.607831 7465 ts_tablet_manager.cc:1909] T da50d2c38f2b4b1aa91ca1d5b92d32f8 P 63d49bebfcb9422b87754fb258e16a97: Deleting consensus metadata
W20240503 23:27:55.608407 7467 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 23:27:55.608582 7068 catalog_manager.cc:4573] TS 9e3668e37348496ca15ab4da57534a53 (127.0.100.1:41837): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 8269340eb54c455c95d80065a480be0f: Network error: Client connection negotiation failed: client connection to 127.0.100.1:41837: connect: Connection refused (error 111)
I20240503 23:27:55.606168 7464 ts_tablet_manager.cc:1888] T da50d2c38f2b4b1aa91ca1d5b92d32f8 P ad2dd25e957d4536bdc22a0df2b7263f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240503 23:27:55.609119 7464 log.cc:1196] T da50d2c38f2b4b1aa91ca1d5b92d32f8 P ad2dd25e957d4536bdc22a0df2b7263f: Deleting WAL directory at /tmp/dist-test-taskZhIiPz/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestDeletedTables.1714778688451250-400-0/minicluster-data/ts-1-root/wals/da50d2c38f2b4b1aa91ca1d5b92d32f8
W20240503 23:27:55.609383 7468 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 23:27:55.609508 7066 catalog_manager.cc:4830] TS 63d49bebfcb9422b87754fb258e16a97 (127.0.100.3:41385): tablet da50d2c38f2b4b1aa91ca1d5b92d32f8 (table dugtrio [id=1e717799f69545008dc79566448dcf9f]) successfully deleted
I20240503 23:27:55.609586 7464 ts_tablet_manager.cc:1909] T da50d2c38f2b4b1aa91ca1d5b92d32f8 P ad2dd25e957d4536bdc22a0df2b7263f: Deleting consensus metadata
I20240503 23:27:55.609872 7465 tablet_replica.cc:330] T 01ffeaa637ee4aa7a172de6b3e87e2e7 P 63d49bebfcb9422b87754fb258e16a97: stopping tablet replica
I20240503 23:27:55.610103 7465 raft_consensus.cc:2229] T 01ffeaa637ee4aa7a172de6b3e87e2e7 P 63d49bebfcb9422b87754fb258e16a97 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 23:27:55.610306 7465 raft_consensus.cc:2258] T 01ffeaa637ee4aa7a172de6b3e87e2e7 P 63d49bebfcb9422b87754fb258e16a97 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 23:27:55.610611 7464 tablet_replica.cc:330] T 8269340eb54c455c95d80065a480be0f P ad2dd25e957d4536bdc22a0df2b7263f: stopping tablet replica
I20240503 23:27:55.610831 7069 catalog_manager.cc:4830] TS ad2dd25e957d4536bdc22a0df2b7263f (127.0.100.2:38457): tablet da50d2c38f2b4b1aa91ca1d5b92d32f8 (table dugtrio [id=1e717799f69545008dc79566448dcf9f]) successfully deleted
I20240503 23:27:55.611083 7465 ts_tablet_manager.cc:1875] T 01ffeaa637ee4aa7a172de6b3e87e2e7 P 63d49bebfcb9422b87754fb258e16a97: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 23:27:55.610832 7464 raft_consensus.cc:2229] T 8269340eb54c455c95d80065a480be0f P ad2dd25e957d4536bdc22a0df2b7263f [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 23:27:55.611335 7464 raft_consensus.cc:2258] T 8269340eb54c455c95d80065a480be0f P ad2dd25e957d4536bdc22a0df2b7263f [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 23:27:55.612031 7464 ts_tablet_manager.cc:1875] T 8269340eb54c455c95d80065a480be0f P ad2dd25e957d4536bdc22a0df2b7263f: Deleting tablet data with delete state TABLET_DATA_DELETED
W20240503 23:27:55.612463 7470 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 23:27:55.613363 400 server_base.cc:1034] running on GCE node
I20240503 23:27:55.613700 400 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240503 23:27:55.613775 400 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240503 23:27:55.613809 400 hybrid_clock.cc:648] HybridClock initialized: now 1714778875613804 us; error 0 us; skew 500 ppm
I20240503 23:27:55.614033 400 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 23:27:55.615866 400 webserver.cc:460] Webserver started at http://127.0.100.4:34145/ using document root <none> and password file <none>
I20240503 23:27:55.616132 400 fs_manager.cc:359] Metadata directory not provided
I20240503 23:27:55.616201 400 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 23:27:55.616381 400 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 23:27:55.617141 400 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZhIiPz/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestDeletedTables.1714778688451250-400-0/minicluster-data/ts-3-root/instance:
uuid: "9c15c9d698fe4fa4a7c458f2e51ab5f9"
format_stamp: "Formatted at 2024-05-03 23:27:55 on dist-test-slave-jvbp"
I20240503 23:27:55.617317 7465 ts_tablet_manager.cc:1888] T 01ffeaa637ee4aa7a172de6b3e87e2e7 P 63d49bebfcb9422b87754fb258e16a97: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240503 23:27:55.617442 7465 log.cc:1196] T 01ffeaa637ee4aa7a172de6b3e87e2e7 P 63d49bebfcb9422b87754fb258e16a97: Deleting WAL directory at /tmp/dist-test-taskZhIiPz/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestDeletedTables.1714778688451250-400-0/minicluster-data/ts-2-root/wals/01ffeaa637ee4aa7a172de6b3e87e2e7
I20240503 23:27:55.617766 7465 ts_tablet_manager.cc:1909] T 01ffeaa637ee4aa7a172de6b3e87e2e7 P 63d49bebfcb9422b87754fb258e16a97: Deleting consensus metadata
I20240503 23:27:55.618194 7464 ts_tablet_manager.cc:1888] T 8269340eb54c455c95d80065a480be0f P ad2dd25e957d4536bdc22a0df2b7263f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240503 23:27:55.618302 7464 log.cc:1196] T 8269340eb54c455c95d80065a480be0f P ad2dd25e957d4536bdc22a0df2b7263f: Deleting WAL directory at /tmp/dist-test-taskZhIiPz/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestDeletedTables.1714778688451250-400-0/minicluster-data/ts-1-root/wals/8269340eb54c455c95d80065a480be0f
I20240503 23:27:55.618592 7464 ts_tablet_manager.cc:1909] T 8269340eb54c455c95d80065a480be0f P ad2dd25e957d4536bdc22a0df2b7263f: Deleting consensus metadata
I20240503 23:27:55.618888 7066 catalog_manager.cc:4830] TS 63d49bebfcb9422b87754fb258e16a97 (127.0.100.3:41385): tablet 01ffeaa637ee4aa7a172de6b3e87e2e7 (table dugtrio [id=1e717799f69545008dc79566448dcf9f]) successfully deleted
I20240503 23:27:55.619007 7465 tablet_replica.cc:330] T 6cfc00700ec04e688a83c434db278210 P 63d49bebfcb9422b87754fb258e16a97: stopping tablet replica
I20240503 23:27:55.619194 7465 raft_consensus.cc:2229] T 6cfc00700ec04e688a83c434db278210 P 63d49bebfcb9422b87754fb258e16a97 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 23:27:55.619367 7465 raft_consensus.cc:2258] T 6cfc00700ec04e688a83c434db278210 P 63d49bebfcb9422b87754fb258e16a97 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 23:27:55.619518 7464 tablet_replica.cc:330] T 6cfc00700ec04e688a83c434db278210 P ad2dd25e957d4536bdc22a0df2b7263f: stopping tablet replica
I20240503 23:27:55.619653 7069 catalog_manager.cc:4830] TS ad2dd25e957d4536bdc22a0df2b7263f (127.0.100.2:38457): tablet 8269340eb54c455c95d80065a480be0f (table dugtrio [id=1e717799f69545008dc79566448dcf9f]) successfully deleted
I20240503 23:27:55.620041 7465 ts_tablet_manager.cc:1875] T 6cfc00700ec04e688a83c434db278210 P 63d49bebfcb9422b87754fb258e16a97: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 23:27:55.620126 7464 raft_consensus.cc:2229] T 6cfc00700ec04e688a83c434db278210 P ad2dd25e957d4536bdc22a0df2b7263f [term 1 LEADER]: Raft consensus shutting down.
I20240503 23:27:55.620528 7464 raft_consensus.cc:2258] T 6cfc00700ec04e688a83c434db278210 P ad2dd25e957d4536bdc22a0df2b7263f [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 23:27:55.620800 400 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.003s
I20240503 23:27:55.621234 7464 ts_tablet_manager.cc:1875] T 6cfc00700ec04e688a83c434db278210 P ad2dd25e957d4536bdc22a0df2b7263f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 23:27:55.622989 7475 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 23:27:55.623456 400 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.001s
I20240503 23:27:55.623607 400 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZhIiPz/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestDeletedTables.1714778688451250-400-0/minicluster-data/ts-3-root
uuid: "9c15c9d698fe4fa4a7c458f2e51ab5f9"
format_stamp: "Formatted at 2024-05-03 23:27:55 on dist-test-slave-jvbp"
I20240503 23:27:55.623849 400 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZhIiPz/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestDeletedTables.1714778688451250-400-0/minicluster-data/ts-3-root
metadata directory: /tmp/dist-test-taskZhIiPz/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestDeletedTables.1714778688451250-400-0/minicluster-data/ts-3-root
1 data directories: /tmp/dist-test-taskZhIiPz/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestDeletedTables.1714778688451250-400-0/minicluster-data/ts-3-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 23:27:55.625103 7465 ts_tablet_manager.cc:1888] T 6cfc00700ec04e688a83c434db278210 P 63d49bebfcb9422b87754fb258e16a97: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240503 23:27:55.625245 7465 log.cc:1196] T 6cfc00700ec04e688a83c434db278210 P 63d49bebfcb9422b87754fb258e16a97: Deleting WAL directory at /tmp/dist-test-taskZhIiPz/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestDeletedTables.1714778688451250-400-0/minicluster-data/ts-2-root/wals/6cfc00700ec04e688a83c434db278210
I20240503 23:27:55.625684 7465 ts_tablet_manager.cc:1909] T 6cfc00700ec04e688a83c434db278210 P 63d49bebfcb9422b87754fb258e16a97: Deleting consensus metadata
I20240503 23:27:55.626952 7465 tablet_replica.cc:330] T 8269340eb54c455c95d80065a480be0f P 63d49bebfcb9422b87754fb258e16a97: stopping tablet replica
I20240503 23:27:55.627152 7066 catalog_manager.cc:4830] TS 63d49bebfcb9422b87754fb258e16a97 (127.0.100.3:41385): tablet 6cfc00700ec04e688a83c434db278210 (table dugtrio [id=1e717799f69545008dc79566448dcf9f]) successfully deleted
I20240503 23:27:55.627218 7465 raft_consensus.cc:2229] T 8269340eb54c455c95d80065a480be0f P 63d49bebfcb9422b87754fb258e16a97 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 23:27:55.627389 7465 raft_consensus.cc:2258] T 8269340eb54c455c95d80065a480be0f P 63d49bebfcb9422b87754fb258e16a97 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 23:27:55.628028 7465 ts_tablet_manager.cc:1875] T 8269340eb54c455c95d80065a480be0f P 63d49bebfcb9422b87754fb258e16a97: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 23:27:55.628206 7464 ts_tablet_manager.cc:1888] T 6cfc00700ec04e688a83c434db278210 P ad2dd25e957d4536bdc22a0df2b7263f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240503 23:27:55.628311 7464 log.cc:1196] T 6cfc00700ec04e688a83c434db278210 P ad2dd25e957d4536bdc22a0df2b7263f: Deleting WAL directory at /tmp/dist-test-taskZhIiPz/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestDeletedTables.1714778688451250-400-0/minicluster-data/ts-1-root/wals/6cfc00700ec04e688a83c434db278210
I20240503 23:27:55.628693 7464 ts_tablet_manager.cc:1909] T 6cfc00700ec04e688a83c434db278210 P ad2dd25e957d4536bdc22a0df2b7263f: Deleting consensus metadata
I20240503 23:27:55.629652 7464 tablet_replica.cc:330] T 01ffeaa637ee4aa7a172de6b3e87e2e7 P ad2dd25e957d4536bdc22a0df2b7263f: stopping tablet replica
I20240503 23:27:55.629871 7464 raft_consensus.cc:2229] T 01ffeaa637ee4aa7a172de6b3e87e2e7 P ad2dd25e957d4536bdc22a0df2b7263f [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 23:27:55.629909 7069 catalog_manager.cc:4830] TS ad2dd25e957d4536bdc22a0df2b7263f (127.0.100.2:38457): tablet 6cfc00700ec04e688a83c434db278210 (table dugtrio [id=1e717799f69545008dc79566448dcf9f]) successfully deleted
I20240503 23:27:55.630031 7464 raft_consensus.cc:2258] T 01ffeaa637ee4aa7a172de6b3e87e2e7 P ad2dd25e957d4536bdc22a0df2b7263f [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 23:27:55.630631 7464 ts_tablet_manager.cc:1875] T 01ffeaa637ee4aa7a172de6b3e87e2e7 P ad2dd25e957d4536bdc22a0df2b7263f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 23:27:55.634146 7465 ts_tablet_manager.cc:1888] T 8269340eb54c455c95d80065a480be0f P 63d49bebfcb9422b87754fb258e16a97: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240503 23:27:55.634299 7465 log.cc:1196] T 8269340eb54c455c95d80065a480be0f P 63d49bebfcb9422b87754fb258e16a97: Deleting WAL directory at /tmp/dist-test-taskZhIiPz/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestDeletedTables.1714778688451250-400-0/minicluster-data/ts-2-root/wals/8269340eb54c455c95d80065a480be0f
I20240503 23:27:55.634624 7465 ts_tablet_manager.cc:1909] T 8269340eb54c455c95d80065a480be0f P 63d49bebfcb9422b87754fb258e16a97: Deleting consensus metadata
I20240503 23:27:55.635739 7066 catalog_manager.cc:4830] TS 63d49bebfcb9422b87754fb258e16a97 (127.0.100.3:41385): tablet 8269340eb54c455c95d80065a480be0f (table dugtrio [id=1e717799f69545008dc79566448dcf9f]) successfully deleted
I20240503 23:27:55.637557 7464 ts_tablet_manager.cc:1888] T 01ffeaa637ee4aa7a172de6b3e87e2e7 P ad2dd25e957d4536bdc22a0df2b7263f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240503 23:27:55.637706 7464 log.cc:1196] T 01ffeaa637ee4aa7a172de6b3e87e2e7 P ad2dd25e957d4536bdc22a0df2b7263f: Deleting WAL directory at /tmp/dist-test-taskZhIiPz/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestDeletedTables.1714778688451250-400-0/minicluster-data/ts-1-root/wals/01ffeaa637ee4aa7a172de6b3e87e2e7
I20240503 23:27:55.638000 7464 ts_tablet_manager.cc:1909] T 01ffeaa637ee4aa7a172de6b3e87e2e7 P ad2dd25e957d4536bdc22a0df2b7263f: Deleting consensus metadata
I20240503 23:27:55.639173 7069 catalog_manager.cc:4830] TS ad2dd25e957d4536bdc22a0df2b7263f (127.0.100.2:38457): tablet 01ffeaa637ee4aa7a172de6b3e87e2e7 (table dugtrio [id=1e717799f69545008dc79566448dcf9f]) successfully deleted
I20240503 23:27:55.645592 400 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 23:27:55.646200 400 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 23:27:55.646873 400 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 23:27:55.647903 400 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 23:27:55.647967 400 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 23:27:55.648280 400 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 23:27:55.648408 400 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 23:27:55.664237 400 rpc_server.cc:307] RPC server started. Bound to: 127.0.100.4:33075
I20240503 23:27:55.664258 7537 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.100.4:33075 every 8 connection(s)
I20240503 23:27:55.665920 400 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 23:27:55.669471 7541 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 23:27:55.670269 7542 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 23:27:55.670790 7544 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 23:27:55.671890 400 server_base.cc:1034] running on GCE node
I20240503 23:27:55.672163 400 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240503 23:27:55.672221 400 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240503 23:27:55.672271 400 hybrid_clock.cc:648] HybridClock initialized: now 1714778875672264 us; error 0 us; skew 500 ppm
I20240503 23:27:55.672443 400 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 23:27:55.672683 7538 heartbeater.cc:347] Connected to a master server at 127.0.100.62:43625
I20240503 23:27:55.672816 7538 heartbeater.cc:464] Registering TS with master...
I20240503 23:27:55.673274 7538 heartbeater.cc:511] Master 127.0.100.62:43625 requested a full tablet report, sending...
I20240503 23:27:55.674122 400 webserver.cc:460] Webserver started at http://127.0.100.1:34279/ using document root <none> and password file <none>
I20240503 23:27:55.674379 7082 ts_manager.cc:194] Registered new tserver with Master: 9c15c9d698fe4fa4a7c458f2e51ab5f9 (127.0.100.4:33075)
I20240503 23:27:55.674405 400 fs_manager.cc:359] Metadata directory not provided
I20240503 23:27:55.674492 400 fs_manager.cc:362] Using existing metadata directory in first data directory
I20240503 23:27:55.675520 7082 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:34384
I20240503 23:27:55.677119 400 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.000s sys 0.002s
I20240503 23:27:55.679147 7549 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 23:27:55.679548 400 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.002s
I20240503 23:27:55.679656 400 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZhIiPz/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestDeletedTables.1714778688451250-400-0/minicluster-data/ts-0-root
uuid: "9e3668e37348496ca15ab4da57534a53"
format_stamp: "Formatted at 2024-05-03 23:27:52 on dist-test-slave-jvbp"
I20240503 23:27:55.679823 400 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZhIiPz/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestDeletedTables.1714778688451250-400-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskZhIiPz/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestDeletedTables.1714778688451250-400-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskZhIiPz/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestDeletedTables.1714778688451250-400-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 23:27:55.687368 400 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 23:27:55.687971 400 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 23:27:55.688611 400 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 23:27:55.690097 7556 ts_tablet_manager.cc:522] Loading tablet metadata (0/8 complete)
I20240503 23:27:55.709132 400 ts_tablet_manager.cc:559] Loaded tablet metadata (8 total tablets, 8 live tablets)
I20240503 23:27:55.709254 400 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.019s user 0.001s sys 0.000s
I20240503 23:27:55.709379 400 ts_tablet_manager.cc:574] Registering tablets (0/8 complete)
I20240503 23:27:55.711552 7556 tablet_bootstrap.cc:492] T 27c38e75002d42688eac2cb1f13c66d4 P 9e3668e37348496ca15ab4da57534a53: Bootstrap starting.
I20240503 23:27:55.716500 7556 tablet_bootstrap.cc:492] T 27c38e75002d42688eac2cb1f13c66d4 P 9e3668e37348496ca15ab4da57534a53: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 23:27:55.716974 7556 tablet_bootstrap.cc:492] T 27c38e75002d42688eac2cb1f13c66d4 P 9e3668e37348496ca15ab4da57534a53: Bootstrap complete.
I20240503 23:27:55.717227 7556 ts_tablet_manager.cc:1367] T 27c38e75002d42688eac2cb1f13c66d4 P 9e3668e37348496ca15ab4da57534a53: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20240503 23:27:55.717905 7556 raft_consensus.cc:348] T 27c38e75002d42688eac2cb1f13c66d4 P 9e3668e37348496ca15ab4da57534a53 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e3668e37348496ca15ab4da57534a53" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 41837 } } peers { permanent_uuid: "ad2dd25e957d4536bdc22a0df2b7263f" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 38457 } } peers { permanent_uuid: "63d49bebfcb9422b87754fb258e16a97" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 41385 } }
I20240503 23:27:55.718122 7556 raft_consensus.cc:729] T 27c38e75002d42688eac2cb1f13c66d4 P 9e3668e37348496ca15ab4da57534a53 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9e3668e37348496ca15ab4da57534a53, State: Initialized, Role: FOLLOWER
I20240503 23:27:55.718397 7556 consensus_queue.cc:260] T 27c38e75002d42688eac2cb1f13c66d4 P 9e3668e37348496ca15ab4da57534a53 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e3668e37348496ca15ab4da57534a53" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 41837 } } peers { permanent_uuid: "ad2dd25e957d4536bdc22a0df2b7263f" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 38457 } } peers { permanent_uuid: "63d49bebfcb9422b87754fb258e16a97" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 41385 } }
I20240503 23:27:55.719151 7556 ts_tablet_manager.cc:1398] T 27c38e75002d42688eac2cb1f13c66d4 P 9e3668e37348496ca15ab4da57534a53: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 23:27:55.719394 7556 tablet_bootstrap.cc:492] T 01ffeaa637ee4aa7a172de6b3e87e2e7 P 9e3668e37348496ca15ab4da57534a53: Bootstrap starting.
I20240503 23:27:55.723714 7556 tablet_bootstrap.cc:492] T 01ffeaa637ee4aa7a172de6b3e87e2e7 P 9e3668e37348496ca15ab4da57534a53: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=0 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20240503 23:27:55.723960 400 ts_tablet_manager.cc:590] Registered 8 tablets
I20240503 23:27:55.724045 400 ts_tablet_manager.cc:569] Time spent register tablets: real 0.015s user 0.011s sys 0.000s
I20240503 23:27:55.724215 7556 tablet_bootstrap.cc:492] T 01ffeaa637ee4aa7a172de6b3e87e2e7 P 9e3668e37348496ca15ab4da57534a53: Bootstrap complete.
I20240503 23:27:55.724462 7556 ts_tablet_manager.cc:1367] T 01ffeaa637ee4aa7a172de6b3e87e2e7 P 9e3668e37348496ca15ab4da57534a53: Time spent bootstrapping tablet: real 0.005s user 0.005s sys 0.000s
I20240503 23:27:55.725180 7556 raft_consensus.cc:348] T 01ffeaa637ee4aa7a172de6b3e87e2e7 P 9e3668e37348496ca15ab4da57534a53 [term 1 FOLLOWER]: Replica starting. Triggering 1 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e3668e37348496ca15ab4da57534a53" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 41837 } } peers { permanent_uuid: "ad2dd25e957d4536bdc22a0df2b7263f" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 38457 } } peers { permanent_uuid: "63d49bebfcb9422b87754fb258e16a97" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 41385 } }
I20240503 23:27:55.725452 7556 raft_consensus.cc:729] T 01ffeaa637ee4aa7a172de6b3e87e2e7 P 9e3668e37348496ca15ab4da57534a53 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9e3668e37348496ca15ab4da57534a53, State: Initialized, Role: FOLLOWER
I20240503 23:27:55.725697 7556 consensus_queue.cc:260] T 01ffeaa637ee4aa7a172de6b3e87e2e7 P 9e3668e37348496ca15ab4da57534a53 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e3668e37348496ca15ab4da57534a53" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 41837 } } peers { permanent_uuid: "ad2dd25e957d4536bdc22a0df2b7263f" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 38457 } } peers { permanent_uuid: "63d49bebfcb9422b87754fb258e16a97" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 41385 } }
I20240503 23:27:55.726166 7556 ts_tablet_manager.cc:1398] T 01ffeaa637ee4aa7a172de6b3e87e2e7 P 9e3668e37348496ca15ab4da57534a53: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 23:27:55.726392 7556 tablet_bootstrap.cc:492] T 8269340eb54c455c95d80065a480be0f P 9e3668e37348496ca15ab4da57534a53: Bootstrap starting.
I20240503 23:27:55.731024 7556 tablet_bootstrap.cc:492] T 8269340eb54c455c95d80065a480be0f P 9e3668e37348496ca15ab4da57534a53: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=0 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20240503 23:27:55.731427 7556 tablet_bootstrap.cc:492] T 8269340eb54c455c95d80065a480be0f P 9e3668e37348496ca15ab4da57534a53: Bootstrap complete.
I20240503 23:27:55.731658 7556 ts_tablet_manager.cc:1367] T 8269340eb54c455c95d80065a480be0f P 9e3668e37348496ca15ab4da57534a53: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20240503 23:27:55.732304 7556 raft_consensus.cc:348] T 8269340eb54c455c95d80065a480be0f P 9e3668e37348496ca15ab4da57534a53 [term 1 FOLLOWER]: Replica starting. Triggering 1 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e3668e37348496ca15ab4da57534a53" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 41837 } } peers { permanent_uuid: "ad2dd25e957d4536bdc22a0df2b7263f" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 38457 } } peers { permanent_uuid: "63d49bebfcb9422b87754fb258e16a97" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 41385 } }
I20240503 23:27:55.732605 7556 raft_consensus.cc:729] T 8269340eb54c455c95d80065a480be0f P 9e3668e37348496ca15ab4da57534a53 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9e3668e37348496ca15ab4da57534a53, State: Initialized, Role: FOLLOWER
I20240503 23:27:55.732892 7556 consensus_queue.cc:260] T 8269340eb54c455c95d80065a480be0f P 9e3668e37348496ca15ab4da57534a53 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e3668e37348496ca15ab4da57534a53" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 41837 } } peers { permanent_uuid: "ad2dd25e957d4536bdc22a0df2b7263f" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 38457 } } peers { permanent_uuid: "63d49bebfcb9422b87754fb258e16a97" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 41385 } }
I20240503 23:27:55.733407 7556 ts_tablet_manager.cc:1398] T 8269340eb54c455c95d80065a480be0f P 9e3668e37348496ca15ab4da57534a53: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20240503 23:27:55.733664 7556 tablet_bootstrap.cc:492] T fe71c54bf82048669579628b317224e7 P 9e3668e37348496ca15ab4da57534a53: Bootstrap starting.
I20240503 23:27:55.738361 7556 tablet_bootstrap.cc:492] T fe71c54bf82048669579628b317224e7 P 9e3668e37348496ca15ab4da57534a53: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 23:27:55.738847 7556 tablet_bootstrap.cc:492] T fe71c54bf82048669579628b317224e7 P 9e3668e37348496ca15ab4da57534a53: Bootstrap complete.
I20240503 23:27:55.739153 7556 ts_tablet_manager.cc:1367] T fe71c54bf82048669579628b317224e7 P 9e3668e37348496ca15ab4da57534a53: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.003s
I20240503 23:27:55.739845 7556 raft_consensus.cc:348] T fe71c54bf82048669579628b317224e7 P 9e3668e37348496ca15ab4da57534a53 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad2dd25e957d4536bdc22a0df2b7263f" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 38457 } } peers { permanent_uuid: "9e3668e37348496ca15ab4da57534a53" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 41837 } } peers { permanent_uuid: "63d49bebfcb9422b87754fb258e16a97" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 41385 } }
I20240503 23:27:55.740026 7556 raft_consensus.cc:729] T fe71c54bf82048669579628b317224e7 P 9e3668e37348496ca15ab4da57534a53 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9e3668e37348496ca15ab4da57534a53, State: Initialized, Role: FOLLOWER
I20240503 23:27:55.740289 7556 consensus_queue.cc:260] T fe71c54bf82048669579628b317224e7 P 9e3668e37348496ca15ab4da57534a53 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad2dd25e957d4536bdc22a0df2b7263f" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 38457 } } peers { permanent_uuid: "9e3668e37348496ca15ab4da57534a53" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 41837 } } peers { permanent_uuid: "63d49bebfcb9422b87754fb258e16a97" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 41385 } }
I20240503 23:27:55.740787 7556 ts_tablet_manager.cc:1398] T fe71c54bf82048669579628b317224e7 P 9e3668e37348496ca15ab4da57534a53: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20240503 23:27:55.741042 7556 tablet_bootstrap.cc:492] T 6cfc00700ec04e688a83c434db278210 P 9e3668e37348496ca15ab4da57534a53: Bootstrap starting.
I20240503 23:27:55.745777 7556 tablet_bootstrap.cc:492] T 6cfc00700ec04e688a83c434db278210 P 9e3668e37348496ca15ab4da57534a53: Bootstrap replayed 1/1 log segments. Stats: ops{read=0 overwritten=0 applied=0 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 23:27:55.746186 7556 tablet_bootstrap.cc:492] T 6cfc00700ec04e688a83c434db278210 P 9e3668e37348496ca15ab4da57534a53: Bootstrap complete.
I20240503 23:27:55.746449 7556 ts_tablet_manager.cc:1367] T 6cfc00700ec04e688a83c434db278210 P 9e3668e37348496ca15ab4da57534a53: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.004s
I20240503 23:27:55.747150 7556 raft_consensus.cc:348] T 6cfc00700ec04e688a83c434db278210 P 9e3668e37348496ca15ab4da57534a53 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e3668e37348496ca15ab4da57534a53" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 41837 } } peers { permanent_uuid: "ad2dd25e957d4536bdc22a0df2b7263f" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 38457 } } peers { permanent_uuid: "63d49bebfcb9422b87754fb258e16a97" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 41385 } }
I20240503 23:27:55.747345 7556 raft_consensus.cc:729] T 6cfc00700ec04e688a83c434db278210 P 9e3668e37348496ca15ab4da57534a53 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9e3668e37348496ca15ab4da57534a53, State: Initialized, Role: FOLLOWER
I20240503 23:27:55.747654 7556 consensus_queue.cc:260] T 6cfc00700ec04e688a83c434db278210 P 9e3668e37348496ca15ab4da57534a53 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e3668e37348496ca15ab4da57534a53" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 41837 } } peers { permanent_uuid: "ad2dd25e957d4536bdc22a0df2b7263f" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 38457 } } peers { permanent_uuid: "63d49bebfcb9422b87754fb258e16a97" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 41385 } }
I20240503 23:27:55.748114 7556 ts_tablet_manager.cc:1398] T 6cfc00700ec04e688a83c434db278210 P 9e3668e37348496ca15ab4da57534a53: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 23:27:55.748383 7556 tablet_bootstrap.cc:492] T 95f3282b21394b11807125a214e8697a P 9e3668e37348496ca15ab4da57534a53: Bootstrap starting.
I20240503 23:27:55.754756 7556 tablet_bootstrap.cc:492] T 95f3282b21394b11807125a214e8697a P 9e3668e37348496ca15ab4da57534a53: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 23:27:55.755478 7556 tablet_bootstrap.cc:492] T 95f3282b21394b11807125a214e8697a P 9e3668e37348496ca15ab4da57534a53: Bootstrap complete.
I20240503 23:27:55.755844 7556 ts_tablet_manager.cc:1367] T 95f3282b21394b11807125a214e8697a P 9e3668e37348496ca15ab4da57534a53: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20240503 23:27:55.756841 7556 raft_consensus.cc:348] T 95f3282b21394b11807125a214e8697a P 9e3668e37348496ca15ab4da57534a53 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e3668e37348496ca15ab4da57534a53" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 41837 } } peers { permanent_uuid: "ad2dd25e957d4536bdc22a0df2b7263f" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 38457 } } peers { permanent_uuid: "63d49bebfcb9422b87754fb258e16a97" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 41385 } }
I20240503 23:27:55.757161 7556 raft_consensus.cc:729] T 95f3282b21394b11807125a214e8697a P 9e3668e37348496ca15ab4da57534a53 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9e3668e37348496ca15ab4da57534a53, State: Initialized, Role: FOLLOWER
I20240503 23:27:55.757493 7556 consensus_queue.cc:260] T 95f3282b21394b11807125a214e8697a P 9e3668e37348496ca15ab4da57534a53 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e3668e37348496ca15ab4da57534a53" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 41837 } } peers { permanent_uuid: "ad2dd25e957d4536bdc22a0df2b7263f" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 38457 } } peers { permanent_uuid: "63d49bebfcb9422b87754fb258e16a97" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 41385 } }
I20240503 23:27:55.757897 400 rpc_server.cc:307] RPC server started. Bound to: 127.0.100.1:41837
I20240503 23:27:55.757957 7619 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.100.1:41837 every 8 connection(s)
I20240503 23:27:55.758309 7556 ts_tablet_manager.cc:1398] T 95f3282b21394b11807125a214e8697a P 9e3668e37348496ca15ab4da57534a53: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20240503 23:27:55.758601 7556 tablet_bootstrap.cc:492] T da50d2c38f2b4b1aa91ca1d5b92d32f8 P 9e3668e37348496ca15ab4da57534a53: Bootstrap starting.
I20240503 23:27:55.763947 7556 tablet_bootstrap.cc:492] T da50d2c38f2b4b1aa91ca1d5b92d32f8 P 9e3668e37348496ca15ab4da57534a53: Bootstrap replayed 1/1 log segments. Stats: ops{read=0 overwritten=0 applied=0 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 23:27:55.764326 7556 tablet_bootstrap.cc:492] T da50d2c38f2b4b1aa91ca1d5b92d32f8 P 9e3668e37348496ca15ab4da57534a53: Bootstrap complete.
I20240503 23:27:55.764636 7556 ts_tablet_manager.cc:1367] T da50d2c38f2b4b1aa91ca1d5b92d32f8 P 9e3668e37348496ca15ab4da57534a53: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.004s
I20240503 23:27:55.765359 7556 raft_consensus.cc:348] T da50d2c38f2b4b1aa91ca1d5b92d32f8 P 9e3668e37348496ca15ab4da57534a53 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e3668e37348496ca15ab4da57534a53" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 41837 } } peers { permanent_uuid: "ad2dd25e957d4536bdc22a0df2b7263f" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 38457 } } peers { permanent_uuid: "63d49bebfcb9422b87754fb258e16a97" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 41385 } }
I20240503 23:27:55.765622 7556 raft_consensus.cc:729] T da50d2c38f2b4b1aa91ca1d5b92d32f8 P 9e3668e37348496ca15ab4da57534a53 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9e3668e37348496ca15ab4da57534a53, State: Initialized, Role: FOLLOWER
I20240503 23:27:55.765913 7556 consensus_queue.cc:260] T da50d2c38f2b4b1aa91ca1d5b92d32f8 P 9e3668e37348496ca15ab4da57534a53 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e3668e37348496ca15ab4da57534a53" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 41837 } } peers { permanent_uuid: "ad2dd25e957d4536bdc22a0df2b7263f" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 38457 } } peers { permanent_uuid: "63d49bebfcb9422b87754fb258e16a97" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 41385 } }
I20240503 23:27:55.766469 7556 ts_tablet_manager.cc:1398] T da50d2c38f2b4b1aa91ca1d5b92d32f8 P 9e3668e37348496ca15ab4da57534a53: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 23:27:55.766748 7556 tablet_bootstrap.cc:492] T 28c504576e8147d2ab6418920edf5e2f P 9e3668e37348496ca15ab4da57534a53: Bootstrap starting.
I20240503 23:27:55.767580 7620 heartbeater.cc:347] Connected to a master server at 127.0.100.62:43625
I20240503 23:27:55.767740 7620 heartbeater.cc:464] Registering TS with master...
I20240503 23:27:55.768179 7620 heartbeater.cc:511] Master 127.0.100.62:43625 requested a full tablet report, sending...
I20240503 23:27:55.770093 7082 ts_manager.cc:194] Re-registered known tserver with Master: 9e3668e37348496ca15ab4da57534a53 (127.0.100.1:41837)
I20240503 23:27:55.772073 7556 tablet_bootstrap.cc:492] T 28c504576e8147d2ab6418920edf5e2f P 9e3668e37348496ca15ab4da57534a53: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 23:27:55.772459 7556 tablet_bootstrap.cc:492] T 28c504576e8147d2ab6418920edf5e2f P 9e3668e37348496ca15ab4da57534a53: Bootstrap complete.
I20240503 23:27:55.772727 7556 ts_tablet_manager.cc:1367] T 28c504576e8147d2ab6418920edf5e2f P 9e3668e37348496ca15ab4da57534a53: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.004s
I20240503 23:27:55.773286 7082 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:34388
I20240503 23:27:55.773399 7556 raft_consensus.cc:348] T 28c504576e8147d2ab6418920edf5e2f P 9e3668e37348496ca15ab4da57534a53 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e3668e37348496ca15ab4da57534a53" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 41837 } } peers { permanent_uuid: "ad2dd25e957d4536bdc22a0df2b7263f" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 38457 } } peers { permanent_uuid: "63d49bebfcb9422b87754fb258e16a97" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 41385 } }
I20240503 23:27:55.773594 7556 raft_consensus.cc:729] T 28c504576e8147d2ab6418920edf5e2f P 9e3668e37348496ca15ab4da57534a53 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9e3668e37348496ca15ab4da57534a53, State: Initialized, Role: FOLLOWER
I20240503 23:27:55.773847 7556 consensus_queue.cc:260] T 28c504576e8147d2ab6418920edf5e2f P 9e3668e37348496ca15ab4da57534a53 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e3668e37348496ca15ab4da57534a53" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 41837 } } peers { permanent_uuid: "ad2dd25e957d4536bdc22a0df2b7263f" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 38457 } } peers { permanent_uuid: "63d49bebfcb9422b87754fb258e16a97" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 41385 } }
I20240503 23:27:55.774367 7556 ts_tablet_manager.cc:1398] T 28c504576e8147d2ab6418920edf5e2f P 9e3668e37348496ca15ab4da57534a53: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240503 23:27:55.774865 7620 heartbeater.cc:503] Master 127.0.100.62:43625 was elected leader, sending a full tablet report...
I20240503 23:27:55.799443 7583 tablet_service.cc:1514] Processing DeleteTablet for tablet 01ffeaa637ee4aa7a172de6b3e87e2e7 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 23:27:55 UTC) from {username='slave'} at 127.0.0.1:55522
I20240503 23:27:55.799491 7582 tablet_service.cc:1514] Processing DeleteTablet for tablet da50d2c38f2b4b1aa91ca1d5b92d32f8 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 23:27:55 UTC) from {username='slave'} at 127.0.0.1:55522
I20240503 23:27:55.800277 7626 tablet_replica.cc:330] stopping tablet replica
I20240503 23:27:55.800521 7626 raft_consensus.cc:2229] T 01ffeaa637ee4aa7a172de6b3e87e2e7 P 9e3668e37348496ca15ab4da57534a53 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 23:27:55.800724 7626 pending_rounds.cc:62] T 01ffeaa637ee4aa7a172de6b3e87e2e7 P 9e3668e37348496ca15ab4da57534a53: Trying to abort 1 pending ops.
I20240503 23:27:55.800769 7626 pending_rounds.cc:68] T 01ffeaa637ee4aa7a172de6b3e87e2e7 P 9e3668e37348496ca15ab4da57534a53: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7023734269089669120 op_type: NO_OP noop_request { }
I20240503 23:27:55.800941 7626 raft_consensus.cc:2866] T 01ffeaa637ee4aa7a172de6b3e87e2e7 P 9e3668e37348496ca15ab4da57534a53 [term 1 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted
I20240503 23:27:55.801043 7626 raft_consensus.cc:2258] T 01ffeaa637ee4aa7a172de6b3e87e2e7 P 9e3668e37348496ca15ab4da57534a53 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 23:27:55.801908 7626 ts_tablet_manager.cc:1875] T 01ffeaa637ee4aa7a172de6b3e87e2e7 P 9e3668e37348496ca15ab4da57534a53: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 23:27:55.807057 7626 ts_tablet_manager.cc:1888] T 01ffeaa637ee4aa7a172de6b3e87e2e7 P 9e3668e37348496ca15ab4da57534a53: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240503 23:27:55.807228 7626 log.cc:1196] T 01ffeaa637ee4aa7a172de6b3e87e2e7 P 9e3668e37348496ca15ab4da57534a53: Deleting WAL directory at /tmp/dist-test-taskZhIiPz/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestDeletedTables.1714778688451250-400-0/minicluster-data/ts-0-root/wals/01ffeaa637ee4aa7a172de6b3e87e2e7
I20240503 23:27:55.807574 7626 ts_tablet_manager.cc:1909] T 01ffeaa637ee4aa7a172de6b3e87e2e7 P 9e3668e37348496ca15ab4da57534a53: Deleting consensus metadata
I20240503 23:27:55.808672 7626 tablet_replica.cc:330] stopping tablet replica
I20240503 23:27:55.808892 7626 raft_consensus.cc:2229] T da50d2c38f2b4b1aa91ca1d5b92d32f8 P 9e3668e37348496ca15ab4da57534a53 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 23:27:55.808884 7068 catalog_manager.cc:4830] TS 9e3668e37348496ca15ab4da57534a53 (127.0.100.1:41837): tablet 01ffeaa637ee4aa7a172de6b3e87e2e7 (table dugtrio [id=1e717799f69545008dc79566448dcf9f]) successfully deleted
I20240503 23:27:55.809139 7626 raft_consensus.cc:2258] T da50d2c38f2b4b1aa91ca1d5b92d32f8 P 9e3668e37348496ca15ab4da57534a53 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 23:27:55.809815 7626 ts_tablet_manager.cc:1875] T da50d2c38f2b4b1aa91ca1d5b92d32f8 P 9e3668e37348496ca15ab4da57534a53: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 23:27:55.811120 7583 tablet_service.cc:1514] Processing DeleteTablet for tablet 6cfc00700ec04e688a83c434db278210 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 23:27:55 UTC) from {username='slave'} at 127.0.0.1:55522
I20240503 23:27:55.814857 7626 ts_tablet_manager.cc:1888] T da50d2c38f2b4b1aa91ca1d5b92d32f8 P 9e3668e37348496ca15ab4da57534a53: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240503 23:27:55.814982 7626 log.cc:1196] T da50d2c38f2b4b1aa91ca1d5b92d32f8 P 9e3668e37348496ca15ab4da57534a53: Deleting WAL directory at /tmp/dist-test-taskZhIiPz/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestDeletedTables.1714778688451250-400-0/minicluster-data/ts-0-root/wals/da50d2c38f2b4b1aa91ca1d5b92d32f8
I20240503 23:27:55.815297 7626 ts_tablet_manager.cc:1909] T da50d2c38f2b4b1aa91ca1d5b92d32f8 P 9e3668e37348496ca15ab4da57534a53: Deleting consensus metadata
I20240503 23:27:55.816273 7626 tablet_replica.cc:330] stopping tablet replica
I20240503 23:27:55.816435 7068 catalog_manager.cc:4830] TS 9e3668e37348496ca15ab4da57534a53 (127.0.100.1:41837): tablet da50d2c38f2b4b1aa91ca1d5b92d32f8 (table dugtrio [id=1e717799f69545008dc79566448dcf9f]) successfully deleted
I20240503 23:27:55.816498 7626 raft_consensus.cc:2229] T 6cfc00700ec04e688a83c434db278210 P 9e3668e37348496ca15ab4da57534a53 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 23:27:55.816671 7626 raft_consensus.cc:2258] T 6cfc00700ec04e688a83c434db278210 P 9e3668e37348496ca15ab4da57534a53 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 23:27:55.817353 7626 ts_tablet_manager.cc:1875] T 6cfc00700ec04e688a83c434db278210 P 9e3668e37348496ca15ab4da57534a53: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 23:27:55.822496 7626 ts_tablet_manager.cc:1888] T 6cfc00700ec04e688a83c434db278210 P 9e3668e37348496ca15ab4da57534a53: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240503 23:27:55.822631 7626 log.cc:1196] T 6cfc00700ec04e688a83c434db278210 P 9e3668e37348496ca15ab4da57534a53: Deleting WAL directory at /tmp/dist-test-taskZhIiPz/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestDeletedTables.1714778688451250-400-0/minicluster-data/ts-0-root/wals/6cfc00700ec04e688a83c434db278210
I20240503 23:27:55.822893 7626 ts_tablet_manager.cc:1909] T 6cfc00700ec04e688a83c434db278210 P 9e3668e37348496ca15ab4da57534a53: Deleting consensus metadata
I20240503 23:27:55.824054 7068 catalog_manager.cc:4830] TS 9e3668e37348496ca15ab4da57534a53 (127.0.100.1:41837): tablet 6cfc00700ec04e688a83c434db278210 (table dugtrio [id=1e717799f69545008dc79566448dcf9f]) successfully deleted
I20240503 23:27:55.828052 7448 raft_consensus.cc:482] T fe71c54bf82048669579628b317224e7 P ad2dd25e957d4536bdc22a0df2b7263f [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 9e3668e37348496ca15ab4da57534a53)
I20240503 23:27:55.828256 7448 raft_consensus.cc:504] T fe71c54bf82048669579628b317224e7 P ad2dd25e957d4536bdc22a0df2b7263f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad2dd25e957d4536bdc22a0df2b7263f" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 38457 } } peers { permanent_uuid: "9e3668e37348496ca15ab4da57534a53" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 41837 } } peers { permanent_uuid: "63d49bebfcb9422b87754fb258e16a97" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 41385 } }
I20240503 23:27:55.829130 7448 leader_election.cc:288] T fe71c54bf82048669579628b317224e7 P ad2dd25e957d4536bdc22a0df2b7263f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9e3668e37348496ca15ab4da57534a53 (127.0.100.1:41837), 63d49bebfcb9422b87754fb258e16a97 (127.0.100.3:41385)
I20240503 23:27:55.829667 7386 raft_consensus.cc:482] T fe71c54bf82048669579628b317224e7 P 63d49bebfcb9422b87754fb258e16a97 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 9e3668e37348496ca15ab4da57534a53)
I20240503 23:27:55.829816 7332 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe71c54bf82048669579628b317224e7" candidate_uuid: "ad2dd25e957d4536bdc22a0df2b7263f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "63d49bebfcb9422b87754fb258e16a97" is_pre_election: true
I20240503 23:27:55.829856 7386 raft_consensus.cc:504] T fe71c54bf82048669579628b317224e7 P 63d49bebfcb9422b87754fb258e16a97 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad2dd25e957d4536bdc22a0df2b7263f" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 38457 } } peers { permanent_uuid: "9e3668e37348496ca15ab4da57534a53" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 41837 } } peers { permanent_uuid: "63d49bebfcb9422b87754fb258e16a97" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 41385 } }
I20240503 23:27:55.830096 7332 raft_consensus.cc:2454] T fe71c54bf82048669579628b317224e7 P 63d49bebfcb9422b87754fb258e16a97 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ad2dd25e957d4536bdc22a0df2b7263f in term 1.
I20240503 23:27:55.830456 7386 leader_election.cc:288] T fe71c54bf82048669579628b317224e7 P 63d49bebfcb9422b87754fb258e16a97 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ad2dd25e957d4536bdc22a0df2b7263f (127.0.100.2:38457), 9e3668e37348496ca15ab4da57534a53 (127.0.100.1:41837)
I20240503 23:27:55.830627 7222 leader_election.cc:302] T fe71c54bf82048669579628b317224e7 P ad2dd25e957d4536bdc22a0df2b7263f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 63d49bebfcb9422b87754fb258e16a97, ad2dd25e957d4536bdc22a0df2b7263f; no voters:
I20240503 23:27:55.830899 7448 raft_consensus.cc:2781] T fe71c54bf82048669579628b317224e7 P ad2dd25e957d4536bdc22a0df2b7263f [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240503 23:27:55.830966 7448 raft_consensus.cc:482] T fe71c54bf82048669579628b317224e7 P ad2dd25e957d4536bdc22a0df2b7263f [term 1 FOLLOWER]: Starting leader election (detected failure of leader 9e3668e37348496ca15ab4da57534a53)
I20240503 23:27:55.831053 7448 raft_consensus.cc:3037] T fe71c54bf82048669579628b317224e7 P ad2dd25e957d4536bdc22a0df2b7263f [term 1 FOLLOWER]: Advancing to term 2
I20240503 23:27:55.831568 7258 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe71c54bf82048669579628b317224e7" candidate_uuid: "63d49bebfcb9422b87754fb258e16a97" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ad2dd25e957d4536bdc22a0df2b7263f" is_pre_election: true
I20240503 23:27:55.835606 7448 raft_consensus.cc:504] T fe71c54bf82048669579628b317224e7 P ad2dd25e957d4536bdc22a0df2b7263f [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad2dd25e957d4536bdc22a0df2b7263f" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 38457 } } peers { permanent_uuid: "9e3668e37348496ca15ab4da57534a53" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 41837 } } peers { permanent_uuid: "63d49bebfcb9422b87754fb258e16a97" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 41385 } }
I20240503 23:27:55.836230 7258 raft_consensus.cc:2379] T fe71c54bf82048669579628b317224e7 P ad2dd25e957d4536bdc22a0df2b7263f [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 63d49bebfcb9422b87754fb258e16a97 in current term 2: Already voted for candidate ad2dd25e957d4536bdc22a0df2b7263f in this term.
I20240503 23:27:55.838838 7332 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe71c54bf82048669579628b317224e7" candidate_uuid: "ad2dd25e957d4536bdc22a0df2b7263f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "63d49bebfcb9422b87754fb258e16a97"
I20240503 23:27:55.839174 7332 raft_consensus.cc:3037] T fe71c54bf82048669579628b317224e7 P 63d49bebfcb9422b87754fb258e16a97 [term 1 FOLLOWER]: Advancing to term 2
I20240503 23:27:55.842016 7332 raft_consensus.cc:2454] T fe71c54bf82048669579628b317224e7 P 63d49bebfcb9422b87754fb258e16a97 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ad2dd25e957d4536bdc22a0df2b7263f in term 2.
I20240503 23:27:55.842094 7448 leader_election.cc:288] T fe71c54bf82048669579628b317224e7 P ad2dd25e957d4536bdc22a0df2b7263f [CANDIDATE]: Term 2 election: Requested vote from peers 9e3668e37348496ca15ab4da57534a53 (127.0.100.1:41837), 63d49bebfcb9422b87754fb258e16a97 (127.0.100.3:41385)
I20240503 23:27:55.842559 7222 leader_election.cc:302] T fe71c54bf82048669579628b317224e7 P ad2dd25e957d4536bdc22a0df2b7263f [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 63d49bebfcb9422b87754fb258e16a97, ad2dd25e957d4536bdc22a0df2b7263f; no voters:
I20240503 23:27:55.842911 7448 raft_consensus.cc:2781] T fe71c54bf82048669579628b317224e7 P ad2dd25e957d4536bdc22a0df2b7263f [term 2 FOLLOWER]: Leader election won for term 2
I20240503 23:27:55.843078 7448 raft_consensus.cc:686] T fe71c54bf82048669579628b317224e7 P ad2dd25e957d4536bdc22a0df2b7263f [term 2 LEADER]: Becoming Leader. State: Replica: ad2dd25e957d4536bdc22a0df2b7263f, State: Running, Role: LEADER
I20240503 23:27:55.843400 7448 consensus_queue.cc:237] T fe71c54bf82048669579628b317224e7 P ad2dd25e957d4536bdc22a0df2b7263f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad2dd25e957d4536bdc22a0df2b7263f" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 38457 } } peers { permanent_uuid: "9e3668e37348496ca15ab4da57534a53" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 41837 } } peers { permanent_uuid: "63d49bebfcb9422b87754fb258e16a97" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 41385 } }
I20240503 23:27:55.843628 7594 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe71c54bf82048669579628b317224e7" candidate_uuid: "ad2dd25e957d4536bdc22a0df2b7263f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9e3668e37348496ca15ab4da57534a53" is_pre_election: true
I20240503 23:27:55.843838 7594 raft_consensus.cc:2454] T fe71c54bf82048669579628b317224e7 P 9e3668e37348496ca15ab4da57534a53 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ad2dd25e957d4536bdc22a0df2b7263f in term 1.
I20240503 23:27:55.844369 7594 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe71c54bf82048669579628b317224e7" candidate_uuid: "ad2dd25e957d4536bdc22a0df2b7263f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9e3668e37348496ca15ab4da57534a53"
I20240503 23:27:55.844583 7594 raft_consensus.cc:3037] T fe71c54bf82048669579628b317224e7 P 9e3668e37348496ca15ab4da57534a53 [term 1 FOLLOWER]: Advancing to term 2
I20240503 23:27:55.846922 7386 raft_consensus.cc:482] T 95f3282b21394b11807125a214e8697a P 63d49bebfcb9422b87754fb258e16a97 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 9e3668e37348496ca15ab4da57534a53)
I20240503 23:27:55.847069 7386 raft_consensus.cc:504] T 95f3282b21394b11807125a214e8697a P 63d49bebfcb9422b87754fb258e16a97 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e3668e37348496ca15ab4da57534a53" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 41837 } } peers { permanent_uuid: "ad2dd25e957d4536bdc22a0df2b7263f" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 38457 } } peers { permanent_uuid: "63d49bebfcb9422b87754fb258e16a97" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 41385 } }
I20240503 23:27:55.847749 7386 leader_election.cc:288] T 95f3282b21394b11807125a214e8697a P 63d49bebfcb9422b87754fb258e16a97 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9e3668e37348496ca15ab4da57534a53 (127.0.100.1:41837), ad2dd25e957d4536bdc22a0df2b7263f (127.0.100.2:38457)
I20240503 23:27:55.847879 7594 raft_consensus.cc:2454] T fe71c54bf82048669579628b317224e7 P 9e3668e37348496ca15ab4da57534a53 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ad2dd25e957d4536bdc22a0df2b7263f in term 2.
I20240503 23:27:55.848183 7258 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "95f3282b21394b11807125a214e8697a" candidate_uuid: "63d49bebfcb9422b87754fb258e16a97" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ad2dd25e957d4536bdc22a0df2b7263f" is_pre_election: true
I20240503 23:27:55.848393 7258 raft_consensus.cc:2454] T 95f3282b21394b11807125a214e8697a P ad2dd25e957d4536bdc22a0df2b7263f [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 63d49bebfcb9422b87754fb258e16a97 in term 1.
I20240503 23:27:55.848858 7299 leader_election.cc:302] T 95f3282b21394b11807125a214e8697a P 63d49bebfcb9422b87754fb258e16a97 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 63d49bebfcb9422b87754fb258e16a97, ad2dd25e957d4536bdc22a0df2b7263f; no voters:
I20240503 23:27:55.848919 7082 catalog_manager.cc:5484] T fe71c54bf82048669579628b317224e7 P ad2dd25e957d4536bdc22a0df2b7263f reported cstate change: term changed from 1 to 2, leader changed from 9e3668e37348496ca15ab4da57534a53 (127.0.100.1) to ad2dd25e957d4536bdc22a0df2b7263f (127.0.100.2). New cstate: current_term: 2 leader_uuid: "ad2dd25e957d4536bdc22a0df2b7263f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad2dd25e957d4536bdc22a0df2b7263f" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 38457 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9e3668e37348496ca15ab4da57534a53" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 41837 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "63d49bebfcb9422b87754fb258e16a97" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 41385 } health_report { overall_health: UNKNOWN } } }
I20240503 23:27:55.849164 7386 raft_consensus.cc:2781] T 95f3282b21394b11807125a214e8697a P 63d49bebfcb9422b87754fb258e16a97 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240503 23:27:55.849321 7386 raft_consensus.cc:482] T 95f3282b21394b11807125a214e8697a P 63d49bebfcb9422b87754fb258e16a97 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 9e3668e37348496ca15ab4da57534a53)
I20240503 23:27:55.849409 7386 raft_consensus.cc:3037] T 95f3282b21394b11807125a214e8697a P 63d49bebfcb9422b87754fb258e16a97 [term 1 FOLLOWER]: Advancing to term 2
I20240503 23:27:55.850371 7594 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe71c54bf82048669579628b317224e7" candidate_uuid: "63d49bebfcb9422b87754fb258e16a97" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9e3668e37348496ca15ab4da57534a53" is_pre_election: true
I20240503 23:27:55.850440 7593 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "95f3282b21394b11807125a214e8697a" candidate_uuid: "63d49bebfcb9422b87754fb258e16a97" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9e3668e37348496ca15ab4da57534a53" is_pre_election: true
I20240503 23:27:55.850647 7594 raft_consensus.cc:2379] T fe71c54bf82048669579628b317224e7 P 9e3668e37348496ca15ab4da57534a53 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 63d49bebfcb9422b87754fb258e16a97 in current term 2: Already voted for candidate ad2dd25e957d4536bdc22a0df2b7263f in this term.
I20240503 23:27:55.850625 7593 raft_consensus.cc:2454] T 95f3282b21394b11807125a214e8697a P 9e3668e37348496ca15ab4da57534a53 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 63d49bebfcb9422b87754fb258e16a97 in term 1.
I20240503 23:27:55.851181 7298 leader_election.cc:302] T fe71c54bf82048669579628b317224e7 P 63d49bebfcb9422b87754fb258e16a97 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 63d49bebfcb9422b87754fb258e16a97; no voters: 9e3668e37348496ca15ab4da57534a53, ad2dd25e957d4536bdc22a0df2b7263f
I20240503 23:27:55.852020 7630 raft_consensus.cc:2726] T fe71c54bf82048669579628b317224e7 P 63d49bebfcb9422b87754fb258e16a97 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240503 23:27:55.852728 7386 raft_consensus.cc:504] T 95f3282b21394b11807125a214e8697a P 63d49bebfcb9422b87754fb258e16a97 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e3668e37348496ca15ab4da57534a53" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 41837 } } peers { permanent_uuid: "ad2dd25e957d4536bdc22a0df2b7263f" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 38457 } } peers { permanent_uuid: "63d49bebfcb9422b87754fb258e16a97" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 41385 } }
I20240503 23:27:55.853447 7386 leader_election.cc:288] T 95f3282b21394b11807125a214e8697a P 63d49bebfcb9422b87754fb258e16a97 [CANDIDATE]: Term 2 election: Requested vote from peers 9e3668e37348496ca15ab4da57534a53 (127.0.100.1:41837), ad2dd25e957d4536bdc22a0df2b7263f (127.0.100.2:38457)
I20240503 23:27:55.853888 7593 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "95f3282b21394b11807125a214e8697a" candidate_uuid: "63d49bebfcb9422b87754fb258e16a97" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9e3668e37348496ca15ab4da57534a53"
I20240503 23:27:55.854081 7593 raft_consensus.cc:3037] T 95f3282b21394b11807125a214e8697a P 9e3668e37348496ca15ab4da57534a53 [term 1 FOLLOWER]: Advancing to term 2
I20240503 23:27:55.856350 7593 raft_consensus.cc:2454] T 95f3282b21394b11807125a214e8697a P 9e3668e37348496ca15ab4da57534a53 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 63d49bebfcb9422b87754fb258e16a97 in term 2.
I20240503 23:27:55.856734 7258 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "95f3282b21394b11807125a214e8697a" candidate_uuid: "63d49bebfcb9422b87754fb258e16a97" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ad2dd25e957d4536bdc22a0df2b7263f"
I20240503 23:27:55.857012 7258 raft_consensus.cc:3037] T 95f3282b21394b11807125a214e8697a P ad2dd25e957d4536bdc22a0df2b7263f [term 1 FOLLOWER]: Advancing to term 2
I20240503 23:27:55.857050 7298 leader_election.cc:302] T 95f3282b21394b11807125a214e8697a P 63d49bebfcb9422b87754fb258e16a97 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 63d49bebfcb9422b87754fb258e16a97, 9e3668e37348496ca15ab4da57534a53; no voters:
I20240503 23:27:55.857393 7386 raft_consensus.cc:2781] T 95f3282b21394b11807125a214e8697a P 63d49bebfcb9422b87754fb258e16a97 [term 2 FOLLOWER]: Leader election won for term 2
I20240503 23:27:55.857568 7386 raft_consensus.cc:686] T 95f3282b21394b11807125a214e8697a P 63d49bebfcb9422b87754fb258e16a97 [term 2 LEADER]: Becoming Leader. State: Replica: 63d49bebfcb9422b87754fb258e16a97, State: Running, Role: LEADER
I20240503 23:27:55.857929 7386 consensus_queue.cc:237] T 95f3282b21394b11807125a214e8697a P 63d49bebfcb9422b87754fb258e16a97 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e3668e37348496ca15ab4da57534a53" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 41837 } } peers { permanent_uuid: "ad2dd25e957d4536bdc22a0df2b7263f" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 38457 } } peers { permanent_uuid: "63d49bebfcb9422b87754fb258e16a97" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 41385 } }
I20240503 23:27:55.860020 7258 raft_consensus.cc:2454] T 95f3282b21394b11807125a214e8697a P ad2dd25e957d4536bdc22a0df2b7263f [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 63d49bebfcb9422b87754fb258e16a97 in term 2.
I20240503 23:27:55.861122 7082 catalog_manager.cc:5484] T 95f3282b21394b11807125a214e8697a P 63d49bebfcb9422b87754fb258e16a97 reported cstate change: term changed from 1 to 2, leader changed from 9e3668e37348496ca15ab4da57534a53 (127.0.100.1) to 63d49bebfcb9422b87754fb258e16a97 (127.0.100.3). New cstate: current_term: 2 leader_uuid: "63d49bebfcb9422b87754fb258e16a97" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e3668e37348496ca15ab4da57534a53" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 41837 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ad2dd25e957d4536bdc22a0df2b7263f" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 38457 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "63d49bebfcb9422b87754fb258e16a97" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 41385 } health_report { overall_health: HEALTHY } } }
I20240503 23:27:55.916044 7386 consensus_queue.cc:579] T 27c38e75002d42688eac2cb1f13c66d4 P 63d49bebfcb9422b87754fb258e16a97 [LEADER]: Leader has been unable to successfully communicate with peer 9e3668e37348496ca15ab4da57534a53 for more than 1 seconds (1.578s)
I20240503 23:27:55.920658 7332 consensus_queue.cc:237] T 27c38e75002d42688eac2cb1f13c66d4 P 63d49bebfcb9422b87754fb258e16a97 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9e3668e37348496ca15ab4da57534a53" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 41837 } } peers { permanent_uuid: "ad2dd25e957d4536bdc22a0df2b7263f" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 38457 } } peers { permanent_uuid: "63d49bebfcb9422b87754fb258e16a97" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 41385 } } peers { permanent_uuid: "9c15c9d698fe4fa4a7c458f2e51ab5f9" member_type: NON_VOTER last_known_addr { host: "127.0.100.4" port: 33075 } attrs { promote: true } }
I20240503 23:27:55.921051 7583 tablet_service.cc:1514] Processing DeleteTablet for tablet 8269340eb54c455c95d80065a480be0f with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 23:27:55 UTC) from {username='slave'} at 127.0.0.1:55522
I20240503 23:27:55.921425 7626 tablet_replica.cc:330] stopping tablet replica
I20240503 23:27:55.921684 7626 raft_consensus.cc:2229] T 8269340eb54c455c95d80065a480be0f P 9e3668e37348496ca15ab4da57534a53 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 23:27:55.921911 7626 pending_rounds.cc:62] T 8269340eb54c455c95d80065a480be0f P 9e3668e37348496ca15ab4da57534a53: Trying to abort 1 pending ops.
I20240503 23:27:55.921979 7626 pending_rounds.cc:68] T 8269340eb54c455c95d80065a480be0f P 9e3668e37348496ca15ab4da57534a53: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7023734268728868864 op_type: NO_OP noop_request { }
I20240503 23:27:55.922164 7626 raft_consensus.cc:2866] T 8269340eb54c455c95d80065a480be0f P 9e3668e37348496ca15ab4da57534a53 [term 1 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted
I20240503 23:27:55.922312 7626 raft_consensus.cc:2258] T 8269340eb54c455c95d80065a480be0f P 9e3668e37348496ca15ab4da57534a53 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 23:27:55.923436 7626 ts_tablet_manager.cc:1875] T 8269340eb54c455c95d80065a480be0f P 9e3668e37348496ca15ab4da57534a53: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 23:27:55.924335 7593 raft_consensus.cc:1261] T 27c38e75002d42688eac2cb1f13c66d4 P 9e3668e37348496ca15ab4da57534a53 [term 1 FOLLOWER]: Refusing update from remote peer 63d49bebfcb9422b87754fb258e16a97: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 23:27:55.924345 7258 raft_consensus.cc:1261] T 27c38e75002d42688eac2cb1f13c66d4 P ad2dd25e957d4536bdc22a0df2b7263f [term 1 FOLLOWER]: Refusing update from remote peer 63d49bebfcb9422b87754fb258e16a97: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 23:27:55.925060 7632 consensus_queue.cc:1035] T 27c38e75002d42688eac2cb1f13c66d4 P 63d49bebfcb9422b87754fb258e16a97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9e3668e37348496ca15ab4da57534a53" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 41837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 23:27:55.925556 7630 consensus_queue.cc:1035] T 27c38e75002d42688eac2cb1f13c66d4 P 63d49bebfcb9422b87754fb258e16a97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ad2dd25e957d4536bdc22a0df2b7263f" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 38457 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 23:27:55.931636 7626 ts_tablet_manager.cc:1888] T 8269340eb54c455c95d80065a480be0f P 9e3668e37348496ca15ab4da57534a53: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240503 23:27:55.932008 7626 log.cc:1196] T 8269340eb54c455c95d80065a480be0f P 9e3668e37348496ca15ab4da57534a53: Deleting WAL directory at /tmp/dist-test-taskZhIiPz/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestDeletedTables.1714778688451250-400-0/minicluster-data/ts-0-root/wals/8269340eb54c455c95d80065a480be0f
I20240503 23:27:55.932636 7626 ts_tablet_manager.cc:1909] T 8269340eb54c455c95d80065a480be0f P 9e3668e37348496ca15ab4da57534a53: Deleting consensus metadata
W20240503 23:27:55.932860 7296 consensus_peers.cc:485] T 27c38e75002d42688eac2cb1f13c66d4 P 63d49bebfcb9422b87754fb258e16a97 -> Peer 9c15c9d698fe4fa4a7c458f2e51ab5f9 (127.0.100.4:33075): Couldn't send request to peer 9c15c9d698fe4fa4a7c458f2e51ab5f9. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 27c38e75002d42688eac2cb1f13c66d4. This is attempt 1: this message will repeat every 5th retry.
I20240503 23:27:55.933949 7068 catalog_manager.cc:4830] TS 9e3668e37348496ca15ab4da57534a53 (127.0.100.1:41837): tablet 8269340eb54c455c95d80065a480be0f (table dugtrio [id=1e717799f69545008dc79566448dcf9f]) successfully deleted
I20240503 23:27:55.934937 7632 raft_consensus.cc:2932] T 27c38e75002d42688eac2cb1f13c66d4 P 63d49bebfcb9422b87754fb258e16a97 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 9c15c9d698fe4fa4a7c458f2e51ab5f9 (127.0.100.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9e3668e37348496ca15ab4da57534a53" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 41837 } } peers { permanent_uuid: "ad2dd25e957d4536bdc22a0df2b7263f" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 38457 } } peers { permanent_uuid: "63d49bebfcb9422b87754fb258e16a97" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 41385 } } peers { permanent_uuid: "9c15c9d698fe4fa4a7c458f2e51ab5f9" member_type: NON_VOTER last_known_addr { host: "127.0.100.4" port: 33075 } attrs { promote: true } } }
I20240503 23:27:55.935698 7593 raft_consensus.cc:2932] T 27c38e75002d42688eac2cb1f13c66d4 P 9e3668e37348496ca15ab4da57534a53 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 9c15c9d698fe4fa4a7c458f2e51ab5f9 (127.0.100.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9e3668e37348496ca15ab4da57534a53" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 41837 } } peers { permanent_uuid: "ad2dd25e957d4536bdc22a0df2b7263f" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 38457 } } peers { permanent_uuid: "63d49bebfcb9422b87754fb258e16a97" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 41385 } } peers { permanent_uuid: "9c15c9d698fe4fa4a7c458f2e51ab5f9" member_type: NON_VOTER last_known_addr { host: "127.0.100.4" port: 33075 } attrs { promote: true } } }
I20240503 23:27:55.935684 7258 raft_consensus.cc:2932] T 27c38e75002d42688eac2cb1f13c66d4 P ad2dd25e957d4536bdc22a0df2b7263f [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 9c15c9d698fe4fa4a7c458f2e51ab5f9 (127.0.100.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9e3668e37348496ca15ab4da57534a53" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 41837 } } peers { permanent_uuid: "ad2dd25e957d4536bdc22a0df2b7263f" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 38457 } } peers { permanent_uuid: "63d49bebfcb9422b87754fb258e16a97" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 41385 } } peers { permanent_uuid: "9c15c9d698fe4fa4a7c458f2e51ab5f9" member_type: NON_VOTER last_known_addr { host: "127.0.100.4" port: 33075 } attrs { promote: true } } }
I20240503 23:27:55.939851 7066 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 27c38e75002d42688eac2cb1f13c66d4 with cas_config_opid_index -1: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20240503 23:27:55.941044 7082 catalog_manager.cc:5484] T 27c38e75002d42688eac2cb1f13c66d4 P 9e3668e37348496ca15ab4da57534a53 reported cstate change: config changed from index -1 to 2, NON_VOTER 9c15c9d698fe4fa4a7c458f2e51ab5f9 (127.0.100.4) added. New cstate: current_term: 1 leader_uuid: "63d49bebfcb9422b87754fb258e16a97" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9e3668e37348496ca15ab4da57534a53" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 41837 } } peers { permanent_uuid: "ad2dd25e957d4536bdc22a0df2b7263f" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 38457 } } peers { permanent_uuid: "63d49bebfcb9422b87754fb258e16a97" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 41385 } } peers { permanent_uuid: "9c15c9d698fe4fa4a7c458f2e51ab5f9" member_type: NON_VOTER last_known_addr { host: "127.0.100.4" port: 33075 } attrs { promote: true } } }
I20240503 23:27:55.947042 7332 consensus_queue.cc:237] T 27c38e75002d42688eac2cb1f13c66d4 P 63d49bebfcb9422b87754fb258e16a97 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "9e3668e37348496ca15ab4da57534a53" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 41837 } } peers { permanent_uuid: "ad2dd25e957d4536bdc22a0df2b7263f" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 38457 } } peers { permanent_uuid: "63d49bebfcb9422b87754fb258e16a97" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 41385 } }
I20240503 23:27:55.949383 7258 raft_consensus.cc:1261] T 27c38e75002d42688eac2cb1f13c66d4 P ad2dd25e957d4536bdc22a0df2b7263f [term 1 FOLLOWER]: Refusing update from remote peer 63d49bebfcb9422b87754fb258e16a97: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 23:27:55.950366 7630 consensus_queue.cc:1035] T 27c38e75002d42688eac2cb1f13c66d4 P 63d49bebfcb9422b87754fb258e16a97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ad2dd25e957d4536bdc22a0df2b7263f" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 38457 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 23:27:55.950624 7593 raft_consensus.cc:1261] T 27c38e75002d42688eac2cb1f13c66d4 P 9e3668e37348496ca15ab4da57534a53 [term 1 FOLLOWER]: Refusing update from remote peer 63d49bebfcb9422b87754fb258e16a97: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 23:27:55.951143 7630 consensus_queue.cc:1035] T 27c38e75002d42688eac2cb1f13c66d4 P 63d49bebfcb9422b87754fb258e16a97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9e3668e37348496ca15ab4da57534a53" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 41837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 23:27:55.952914 7386 raft_consensus.cc:2932] T 27c38e75002d42688eac2cb1f13c66d4 P 63d49bebfcb9422b87754fb258e16a97 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, NON_VOTER 9c15c9d698fe4fa4a7c458f2e51ab5f9 (127.0.100.4) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "9e3668e37348496ca15ab4da57534a53" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 41837 } } peers { permanent_uuid: "ad2dd25e957d4536bdc22a0df2b7263f" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 38457 } } peers { permanent_uuid: "63d49bebfcb9422b87754fb258e16a97" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 41385 } } }
I20240503 23:27:55.953575 7258 raft_consensus.cc:2932] T 27c38e75002d42688eac2cb1f13c66d4 P ad2dd25e957d4536bdc22a0df2b7263f [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, NON_VOTER 9c15c9d698fe4fa4a7c458f2e51ab5f9 (127.0.100.4) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "9e3668e37348496ca15ab4da57534a53" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 41837 } } peers { permanent_uuid: "ad2dd25e957d4536bdc22a0df2b7263f" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 38457 } } peers { permanent_uuid: "63d49bebfcb9422b87754fb258e16a97" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 41385 } } }
I20240503 23:27:55.953917 7593 raft_consensus.cc:2932] T 27c38e75002d42688eac2cb1f13c66d4 P 9e3668e37348496ca15ab4da57534a53 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, NON_VOTER 9c15c9d698fe4fa4a7c458f2e51ab5f9 (127.0.100.4) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "9e3668e37348496ca15ab4da57534a53" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 41837 } } peers { permanent_uuid: "ad2dd25e957d4536bdc22a0df2b7263f" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 38457 } } peers { permanent_uuid: "63d49bebfcb9422b87754fb258e16a97" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 41385 } } }
I20240503 23:27:55.956310 7066 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 27c38e75002d42688eac2cb1f13c66d4 with cas_config_opid_index 2: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 23:27:55.958343 7079 catalog_manager.cc:5484] T 27c38e75002d42688eac2cb1f13c66d4 P 63d49bebfcb9422b87754fb258e16a97 reported cstate change: config changed from index 2 to 3, NON_VOTER 9c15c9d698fe4fa4a7c458f2e51ab5f9 (127.0.100.4) evicted. New cstate: current_term: 1 leader_uuid: "63d49bebfcb9422b87754fb258e16a97" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "9e3668e37348496ca15ab4da57534a53" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 41837 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ad2dd25e957d4536bdc22a0df2b7263f" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 38457 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "63d49bebfcb9422b87754fb258e16a97" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 41385 } health_report { overall_health: HEALTHY } } }
I20240503 23:27:55.968739 7503 tablet_service.cc:1514] Processing DeleteTablet for tablet 27c38e75002d42688eac2cb1f13c66d4 with delete_type TABLET_DATA_TOMBSTONED (TS 9c15c9d698fe4fa4a7c458f2e51ab5f9 not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:44280
W20240503 23:27:55.970129 7066 catalog_manager.cc:4797] TS 9c15c9d698fe4fa4a7c458f2e51ab5f9 (127.0.100.4:33075): delete failed for tablet 27c38e75002d42688eac2cb1f13c66d4 because the tablet was not found. No further retry: Not found: Tablet not found: 27c38e75002d42688eac2cb1f13c66d4
I20240503 23:27:55.986214 7630 raft_consensus.cc:482] T 28c504576e8147d2ab6418920edf5e2f P 63d49bebfcb9422b87754fb258e16a97 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 9e3668e37348496ca15ab4da57534a53)
I20240503 23:27:55.986482 7630 raft_consensus.cc:504] T 28c504576e8147d2ab6418920edf5e2f P 63d49bebfcb9422b87754fb258e16a97 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e3668e37348496ca15ab4da57534a53" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 41837 } } peers { permanent_uuid: "ad2dd25e957d4536bdc22a0df2b7263f" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 38457 } } peers { permanent_uuid: "63d49bebfcb9422b87754fb258e16a97" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 41385 } }
I20240503 23:27:55.987371 7630 leader_election.cc:288] T 28c504576e8147d2ab6418920edf5e2f P 63d49bebfcb9422b87754fb258e16a97 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9e3668e37348496ca15ab4da57534a53 (127.0.100.1:41837), ad2dd25e957d4536bdc22a0df2b7263f (127.0.100.2:38457)
I20240503 23:27:55.988072 7593 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "28c504576e8147d2ab6418920edf5e2f" candidate_uuid: "63d49bebfcb9422b87754fb258e16a97" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9e3668e37348496ca15ab4da57534a53" is_pre_election: true
I20240503 23:27:55.988072 7258 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "28c504576e8147d2ab6418920edf5e2f" candidate_uuid: "63d49bebfcb9422b87754fb258e16a97" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ad2dd25e957d4536bdc22a0df2b7263f" is_pre_election: true
I20240503 23:27:55.988520 7258 raft_consensus.cc:2454] T 28c504576e8147d2ab6418920edf5e2f P ad2dd25e957d4536bdc22a0df2b7263f [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 63d49bebfcb9422b87754fb258e16a97 in term 1.
I20240503 23:27:55.988528 7593 raft_consensus.cc:2454] T 28c504576e8147d2ab6418920edf5e2f P 9e3668e37348496ca15ab4da57534a53 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 63d49bebfcb9422b87754fb258e16a97 in term 1.
I20240503 23:27:55.989209 7299 leader_election.cc:302] T 28c504576e8147d2ab6418920edf5e2f P 63d49bebfcb9422b87754fb258e16a97 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 63d49bebfcb9422b87754fb258e16a97, ad2dd25e957d4536bdc22a0df2b7263f; no voters:
I20240503 23:27:55.989553 7630 raft_consensus.cc:2781] T 28c504576e8147d2ab6418920edf5e2f P 63d49bebfcb9422b87754fb258e16a97 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240503 23:27:55.989701 7630 raft_consensus.cc:482] T 28c504576e8147d2ab6418920edf5e2f P 63d49bebfcb9422b87754fb258e16a97 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 9e3668e37348496ca15ab4da57534a53)
I20240503 23:27:55.989820 7630 raft_consensus.cc:3037] T 28c504576e8147d2ab6418920edf5e2f P 63d49bebfcb9422b87754fb258e16a97 [term 1 FOLLOWER]: Advancing to term 2
I20240503 23:27:55.992225 7630 raft_consensus.cc:504] T 28c504576e8147d2ab6418920edf5e2f P 63d49bebfcb9422b87754fb258e16a97 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e3668e37348496ca15ab4da57534a53" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 41837 } } peers { permanent_uuid: "ad2dd25e957d4536bdc22a0df2b7263f" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 38457 } } peers { permanent_uuid: "63d49bebfcb9422b87754fb258e16a97" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 41385 } }
I20240503 23:27:55.993041 7630 leader_election.cc:288] T 28c504576e8147d2ab6418920edf5e2f P 63d49bebfcb9422b87754fb258e16a97 [CANDIDATE]: Term 2 election: Requested vote from peers 9e3668e37348496ca15ab4da57534a53 (127.0.100.1:41837), ad2dd25e957d4536bdc22a0df2b7263f (127.0.100.2:38457)
I20240503 23:27:55.993780 7258 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "28c504576e8147d2ab6418920edf5e2f" candidate_uuid: "63d49bebfcb9422b87754fb258e16a97" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ad2dd25e957d4536bdc22a0df2b7263f"
I20240503 23:27:55.993780 7593 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "28c504576e8147d2ab6418920edf5e2f" candidate_uuid: "63d49bebfcb9422b87754fb258e16a97" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9e3668e37348496ca15ab4da57534a53"
I20240503 23:27:55.994084 7258 raft_consensus.cc:3037] T 28c504576e8147d2ab6418920edf5e2f P ad2dd25e957d4536bdc22a0df2b7263f [term 1 FOLLOWER]: Advancing to term 2
I20240503 23:27:55.994093 7593 raft_consensus.cc:3037] T 28c504576e8147d2ab6418920edf5e2f P 9e3668e37348496ca15ab4da57534a53 [term 1 FOLLOWER]: Advancing to term 2
I20240503 23:27:55.997471 7258 raft_consensus.cc:2454] T 28c504576e8147d2ab6418920edf5e2f P ad2dd25e957d4536bdc22a0df2b7263f [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 63d49bebfcb9422b87754fb258e16a97 in term 2.
I20240503 23:27:55.997471 7593 raft_consensus.cc:2454] T 28c504576e8147d2ab6418920edf5e2f P 9e3668e37348496ca15ab4da57534a53 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 63d49bebfcb9422b87754fb258e16a97 in term 2.
I20240503 23:27:55.998231 7299 leader_election.cc:302] T 28c504576e8147d2ab6418920edf5e2f P 63d49bebfcb9422b87754fb258e16a97 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 63d49bebfcb9422b87754fb258e16a97, ad2dd25e957d4536bdc22a0df2b7263f; no voters:
I20240503 23:27:55.998616 7630 raft_consensus.cc:2781] T 28c504576e8147d2ab6418920edf5e2f P 63d49bebfcb9422b87754fb258e16a97 [term 2 FOLLOWER]: Leader election won for term 2
I20240503 23:27:55.998791 7630 raft_consensus.cc:686] T 28c504576e8147d2ab6418920edf5e2f P 63d49bebfcb9422b87754fb258e16a97 [term 2 LEADER]: Becoming Leader. State: Replica: 63d49bebfcb9422b87754fb258e16a97, State: Running, Role: LEADER
I20240503 23:27:55.999388 7630 consensus_queue.cc:237] T 28c504576e8147d2ab6418920edf5e2f P 63d49bebfcb9422b87754fb258e16a97 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e3668e37348496ca15ab4da57534a53" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 41837 } } peers { permanent_uuid: "ad2dd25e957d4536bdc22a0df2b7263f" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 38457 } } peers { permanent_uuid: "63d49bebfcb9422b87754fb258e16a97" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 41385 } }
I20240503 23:27:56.002454 7082 catalog_manager.cc:5484] T 28c504576e8147d2ab6418920edf5e2f P 63d49bebfcb9422b87754fb258e16a97 reported cstate change: term changed from 1 to 2, leader changed from 9e3668e37348496ca15ab4da57534a53 (127.0.100.1) to 63d49bebfcb9422b87754fb258e16a97 (127.0.100.3). New cstate: current_term: 2 leader_uuid: "63d49bebfcb9422b87754fb258e16a97" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e3668e37348496ca15ab4da57534a53" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 41837 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ad2dd25e957d4536bdc22a0df2b7263f" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 38457 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "63d49bebfcb9422b87754fb258e16a97" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 41385 } health_report { overall_health: HEALTHY } } }
I20240503 23:27:56.258648 7593 raft_consensus.cc:1261] T fe71c54bf82048669579628b317224e7 P 9e3668e37348496ca15ab4da57534a53 [term 2 FOLLOWER]: Refusing update from remote peer ad2dd25e957d4536bdc22a0df2b7263f: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240503 23:27:56.259479 7448 consensus_queue.cc:1035] T fe71c54bf82048669579628b317224e7 P ad2dd25e957d4536bdc22a0df2b7263f [LEADER]: Connected to new peer: Peer: permanent_uuid: "9e3668e37348496ca15ab4da57534a53" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 41837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 23:27:56.264891 7332 raft_consensus.cc:1261] T fe71c54bf82048669579628b317224e7 P 63d49bebfcb9422b87754fb258e16a97 [term 2 FOLLOWER]: Refusing update from remote peer ad2dd25e957d4536bdc22a0df2b7263f: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240503 23:27:56.265594 7641 consensus_queue.cc:1035] T fe71c54bf82048669579628b317224e7 P ad2dd25e957d4536bdc22a0df2b7263f [LEADER]: Connected to new peer: Peer: permanent_uuid: "63d49bebfcb9422b87754fb258e16a97" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 41385 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 23:27:56.285847 7258 raft_consensus.cc:1261] T 95f3282b21394b11807125a214e8697a P ad2dd25e957d4536bdc22a0df2b7263f [term 2 FOLLOWER]: Refusing update from remote peer 63d49bebfcb9422b87754fb258e16a97: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240503 23:27:56.286772 7630 consensus_queue.cc:1035] T 95f3282b21394b11807125a214e8697a P 63d49bebfcb9422b87754fb258e16a97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ad2dd25e957d4536bdc22a0df2b7263f" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 38457 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 23:27:56.290959 7593 raft_consensus.cc:1261] T 95f3282b21394b11807125a214e8697a P 9e3668e37348496ca15ab4da57534a53 [term 2 FOLLOWER]: Refusing update from remote peer 63d49bebfcb9422b87754fb258e16a97: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240503 23:27:56.292227 7632 consensus_queue.cc:1035] T 95f3282b21394b11807125a214e8697a P 63d49bebfcb9422b87754fb258e16a97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9e3668e37348496ca15ab4da57534a53" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 41837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20240503 23:27:56.338027 7130 auto_rebalancer.cc:249] could not retrieve auto-rebalancing replica moves: Not found: table 23b3c6b2809b4e18a2ef0440c24e7d9e: could not find any suitable replica to move from server 63d49bebfcb9422b87754fb258e16a97 to server 9c15c9d698fe4fa4a7c458f2e51ab5f9
I20240503 23:27:56.459378 7593 raft_consensus.cc:1261] T 28c504576e8147d2ab6418920edf5e2f P 9e3668e37348496ca15ab4da57534a53 [term 2 FOLLOWER]: Refusing update from remote peer 63d49bebfcb9422b87754fb258e16a97: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240503 23:27:56.460139 7632 consensus_queue.cc:1035] T 28c504576e8147d2ab6418920edf5e2f P 63d49bebfcb9422b87754fb258e16a97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9e3668e37348496ca15ab4da57534a53" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 41837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 23:27:56.464484 7258 raft_consensus.cc:1261] T 28c504576e8147d2ab6418920edf5e2f P ad2dd25e957d4536bdc22a0df2b7263f [term 2 FOLLOWER]: Refusing update from remote peer 63d49bebfcb9422b87754fb258e16a97: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240503 23:27:56.465376 7386 consensus_queue.cc:1035] T 28c504576e8147d2ab6418920edf5e2f P 63d49bebfcb9422b87754fb258e16a97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ad2dd25e957d4536bdc22a0df2b7263f" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 38457 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 23:27:56.677070 7538 heartbeater.cc:503] Master 127.0.100.62:43625 was elected leader, sending a full tablet report...
I20240503 23:27:57.349436 7332 consensus_queue.cc:237] T 27c38e75002d42688eac2cb1f13c66d4 P 63d49bebfcb9422b87754fb258e16a97 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9e3668e37348496ca15ab4da57534a53" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 41837 } } peers { permanent_uuid: "ad2dd25e957d4536bdc22a0df2b7263f" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 38457 } attrs { replace: true } } peers { permanent_uuid: "63d49bebfcb9422b87754fb258e16a97" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 41385 } } peers { permanent_uuid: "9c15c9d698fe4fa4a7c458f2e51ab5f9" member_type: NON_VOTER last_known_addr { host: "127.0.100.4" port: 33075 } attrs { promote: true } }
I20240503 23:27:57.352353 7593 raft_consensus.cc:1261] T 27c38e75002d42688eac2cb1f13c66d4 P 9e3668e37348496ca15ab4da57534a53 [term 1 FOLLOWER]: Refusing update from remote peer 63d49bebfcb9422b87754fb258e16a97: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240503 23:27:57.352420 7296 consensus_peers.cc:485] T 27c38e75002d42688eac2cb1f13c66d4 P 63d49bebfcb9422b87754fb258e16a97 -> Peer 9c15c9d698fe4fa4a7c458f2e51ab5f9 (127.0.100.4:33075): Couldn't send request to peer 9c15c9d698fe4fa4a7c458f2e51ab5f9. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 27c38e75002d42688eac2cb1f13c66d4. This is attempt 1: this message will repeat every 5th retry.
I20240503 23:27:57.352499 7258 raft_consensus.cc:1261] T 27c38e75002d42688eac2cb1f13c66d4 P ad2dd25e957d4536bdc22a0df2b7263f [term 1 FOLLOWER]: Refusing update from remote peer 63d49bebfcb9422b87754fb258e16a97: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 23:27:57.353371 7632 consensus_queue.cc:1035] T 27c38e75002d42688eac2cb1f13c66d4 P 63d49bebfcb9422b87754fb258e16a97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ad2dd25e957d4536bdc22a0df2b7263f" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 38457 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 23:27:57.356518 7386 raft_consensus.cc:2932] T 27c38e75002d42688eac2cb1f13c66d4 P 63d49bebfcb9422b87754fb258e16a97 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, NON_VOTER 9c15c9d698fe4fa4a7c458f2e51ab5f9 (127.0.100.4) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9e3668e37348496ca15ab4da57534a53" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 41837 } } peers { permanent_uuid: "ad2dd25e957d4536bdc22a0df2b7263f" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 38457 } attrs { replace: true } } peers { permanent_uuid: "63d49bebfcb9422b87754fb258e16a97" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 41385 } } peers { permanent_uuid: "9c15c9d698fe4fa4a7c458f2e51ab5f9" member_type: NON_VOTER last_known_addr { host: "127.0.100.4" port: 33075 } attrs { promote: true } } }
I20240503 23:27:57.357251 7258 raft_consensus.cc:2932] T 27c38e75002d42688eac2cb1f13c66d4 P ad2dd25e957d4536bdc22a0df2b7263f [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, NON_VOTER 9c15c9d698fe4fa4a7c458f2e51ab5f9 (127.0.100.4) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9e3668e37348496ca15ab4da57534a53" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 41837 } } peers { permanent_uuid: "ad2dd25e957d4536bdc22a0df2b7263f" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 38457 } attrs { replace: true } } peers { permanent_uuid: "63d49bebfcb9422b87754fb258e16a97" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 41385 } } peers { permanent_uuid: "9c15c9d698fe4fa4a7c458f2e51ab5f9" member_type: NON_VOTER last_known_addr { host: "127.0.100.4" port: 33075 } attrs { promote: true } } }
I20240503 23:27:57.362710 7082 catalog_manager.cc:5484] T 27c38e75002d42688eac2cb1f13c66d4 P ad2dd25e957d4536bdc22a0df2b7263f reported cstate change: config changed from index 3 to 4, NON_VOTER 9c15c9d698fe4fa4a7c458f2e51ab5f9 (127.0.100.4) added. New cstate: current_term: 1 leader_uuid: "63d49bebfcb9422b87754fb258e16a97" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9e3668e37348496ca15ab4da57534a53" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 41837 } } peers { permanent_uuid: "ad2dd25e957d4536bdc22a0df2b7263f" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 38457 } attrs { replace: true } } peers { permanent_uuid: "63d49bebfcb9422b87754fb258e16a97" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 41385 } } peers { permanent_uuid: "9c15c9d698fe4fa4a7c458f2e51ab5f9" member_type: NON_VOTER last_known_addr { host: "127.0.100.4" port: 33075 } attrs { promote: true } } }
I20240503 23:27:57.362943 7332 consensus_queue.cc:237] T 95f3282b21394b11807125a214e8697a P 63d49bebfcb9422b87754fb258e16a97 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "9e3668e37348496ca15ab4da57534a53" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 41837 } attrs { replace: true } } peers { permanent_uuid: "ad2dd25e957d4536bdc22a0df2b7263f" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 38457 } } peers { permanent_uuid: "63d49bebfcb9422b87754fb258e16a97" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 41385 } } peers { permanent_uuid: "9c15c9d698fe4fa4a7c458f2e51ab5f9" member_type: NON_VOTER last_known_addr { host: "127.0.100.4" port: 33075 } attrs { promote: true } }
I20240503 23:27:57.370571 7632 consensus_queue.cc:1035] T 27c38e75002d42688eac2cb1f13c66d4 P 63d49bebfcb9422b87754fb258e16a97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9e3668e37348496ca15ab4da57534a53" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 41837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 23:27:57.371809 7593 raft_consensus.cc:1261] T 95f3282b21394b11807125a214e8697a P 9e3668e37348496ca15ab4da57534a53 [term 2 FOLLOWER]: Refusing update from remote peer 63d49bebfcb9422b87754fb258e16a97: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20240503 23:27:57.371932 7258 raft_consensus.cc:1261] T 95f3282b21394b11807125a214e8697a P ad2dd25e957d4536bdc22a0df2b7263f [term 2 FOLLOWER]: Refusing update from remote peer 63d49bebfcb9422b87754fb258e16a97: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
W20240503 23:27:57.372596 7296 consensus_peers.cc:485] T 95f3282b21394b11807125a214e8697a P 63d49bebfcb9422b87754fb258e16a97 -> Peer 9c15c9d698fe4fa4a7c458f2e51ab5f9 (127.0.100.4:33075): Couldn't send request to peer 9c15c9d698fe4fa4a7c458f2e51ab5f9. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 95f3282b21394b11807125a214e8697a. This is attempt 1: this message will repeat every 5th retry.
I20240503 23:27:57.372668 7632 consensus_queue.cc:1035] T 95f3282b21394b11807125a214e8697a P 63d49bebfcb9422b87754fb258e16a97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ad2dd25e957d4536bdc22a0df2b7263f" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 38457 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 23:27:57.373044 7630 consensus_queue.cc:1035] T 95f3282b21394b11807125a214e8697a P 63d49bebfcb9422b87754fb258e16a97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9e3668e37348496ca15ab4da57534a53" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 41837 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 23:27:57.375694 7593 raft_consensus.cc:2932] T 27c38e75002d42688eac2cb1f13c66d4 P 9e3668e37348496ca15ab4da57534a53 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, NON_VOTER 9c15c9d698fe4fa4a7c458f2e51ab5f9 (127.0.100.4) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9e3668e37348496ca15ab4da57534a53" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 41837 } } peers { permanent_uuid: "ad2dd25e957d4536bdc22a0df2b7263f" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 38457 } attrs { replace: true } } peers { permanent_uuid: "63d49bebfcb9422b87754fb258e16a97" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 41385 } } peers { permanent_uuid: "9c15c9d698fe4fa4a7c458f2e51ab5f9" member_type: NON_VOTER last_known_addr { host: "127.0.100.4" port: 33075 } attrs { promote: true } } }
I20240503 23:27:57.376143 7386 raft_consensus.cc:2932] T 95f3282b21394b11807125a214e8697a P 63d49bebfcb9422b87754fb258e16a97 [term 2 LEADER]: Committing config change with OpId 2.3: config changed from index -1 to 3, NON_VOTER 9c15c9d698fe4fa4a7c458f2e51ab5f9 (127.0.100.4) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "9e3668e37348496ca15ab4da57534a53" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 41837 } attrs { replace: true } } peers { permanent_uuid: "ad2dd25e957d4536bdc22a0df2b7263f" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 38457 } } peers { permanent_uuid: "63d49bebfcb9422b87754fb258e16a97" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 41385 } } peers { permanent_uuid: "9c15c9d698fe4fa4a7c458f2e51ab5f9" member_type: NON_VOTER last_known_addr { host: "127.0.100.4" port: 33075 } attrs { promote: true } } }
I20240503 23:27:57.376981 7258 raft_consensus.cc:2932] T 95f3282b21394b11807125a214e8697a P ad2dd25e957d4536bdc22a0df2b7263f [term 2 FOLLOWER]: Committing config change with OpId 2.3: config changed from index -1 to 3, NON_VOTER 9c15c9d698fe4fa4a7c458f2e51ab5f9 (127.0.100.4) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "9e3668e37348496ca15ab4da57534a53" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 41837 } attrs { replace: true } } peers { permanent_uuid: "ad2dd25e957d4536bdc22a0df2b7263f" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 38457 } } peers { permanent_uuid: "63d49bebfcb9422b87754fb258e16a97" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 41385 } } peers { permanent_uuid: "9c15c9d698fe4fa4a7c458f2e51ab5f9" member_type: NON_VOTER last_known_addr { host: "127.0.100.4" port: 33075 } attrs { promote: true } } }
I20240503 23:27:57.377070 7594 raft_consensus.cc:2932] T 95f3282b21394b11807125a214e8697a P 9e3668e37348496ca15ab4da57534a53 [term 2 FOLLOWER]: Committing config change with OpId 2.3: config changed from index -1 to 3, NON_VOTER 9c15c9d698fe4fa4a7c458f2e51ab5f9 (127.0.100.4) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "9e3668e37348496ca15ab4da57534a53" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 41837 } attrs { replace: true } } peers { permanent_uuid: "ad2dd25e957d4536bdc22a0df2b7263f" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 38457 } } peers { permanent_uuid: "63d49bebfcb9422b87754fb258e16a97" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 41385 } } peers { permanent_uuid: "9c15c9d698fe4fa4a7c458f2e51ab5f9" member_type: NON_VOTER last_known_addr { host: "127.0.100.4" port: 33075 } attrs { promote: true } } }
I20240503 23:27:57.382570 7079 catalog_manager.cc:5484] T 95f3282b21394b11807125a214e8697a P 63d49bebfcb9422b87754fb258e16a97 reported cstate change: config changed from index -1 to 3, NON_VOTER 9c15c9d698fe4fa4a7c458f2e51ab5f9 (127.0.100.4) added. New cstate: current_term: 2 leader_uuid: "63d49bebfcb9422b87754fb258e16a97" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "9e3668e37348496ca15ab4da57534a53" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 41837 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ad2dd25e957d4536bdc22a0df2b7263f" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 38457 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "63d49bebfcb9422b87754fb258e16a97" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 41385 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9c15c9d698fe4fa4a7c458f2e51ab5f9" member_type: NON_VOTER last_known_addr { host: "127.0.100.4" port: 33075 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 23:27:57.392843 7258 consensus_queue.cc:237] T fe71c54bf82048669579628b317224e7 P ad2dd25e957d4536bdc22a0df2b7263f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ad2dd25e957d4536bdc22a0df2b7263f" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 38457 } } peers { permanent_uuid: "9e3668e37348496ca15ab4da57534a53" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 41837 } } peers { permanent_uuid: "63d49bebfcb9422b87754fb258e16a97" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 41385 } attrs { replace: true } } peers { permanent_uuid: "9c15c9d698fe4fa4a7c458f2e51ab5f9" member_type: NON_VOTER last_known_addr { host: "127.0.100.4" port: 33075 } attrs { promote: true } }
I20240503 23:27:57.395763 7593 raft_consensus.cc:1261] T fe71c54bf82048669579628b317224e7 P 9e3668e37348496ca15ab4da57534a53 [term 2 FOLLOWER]: Refusing update from remote peer ad2dd25e957d4536bdc22a0df2b7263f: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20240503 23:27:57.396818 7332 raft_consensus.cc:1261] T fe71c54bf82048669579628b317224e7 P 63d49bebfcb9422b87754fb258e16a97 [term 2 FOLLOWER]: Refusing update from remote peer ad2dd25e957d4536bdc22a0df2b7263f: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20240503 23:27:57.396839 7666 consensus_queue.cc:1035] T fe71c54bf82048669579628b317224e7 P ad2dd25e957d4536bdc22a0df2b7263f [LEADER]: Connected to new peer: Peer: permanent_uuid: "9e3668e37348496ca15ab4da57534a53" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 41837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 23:27:57.397512 7641 consensus_queue.cc:1035] T fe71c54bf82048669579628b317224e7 P ad2dd25e957d4536bdc22a0df2b7263f [LEADER]: Connected to new peer: Peer: permanent_uuid: "63d49bebfcb9422b87754fb258e16a97" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 41385 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 23:27:57.399986 7666 raft_consensus.cc:2932] T fe71c54bf82048669579628b317224e7 P ad2dd25e957d4536bdc22a0df2b7263f [term 2 LEADER]: Committing config change with OpId 2.3: config changed from index -1 to 3, NON_VOTER 9c15c9d698fe4fa4a7c458f2e51ab5f9 (127.0.100.4) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ad2dd25e957d4536bdc22a0df2b7263f" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 38457 } } peers { permanent_uuid: "9e3668e37348496ca15ab4da57534a53" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 41837 } } peers { permanent_uuid: "63d49bebfcb9422b87754fb258e16a97" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 41385 } attrs { replace: true } } peers { permanent_uuid: "9c15c9d698fe4fa4a7c458f2e51ab5f9" member_type: NON_VOTER last_known_addr { host: "127.0.100.4" port: 33075 } attrs { promote: true } } }
I20240503 23:27:57.401055 7332 raft_consensus.cc:2932] T fe71c54bf82048669579628b317224e7 P 63d49bebfcb9422b87754fb258e16a97 [term 2 FOLLOWER]: Committing config change with OpId 2.3: config changed from index -1 to 3, NON_VOTER 9c15c9d698fe4fa4a7c458f2e51ab5f9 (127.0.100.4) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ad2dd25e957d4536bdc22a0df2b7263f" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 38457 } } peers { permanent_uuid: "9e3668e37348496ca15ab4da57534a53" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 41837 } } peers { permanent_uuid: "63d49bebfcb9422b87754fb258e16a97" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 41385 } attrs { replace: true } } peers { permanent_uuid: "9c15c9d698fe4fa4a7c458f2e51ab5f9" member_type: NON_VOTER last_known_addr { host: "127.0.100.4" port: 33075 } attrs { promote: true } } }
I20240503 23:27:57.401059 7593 raft_consensus.cc:2932] T fe71c54bf82048669579628b317224e7 P 9e3668e37348496ca15ab4da57534a53 [term 2 FOLLOWER]: Committing config change with OpId 2.3: config changed from index -1 to 3, NON_VOTER 9c15c9d698fe4fa4a7c458f2e51ab5f9 (127.0.100.4) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ad2dd25e957d4536bdc22a0df2b7263f" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 38457 } } peers { permanent_uuid: "9e3668e37348496ca15ab4da57534a53" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 41837 } } peers { permanent_uuid: "63d49bebfcb9422b87754fb258e16a97" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 41385 } attrs { replace: true } } peers { permanent_uuid: "9c15c9d698fe4fa4a7c458f2e51ab5f9" member_type: NON_VOTER last_known_addr { host: "127.0.100.4" port: 33075 } attrs { promote: true } } }
W20240503 23:27:57.403676 7222 consensus_peers.cc:485] T fe71c54bf82048669579628b317224e7 P ad2dd25e957d4536bdc22a0df2b7263f -> Peer 9c15c9d698fe4fa4a7c458f2e51ab5f9 (127.0.100.4:33075): Couldn't send request to peer 9c15c9d698fe4fa4a7c458f2e51ab5f9. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: fe71c54bf82048669579628b317224e7. This is attempt 1: this message will repeat every 5th retry.
I20240503 23:27:57.406565 7079 catalog_manager.cc:5484] T fe71c54bf82048669579628b317224e7 P ad2dd25e957d4536bdc22a0df2b7263f reported cstate change: config changed from index -1 to 3, NON_VOTER 9c15c9d698fe4fa4a7c458f2e51ab5f9 (127.0.100.4) added. New cstate: current_term: 2 leader_uuid: "ad2dd25e957d4536bdc22a0df2b7263f" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ad2dd25e957d4536bdc22a0df2b7263f" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 38457 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9e3668e37348496ca15ab4da57534a53" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 41837 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "63d49bebfcb9422b87754fb258e16a97" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 41385 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9c15c9d698fe4fa4a7c458f2e51ab5f9" member_type: NON_VOTER last_known_addr { host: "127.0.100.4" port: 33075 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 23:27:57.836436 7671 ts_tablet_manager.cc:902] T 27c38e75002d42688eac2cb1f13c66d4 P 9c15c9d698fe4fa4a7c458f2e51ab5f9: Initiating tablet copy from peer 63d49bebfcb9422b87754fb258e16a97 (127.0.100.3:41385)
I20240503 23:27:57.837208 7671 tablet_copy_client.cc:316] T 27c38e75002d42688eac2cb1f13c66d4 P 9c15c9d698fe4fa4a7c458f2e51ab5f9: tablet copy: Beginning tablet copy session from remote peer at address 127.0.100.3:41385
I20240503 23:27:57.844156 7343 tablet_copy_service.cc:140] P 63d49bebfcb9422b87754fb258e16a97: Received BeginTabletCopySession request for tablet 27c38e75002d42688eac2cb1f13c66d4 from peer 9c15c9d698fe4fa4a7c458f2e51ab5f9 ({username='slave'} at 127.0.0.1:45974)
I20240503 23:27:57.844337 7343 tablet_copy_service.cc:161] P 63d49bebfcb9422b87754fb258e16a97: Beginning new tablet copy session on tablet 27c38e75002d42688eac2cb1f13c66d4 from peer 9c15c9d698fe4fa4a7c458f2e51ab5f9 at {username='slave'} at 127.0.0.1:45974: session id = 9c15c9d698fe4fa4a7c458f2e51ab5f9-27c38e75002d42688eac2cb1f13c66d4
I20240503 23:27:57.847119 7343 tablet_copy_source_session.cc:205] T 27c38e75002d42688eac2cb1f13c66d4 P 63d49bebfcb9422b87754fb258e16a97: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 23:27:57.848526 7671 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 27c38e75002d42688eac2cb1f13c66d4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 23:27:57.855082 7671 tablet_copy_client.cc:791] T 27c38e75002d42688eac2cb1f13c66d4 P 9c15c9d698fe4fa4a7c458f2e51ab5f9: tablet copy: Starting download of 0 data blocks...
I20240503 23:27:57.855562 7671 tablet_copy_client.cc:655] T 27c38e75002d42688eac2cb1f13c66d4 P 9c15c9d698fe4fa4a7c458f2e51ab5f9: tablet copy: Starting download of 1 WAL segments...
I20240503 23:27:57.857461 7671 tablet_copy_client.cc:531] T 27c38e75002d42688eac2cb1f13c66d4 P 9c15c9d698fe4fa4a7c458f2e51ab5f9: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 23:27:57.861416 7671 tablet_bootstrap.cc:492] T 27c38e75002d42688eac2cb1f13c66d4 P 9c15c9d698fe4fa4a7c458f2e51ab5f9: Bootstrap starting.
I20240503 23:27:57.869467 7671 tablet_bootstrap.cc:492] T 27c38e75002d42688eac2cb1f13c66d4 P 9c15c9d698fe4fa4a7c458f2e51ab5f9: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 23:27:57.869922 7671 tablet_bootstrap.cc:492] T 27c38e75002d42688eac2cb1f13c66d4 P 9c15c9d698fe4fa4a7c458f2e51ab5f9: Bootstrap complete.
I20240503 23:27:57.870213 7671 ts_tablet_manager.cc:1367] T 27c38e75002d42688eac2cb1f13c66d4 P 9c15c9d698fe4fa4a7c458f2e51ab5f9: Time spent bootstrapping tablet: real 0.009s user 0.004s sys 0.004s
I20240503 23:27:57.871021 7671 raft_consensus.cc:348] T 27c38e75002d42688eac2cb1f13c66d4 P 9c15c9d698fe4fa4a7c458f2e51ab5f9 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9e3668e37348496ca15ab4da57534a53" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 41837 } } peers { permanent_uuid: "ad2dd25e957d4536bdc22a0df2b7263f" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 38457 } attrs { replace: true } } peers { permanent_uuid: "63d49bebfcb9422b87754fb258e16a97" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 41385 } } peers { permanent_uuid: "9c15c9d698fe4fa4a7c458f2e51ab5f9" member_type: NON_VOTER last_known_addr { host: "127.0.100.4" port: 33075 } attrs { promote: true } }
I20240503 23:27:57.871308 7671 raft_consensus.cc:729] T 27c38e75002d42688eac2cb1f13c66d4 P 9c15c9d698fe4fa4a7c458f2e51ab5f9 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 9c15c9d698fe4fa4a7c458f2e51ab5f9, State: Initialized, Role: LEARNER
I20240503 23:27:57.871522 7671 consensus_queue.cc:260] T 27c38e75002d42688eac2cb1f13c66d4 P 9c15c9d698fe4fa4a7c458f2e51ab5f9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9e3668e37348496ca15ab4da57534a53" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 41837 } } peers { permanent_uuid: "ad2dd25e957d4536bdc22a0df2b7263f" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 38457 } attrs { replace: true } } peers { permanent_uuid: "63d49bebfcb9422b87754fb258e16a97" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 41385 } } peers { permanent_uuid: "9c15c9d698fe4fa4a7c458f2e51ab5f9" member_type: NON_VOTER last_known_addr { host: "127.0.100.4" port: 33075 } attrs { promote: true } }
I20240503 23:27:57.872474 7671 ts_tablet_manager.cc:1398] T 27c38e75002d42688eac2cb1f13c66d4 P 9c15c9d698fe4fa4a7c458f2e51ab5f9: Time spent starting tablet: real 0.002s user 0.001s sys 0.003s
I20240503 23:27:57.873242 7343 tablet_copy_service.cc:342] P 63d49bebfcb9422b87754fb258e16a97: Request end of tablet copy session 9c15c9d698fe4fa4a7c458f2e51ab5f9-27c38e75002d42688eac2cb1f13c66d4 received from {username='slave'} at 127.0.0.1:45974
I20240503 23:27:57.873364 7343 tablet_copy_service.cc:433] P 63d49bebfcb9422b87754fb258e16a97: ending tablet copy session 9c15c9d698fe4fa4a7c458f2e51ab5f9-27c38e75002d42688eac2cb1f13c66d4 on tablet 27c38e75002d42688eac2cb1f13c66d4 with peer 9c15c9d698fe4fa4a7c458f2e51ab5f9
I20240503 23:27:57.939340 7671 ts_tablet_manager.cc:902] T fe71c54bf82048669579628b317224e7 P 9c15c9d698fe4fa4a7c458f2e51ab5f9: Initiating tablet copy from peer ad2dd25e957d4536bdc22a0df2b7263f (127.0.100.2:38457)
I20240503 23:27:57.940136 7671 tablet_copy_client.cc:316] T fe71c54bf82048669579628b317224e7 P 9c15c9d698fe4fa4a7c458f2e51ab5f9: tablet copy: Beginning tablet copy session from remote peer at address 127.0.100.2:38457
I20240503 23:27:57.947243 7269 tablet_copy_service.cc:140] P ad2dd25e957d4536bdc22a0df2b7263f: Received BeginTabletCopySession request for tablet fe71c54bf82048669579628b317224e7 from peer 9c15c9d698fe4fa4a7c458f2e51ab5f9 ({username='slave'} at 127.0.0.1:48586)
I20240503 23:27:57.947471 7269 tablet_copy_service.cc:161] P ad2dd25e957d4536bdc22a0df2b7263f: Beginning new tablet copy session on tablet fe71c54bf82048669579628b317224e7 from peer 9c15c9d698fe4fa4a7c458f2e51ab5f9 at {username='slave'} at 127.0.0.1:48586: session id = 9c15c9d698fe4fa4a7c458f2e51ab5f9-fe71c54bf82048669579628b317224e7
I20240503 23:27:57.950342 7269 tablet_copy_source_session.cc:205] T fe71c54bf82048669579628b317224e7 P ad2dd25e957d4536bdc22a0df2b7263f: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 23:27:57.951712 7671 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fe71c54bf82048669579628b317224e7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 23:27:57.957829 7671 tablet_copy_client.cc:791] T fe71c54bf82048669579628b317224e7 P 9c15c9d698fe4fa4a7c458f2e51ab5f9: tablet copy: Starting download of 0 data blocks...
I20240503 23:27:57.958145 7671 tablet_copy_client.cc:655] T fe71c54bf82048669579628b317224e7 P 9c15c9d698fe4fa4a7c458f2e51ab5f9: tablet copy: Starting download of 1 WAL segments...
I20240503 23:27:57.959705 7671 tablet_copy_client.cc:531] T fe71c54bf82048669579628b317224e7 P 9c15c9d698fe4fa4a7c458f2e51ab5f9: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 23:27:57.964668 7671 tablet_bootstrap.cc:492] T fe71c54bf82048669579628b317224e7 P 9c15c9d698fe4fa4a7c458f2e51ab5f9: Bootstrap starting.
I20240503 23:27:57.971542 7671 tablet_bootstrap.cc:492] T fe71c54bf82048669579628b317224e7 P 9c15c9d698fe4fa4a7c458f2e51ab5f9: Bootstrap replayed 1/1 log segments. Stats: ops{read=3 overwritten=0 applied=3 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 23:27:57.972162 7671 tablet_bootstrap.cc:492] T fe71c54bf82048669579628b317224e7 P 9c15c9d698fe4fa4a7c458f2e51ab5f9: Bootstrap complete.
I20240503 23:27:57.972513 7671 ts_tablet_manager.cc:1367] T fe71c54bf82048669579628b317224e7 P 9c15c9d698fe4fa4a7c458f2e51ab5f9: Time spent bootstrapping tablet: real 0.008s user 0.004s sys 0.005s
I20240503 23:27:57.972599 7680 ts_tablet_manager.cc:902] T 95f3282b21394b11807125a214e8697a P 9c15c9d698fe4fa4a7c458f2e51ab5f9: Initiating tablet copy from peer 63d49bebfcb9422b87754fb258e16a97 (127.0.100.3:41385)
I20240503 23:27:57.973544 7680 tablet_copy_client.cc:316] T 95f3282b21394b11807125a214e8697a P 9c15c9d698fe4fa4a7c458f2e51ab5f9: tablet copy: Beginning tablet copy session from remote peer at address 127.0.100.3:41385
I20240503 23:27:57.973551 7671 raft_consensus.cc:348] T fe71c54bf82048669579628b317224e7 P 9c15c9d698fe4fa4a7c458f2e51ab5f9 [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ad2dd25e957d4536bdc22a0df2b7263f" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 38457 } } peers { permanent_uuid: "9e3668e37348496ca15ab4da57534a53" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 41837 } } peers { permanent_uuid: "63d49bebfcb9422b87754fb258e16a97" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 41385 } attrs { replace: true } } peers { permanent_uuid: "9c15c9d698fe4fa4a7c458f2e51ab5f9" member_type: NON_VOTER last_known_addr { host: "127.0.100.4" port: 33075 } attrs { promote: true } }
I20240503 23:27:57.974013 7671 raft_consensus.cc:729] T fe71c54bf82048669579628b317224e7 P 9c15c9d698fe4fa4a7c458f2e51ab5f9 [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: 9c15c9d698fe4fa4a7c458f2e51ab5f9, State: Initialized, Role: LEARNER
I20240503 23:27:57.974263 7671 consensus_queue.cc:260] T fe71c54bf82048669579628b317224e7 P 9c15c9d698fe4fa4a7c458f2e51ab5f9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 2.3, Last appended by leader: 3, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ad2dd25e957d4536bdc22a0df2b7263f" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 38457 } } peers { permanent_uuid: "9e3668e37348496ca15ab4da57534a53" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 41837 } } peers { permanent_uuid: "63d49bebfcb9422b87754fb258e16a97" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 41385 } attrs { replace: true } } peers { permanent_uuid: "9c15c9d698fe4fa4a7c458f2e51ab5f9" member_type: NON_VOTER last_known_addr { host: "127.0.100.4" port: 33075 } attrs { promote: true } }
I20240503 23:27:57.974411 7343 tablet_copy_service.cc:140] P 63d49bebfcb9422b87754fb258e16a97: Received BeginTabletCopySession request for tablet 95f3282b21394b11807125a214e8697a from peer 9c15c9d698fe4fa4a7c458f2e51ab5f9 ({username='slave'} at 127.0.0.1:45974)
I20240503 23:27:57.974591 7343 tablet_copy_service.cc:161] P 63d49bebfcb9422b87754fb258e16a97: Beginning new tablet copy session on tablet 95f3282b21394b11807125a214e8697a from peer 9c15c9d698fe4fa4a7c458f2e51ab5f9 at {username='slave'} at 127.0.0.1:45974: session id = 9c15c9d698fe4fa4a7c458f2e51ab5f9-95f3282b21394b11807125a214e8697a
I20240503 23:27:57.974923 7671 ts_tablet_manager.cc:1398] T fe71c54bf82048669579628b317224e7 P 9c15c9d698fe4fa4a7c458f2e51ab5f9: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 23:27:57.975785 7269 tablet_copy_service.cc:342] P ad2dd25e957d4536bdc22a0df2b7263f: Request end of tablet copy session 9c15c9d698fe4fa4a7c458f2e51ab5f9-fe71c54bf82048669579628b317224e7 received from {username='slave'} at 127.0.0.1:48586
I20240503 23:27:57.975914 7269 tablet_copy_service.cc:433] P ad2dd25e957d4536bdc22a0df2b7263f: ending tablet copy session 9c15c9d698fe4fa4a7c458f2e51ab5f9-fe71c54bf82048669579628b317224e7 on tablet fe71c54bf82048669579628b317224e7 with peer 9c15c9d698fe4fa4a7c458f2e51ab5f9
I20240503 23:27:57.977675 7343 tablet_copy_source_session.cc:205] T 95f3282b21394b11807125a214e8697a P 63d49bebfcb9422b87754fb258e16a97: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 23:27:57.979174 7680 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 95f3282b21394b11807125a214e8697a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 23:27:57.985664 7680 tablet_copy_client.cc:791] T 95f3282b21394b11807125a214e8697a P 9c15c9d698fe4fa4a7c458f2e51ab5f9: tablet copy: Starting download of 0 data blocks...
I20240503 23:27:57.985998 7680 tablet_copy_client.cc:655] T 95f3282b21394b11807125a214e8697a P 9c15c9d698fe4fa4a7c458f2e51ab5f9: tablet copy: Starting download of 1 WAL segments...
I20240503 23:27:57.987839 7680 tablet_copy_client.cc:531] T 95f3282b21394b11807125a214e8697a P 9c15c9d698fe4fa4a7c458f2e51ab5f9: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 23:27:57.991217 7680 tablet_bootstrap.cc:492] T 95f3282b21394b11807125a214e8697a P 9c15c9d698fe4fa4a7c458f2e51ab5f9: Bootstrap starting.
I20240503 23:27:57.998337 7680 tablet_bootstrap.cc:492] T 95f3282b21394b11807125a214e8697a P 9c15c9d698fe4fa4a7c458f2e51ab5f9: Bootstrap replayed 1/1 log segments. Stats: ops{read=3 overwritten=0 applied=3 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 23:27:57.998798 7680 tablet_bootstrap.cc:492] T 95f3282b21394b11807125a214e8697a P 9c15c9d698fe4fa4a7c458f2e51ab5f9: Bootstrap complete.
I20240503 23:27:57.999131 7680 ts_tablet_manager.cc:1367] T 95f3282b21394b11807125a214e8697a P 9c15c9d698fe4fa4a7c458f2e51ab5f9: Time spent bootstrapping tablet: real 0.008s user 0.007s sys 0.001s
I20240503 23:27:58.000067 7680 raft_consensus.cc:348] T 95f3282b21394b11807125a214e8697a P 9c15c9d698fe4fa4a7c458f2e51ab5f9 [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "9e3668e37348496ca15ab4da57534a53" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 41837 } attrs { replace: true } } peers { permanent_uuid: "ad2dd25e957d4536bdc22a0df2b7263f" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 38457 } } peers { permanent_uuid: "63d49bebfcb9422b87754fb258e16a97" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 41385 } } peers { permanent_uuid: "9c15c9d698fe4fa4a7c458f2e51ab5f9" member_type: NON_VOTER last_known_addr { host: "127.0.100.4" port: 33075 } attrs { promote: true } }
I20240503 23:27:58.000336 7680 raft_consensus.cc:729] T 95f3282b21394b11807125a214e8697a P 9c15c9d698fe4fa4a7c458f2e51ab5f9 [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: 9c15c9d698fe4fa4a7c458f2e51ab5f9, State: Initialized, Role: LEARNER
I20240503 23:27:58.000591 7680 consensus_queue.cc:260] T 95f3282b21394b11807125a214e8697a P 9c15c9d698fe4fa4a7c458f2e51ab5f9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 2.3, Last appended by leader: 3, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "9e3668e37348496ca15ab4da57534a53" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 41837 } attrs { replace: true } } peers { permanent_uuid: "ad2dd25e957d4536bdc22a0df2b7263f" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 38457 } } peers { permanent_uuid: "63d49bebfcb9422b87754fb258e16a97" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 41385 } } peers { permanent_uuid: "9c15c9d698fe4fa4a7c458f2e51ab5f9" member_type: NON_VOTER last_known_addr { host: "127.0.100.4" port: 33075 } attrs { promote: true } }
I20240503 23:27:58.001435 7680 ts_tablet_manager.cc:1398] T 95f3282b21394b11807125a214e8697a P 9c15c9d698fe4fa4a7c458f2e51ab5f9: Time spent starting tablet: real 0.002s user 0.002s sys 0.002s
I20240503 23:27:58.002147 7343 tablet_copy_service.cc:342] P 63d49bebfcb9422b87754fb258e16a97: Request end of tablet copy session 9c15c9d698fe4fa4a7c458f2e51ab5f9-95f3282b21394b11807125a214e8697a received from {username='slave'} at 127.0.0.1:45974
I20240503 23:27:58.002303 7343 tablet_copy_service.cc:433] P 63d49bebfcb9422b87754fb258e16a97: ending tablet copy session 9c15c9d698fe4fa4a7c458f2e51ab5f9-95f3282b21394b11807125a214e8697a on tablet 95f3282b21394b11807125a214e8697a with peer 9c15c9d698fe4fa4a7c458f2e51ab5f9
I20240503 23:27:58.308048 7513 raft_consensus.cc:1203] T 27c38e75002d42688eac2cb1f13c66d4 P 9c15c9d698fe4fa4a7c458f2e51ab5f9 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 23:27:58.342948 7513 raft_consensus.cc:1203] T fe71c54bf82048669579628b317224e7 P 9c15c9d698fe4fa4a7c458f2e51ab5f9 [term 2 LEARNER]: Deduplicated request from leader. Original: 2.2->[2.3-2.3] Dedup: 2.3->[]
I20240503 23:27:58.412083 7513 raft_consensus.cc:1203] T 95f3282b21394b11807125a214e8697a P 9c15c9d698fe4fa4a7c458f2e51ab5f9 [term 2 LEARNER]: Deduplicated request from leader. Original: 2.2->[2.3-2.3] Dedup: 2.3->[]
I20240503 23:27:58.839109 7683 raft_consensus.cc:1050] T fe71c54bf82048669579628b317224e7 P ad2dd25e957d4536bdc22a0df2b7263f: attempting to promote NON_VOTER 9c15c9d698fe4fa4a7c458f2e51ab5f9 to VOTER
I20240503 23:27:58.839929 7683 consensus_queue.cc:237] T fe71c54bf82048669579628b317224e7 P ad2dd25e957d4536bdc22a0df2b7263f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.3, Last appended by leader: 1, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ad2dd25e957d4536bdc22a0df2b7263f" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 38457 } } peers { permanent_uuid: "9e3668e37348496ca15ab4da57534a53" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 41837 } } peers { permanent_uuid: "63d49bebfcb9422b87754fb258e16a97" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 41385 } attrs { replace: true } } peers { permanent_uuid: "9c15c9d698fe4fa4a7c458f2e51ab5f9" member_type: VOTER last_known_addr { host: "127.0.100.4" port: 33075 } attrs { promote: false } }
I20240503 23:27:58.842377 7513 raft_consensus.cc:1261] T fe71c54bf82048669579628b317224e7 P 9c15c9d698fe4fa4a7c458f2e51ab5f9 [term 2 LEARNER]: Refusing update from remote peer ad2dd25e957d4536bdc22a0df2b7263f: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240503 23:27:58.842577 7332 raft_consensus.cc:1261] T fe71c54bf82048669579628b317224e7 P 63d49bebfcb9422b87754fb258e16a97 [term 2 FOLLOWER]: Refusing update from remote peer ad2dd25e957d4536bdc22a0df2b7263f: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240503 23:27:58.842675 7593 raft_consensus.cc:1261] T fe71c54bf82048669579628b317224e7 P 9e3668e37348496ca15ab4da57534a53 [term 2 FOLLOWER]: Refusing update from remote peer ad2dd25e957d4536bdc22a0df2b7263f: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240503 23:27:58.843236 7676 consensus_queue.cc:1035] T fe71c54bf82048669579628b317224e7 P ad2dd25e957d4536bdc22a0df2b7263f [LEADER]: Connected to new peer: Peer: permanent_uuid: "63d49bebfcb9422b87754fb258e16a97" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 41385 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 23:27:58.843780 7676 consensus_queue.cc:1035] T fe71c54bf82048669579628b317224e7 P ad2dd25e957d4536bdc22a0df2b7263f [LEADER]: Connected to new peer: Peer: permanent_uuid: "9e3668e37348496ca15ab4da57534a53" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 41837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 23:27:58.844202 7684 consensus_queue.cc:1035] T fe71c54bf82048669579628b317224e7 P ad2dd25e957d4536bdc22a0df2b7263f [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c15c9d698fe4fa4a7c458f2e51ab5f9" member_type: VOTER last_known_addr { host: "127.0.100.4" port: 33075 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 23:27:58.858459 7684 raft_consensus.cc:2932] T fe71c54bf82048669579628b317224e7 P ad2dd25e957d4536bdc22a0df2b7263f [term 2 LEADER]: Committing config change with OpId 2.4: config changed from index 3 to 4, 9c15c9d698fe4fa4a7c458f2e51ab5f9 (127.0.100.4) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ad2dd25e957d4536bdc22a0df2b7263f" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 38457 } } peers { permanent_uuid: "9e3668e37348496ca15ab4da57534a53" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 41837 } } peers { permanent_uuid: "63d49bebfcb9422b87754fb258e16a97" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 41385 } attrs { replace: true } } peers { permanent_uuid: "9c15c9d698fe4fa4a7c458f2e51ab5f9" member_type: VOTER last_known_addr { host: "127.0.100.4" port: 33075 } attrs { promote: false } } }
I20240503 23:27:58.859014 7593 raft_consensus.cc:2932] T fe71c54bf82048669579628b317224e7 P 9e3668e37348496ca15ab4da57534a53 [term 2 FOLLOWER]: Committing config change with OpId 2.4: config changed from index 3 to 4, 9c15c9d698fe4fa4a7c458f2e51ab5f9 (127.0.100.4) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ad2dd25e957d4536bdc22a0df2b7263f" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 38457 } } peers { permanent_uuid: "9e3668e37348496ca15ab4da57534a53" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 41837 } } peers { permanent_uuid: "63d49bebfcb9422b87754fb258e16a97" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 41385 } attrs { replace: true } } peers { permanent_uuid: "9c15c9d698fe4fa4a7c458f2e51ab5f9" member_type: VOTER last_known_addr { host: "127.0.100.4" port: 33075 } attrs { promote: false } } }
I20240503 23:27:58.863945 7333 raft_consensus.cc:2932] T fe71c54bf82048669579628b317224e7 P 63d49bebfcb9422b87754fb258e16a97 [term 2 FOLLOWER]: Committing config change with OpId 2.4: config changed from index 3 to 4, 9c15c9d698fe4fa4a7c458f2e51ab5f9 (127.0.100.4) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ad2dd25e957d4536bdc22a0df2b7263f" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 38457 } } peers { permanent_uuid: "9e3668e37348496ca15ab4da57534a53" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 41837 } } peers { permanent_uuid: "63d49bebfcb9422b87754fb258e16a97" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 41385 } attrs { replace: true } } peers { permanent_uuid: "9c15c9d698fe4fa4a7c458f2e51ab5f9" member_type: VOTER last_known_addr { host: "127.0.100.4" port: 33075 } attrs { promote: false } } }
I20240503 23:27:58.864022 7513 raft_consensus.cc:2932] T fe71c54bf82048669579628b317224e7 P 9c15c9d698fe4fa4a7c458f2e51ab5f9 [term 2 FOLLOWER]: Committing config change with OpId 2.4: config changed from index 3 to 4, 9c15c9d698fe4fa4a7c458f2e51ab5f9 (127.0.100.4) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ad2dd25e957d4536bdc22a0df2b7263f" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 38457 } } peers { permanent_uuid: "9e3668e37348496ca15ab4da57534a53" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 41837 } } peers { permanent_uuid: "63d49bebfcb9422b87754fb258e16a97" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 41385 } attrs { replace: true } } peers { permanent_uuid: "9c15c9d698fe4fa4a7c458f2e51ab5f9" member_type: VOTER last_known_addr { host: "127.0.100.4" port: 33075 } attrs { promote: false } } }
I20240503 23:27:58.864825 7079 catalog_manager.cc:5484] T fe71c54bf82048669579628b317224e7 P ad2dd25e957d4536bdc22a0df2b7263f reported cstate change: config changed from index 3 to 4, 9c15c9d698fe4fa4a7c458f2e51ab5f9 (127.0.100.4) changed from NON_VOTER to VOTER. New cstate: current_term: 2 leader_uuid: "ad2dd25e957d4536bdc22a0df2b7263f" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ad2dd25e957d4536bdc22a0df2b7263f" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 38457 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9e3668e37348496ca15ab4da57534a53" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 41837 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "63d49bebfcb9422b87754fb258e16a97" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 41385 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9c15c9d698fe4fa4a7c458f2e51ab5f9" member_type: VOTER last_known_addr { host: "127.0.100.4" port: 33075 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 23:27:58.881594 7258 consensus_queue.cc:237] T fe71c54bf82048669579628b317224e7 P ad2dd25e957d4536bdc22a0df2b7263f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.4, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ad2dd25e957d4536bdc22a0df2b7263f" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 38457 } } peers { permanent_uuid: "9e3668e37348496ca15ab4da57534a53" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 41837 } } peers { permanent_uuid: "9c15c9d698fe4fa4a7c458f2e51ab5f9" member_type: VOTER last_known_addr { host: "127.0.100.4" port: 33075 } attrs { promote: false } }
I20240503 23:27:58.883543 7513 raft_consensus.cc:1261] T fe71c54bf82048669579628b317224e7 P 9c15c9d698fe4fa4a7c458f2e51ab5f9 [term 2 FOLLOWER]: Refusing update from remote peer ad2dd25e957d4536bdc22a0df2b7263f: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240503 23:27:58.883695 7593 raft_consensus.cc:1261] T fe71c54bf82048669579628b317224e7 P 9e3668e37348496ca15ab4da57534a53 [term 2 FOLLOWER]: Refusing update from remote peer ad2dd25e957d4536bdc22a0df2b7263f: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240503 23:27:58.884248 7683 consensus_queue.cc:1035] T fe71c54bf82048669579628b317224e7 P ad2dd25e957d4536bdc22a0df2b7263f [LEADER]: Connected to new peer: Peer: permanent_uuid: "9e3668e37348496ca15ab4da57534a53" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 41837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 23:27:58.884532 7686 consensus_queue.cc:1035] T fe71c54bf82048669579628b317224e7 P ad2dd25e957d4536bdc22a0df2b7263f [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c15c9d698fe4fa4a7c458f2e51ab5f9" member_type: VOTER last_known_addr { host: "127.0.100.4" port: 33075 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 23:27:58.887025 7684 raft_consensus.cc:2932] T fe71c54bf82048669579628b317224e7 P ad2dd25e957d4536bdc22a0df2b7263f [term 2 LEADER]: Committing config change with OpId 2.5: config changed from index 4 to 5, VOTER 63d49bebfcb9422b87754fb258e16a97 (127.0.100.3) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ad2dd25e957d4536bdc22a0df2b7263f" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 38457 } } peers { permanent_uuid: "9e3668e37348496ca15ab4da57534a53" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 41837 } } peers { permanent_uuid: "9c15c9d698fe4fa4a7c458f2e51ab5f9" member_type: VOTER last_known_addr { host: "127.0.100.4" port: 33075 } attrs { promote: false } } }
I20240503 23:27:58.888032 7513 raft_consensus.cc:2932] T fe71c54bf82048669579628b317224e7 P 9c15c9d698fe4fa4a7c458f2e51ab5f9 [term 2 FOLLOWER]: Committing config change with OpId 2.5: config changed from index 4 to 5, VOTER 63d49bebfcb9422b87754fb258e16a97 (127.0.100.3) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ad2dd25e957d4536bdc22a0df2b7263f" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 38457 } } peers { permanent_uuid: "9e3668e37348496ca15ab4da57534a53" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 41837 } } peers { permanent_uuid: "9c15c9d698fe4fa4a7c458f2e51ab5f9" member_type: VOTER last_known_addr { host: "127.0.100.4" port: 33075 } attrs { promote: false } } }
I20240503 23:27:58.889081 7593 raft_consensus.cc:2932] T fe71c54bf82048669579628b317224e7 P 9e3668e37348496ca15ab4da57534a53 [term 2 FOLLOWER]: Committing config change with OpId 2.5: config changed from index 4 to 5, VOTER 63d49bebfcb9422b87754fb258e16a97 (127.0.100.3) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ad2dd25e957d4536bdc22a0df2b7263f" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 38457 } } peers { permanent_uuid: "9e3668e37348496ca15ab4da57534a53" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 41837 } } peers { permanent_uuid: "9c15c9d698fe4fa4a7c458f2e51ab5f9" member_type: VOTER last_known_addr { host: "127.0.100.4" port: 33075 } attrs { promote: false } } }
I20240503 23:27:58.891069 7630 raft_consensus.cc:1050] T 27c38e75002d42688eac2cb1f13c66d4 P 63d49bebfcb9422b87754fb258e16a97: attempting to promote NON_VOTER 9c15c9d698fe4fa4a7c458f2e51ab5f9 to VOTER
I20240503 23:27:58.891993 7630 consensus_queue.cc:237] T 27c38e75002d42688eac2cb1f13c66d4 P 63d49bebfcb9422b87754fb258e16a97 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "9e3668e37348496ca15ab4da57534a53" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 41837 } } peers { permanent_uuid: "ad2dd25e957d4536bdc22a0df2b7263f" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 38457 } attrs { replace: true } } peers { permanent_uuid: "63d49bebfcb9422b87754fb258e16a97" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 41385 } } peers { permanent_uuid: "9c15c9d698fe4fa4a7c458f2e51ab5f9" member_type: VOTER last_known_addr { host: "127.0.100.4" port: 33075 } attrs { promote: false } }
I20240503 23:27:58.893885 7081 catalog_manager.cc:5484] T fe71c54bf82048669579628b317224e7 P 9e3668e37348496ca15ab4da57534a53 reported cstate change: config changed from index 4 to 5, VOTER 63d49bebfcb9422b87754fb258e16a97 (127.0.100.3) evicted. New cstate: current_term: 2 leader_uuid: "ad2dd25e957d4536bdc22a0df2b7263f" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ad2dd25e957d4536bdc22a0df2b7263f" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 38457 } } peers { permanent_uuid: "9e3668e37348496ca15ab4da57534a53" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 41837 } } peers { permanent_uuid: "9c15c9d698fe4fa4a7c458f2e51ab5f9" member_type: VOTER last_known_addr { host: "127.0.100.4" port: 33075 } attrs { promote: false } } }
I20240503 23:27:58.894660 7069 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet fe71c54bf82048669579628b317224e7 with cas_config_opid_index 4: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 23:27:58.899251 7322 tablet_service.cc:1514] Processing DeleteTablet for tablet fe71c54bf82048669579628b317224e7 with delete_type TABLET_DATA_TOMBSTONED (TS 63d49bebfcb9422b87754fb258e16a97 not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:45946
I20240503 23:27:58.900254 7696 tablet_replica.cc:330] T fe71c54bf82048669579628b317224e7 P 63d49bebfcb9422b87754fb258e16a97: stopping tablet replica
I20240503 23:27:58.900722 7696 raft_consensus.cc:2229] T fe71c54bf82048669579628b317224e7 P 63d49bebfcb9422b87754fb258e16a97 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 23:27:58.901145 7696 raft_consensus.cc:2258] T fe71c54bf82048669579628b317224e7 P 63d49bebfcb9422b87754fb258e16a97 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 23:27:58.902256 7696 ts_tablet_manager.cc:1875] T fe71c54bf82048669579628b317224e7 P 63d49bebfcb9422b87754fb258e16a97: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 23:27:58.902455 7513 raft_consensus.cc:1261] T 27c38e75002d42688eac2cb1f13c66d4 P 9c15c9d698fe4fa4a7c458f2e51ab5f9 [term 1 LEARNER]: Refusing update from remote peer 63d49bebfcb9422b87754fb258e16a97: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 23:27:58.902626 7258 raft_consensus.cc:1261] T 27c38e75002d42688eac2cb1f13c66d4 P ad2dd25e957d4536bdc22a0df2b7263f [term 1 FOLLOWER]: Refusing update from remote peer 63d49bebfcb9422b87754fb258e16a97: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 23:27:58.903043 7632 consensus_queue.cc:1035] T 27c38e75002d42688eac2cb1f13c66d4 P 63d49bebfcb9422b87754fb258e16a97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c15c9d698fe4fa4a7c458f2e51ab5f9" member_type: VOTER last_known_addr { host: "127.0.100.4" port: 33075 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 23:27:58.903363 7658 consensus_queue.cc:1035] T 27c38e75002d42688eac2cb1f13c66d4 P 63d49bebfcb9422b87754fb258e16a97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ad2dd25e957d4536bdc22a0df2b7263f" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 38457 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 23:27:58.905138 7593 raft_consensus.cc:1261] T 27c38e75002d42688eac2cb1f13c66d4 P 9e3668e37348496ca15ab4da57534a53 [term 1 FOLLOWER]: Refusing update from remote peer 63d49bebfcb9422b87754fb258e16a97: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 23:27:58.906736 7632 consensus_queue.cc:1035] T 27c38e75002d42688eac2cb1f13c66d4 P 63d49bebfcb9422b87754fb258e16a97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9e3668e37348496ca15ab4da57534a53" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 41837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 23:27:58.911201 7696 ts_tablet_manager.cc:1888] T fe71c54bf82048669579628b317224e7 P 63d49bebfcb9422b87754fb258e16a97: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.4
I20240503 23:27:58.911810 7696 log.cc:1196] T fe71c54bf82048669579628b317224e7 P 63d49bebfcb9422b87754fb258e16a97: Deleting WAL directory at /tmp/dist-test-taskZhIiPz/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestDeletedTables.1714778688451250-400-0/minicluster-data/ts-2-root/wals/fe71c54bf82048669579628b317224e7
I20240503 23:27:58.914924 7066 catalog_manager.cc:4830] TS 63d49bebfcb9422b87754fb258e16a97 (127.0.100.3:41385): tablet fe71c54bf82048669579628b317224e7 (table test-workload [id=23b3c6b2809b4e18a2ef0440c24e7d9e]) successfully deleted
I20240503 23:27:58.915889 7658 raft_consensus.cc:2932] T 27c38e75002d42688eac2cb1f13c66d4 P 63d49bebfcb9422b87754fb258e16a97 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 9c15c9d698fe4fa4a7c458f2e51ab5f9 (127.0.100.4) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "9e3668e37348496ca15ab4da57534a53" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 41837 } } peers { permanent_uuid: "ad2dd25e957d4536bdc22a0df2b7263f" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 38457 } attrs { replace: true } } peers { permanent_uuid: "63d49bebfcb9422b87754fb258e16a97" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 41385 } } peers { permanent_uuid: "9c15c9d698fe4fa4a7c458f2e51ab5f9" member_type: VOTER last_known_addr { host: "127.0.100.4" port: 33075 } attrs { promote: false } } }
I20240503 23:27:58.916604 7594 raft_consensus.cc:2932] T 27c38e75002d42688eac2cb1f13c66d4 P 9e3668e37348496ca15ab4da57534a53 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 9c15c9d698fe4fa4a7c458f2e51ab5f9 (127.0.100.4) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "9e3668e37348496ca15ab4da57534a53" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 41837 } } peers { permanent_uuid: "ad2dd25e957d4536bdc22a0df2b7263f" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 38457 } attrs { replace: true } } peers { permanent_uuid: "63d49bebfcb9422b87754fb258e16a97" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 41385 } } peers { permanent_uuid: "9c15c9d698fe4fa4a7c458f2e51ab5f9" member_type: VOTER last_known_addr { host: "127.0.100.4" port: 33075 } attrs { promote: false } } }
I20240503 23:27:58.921438 7258 raft_consensus.cc:2932] T 27c38e75002d42688eac2cb1f13c66d4 P ad2dd25e957d4536bdc22a0df2b7263f [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 9c15c9d698fe4fa4a7c458f2e51ab5f9 (127.0.100.4) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "9e3668e37348496ca15ab4da57534a53" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 41837 } } peers { permanent_uuid: "ad2dd25e957d4536bdc22a0df2b7263f" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 38457 } attrs { replace: true } } peers { permanent_uuid: "63d49bebfcb9422b87754fb258e16a97" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 41385 } } peers { permanent_uuid: "9c15c9d698fe4fa4a7c458f2e51ab5f9" member_type: VOTER last_known_addr { host: "127.0.100.4" port: 33075 } attrs { promote: false } } }
I20240503 23:27:58.923489 7080 catalog_manager.cc:5484] T 27c38e75002d42688eac2cb1f13c66d4 P 9e3668e37348496ca15ab4da57534a53 reported cstate change: config changed from index 4 to 5, 9c15c9d698fe4fa4a7c458f2e51ab5f9 (127.0.100.4) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "63d49bebfcb9422b87754fb258e16a97" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "9e3668e37348496ca15ab4da57534a53" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 41837 } } peers { permanent_uuid: "ad2dd25e957d4536bdc22a0df2b7263f" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 38457 } attrs { replace: true } } peers { permanent_uuid: "63d49bebfcb9422b87754fb258e16a97" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 41385 } } peers { permanent_uuid: "9c15c9d698fe4fa4a7c458f2e51ab5f9" member_type: VOTER last_known_addr { host: "127.0.100.4" port: 33075 } attrs { promote: false } } }
I20240503 23:27:58.924108 7513 raft_consensus.cc:2932] T 27c38e75002d42688eac2cb1f13c66d4 P 9c15c9d698fe4fa4a7c458f2e51ab5f9 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 9c15c9d698fe4fa4a7c458f2e51ab5f9 (127.0.100.4) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "9e3668e37348496ca15ab4da57534a53" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 41837 } } peers { permanent_uuid: "ad2dd25e957d4536bdc22a0df2b7263f" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 38457 } attrs { replace: true } } peers { permanent_uuid: "63d49bebfcb9422b87754fb258e16a97" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 41385 } } peers { permanent_uuid: "9c15c9d698fe4fa4a7c458f2e51ab5f9" member_type: VOTER last_known_addr { host: "127.0.100.4" port: 33075 } attrs { promote: false } } }
I20240503 23:27:58.936933 7630 raft_consensus.cc:1050] T 95f3282b21394b11807125a214e8697a P 63d49bebfcb9422b87754fb258e16a97: attempting to promote NON_VOTER 9c15c9d698fe4fa4a7c458f2e51ab5f9 to VOTER
I20240503 23:27:58.941326 7332 consensus_queue.cc:237] T 27c38e75002d42688eac2cb1f13c66d4 P 63d49bebfcb9422b87754fb258e16a97 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "9e3668e37348496ca15ab4da57534a53" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 41837 } } peers { permanent_uuid: "63d49bebfcb9422b87754fb258e16a97" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 41385 } } peers { permanent_uuid: "9c15c9d698fe4fa4a7c458f2e51ab5f9" member_type: VOTER last_known_addr { host: "127.0.100.4" port: 33075 } attrs { promote: false } }
I20240503 23:27:58.941326 7630 consensus_queue.cc:237] T 95f3282b21394b11807125a214e8697a P 63d49bebfcb9422b87754fb258e16a97 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.3, Last appended by leader: 1, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9e3668e37348496ca15ab4da57534a53" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 41837 } attrs { replace: true } } peers { permanent_uuid: "ad2dd25e957d4536bdc22a0df2b7263f" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 38457 } } peers { permanent_uuid: "63d49bebfcb9422b87754fb258e16a97" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 41385 } } peers { permanent_uuid: "9c15c9d698fe4fa4a7c458f2e51ab5f9" member_type: VOTER last_known_addr { host: "127.0.100.4" port: 33075 } attrs { promote: false } }
I20240503 23:27:58.943924 7594 raft_consensus.cc:1261] T 27c38e75002d42688eac2cb1f13c66d4 P 9e3668e37348496ca15ab4da57534a53 [term 1 FOLLOWER]: Refusing update from remote peer 63d49bebfcb9422b87754fb258e16a97: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 23:27:58.944597 7632 consensus_queue.cc:1035] T 27c38e75002d42688eac2cb1f13c66d4 P 63d49bebfcb9422b87754fb258e16a97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9e3668e37348496ca15ab4da57534a53" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 41837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 23:27:58.946203 7513 raft_consensus.cc:1261] T 27c38e75002d42688eac2cb1f13c66d4 P 9c15c9d698fe4fa4a7c458f2e51ab5f9 [term 1 FOLLOWER]: Refusing update from remote peer 63d49bebfcb9422b87754fb258e16a97: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 23:27:58.946808 7632 consensus_queue.cc:1035] T 27c38e75002d42688eac2cb1f13c66d4 P 63d49bebfcb9422b87754fb258e16a97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c15c9d698fe4fa4a7c458f2e51ab5f9" member_type: VOTER last_known_addr { host: "127.0.100.4" port: 33075 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 23:27:58.947429 7632 raft_consensus.cc:2932] T 27c38e75002d42688eac2cb1f13c66d4 P 63d49bebfcb9422b87754fb258e16a97 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER ad2dd25e957d4536bdc22a0df2b7263f (127.0.100.2) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "9e3668e37348496ca15ab4da57534a53" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 41837 } } peers { permanent_uuid: "63d49bebfcb9422b87754fb258e16a97" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 41385 } } peers { permanent_uuid: "9c15c9d698fe4fa4a7c458f2e51ab5f9" member_type: VOTER last_known_addr { host: "127.0.100.4" port: 33075 } attrs { promote: false } } }
I20240503 23:27:58.948077 7594 raft_consensus.cc:2932] T 27c38e75002d42688eac2cb1f13c66d4 P 9e3668e37348496ca15ab4da57534a53 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER ad2dd25e957d4536bdc22a0df2b7263f (127.0.100.2) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "9e3668e37348496ca15ab4da57534a53" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 41837 } } peers { permanent_uuid: "63d49bebfcb9422b87754fb258e16a97" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 41385 } } peers { permanent_uuid: "9c15c9d698fe4fa4a7c458f2e51ab5f9" member_type: VOTER last_known_addr { host: "127.0.100.4" port: 33075 } attrs { promote: false } } }
I20240503 23:27:58.952710 7513 raft_consensus.cc:2932] T 27c38e75002d42688eac2cb1f13c66d4 P 9c15c9d698fe4fa4a7c458f2e51ab5f9 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER ad2dd25e957d4536bdc22a0df2b7263f (127.0.100.2) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "9e3668e37348496ca15ab4da57534a53" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 41837 } } peers { permanent_uuid: "63d49bebfcb9422b87754fb258e16a97" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 41385 } } peers { permanent_uuid: "9c15c9d698fe4fa4a7c458f2e51ab5f9" member_type: VOTER last_known_addr { host: "127.0.100.4" port: 33075 } attrs { promote: false } } }
I20240503 23:27:58.954595 7512 raft_consensus.cc:1261] T 95f3282b21394b11807125a214e8697a P 9c15c9d698fe4fa4a7c458f2e51ab5f9 [term 2 LEARNER]: Refusing update from remote peer 63d49bebfcb9422b87754fb258e16a97: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240503 23:27:58.954939 7593 raft_consensus.cc:1261] T 95f3282b21394b11807125a214e8697a P 9e3668e37348496ca15ab4da57534a53 [term 2 FOLLOWER]: Refusing update from remote peer 63d49bebfcb9422b87754fb258e16a97: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240503 23:27:58.955278 7066 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 27c38e75002d42688eac2cb1f13c66d4 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 23:27:58.955500 7702 consensus_queue.cc:1035] T 95f3282b21394b11807125a214e8697a P 63d49bebfcb9422b87754fb258e16a97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c15c9d698fe4fa4a7c458f2e51ab5f9" member_type: VOTER last_known_addr { host: "127.0.100.4" port: 33075 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 23:27:58.956369 7702 consensus_queue.cc:1035] T 95f3282b21394b11807125a214e8697a P 63d49bebfcb9422b87754fb258e16a97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9e3668e37348496ca15ab4da57534a53" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 41837 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 23:27:58.960002 7081 catalog_manager.cc:5484] T 27c38e75002d42688eac2cb1f13c66d4 P 63d49bebfcb9422b87754fb258e16a97 reported cstate change: config changed from index 5 to 6, VOTER ad2dd25e957d4536bdc22a0df2b7263f (127.0.100.2) evicted. New cstate: current_term: 1 leader_uuid: "63d49bebfcb9422b87754fb258e16a97" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "9e3668e37348496ca15ab4da57534a53" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 41837 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "63d49bebfcb9422b87754fb258e16a97" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 41385 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9c15c9d698fe4fa4a7c458f2e51ab5f9" member_type: VOTER last_known_addr { host: "127.0.100.4" port: 33075 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 23:27:58.963021 7258 raft_consensus.cc:1261] T 95f3282b21394b11807125a214e8697a P ad2dd25e957d4536bdc22a0df2b7263f [term 2 FOLLOWER]: Refusing update from remote peer 63d49bebfcb9422b87754fb258e16a97: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240503 23:27:58.965655 7248 tablet_service.cc:1514] Processing DeleteTablet for tablet 27c38e75002d42688eac2cb1f13c66d4 with delete_type TABLET_DATA_TOMBSTONED (TS ad2dd25e957d4536bdc22a0df2b7263f not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:48548
I20240503 23:27:58.966514 7706 tablet_replica.cc:330] T 27c38e75002d42688eac2cb1f13c66d4 P ad2dd25e957d4536bdc22a0df2b7263f: stopping tablet replica
I20240503 23:27:58.966765 7706 raft_consensus.cc:2229] T 27c38e75002d42688eac2cb1f13c66d4 P ad2dd25e957d4536bdc22a0df2b7263f [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 23:27:58.967037 7706 raft_consensus.cc:2258] T 27c38e75002d42688eac2cb1f13c66d4 P ad2dd25e957d4536bdc22a0df2b7263f [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 23:27:58.967263 7701 consensus_queue.cc:1035] T 95f3282b21394b11807125a214e8697a P 63d49bebfcb9422b87754fb258e16a97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ad2dd25e957d4536bdc22a0df2b7263f" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 38457 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 23:27:58.970530 7706 ts_tablet_manager.cc:1875] T 27c38e75002d42688eac2cb1f13c66d4 P ad2dd25e957d4536bdc22a0df2b7263f: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 23:27:58.979586 7706 ts_tablet_manager.cc:1888] T 27c38e75002d42688eac2cb1f13c66d4 P ad2dd25e957d4536bdc22a0df2b7263f: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240503 23:27:58.979790 7706 log.cc:1196] T 27c38e75002d42688eac2cb1f13c66d4 P ad2dd25e957d4536bdc22a0df2b7263f: Deleting WAL directory at /tmp/dist-test-taskZhIiPz/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestDeletedTables.1714778688451250-400-0/minicluster-data/ts-1-root/wals/27c38e75002d42688eac2cb1f13c66d4
I20240503 23:27:58.979931 7632 raft_consensus.cc:2932] T 95f3282b21394b11807125a214e8697a P 63d49bebfcb9422b87754fb258e16a97 [term 2 LEADER]: Committing config change with OpId 2.4: config changed from index 3 to 4, 9c15c9d698fe4fa4a7c458f2e51ab5f9 (127.0.100.4) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9e3668e37348496ca15ab4da57534a53" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 41837 } attrs { replace: true } } peers { permanent_uuid: "ad2dd25e957d4536bdc22a0df2b7263f" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 38457 } } peers { permanent_uuid: "63d49bebfcb9422b87754fb258e16a97" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 41385 } } peers { permanent_uuid: "9c15c9d698fe4fa4a7c458f2e51ab5f9" member_type: VOTER last_known_addr { host: "127.0.100.4" port: 33075 } attrs { promote: false } } }
I20240503 23:27:58.980885 7069 catalog_manager.cc:4830] TS ad2dd25e957d4536bdc22a0df2b7263f (127.0.100.2:38457): tablet 27c38e75002d42688eac2cb1f13c66d4 (table test-workload [id=23b3c6b2809b4e18a2ef0440c24e7d9e]) successfully deleted
I20240503 23:27:58.981827 7258 raft_consensus.cc:2932] T 95f3282b21394b11807125a214e8697a P ad2dd25e957d4536bdc22a0df2b7263f [term 2 FOLLOWER]: Committing config change with OpId 2.4: config changed from index 3 to 4, 9c15c9d698fe4fa4a7c458f2e51ab5f9 (127.0.100.4) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9e3668e37348496ca15ab4da57534a53" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 41837 } attrs { replace: true } } peers { permanent_uuid: "ad2dd25e957d4536bdc22a0df2b7263f" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 38457 } } peers { permanent_uuid: "63d49bebfcb9422b87754fb258e16a97" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 41385 } } peers { permanent_uuid: "9c15c9d698fe4fa4a7c458f2e51ab5f9" member_type: VOTER last_known_addr { host: "127.0.100.4" port: 33075 } attrs { promote: false } } }
I20240503 23:27:58.984846 7512 raft_consensus.cc:2932] T 95f3282b21394b11807125a214e8697a P 9c15c9d698fe4fa4a7c458f2e51ab5f9 [term 2 FOLLOWER]: Committing config change with OpId 2.4: config changed from index 3 to 4, 9c15c9d698fe4fa4a7c458f2e51ab5f9 (127.0.100.4) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9e3668e37348496ca15ab4da57534a53" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 41837 } attrs { replace: true } } peers { permanent_uuid: "ad2dd25e957d4536bdc22a0df2b7263f" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 38457 } } peers { permanent_uuid: "63d49bebfcb9422b87754fb258e16a97" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 41385 } } peers { permanent_uuid: "9c15c9d698fe4fa4a7c458f2e51ab5f9" member_type: VOTER last_known_addr { host: "127.0.100.4" port: 33075 } attrs { promote: false } } }
I20240503 23:27:58.985455 7080 catalog_manager.cc:5484] T 95f3282b21394b11807125a214e8697a P 63d49bebfcb9422b87754fb258e16a97 reported cstate change: config changed from index 3 to 4, 9c15c9d698fe4fa4a7c458f2e51ab5f9 (127.0.100.4) changed from NON_VOTER to VOTER. New cstate: current_term: 2 leader_uuid: "63d49bebfcb9422b87754fb258e16a97" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9e3668e37348496ca15ab4da57534a53" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 41837 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ad2dd25e957d4536bdc22a0df2b7263f" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 38457 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "63d49bebfcb9422b87754fb258e16a97" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 41385 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9c15c9d698fe4fa4a7c458f2e51ab5f9" member_type: VOTER last_known_addr { host: "127.0.100.4" port: 33075 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240503 23:27:58.992003 7593 raft_consensus.cc:2932] T 95f3282b21394b11807125a214e8697a P 9e3668e37348496ca15ab4da57534a53 [term 2 FOLLOWER]: Committing config change with OpId 2.4: config changed from index 3 to 4, 9c15c9d698fe4fa4a7c458f2e51ab5f9 (127.0.100.4) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9e3668e37348496ca15ab4da57534a53" member_type: VOTER last_known_addr { host: "127.0.100.1" port: 41837 } attrs { replace: true } } peers { permanent_uuid: "ad2dd25e957d4536bdc22a0df2b7263f" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 38457 } } peers { permanent_uuid: "63d49bebfcb9422b87754fb258e16a97" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 41385 } } peers { permanent_uuid: "9c15c9d698fe4fa4a7c458f2e51ab5f9" member_type: VOTER last_known_addr { host: "127.0.100.4" port: 33075 } attrs { promote: false } } }
I20240503 23:27:58.994511 7332 consensus_queue.cc:237] T 95f3282b21394b11807125a214e8697a P 63d49bebfcb9422b87754fb258e16a97 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.4, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ad2dd25e957d4536bdc22a0df2b7263f" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 38457 } } peers { permanent_uuid: "63d49bebfcb9422b87754fb258e16a97" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 41385 } } peers { permanent_uuid: "9c15c9d698fe4fa4a7c458f2e51ab5f9" member_type: VOTER last_known_addr { host: "127.0.100.4" port: 33075 } attrs { promote: false } }
I20240503 23:27:58.996829 7512 raft_consensus.cc:1261] T 95f3282b21394b11807125a214e8697a P 9c15c9d698fe4fa4a7c458f2e51ab5f9 [term 2 FOLLOWER]: Refusing update from remote peer 63d49bebfcb9422b87754fb258e16a97: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240503 23:27:58.997362 7258 raft_consensus.cc:1261] T 95f3282b21394b11807125a214e8697a P ad2dd25e957d4536bdc22a0df2b7263f [term 2 FOLLOWER]: Refusing update from remote peer 63d49bebfcb9422b87754fb258e16a97: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240503 23:27:58.997439 7701 consensus_queue.cc:1035] T 95f3282b21394b11807125a214e8697a P 63d49bebfcb9422b87754fb258e16a97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c15c9d698fe4fa4a7c458f2e51ab5f9" member_type: VOTER last_known_addr { host: "127.0.100.4" port: 33075 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 23:27:58.997920 7658 consensus_queue.cc:1035] T 95f3282b21394b11807125a214e8697a P 63d49bebfcb9422b87754fb258e16a97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ad2dd25e957d4536bdc22a0df2b7263f" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 38457 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 23:27:59.000365 7658 raft_consensus.cc:2932] T 95f3282b21394b11807125a214e8697a P 63d49bebfcb9422b87754fb258e16a97 [term 2 LEADER]: Committing config change with OpId 2.5: config changed from index 4 to 5, VOTER 9e3668e37348496ca15ab4da57534a53 (127.0.100.1) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ad2dd25e957d4536bdc22a0df2b7263f" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 38457 } } peers { permanent_uuid: "63d49bebfcb9422b87754fb258e16a97" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 41385 } } peers { permanent_uuid: "9c15c9d698fe4fa4a7c458f2e51ab5f9" member_type: VOTER last_known_addr { host: "127.0.100.4" port: 33075 } attrs { promote: false } } }
I20240503 23:27:59.000896 7512 raft_consensus.cc:2932] T 95f3282b21394b11807125a214e8697a P 9c15c9d698fe4fa4a7c458f2e51ab5f9 [term 2 FOLLOWER]: Committing config change with OpId 2.5: config changed from index 4 to 5, VOTER 9e3668e37348496ca15ab4da57534a53 (127.0.100.1) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ad2dd25e957d4536bdc22a0df2b7263f" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 38457 } } peers { permanent_uuid: "63d49bebfcb9422b87754fb258e16a97" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 41385 } } peers { permanent_uuid: "9c15c9d698fe4fa4a7c458f2e51ab5f9" member_type: VOTER last_known_addr { host: "127.0.100.4" port: 33075 } attrs { promote: false } } }
I20240503 23:27:59.001107 7258 raft_consensus.cc:2932] T 95f3282b21394b11807125a214e8697a P ad2dd25e957d4536bdc22a0df2b7263f [term 2 FOLLOWER]: Committing config change with OpId 2.5: config changed from index 4 to 5, VOTER 9e3668e37348496ca15ab4da57534a53 (127.0.100.1) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ad2dd25e957d4536bdc22a0df2b7263f" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 38457 } } peers { permanent_uuid: "63d49bebfcb9422b87754fb258e16a97" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 41385 } } peers { permanent_uuid: "9c15c9d698fe4fa4a7c458f2e51ab5f9" member_type: VOTER last_known_addr { host: "127.0.100.4" port: 33075 } attrs { promote: false } } }
I20240503 23:27:59.005656 7066 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 95f3282b21394b11807125a214e8697a with cas_config_opid_index 4: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 23:27:59.005738 7080 catalog_manager.cc:5484] T 95f3282b21394b11807125a214e8697a P ad2dd25e957d4536bdc22a0df2b7263f reported cstate change: config changed from index 4 to 5, VOTER 9e3668e37348496ca15ab4da57534a53 (127.0.100.1) evicted. New cstate: current_term: 2 leader_uuid: "63d49bebfcb9422b87754fb258e16a97" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ad2dd25e957d4536bdc22a0df2b7263f" member_type: VOTER last_known_addr { host: "127.0.100.2" port: 38457 } } peers { permanent_uuid: "63d49bebfcb9422b87754fb258e16a97" member_type: VOTER last_known_addr { host: "127.0.100.3" port: 41385 } } peers { permanent_uuid: "9c15c9d698fe4fa4a7c458f2e51ab5f9" member_type: VOTER last_known_addr { host: "127.0.100.4" port: 33075 } attrs { promote: false } } }
I20240503 23:27:59.010152 7583 tablet_service.cc:1514] Processing DeleteTablet for tablet 95f3282b21394b11807125a214e8697a with delete_type TABLET_DATA_TOMBSTONED (TS 9e3668e37348496ca15ab4da57534a53 not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:55522
I20240503 23:27:59.010970 7708 tablet_replica.cc:330] stopping tablet replica
I20240503 23:27:59.011253 7708 raft_consensus.cc:2229] T 95f3282b21394b11807125a214e8697a P 9e3668e37348496ca15ab4da57534a53 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 23:27:59.011499 7708 raft_consensus.cc:2258] T 95f3282b21394b11807125a214e8697a P 9e3668e37348496ca15ab4da57534a53 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 23:27:59.012501 7708 ts_tablet_manager.cc:1875] T 95f3282b21394b11807125a214e8697a P 9e3668e37348496ca15ab4da57534a53: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 23:27:59.017951 7708 ts_tablet_manager.cc:1888] T 95f3282b21394b11807125a214e8697a P 9e3668e37348496ca15ab4da57534a53: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.4
I20240503 23:27:59.018132 7708 log.cc:1196] T 95f3282b21394b11807125a214e8697a P 9e3668e37348496ca15ab4da57534a53: Deleting WAL directory at /tmp/dist-test-taskZhIiPz/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestDeletedTables.1714778688451250-400-0/minicluster-data/ts-0-root/wals/95f3282b21394b11807125a214e8697a
I20240503 23:27:59.018985 7068 catalog_manager.cc:4830] TS 9e3668e37348496ca15ab4da57534a53 (127.0.100.1:41837): tablet 95f3282b21394b11807125a214e8697a (table test-workload [id=23b3c6b2809b4e18a2ef0440c24e7d9e]) successfully deleted
W20240503 23:28:00.007719 7130 auto_rebalancer.cc:227] Could not retrieve cluster info: Not found: tserver ad2dd25e957d4536bdc22a0df2b7263f not available for placement
I20240503 23:28:00.981833 400 tablet_server.cc:178] TabletServer@127.0.100.1:41837 shutting down...
I20240503 23:28:00.994748 400 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240503 23:28:00.995069 400 tablet_replica.cc:330] stopping tablet replica
I20240503 23:28:00.995260 400 raft_consensus.cc:2229] T 28c504576e8147d2ab6418920edf5e2f P 9e3668e37348496ca15ab4da57534a53 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 23:28:00.995450 400 raft_consensus.cc:2258] T 28c504576e8147d2ab6418920edf5e2f P 9e3668e37348496ca15ab4da57534a53 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 23:28:00.996109 400 tablet_replica.cc:330] stopping tablet replica
I20240503 23:28:00.996234 400 raft_consensus.cc:2229] T fe71c54bf82048669579628b317224e7 P 9e3668e37348496ca15ab4da57534a53 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 23:28:00.996420 400 raft_consensus.cc:2258] T fe71c54bf82048669579628b317224e7 P 9e3668e37348496ca15ab4da57534a53 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 23:28:00.996922 400 tablet_replica.cc:330] stopping tablet replica
I20240503 23:28:00.997037 400 raft_consensus.cc:2229] T 27c38e75002d42688eac2cb1f13c66d4 P 9e3668e37348496ca15ab4da57534a53 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 23:28:00.997242 400 raft_consensus.cc:2258] T 27c38e75002d42688eac2cb1f13c66d4 P 9e3668e37348496ca15ab4da57534a53 [term 1 FOLLOWER]: Raft consensus is shut down!
W20240503 23:28:01.008824 7130 auto_rebalancer.cc:227] Could not retrieve cluster info: Not found: tserver 9e3668e37348496ca15ab4da57534a53 not available for placement
I20240503 23:28:01.012181 400 tablet_server.cc:194] TabletServer@127.0.100.1:41837 shutdown complete.
I20240503 23:28:01.018005 400 tablet_server.cc:178] TabletServer@127.0.100.2:0 shutting down...
I20240503 23:28:01.027124 400 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240503 23:28:01.027454 400 tablet_replica.cc:330] T 28c504576e8147d2ab6418920edf5e2f P ad2dd25e957d4536bdc22a0df2b7263f: stopping tablet replica
I20240503 23:28:01.027645 400 raft_consensus.cc:2229] T 28c504576e8147d2ab6418920edf5e2f P ad2dd25e957d4536bdc22a0df2b7263f [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 23:28:01.027884 400 raft_consensus.cc:2258] T 28c504576e8147d2ab6418920edf5e2f P ad2dd25e957d4536bdc22a0df2b7263f [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 23:28:01.028745 400 tablet_replica.cc:330] T fe71c54bf82048669579628b317224e7 P ad2dd25e957d4536bdc22a0df2b7263f: stopping tablet replica
I20240503 23:28:01.028911 400 raft_consensus.cc:2229] T fe71c54bf82048669579628b317224e7 P ad2dd25e957d4536bdc22a0df2b7263f [term 2 LEADER]: Raft consensus shutting down.
I20240503 23:28:01.029242 400 raft_consensus.cc:2258] T fe71c54bf82048669579628b317224e7 P ad2dd25e957d4536bdc22a0df2b7263f [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 23:28:01.029798 400 tablet_replica.cc:330] T 95f3282b21394b11807125a214e8697a P ad2dd25e957d4536bdc22a0df2b7263f: stopping tablet replica
I20240503 23:28:01.029935 400 raft_consensus.cc:2229] T 95f3282b21394b11807125a214e8697a P ad2dd25e957d4536bdc22a0df2b7263f [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 23:28:01.030186 400 raft_consensus.cc:2258] T 95f3282b21394b11807125a214e8697a P ad2dd25e957d4536bdc22a0df2b7263f [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 23:28:01.045293 400 tablet_server.cc:194] TabletServer@127.0.100.2:0 shutdown complete.
I20240503 23:28:01.059343 400 tablet_server.cc:178] TabletServer@127.0.100.3:0 shutting down...
I20240503 23:28:01.069593 400 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240503 23:28:01.069898 400 tablet_replica.cc:330] T 28c504576e8147d2ab6418920edf5e2f P 63d49bebfcb9422b87754fb258e16a97: stopping tablet replica
I20240503 23:28:01.070106 400 raft_consensus.cc:2229] T 28c504576e8147d2ab6418920edf5e2f P 63d49bebfcb9422b87754fb258e16a97 [term 2 LEADER]: Raft consensus shutting down.
I20240503 23:28:01.070510 400 raft_consensus.cc:2258] T 28c504576e8147d2ab6418920edf5e2f P 63d49bebfcb9422b87754fb258e16a97 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 23:28:01.071220 400 tablet_replica.cc:330] T 27c38e75002d42688eac2cb1f13c66d4 P 63d49bebfcb9422b87754fb258e16a97: stopping tablet replica
I20240503 23:28:01.071368 400 raft_consensus.cc:2229] T 27c38e75002d42688eac2cb1f13c66d4 P 63d49bebfcb9422b87754fb258e16a97 [term 1 LEADER]: Raft consensus shutting down.
I20240503 23:28:01.071645 400 raft_consensus.cc:2258] T 27c38e75002d42688eac2cb1f13c66d4 P 63d49bebfcb9422b87754fb258e16a97 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 23:28:01.072131 400 tablet_replica.cc:330] T 95f3282b21394b11807125a214e8697a P 63d49bebfcb9422b87754fb258e16a97: stopping tablet replica
I20240503 23:28:01.072260 400 raft_consensus.cc:2229] T 95f3282b21394b11807125a214e8697a P 63d49bebfcb9422b87754fb258e16a97 [term 2 LEADER]: Raft consensus shutting down.
I20240503 23:28:01.072508 400 raft_consensus.cc:2258] T 95f3282b21394b11807125a214e8697a P 63d49bebfcb9422b87754fb258e16a97 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 23:28:01.087638 400 tablet_server.cc:194] TabletServer@127.0.100.3:0 shutdown complete.
I20240503 23:28:01.093122 400 tablet_server.cc:178] TabletServer@127.0.100.4:0 shutting down...
I20240503 23:28:01.102525 400 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240503 23:28:01.102933 400 tablet_replica.cc:330] T 27c38e75002d42688eac2cb1f13c66d4 P 9c15c9d698fe4fa4a7c458f2e51ab5f9: stopping tablet replica
I20240503 23:28:01.103196 400 raft_consensus.cc:2229] T 27c38e75002d42688eac2cb1f13c66d4 P 9c15c9d698fe4fa4a7c458f2e51ab5f9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 23:28:01.103591 400 raft_consensus.cc:2258] T 27c38e75002d42688eac2cb1f13c66d4 P 9c15c9d698fe4fa4a7c458f2e51ab5f9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 23:28:01.104701 400 tablet_replica.cc:330] T 95f3282b21394b11807125a214e8697a P 9c15c9d698fe4fa4a7c458f2e51ab5f9: stopping tablet replica
I20240503 23:28:01.104919 400 raft_consensus.cc:2229] T 95f3282b21394b11807125a214e8697a P 9c15c9d698fe4fa4a7c458f2e51ab5f9 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 23:28:01.105232 400 raft_consensus.cc:2258] T 95f3282b21394b11807125a214e8697a P 9c15c9d698fe4fa4a7c458f2e51ab5f9 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 23:28:01.106030 400 tablet_replica.cc:330] T fe71c54bf82048669579628b317224e7 P 9c15c9d698fe4fa4a7c458f2e51ab5f9: stopping tablet replica
I20240503 23:28:01.106230 400 raft_consensus.cc:2229] T fe71c54bf82048669579628b317224e7 P 9c15c9d698fe4fa4a7c458f2e51ab5f9 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 23:28:01.106534 400 raft_consensus.cc:2258] T fe71c54bf82048669579628b317224e7 P 9c15c9d698fe4fa4a7c458f2e51ab5f9 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 23:28:01.121935 400 tablet_server.cc:194] TabletServer@127.0.100.4:0 shutdown complete.
I20240503 23:28:01.126571 400 master.cc:537] Master@127.0.100.62:43625 shutting down...
I20240503 23:28:01.135228 400 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 3e719c8272094dccadee8e7490c155c9 [term 1 LEADER]: Raft consensus shutting down.
I20240503 23:28:01.135538 400 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 3e719c8272094dccadee8e7490c155c9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 23:28:01.135633 400 tablet_replica.cc:330] T 00000000000000000000000000000000 P 3e719c8272094dccadee8e7490c155c9: stopping tablet replica
I20240503 23:28:01.149839 400 master.cc:552] Master@127.0.100.62:43625 shutdown complete.
[ OK ] AutoRebalancerTest.TestDeletedTables (8904 ms)
[----------] 14 tests from AutoRebalancerTest (192637 ms total)
[----------] Global test environment tear-down
[==========] 14 tests from 1 test suite ran. (192638 ms total)
[ PASSED ] 13 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] AutoRebalancerTest.TestMaxMovesPerServer
1 FAILED TEST
I20240503 23:28:01.177764 400 logging.cc:424] LogThrottler /home/jenkins-slave/workspace/build_and_test_flaky/src/kudu/tserver/ts_tablet_manager.cc:574: suppressed but not reported on 7 messages since previous log ~5 seconds ago
I20240503 23:28:01.177873 400 logging.cc:424] LogThrottler /home/jenkins-slave/workspace/build_and_test_flaky/src/kudu/tserver/ts_tablet_manager.cc:522: suppressed but not reported on 7 messages since previous log ~5 seconds ago
I20240503 23:28:01.177932 400 logging.cc:424] LogThrottler /home/jenkins-slave/workspace/build_and_test_flaky/src/kudu/master/catalog_manager.cc:4573: suppressed but not reported on 12 messages since previous log ~5 seconds ago
I20240503 23:28:01.177986 400 logging.cc:424] LogThrottler /home/jenkins-slave/workspace/build_and_test_flaky/src/kudu/master/catalog_manager.cc:5162: suppressed but not reported on 7 messages since previous log ~31 seconds ago
I20240503 23:28:01.178047 400 logging.cc:424] LogThrottler /home/jenkins-slave/workspace/build_and_test_flaky/src/kudu/tserver/heartbeater.cc:644: suppressed but not reported on 8 messages since previous log ~178 seconds ago
I20240503 23:28:01.178157 400 logging.cc:424] LogThrottler /home/jenkins-slave/workspace/build_and_test_flaky/src/kudu/tablet/tablet.cc:2369: suppressed but not reported on 50 messages since previous log ~19 seconds ago
I20240503 23:28:01.178208 400 logging.cc:424] LogThrottler /home/jenkins-slave/workspace/build_and_test_flaky/src/kudu/rpc/proxy.cc:239: suppressed but not reported on 25 messages since previous log ~9 seconds ago